Project

General

Profile

Bug #28560

Removed disk remained in geom database

Added by Richard Kojedzinszky over 1 year ago. Updated 7 months ago.

Status:
Closed
Priority:
Nice to have
Assignee:
Alexander Motin
Category:
OS
Target version:
Seen in:
Severity:
Low Medium
Reason for Closing:
Cannot Reproduce
Reason for Blocked:
Needs QA:
No
Needs Doc:
No
Needs Merging:
No
Needs Automation:
No
Support Suite Ticket:
n/a
Hardware Configuration:
ChangeLog Required:
No

Description

We were in a process of replacing all our disks in a pool to new and larger disks. It affected 24 disks. In each round, we replaced 3 disks, as we had that many empty slots in our chassis. We have a pool of 4x(4+2 raidz2) configuration, each only one disk was replaced in one vdev.

After the replace finished, we just pulled the disks out of the chassis, replaced them with the new ones, and initiated a replace on freenas UI.

After one disk removal, something went wrong. We removed da35, camcontrol devlist shows it does not exist:

# camcontrol devlist
<LSI CORP SAS2X28 0717> at scbus0 target 24 lun 0 (ses0,pass0)
<ATA ST6000VN0001-1SF AN02> at scbus0 target 26 lun 0 (pass1,da0)
<ATA ST6000VN0001-1SF AN02> at scbus0 target 27 lun 0 (pass2,da1)
<ATA ST6000VN0001-1SF AN02> at scbus0 target 28 lun 0 (pass3,da2)
<ATA ST6000VN0001-1SF AN02> at scbus0 target 29 lun 0 (pass4,da3)
<ATA ST6000VN0001-1SF AN02> at scbus0 target 30 lun 0 (pass5,da4)
<ATA ST6000VN0001-1SF AN02> at scbus0 target 31 lun 0 (pass6,da5)
<ATA ST6000VN0001-1SF AN02> at scbus0 target 32 lun 0 (pass7,da6)
<ATA ST6000VN0001-1SF AN02> at scbus0 target 33 lun 0 (pass8,da7)
<ATA ST6000VN0001-1SF AN02> at scbus0 target 34 lun 0 (pass9,da8)
<ATA ST6000VN0001-1SF AN02> at scbus0 target 35 lun 0 (pass10,da9)
<ATA ST6000VN0001-1SF AN02> at scbus0 target 37 lun 0 (pass11,da10)
<ATA ST6000VN0001-1SF AN02> at scbus0 target 38 lun 0 (pass12,da11)
<ATA ST6000VN0001-1SF AN02> at scbus0 target 39 lun 0 (pass13,da12)
<ATA ST6000VN0001-1SF AN02> at scbus0 target 40 lun 0 (pass14,da13)
<ATA ST6000VN0001-1SF AN02> at scbus0 target 41 lun 0 (pass15,da14)
<ATA ST6000VN0001-1SF AN02> at scbus0 target 42 lun 0 (pass16,da15)
<LSI SAS3x36 0601> at scbus1 target 14 lun 0 (ses1,pass19)
<MICRON S630DC-800 M017> at scbus1 target 18 lun 0 (pass20,da18)
<MICRON S630DC-800 M017> at scbus1 target 19 lun 0 (pass21,da19)
<MICRON S630DC-800 M017> at scbus1 target 20 lun 0 (pass22,da20)
<SEAGATE ST10000NM0096 E001> at scbus1 target 21 lun 0 (pass23,da21)
<SEAGATE ST10000NM0096 E001> at scbus1 target 22 lun 0 (pass24,da22)
<SEAGATE ST10000NM0096 E001> at scbus1 target 23 lun 0 (pass25,da23)
<SEAGATE ST10000NM0096 E001> at scbus1 target 24 lun 0 (pass26,da24)
<SEAGATE ST10000NM0096 E001> at scbus1 target 25 lun 0 (pass27,da25)
<SEAGATE ST10000NM0096 E001> at scbus1 target 26 lun 0 (pass28,da26)
<SEAGATE ST10000NM0096 E001> at scbus1 target 27 lun 0 (pass29,da27)
<SEAGATE ST10000NM0096 E001> at scbus1 target 28 lun 0 (pass30,da28)
<SEAGATE ST10000NM0096 E001> at scbus1 target 30 lun 0 (da47,pass52)
<SEAGATE ST10000NM0096 E001> at scbus1 target 31 lun 0 (da48,pass53)
<SEAGATE ST10000NM0096 E001> at scbus1 target 32 lun 0 (da16,pass17)
<SEAGATE ST10000NM0096 E001> at scbus1 target 33 lun 0 (da17,pass18)
<SEAGATE ST10000NM0096 E001> at scbus1 target 34 lun 0 (da49,pass54)
<MICRON S650DC-800 M017> at scbus2 target 10 lun 0 (pass32,da30)
<MICRON S650DC-800 M017> at scbus2 target 11 lun 0 (pass33,da31)
<LSI SAS3x36 0601> at scbus2 target 13 lun 0 (ses2,pass34)
<SEAGATE ST10000NM0096 E001> at scbus2 target 31 lun 0 (da29,pass31)
<SEAGATE ST10000NM0096 E001> at scbus2 target 32 lun 0 (da32,pass35)
<SEAGATE ST10000NM0096 E001> at scbus2 target 33 lun 0 (da37,pass40)
<SEAGATE ST10000NM0096 E001> at scbus2 target 34 lun 0 (da41,pass44)
<SEAGATE ST10000NM0096 E002> at scbus2 target 35 lun 0 (da38,pass38)
<SEAGATE ST10000NM0096 E002> at scbus2 target 36 lun 0 (da42,pass41)
<SEAGATE ST10000NM0096 E002> at scbus2 target 37 lun 0 (da50,pass45)
<SEAGATE ST10000NM0096 E002> at scbus2 target 38 lun 0 (da33,pass36)
<SEAGATE ST10000NM0096 E002> at scbus2 target 39 lun 0 (da39,pass42)
<SEAGATE ST10000NM0096 E002> at scbus2 target 40 lun 0 (da43,pass46)
<SEAGATE ST10000NM0096 E002> at scbus2 target 41 lun 0 (da34,pass37)
<SEAGATE ST10000NM0096 E002> at scbus2 target 42 lun 0 (da36,pass39)
<INTEL SSDSC2BA100G3 5DV10250> at scbus3 target 0 lun 0 (ada0,pass48)
<INTEL SSDSC2BA100G3 5DV10250> at scbus5 target 0 lun 0 (ada1,pass49)
<SanDisk Cruzer Fit 1.27> at scbus10 target 0 lun 0 (pass50,da45)
<SanDisk Cruzer Fit 1.26> at scbus11 target 0 lun 0 (pass51,da46)

While gpart still shows it:
# gpart show da35
=> 40 3907029088 da35 GPT (1.8T)
40 8 - free - (4.0K)
48 4194304 1 freebsd-swap (2.0G)
4194352 3902834768 2 freebsd-zfs (1.8T)
3907029120 8 - free - (4.0K)

I've destroyed the glabel geoms (so glabel status does not return it), removed device entries under /dev (by rm), but as the geom knows something about it, and the freenas UI tries to access it somehow, and it effectively breaks disk replacement tasks for example from the ui.

When I try to modify the gpt, for example with:
# gpart add -t freebsd-zfs da35
gpart: geom 'da35': Device not configured

I get the following kernel message:
# dmesg
g_access(918): provider da35 has error 6 set

For now it only seems to be a leak somewhere, and does not affect normal operation until I can replace disks in the pool from command line.

This happened only once, I cannot reproduce it, but actually the server is in this state.

If needed, I could provide any information which might help getting closer this bug/leak.

Can I somehow make geom forget about this drive?


Related issues

Related to FreeNAS - Bug #32772: Reconfigure swap when a disk vanishes Done

History

#1 Updated by Dru Lavigne over 1 year ago

  • Assignee changed from Release Council to Alexander Motin

Alexander: what are your thoughts on this one? Is there something Richard can provide to help pin down the cause?

#2 Updated by Alexander Motin over 1 year ago

Both CAM and GEOM will not desctroy the device until it is closed. In most cases it should happen automatically, but I am not sure that in all. Take a closer look on open counts for the disk in `geom disk list da35` and `geom part list da35`. Also check swapinfo that swap for that disk is closed.

#3 Updated by Richard Kojedzinszky over 1 year ago

# geom disk list da35
Geom name: da35
Providers:
1. Name: da35
Mediasize: 2000398934016 (1.8T)
Sectorsize: 512
Mode: r0w0e0
wither: (null)

# geom part list da35
Geom name: da35
modified: false
state: OK
fwheads: 255
fwsectors: 63
last: 3907029127
first: 40
entries: 152
scheme: GPT
Providers:
1. Name: da35p1
Mediasize: 2147483648 (2.0G)
Sectorsize: 512
Stripesize: 0
Stripeoffset: 24576
Mode: r0w0e0
rawuuid: 161d7de8-c3b3-11e7-a9d0-0025902b8558
rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
label: (null)
length: 2147483648
offset: 24576
type: freebsd-swap
index: 1
end: 4194351
start: 48
2. Name: da35p2
Mediasize: 1998251401216 (1.8T)
Sectorsize: 512
Stripesize: 0
Stripeoffset: 2147508224
Mode: r0w0e0
rawuuid: 2e0893ca-c3b3-11e7-a9d0-0025902b8558
rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
label: (null)
length: 1998251401216
offset: 2147508224
type: freebsd-zfs
index: 2
end: 3907029119
start: 4194352
Consumers:
1. Name: da35
Mediasize: 2000398934016 (1.8T)
Sectorsize: 512
Mode: r0w0e0

# swapinfo
Device 1K-blocks Used Avail Capacity
/dev/#C:0x2a8 2097152 49780 2047372 2%
/dev/mirror/swap3.eli 2097152 48960 2048192 2%
/dev/mirror/swap1.eli 2097152 49420 2047732 2%
/dev/mirror/swap4.eli 2097152 0 2097152 0%
Total 8388608 148160 8240448 2%

# gmirror status
Name Status Components
mirror/swap3 COMPLETE da16p1 (ACTIVE)
da48p1 (ACTIVE)
mirror/swap1 COMPLETE da17p1 (ACTIVE)
da47p1 (ACTIVE)
mirror/swap4 COMPLETE da41p1 (ACTIVE)
da37p1 (ACTIVE)

In swapinfo the first entry is very strange. Maybe that is the cause. In that state right now, I assume the actually swapped data is gone, so should I force a reboot as soon as possible?

#4 Updated by Alexander Motin over 1 year ago

Richard Kojedzinszky wrote:

In swapinfo the first entry is very strange. Maybe that is the cause. In that state right now, I assume the actually swapped data is gone, so should I force a reboot as soon as possible?

Yes, you probably should reboot ASAP. Otherwise system may panic if any system-critical memory pages are there and system try to get them in.

From "Mode: r0w0e0" it seems like the disk was closed, so I am not sure why it haven't disappeared. May be it is somehow related to mirror and eli om top of it somehow. I haven't tested unplugging multiple disks after then mirrored swap code was implemented in FreeNAS.

#5 Updated by Richard Kojedzinszky over 1 year ago

I've issued a reboot -l -q, but that did not succeed, so I had to issue the reboot with -n also. That did reboot the machine.

I've tried some simple cases regarding the swap code, and could not achieve the bad behaviour: the code always assembled a mirror of active disks, and not used the disks being replaced. Altough I would not say my tests were complete, so I will also do some more research on it.

Thanks,

#6 Updated by Richard Kojedzinszky over 1 year ago

It seems that swap is configured based on swap partition sizes. So it is possible one to create larger swap partitions, later change his mind, and upon replacing disks in a pool, the partitioner will create smaller swap partitions. Thus, the swap will still be configured on the larger partitions, on disks which are planned to be removed.

Maybe, some kind of workaround would be that the swap configuration tool would only consider disks already configured in a pool, not use those that are not used for anything.

#7 Updated by Dru Lavigne over 1 year ago

  • Target version set to 11.2-U2

#8 Updated by Alexander Motin over 1 year ago

  • Assignee changed from Alexander Motin to William Grzybowski

Sounds like a middleware issue. Something went wrong about swap during the replace process. In case of different partition sizes I think it indeed can be more difficult to handle it correctly.

#9 Updated by William Grzybowski about 1 year ago

  • Category changed from Hardware to Middleware
  • Severity set to Low Medium

#10 Updated by William Grzybowski about 1 year ago

  • Related to Bug #32772: Reconfigure swap when a disk vanishes added

#11 Updated by William Grzybowski about 1 year ago

  • Assignee changed from William Grzybowski to Alexander Motin

I have related the ticket which reconfigure swaps (gmirror) when one disks ungracefully disappears.

Passing back to Alexander in case there is something actionable on FreeBSD side.

#12 Updated by Alexander Motin 7 months ago

  • Category changed from Middleware to OS
  • Status changed from Not Started to Closed
  • Target version changed from 11.2-U2 to N/A
  • Reason for Closing set to Cannot Reproduce
  • Needs QA changed from Yes to No
  • Needs Doc changed from Yes to No
  • Needs Merging changed from Yes to No

I am closing this based on lack of ideas, other reports like this and passed time.

Also available in: Atom PDF