Atlassian uses cookies to improve your browsing experience, perform analytics and research, and conduct advertising. Accept all cookies to indicate that you agree to our use of cookies on your device. Atlassian cookies and tracking notice, (opens new window)
12-U1 converts mirrored disks to multipath which make the ZFS pool inaccessible
Description
Upgraded today to 12-U1 in order to test the bhyve VM restart issue.
started existing VM
VM started but said that it writing something to UEFI
My pools were suddenly no longer accessible
Reboot in 12.0-U1 - Pool were UNEXISTENT Rebooted previous version 12.0 same issue (initial ticket was made from within 11.3-U5)
Data is back, but root cause unknown . --> Bug still present : why are (double mirror) disks converted to MULTIPATH
Mellmans comment saved my life : [SOLVED - GPT Corrupt or invalid | TrueNAS Community|https://www.truenas.com/community/threads/gpt-corrupt-or-invalid.81180/]
Symptoms :
zpool disappeared, all disks seemed present dmesg output shows things like : GEOM_MULTIPATH: disk2 created GEOM_MULTIPATH: da1 added to disk2 GEOM_MULTIPATH: da1 is now active path in disk2 GEOM: multipath/disk2: corrupt or invalid GPT detected. GEOM: multipath/disk2: GPT rejected – may not be recoverable .
WARNING/DISCLAIMER : As I had a multi mirror system, I initially remove 2 out of 3 to limit my risk and to avail doing disk clones before starting recovery. I deleted my external back two days ago, so this was an all or nothing situation).
REMINDER: Mirror is not backup . AlLways try to make a backup to a NON connected device (USB drive a other storage server)
commands used for diagnose and repair
zpool import -D
no pools available to import
sysctl kern.disks kern.disks: da3 da2 da1 da0 ada0 (–> all disk I expected where here)
gmultipath destroy disk1 (I was very scared, but this was not needed as I was sure NOT to have multipaths) - Do this command for each disk in your pool (after label verif!).
zpool import -f
Pool wad *FOUND *, but because I had the 2 other disks from the MIRROR physicaly removed, the zpool wasn't yet able to start.
pool: akira_p01 id: 5424572115530985634 state: UNAVAIL status: One or more devices are missing from the system. action: The pool cannot be imported. Attach the missing devices and try again. see: http://illumos.org/msg/ZFS-8000-3C config:
akira_p01 UNAVAIL insufficient replicas 16830735507974382924 UNAVAIL cannot open mirror-1 DEGRADED 18165764508876870888 UNAVAIL cannot open gptid/696b5de4-b09d-11e5-9fe8-00188b1dee45 ONLINE
repeated gmultipath destroy for the two other disks
ran zpool import -f ** again — YES YES YES - data back
zpool status pool: akira_p01 state: ONLINE scan: scrub repaired 0B in 1 days 12:00:06 with 0 errors on Mon Dec 7 13:01:07 2020 config:
Commands were performed while booted in Freenas 11.3-U5, but this could only mount ZFS ReadOnly because of upgraded ZFS parameters. Booting in a previous version is not required
@Jan, smartctl detects presence of USB-SATA bridge and talks directly to the SATA disks via the pass-through commands, while kernel relies on SCSI identification provided by the bridge.
William Gryzbowski
December 18, 2020 at 12:56 PM
I believe SMART uses a different mechanism to retrieve serial, not SCSI IDENTIFY. Could also be because the USB driver doesn't translate it correctly because your device doesn't follow spec.
Upgraded today to 12-U1 in order to test the bhyve VM restart issue.
started existing VM
VM started but said that it writing something to UEFI
My pools were suddenly no longer accessible
Reboot in 12.0-U1 - Pool were UNEXISTENT
Rebooted previous version 12.0 same issue (initial ticket was made from within 11.3-U5)
Data is back, but root cause unknown . --> Bug still present : why are (double mirror) disks converted to MULTIPATH
Mellmans comment saved my life : [SOLVED - GPT Corrupt or invalid | TrueNAS Community|https://www.truenas.com/community/threads/gpt-corrupt-or-invalid.81180/]
Symptoms :
zpool disappeared, all disks seemed present
dmesg output shows things like :
GEOM_MULTIPATH: disk2 created
GEOM_MULTIPATH: da1 added to disk2
GEOM_MULTIPATH: da1 is now active path in disk2
GEOM: multipath/disk2: corrupt or invalid GPT detected.
GEOM: multipath/disk2: GPT rejected – may not be recoverable .
WARNING/DISCLAIMER : As I had a multi mirror system, I initially remove 2 out of 3 to limit my risk and to avail doing disk clones before starting recovery. I deleted my external back two days ago, so this was an all or nothing situation).
REMINDER: Mirror is not backup . AlLways try to make a backup to a NON connected device (USB drive a other storage server)
commands used for diagnose and repair
zpool import -D
no pools available to import
sysctl kern.disks
kern.disks: da3 da2 da1 da0 ada0 (–> all disk I expected where here)
zdb -l /dev/da3p2 (LABELS seemed to be OK )
------------------------------------
LABEL 0
------------------------------------
version: 5000
name: 'akira_p01'
state: 0
txg: 29173828
pool_guid: 5424572115530985634
errata: 0
hostid: 4193025452
hostname: ''
top_guid: 1827022710836996365
guid: 7863535274456000245
vdev_children: 2
vdev_tree:
type: 'mirror'
id: 1
guid: 1827022710836996365
metaslab_array: 35
metaslab_shift: 34
ashift: 12
asize: 2998440558592
is_log: 0
create_txg: 4
children [0] :
type: 'disk'
id: 0
guid: 18165764508876870888
path: '/dev/gptid/66eb141c-b09d-11e5-9fe8-00188b1dee45'
whole_disk: 1
DTL: 4252
create_txg: 4
children [1] :
type: 'disk'
id: 1
guid: 7863535274456000245
path: '/dev/gptid/696b5de4-b09d-11e5-9fe8-00188b1dee45'
whole_disk: 1
DTL: 4251
create_txg: 4
features_for_read:
com.delphix:hole_birth
com.delphix:embedded_data
------------------------------------
LABEL 1
------------------------------------
version: 5000
name: 'akira_p01'
state: 0
txg: 29173828
pool_guid: 5424572115530985634
errata: 0
hostid: 4193025452
hostname: ''
top_guid: 1827022710836996365
guid: 7863535274456000245
vdev_children: 2
vdev_tree:
type: 'mirror'
id: 1
guid: 1827022710836996365
metaslab_array: 35
metaslab_shift: 34
ashift: 12
asize: 2998440558592
is_log: 0
create_txg: 4
children [0] :
type: 'disk'
id: 0
guid: 18165764508876870888
path: '/dev/gptid/66eb141c-b09d-11e5-9fe8-00188b1dee45'
whole_disk: 1
DTL: 4252
create_txg: 4
children [1] :
type: 'disk'
id: 1
guid: 7863535274456000245
path: '/dev/gptid/696b5de4-b09d-11e5-9fe8-00188b1dee45'
whole_disk: 1
DTL: 4251
create_txg: 4
features_for_read:
com.delphix:hole_birth
com.delphix:embedded_data
------------------------------------
LABEL 2
------------------------------------
version: 5000
name: 'akira_p01'
state: 0
txg: 29173828
pool_guid: 5424572115530985634
errata: 0
hostid: 4193025452
hostname: ''
top_guid: 1827022710836996365
guid: 7863535274456000245
vdev_children: 2
vdev_tree:
type: 'mirror'
id: 1
guid: 1827022710836996365
metaslab_array: 35
metaslab_shift: 34
ashift: 12
asize: 2998440558592
is_log: 0
create_txg: 4
children [0] :
type: 'disk'
id: 0
guid: 18165764508876870888
path: '/dev/gptid/66eb141c-b09d-11e5-9fe8-00188b1dee45'
whole_disk: 1
DTL: 4252
create_txg: 4
children [1] :
type: 'disk'
id: 1
guid: 7863535274456000245
path: '/dev/gptid/696b5de4-b09d-11e5-9fe8-00188b1dee45'
whole_disk: 1
DTL: 4251
create_txg: 4
features_for_read:
com.delphix:hole_birth
com.delphix:embedded_data
------------------------------------
LABEL 3
------------------------------------
version: 5000
name: 'akira_p01'
state: 0
txg: 29173828
pool_guid: 5424572115530985634
errata: 0
hostid: 4193025452
hostname: ''
top_guid: 1827022710836996365
guid: 7863535274456000245
vdev_children: 2
vdev_tree:
type: 'mirror'
id: 1
guid: 1827022710836996365
metaslab_array: 35
metaslab_shift: 34
ashift: 12
asize: 2998440558592
is_log: 0
create_txg: 4
children [0] :
type: 'disk'
id: 0
guid: 18165764508876870888
path: '/dev/gptid/66eb141c-b09d-11e5-9fe8-00188b1dee45'
whole_disk: 1
DTL: 4252
create_txg: 4
children [1] :
type: 'disk'
id: 1
guid: 7863535274456000245
path: '/dev/gptid/696b5de4-b09d-11e5-9fe8-00188b1dee45'
whole_disk: 1
DTL: 4251
create_txg: 4
features_for_read:
com.delphix:hole_birth
com.delphix:embedded_data
gmultipath list ( STRANGE - should have been empty, I never created these)
Gives a multipath for each disk that was in my pool. WHY ---> Mellmans comment saved this
Geom name: disk1
Type: AUTOMATIC
Mode: Active/Passive
UUID: d68bb1af-3acc-11eb-9dd0-00012e7acada
State: DEGRADED
Providers:
1. Name: multipath/disk1
Mediasize: 3000592981504 (2.7T)
Sectorsize: 512
Stripesize: 4096
Stripeoffset: 0
Mode: r0w0e0
State: DEGRADED
Consumers:
1. Name: da0
Mediasize: 3000592982016 (2.7T)
Sectorsize: 512
Stripesize: 4096
Stripeoffset: 0
Mode: r1w1e1
State: ACTIVE
gmultipath destroy disk1 (I was very scared, but this was not needed as I was sure NOT to have multipaths) - Do this command for each disk in your pool (after label verif!).
zpool import -f
Pool wad *FOUND *, but because I had the 2 other disks from the MIRROR physicaly removed, the zpool wasn't yet able to start.
pool: akira_p01
id: 5424572115530985634
state: UNAVAIL
status: One or more devices are missing from the system.
action: The pool cannot be imported. Attach the missing
devices and try again.
see: http://illumos.org/msg/ZFS-8000-3C
config:
akira_p01 UNAVAIL insufficient replicas
16830735507974382924 UNAVAIL cannot open
mirror-1 DEGRADED
18165764508876870888 UNAVAIL cannot open
gptid/696b5de4-b09d-11e5-9fe8-00188b1dee45 ONLINE
repeated gmultipath destroy for the two other disks
ran zpool import -f ** again — YES YES YES - data back
zpool status
pool: akira_p01
state: ONLINE
scan: scrub repaired 0B in 1 days 12:00:06 with 0 errors on Mon Dec 7 13:01:07 2020
config:
NAME STATE READ WRITE CKSUM
akira_p01 ONLINE 0 0 0
gptid/64cb349c-b09d-11e5-9fe8-00188b1dee45 ONLINE 0 0 0
mirror-1 ONLINE 0 0 0
gptid/66eb141c-b09d-11e5-9fe8-00188b1dee45 ONLINE 0 0 0
gptid/696b5de4-b09d-11e5-9fe8-00188b1dee45 ONLINE 0 0 0
errors: No known data errors
Commands were performed while booted in Freenas 11.3-U5, but this could only mount ZFS ReadOnly because of upgraded ZFS parameters. Booting in a previous version is not required