Skip to:

  1. Skip to Jira Navigation
  2. Skip to Side Navigation
  3. Skip to Main Content

SCALE: NVME System Dataset Pool Offline After Upgrade

Description

I did an upgrade from 21.10-MASTER-20211004-172926 to 22.02-MASTER-20211020-212923. The upgrade worked, and the machine rebooted just fine. However, my NVME boot pool shows Offline, which is where my System Dataset lives. My configuration is a mirrored 120gb SSD boot pool, a single m.2 NVME (Samsung 970 SSD) for System Dataset (along with two datasets, ix-applications and UbuntuVM), and then a 8 disk ZFS array.

The disk shows up under Storage->Disks, so I can see the nvme0n1 disk there and I know that the system can see it. The only option I have is an Export/Disconnect button – I haven't tried that since I wanted to make sure I didn't lose the data.

I can fall back to the previous version and the NVME drive mounts fine and the syystem works.

Problem/Justification

None

Impact

None

Activity

Show:

Ryan Moeller 
December 6, 2021 at 4:45 PM

Yes once you have a replica of the data you can upgrade and create a new pool, then restore your datasets from the backup.

Brian Sotak 
December 5, 2021 at 4:58 PM

Thanks. I've been too busy to try, but what is the best way to upgrade and get the flag unset? I've gone to the link and followed it to local replication (https://www.truenas.com/docs/scale/dataprotection/replication/#local-replication) and made a local copy of the drive (that includes one filesystem for ix-applications, one zvol for the VM). Do I now do the upgrade, remount the drive as new, and then go back to Data Protection and do a Restore to the newly configured drive?

Ryan Moeller 
November 5, 2021 at 4:59 PM

The NVME pool has feature@xattr_compat active, which was an experimental feature briefly present on the nightly train and activated by default for a short period in early July. You'll have to back up the data and recreate your NVMe pool to move forward, as the feature flag has been removed in response to recent feedback received while upstreaming the feature. There is more information linked in the release notes for RC1 if you are curious: https://www.truenas.com/docs/releasenotes/scale/22.02-rc.1-1/

Brian Sotak 
November 4, 2021 at 5:41 PM

Just tried to do an update from 21.10 to the latest nightly. It went back to showing the NVME offline. Attached debug log here.

 

Brian Sotak 
November 4, 2021 at 5:16 PM

Sorry, was on travel and couldn't get back to this. Interestingly, I rebooted this time and it shows that the System Dataset Pool is now on the main storage array. The NVME is still not mounted (thus no System Dataset Pool on there, or my existing VM). I'll attach both the valid working 21.10 and failed 22.02 debugs.

Known Error

Details

Assignee

Reporter

Labels

Impact

Time remaining

0m

Fix versions

Affects versions

Priority

Katalon Platform

Created October 22, 2021 at 1:32 PM
Updated July 6, 2022 at 9:01 PM
Resolved November 5, 2021 at 4:59 PM