Schrödinger's Datasets

Nothing prevents your datasets from being mounted normally and then one rogue dataset being mounted over /mnt and hiding all of them. Then as you observed, ZFS will be convinced they are mounted because they are, and you'll be confused because you can't see them or interact with them.

I'm closing this ticket as it's not anything we have any control over when you're manually replicating things around, you'll have to get that right yourself or use the replication features within the TrueNAS UI.

Yes, that's a good hypothesis, but doesn't zfs automatically create the mountpoint subdir if needed? And that should have still worked since the other datasets were not mounted ro.

Order of mounting may have something to do with it, but I'm still unsure because:

• If the /mnt dataset was mounted first, then mounting the vol1 and ssd_scratch datasets on top would have zero impact.

• If the vol1 and ssd_scratch datasets were mounted in /mnt first, then the subsequent dataset mount action to /mnt should have failed?

• But vol1 was accessible and ssd_scratch was not.

Another related issue here: this may be what caused the loss of one of my TrueCharts pvc datasets since they seem to sometimes be mounted in /mnt/var for some bizarre reason. If the container dataset mount failed over top of /mnt/var, those files would have been written into the other dataset and then disappeared after I fixed the problem. Still a little strange since presumably a mount of the dataset would have failed, which should have caused the container creation pipeline to fail, but since somethings seem to have mounted and others didn't, it's possible it reported as mounted but writes were actually going to the other dataset instead of the intended dataset.

So I suppose the dataset mounted over /mnt is preventing the other datasets from being unmounted, since their mountpoints are now hidden.

It wasn't using TrueNAS replication; this is manual zfs send | ssh zfs recv, from Ubuntu workstation to TrueNAS CORE (and now TrueNAS SCALE, but I have not done another send since migrating to SCALE) in order to push a backup of the workstation to the NAS.

I was pursuing setting mountpoint=none, but on the forums a user suggested canmount=noauto, and that's much cleaner. I suspect this does prevent me from doing an easy restore if I needed to though, without setting that canmount=auto after the restore, but at least I won't have to maintain any state on the mountpoints.

This is a mess:

 
 
I don't know why this didn't interfere with my other zpool mounted on /mnt/vol1, and I don't know why this behavior is different in Linux/SCALE v. BSD/CORE.
 Now that I have a workaround, I'm unsure how I should proceed backing up my zfs datasets from the workstation without clobbering the TrueNAS SCALE filesystem mounts.

It is a known issue that lsof doesn't work entirely correctly on ZFS on Linux. From the looks of things, the main issue here seems to be the replication behavior changing somehow from CORE. There is no replication information in the debug unfortunately. Are you doing the send/recv manually?