SCALE 22.02.0.1 - import pools on boot hangs

Description

Community post here . This also appears to be very similar to ticket .

Current summary/status:

After upgrading to SCALE 22.02.01, booting to either SCALE 22.02.0.1 or SCALE 22.02.0 results in the boot process being stuck at "A start job is running for Import ZFS pools" for the full 15m8s allowed. It appears that all pools are imported and mounted after this, however this is not ideal. In addition, all installed Apps are missing (probably due to the long boot time) and sharing services (SMB/NFS) are stopped.

Problem/Justification

None

Impact

None

Activity

Ian Cartwright 
April 1, 2022 at 12:20 AM

That is correct. I deleted the datasets on stank/local_backups. The boot process now takes a reasonable amount of time (about 90 seconds). I now have an issue where the apps subsystem won't start up properly on boot (see my comments from 25 March).

Muhammad 
March 31, 2022 at 7:31 PM

can you please confirm if you delete the backed up dataset of ix-applications/docker, does the importing pool works fine and not time out after 15min?

Ian Cartwright 
March 31, 2022 at 6:01 PM

Here is my setup for pools, snapshots, and replications:

I have three pools:

  • apps: (2x SSD in Mirror) - this is the system pool, the Apps pool and the VM pool (all in different datasets)

  • tank: (4x HDD in RAIDZ1) - this is where all of my "permanent" data lives (media, documents, etc.)

  • stank: (4x HDD in RAIDZ1) - this is my "scratch" pool for temporary files, backups, time machine for my Macs, etc.

I have set up snapshot/replication tasks as follows:

  • apps/vm/Homeseer_xxxxx: local replication to stank/local_backups/vm_Homeseer (Windows VM)

  • apps/ix-applications: local replication to stank/local_backups/apps_ix-applications (the standard dataset created by SCALE for apps)

  • apps/app_data: local replication to stank/local_backups/apps_app-data (dataset I created for all my hostpath storage for apps)

 

As for the ix-applications replication specifically, see the below screenshot.

Muhammad 
March 31, 2022 at 4:44 PM

can you please share where you are replicating your ix-applications dataset if you are replicating it(by where i mean which pool, if you can share the details about that replication task - that would be best)? It seems we have seen this bug before already and this might be a duplicate..

Ian Cartwright 
March 25, 2022 at 10:31 PM

By using the following incantation, I am able to get my apps to start. This does not survive a reboot though, so I still need some help figuring this out.

zfs set mountpoint=legacy apps/ix-applications/k3s/kubelet
mount -t zfs apps/ix-applications/k3s/kubelet /var/lib/kubelet
zfs set mountpoint=legacy $(zfs list -o name | grep ix-applications | grep pvc)
midclt call service.stop kubernetes
midclt call service.start kubernetes

This was put together from various sources including (restarting k8s), (remounting PVCs), (figuring mountpoint=legacy could apply to /var/lib/kubelet since that's the error I was getting in middleware.log)

Duplicate

Details

Assignee

Reporter

Labels

Impact

Time remaining

0m

Components

Fix versions

Affects versions

Priority

Katalon Platform

Created March 24, 2022 at 3:07 PM
Updated July 6, 2022 at 8:58 PM
Resolved April 3, 2022 at 6:48 PM