Alert - Failed to check for alert HasUpdate

21.06 PR: https://github.com/truenas/middleware/pull/6881

12.0 PR: https://github.com/truenas/middleware/pull/6869

I think I now understand what triggered these alerts and the sequence of events that lead here.

The server in question is used as a replication target, which lead to the main pool being set as a read-only file system. Originally the system dataset was on the freenas-boot pool, but I recently moved it to the main pool as the SSD that comprises the freenas-boot is getting too many write cycles. The root cause of the alerts was having the system dataset on a read-only file system.

Suggestion - the system should not allow the system dataset to be selected to a read-only file system. The system should also detect if the file system that holds the boot pool is later set to read-only.

Unfortunately, an attempt to create a debug results in the dialog:

[EFAULT] [EFAULT] Failed to generate debug file: mkdir: /var/db/system/ixdiagnose: Read-only file system

More info:
Error: Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/middlewared/job.py", line 367, in run
await self.future
File "/usr/local/lib/python3.8/site-packages/middlewared/job.py", line 405, in __run_body
rv = await self.middleware.run_in_thread(self.method, *([self] + args))
File "/usr/local/lib/python3.8/site-packages/middlewared/utils/run_in_thread.py", line 10, in run_in_thread
return await self.loop.run_in_executor(self.run_in_thread_executor, functools.partial(method, *args, **kwargs))
File "/usr/local/lib/python3.8/site-packages/middlewared/utils/io_thread_pool_executor.py", line 25, in run
result = self.fn(*self.args, **self.kwargs)
File "/usr/local/lib/python3.8/site-packages/middlewared/schema.py", line 977, in nf
return f(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/middlewared/plugins/system.py", line 785, in debug
raise CallError(debug_job.error)
middlewared.service_exception.CallError: [EFAULT] [EFAULT] Failed to generate debug file: mkdir: /var/db/system/ixdiagnose: Read-only file system

The debug that downloads has size zero.

The traceback complains about /var/db/system/ being read-only,

% ls -al /var/db | grep system
drwxr-xr-x 2 root wheel 2 Apr 4 19:54 system

root should be able to create a directory, but no other user. What user runs middleware?

/var/db/system is empty, which seems strange, as my other server, not yet updated to TN 12.0-U3, has many things in there

ls -al /var/db/system
total 2
drwxr-xr-x 2 root wheel 2 Apr 4 19:54 .
drwxr-xr-x 17 root wheel 1344 May 3 12:55 ..

Thank you for the report, . Can you please attach a debug file to this ticket? To generate a debug file on TrueNAS CORE, log in to the TrueNAS web interface, go to System > Advanced, then click SAVE DEBUG and wait for the file to download to your local system.