I was unable to log in to the web UI, so I ssh'd in and got this after logging in
Traceback (most recent call last): File "/usr/local/sbin/hactl", line 171, in <module> main(args.command, args.q) File "/usr/local/sbin/hactl", line 17, in main client = Client() File "/usr/local/lib/python3.8/site-packages/middlewared/client/client.py", line 281, in __init__ self._ws.connect() File "/usr/local/lib/python3.8/site-packages/middlewared/client/client.py", line 124, in connect rv = super(WSClient, self).connect() File "/usr/local/lib/python3.8/site-packages/ws4py/client/__init__.py", line 216, in connect self.sock.connect(self.bind_addr) ConnectionRefusedError: [Errno 61] Connection refused
So I restarted the middlewared from the shell, and the UI came back, then I had this pop up as an alert
The following system core files were found: python3.8.core. Please create a ticket at https://jira.ixsystems.com/ and attach the relevant core files along with a system debug. Once the core files have been archived and attached to the ticket, they may be removed by running the following command in shell: 'rm /var/db/system/cores/*'.
I grabbed the core file from the server but it's 2.8 GB (400 MB compressed) so I can't upload it here. Happy to provide it though if there's a file host you can recommend.
My knowledge of gdb is minimal but here's what I was able to get from it
/var/db/system/cores # gdb python3.8 -c python3.8.core GNU gdb (GDB) 9.1 [GDB v9.1 for FreeBSD] Copyright (C) 2020 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-portbld-freebsd12.0". Type "show configuration" for configuration details. For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from python3.8... (No debugging symbols found in python3.8) [New LWP 102127] [New LWP 101406] [New LWP 101518] [New LWP 101519] [New LWP 101520] [New LWP 101521] [New LWP 101522] [New LWP 101523] [New LWP 101524] [New LWP 101525] [New LWP 101526] [New LWP 101527] [New LWP 101528] [New LWP 101529] [New LWP 101530] [New LWP 101531] [New LWP 101532] [New LWP 101533] [New LWP 101534] [New LWP 101535] [New LWP 101536] [New LWP 101537] [New LWP 101540] [New LWP 101544] [New LWP 101545] [New LWP 101575] [New LWP 101605] [New LWP 101608] [New LWP 101661] [New LWP 101728] [New LWP 102315] [New LWP 103826] [New LWP 105449] [New LWP 105557] [New LWP 105170] [New LWP 105171] [New LWP 105375] Core was generated by `python3.8: middlewared'. Program terminated with signal SIGSEGV, Segmentation fault. #0 0x00000008199444d7 in ?? () from /usr/local/lib/libglib-2.0.so.0 [Current thread is 1 (LWP 102127)]