Mini-XL - 11.2-U8 > 11.3-any Update Fails repeatedly

I have been working this problem (#OHV-435-27627) with Customer Service (Duane Broussard, Nick Bettencourt) for several months, and have been unable to find a solution.

I searched through the issue tracker, and NAS-105336 sounds a bit similar, however I was unable to restart middlewared when I finally got to a BSD login.

The machine is a stock IXSystems Mini-XL with MB replaced under warranty in December; 8 3Tb WD Reds; latest available 11.2 at that time.

• There are no jails.
• 8 SMB shares that apparently work properly
• Rsync is running between this server, and an older iXSystems Mini that has successfully updated to 11.3-U3.2

While the documentation is in the ticket, I will summarize again here:

• After letting the new MB run for several days, attempted update to 11.3-Release by changing trains and proceeding
• 11.3-Release failed by the boot process hanging somewhere in the “middlewared” loads. Repeated 2-3 times for confirmation by changing boot environment back to 11.2 and running the train update again.
• Downloaded the 11.3-Release to a USB and performed manual upgrade install. Same result.
• Returned to 11.2, as the server was needed
• Upgraded to 11.2-U8
• At this point I contacted customer support and demonstrated the complete sequence to Mr. Broussard using ASRock Rack and console redirection.
• Later saved config and performed a fresh install from the USB that was successful.
• Restoring the saved config crashed or hung the boot in some unknown state.
• Back to 11.2-U8
• Thinking the SATA-DOM might be bad, changed the boot drive to a spare 72 GB SSD.
• Performed an 11.3-Release train update. Failed
• Since the server was apparently fully functional on 11.2-U8, I tried the 11.3 updates through 11.3-U3.2 as they became available. All had the same result.
• Mr. Bettencourt reviewed the config file and could find nothing out of order.

Observations – Train upgrade

• Boot hangs at “middlewared: setting up plugins (pwenc) [2/35]”
• If I wait several minutes it will go to a FreeBSD login prompt
• After login there are numerous middlewared errors and tracebacks

Over the holiday weekend I will repeat a fresh installation from USB and reload the config. I will send you those results next week.