Root mount waiting for: CAM after upgrade from 12.0-U7 to 12.0-U8 on multiple systems

Description

System #1: HP 8300 SFF desktop, i5-3470, 24GB RAM, 16GB SSD boot drive connected to onboard SATA, LSI 9212-4i controller for pool of 3x4TB drives as a mirror pair with a hot spare

System #2: HP 8300 CMT desktop, i5-3470, 32GB RAM, 16GB SSD boot drive connected to onboard SATA, LSI 9211-8i controller for pool of 4x8TB drives as RaidZ2.

After upgrading from 12.0-U7 to 12.0-U8 and rebooting, the systems get stuck while booting with repeated messages at the console:

Lots of..

Root mount waiting for: CAM

And a handful of..

(aprobe0:ata2:0:0:0): INQUIRY. CDB: 12 00 00 00 24 00
(aprobe0:ata2:0:0:0): CAM status: Command timeout
(aprobe0:ata2:0:0:0): Retrying command, 0 more tries remain

and

(aprobe0:ata2:0:1:0): INQUIRY. CDB: 12 00 00 00 24 00
(aprobe0:ata2:0:1:0): CAM status: Command timeout
(aprobe0:ata2:0:1:0): Retrying command, 0 more tries remain

After ~10 minutes of this, booti is successful. This happened on the reboot after upgrade and every subsequent upgrade.

I originally thought this was due to soft-failing hardware (drive, cable, controller) on System #1, but it's way too coincidental that both systems exhibit the exact same symptoms after upgrade from U7 to U8.

ata2 is the controller where the boot disk is connected.

I'll attach a debug from one of the systems.

Problem/Justification

None

Impact

None

Activity

Show:

David Grabowski 
February 11, 2022 at 8:41 PM
(edited)

I was thinking the same thing - the possibility of the IDE-R functionality as part of Intel AMT. I was never able to get IDE-R to work correctly with these 8300s (I have others that aren't running TrueNAS)

I checked in the Intel AMT config on both systems, and IDE-R is not enabled. Even tried enabling and disabling and a reboot - same result.

The tunable works! Probably not worth digging into this any further. Thanks for looking into it for me.

Alexander Motin 
February 11, 2022 at 1:50 AM

According to "ata2: reset tp2 stat0=50 stat1=50 devices=0x30000", the ata(4) driver detected ATAPI devices as both master and slave on that ATA channel.  Some sources identify this atapci0 ATA controller device we see as "IDE-r", and I found such description for "IDE-r":

So I'd propose you to investigate iAMT or some other BMC on those systems.

Looking deeper into your logs I see that at least on December 8 boot that atapci0 device was not detected, so it is at least not a regression of ata(4) driver.

As the requested workaround setting such loader tunable should help: hint.atapci.0.disabled=1 .

David Grabowski 
February 10, 2022 at 8:14 PM

debug after enabling verbose boot logging

David Grabowski 
February 10, 2022 at 8:13 PM

Checked into BIOS - there's not a lot there to begin with. Nothing jumps out, and nothing has been changed since forever ago.

I enabled verbose booting and will upload a debug momentarily.

Reading the output of dmesg, it's making me wonder if there's a tunable that I can toss in that will simply skip the scanning of the (presumably phantom?) ata controller(s). While that doesn't technically solve the problem or answer why this happened all of a sudden, it would at least disable it.

David Grabowski 
February 10, 2022 at 6:05 PM

Hi - thank you for your replies.

Both systems are running BIOS version 3.07, which is one behind the latest (and presumably, last) version - 3.08. The only difference between 3.07 and 3.08 was a security fix for WiFi, and some intel microcode updates to fix other security vulnerabilities. Both systems have been running 3.07 since early 2019 when they were built.

NOTHING on either system has been changed on either system between the U7 and U8 updates. Both systems have been operating for a few years in this configuration. This issue didn't appear until the U8 upgrade.

These systems do not have any legacy ATA ports. They have four onboard SATA ports - only one is in use for the boot drive.

I'll reboot one of them now and check into the BIOS settings, and if that doesn't shed any light, I'll add the tunable you suggest and share the results.

Third Party to Resolve

Details

Assignee

Reporter

Labels

Fix versions

Affects versions

Priority

More fields

Katalon Platform

Created February 6, 2022 at 8:14 PM
Updated July 6, 2022 at 8:57 PM
Resolved February 14, 2022 at 3:17 PM