Encryption crashes during high activity
Description
Problem/Justification
Impact
relates to
SmartDraw Connector
Katalon Manual Tests (BETA)
Activity
Chris McLeod June 25, 2021 at 1:40 PM
Alexander,
I have hit this issue twice, once with 12.0u3 in early May and last night with 12.0u4. I see that the fix is due in 12.0u5 but I would like to avoid another outage between now and then. Are the attached files for the /boot/modules directory compatible with 12.0u4 and if so do you think it would be a good idea for me to use them?
Thanks,,
Alexander Motin May 24, 2021 at 11:58 PM(edited)
The patch is committed upstream. We'll merge it into U5. Thank you for testing.
Michele Lucini May 24, 2021 at 10:42 PM
Hi Alex, my bad, I noticed that I had connected to a standard FreeBSD box instead of the TrueNAS server, that's why the folder was empty.
Anyways, I've replaced the modules on the correct server and ran some tests over the last few days. The results are good, I didn't experience any crashes...
Alexander Motin May 22, 2021 at 2:04 AM(edited)
Michele, it is weird. There should be ~600 files in /boot/modules directory, and there should be about 29 lines in kldstat output. And openzfs.ko should already be in both. Are you talking about TrueNAS 12.0-U3 at all?
Michele Lucini May 22, 2021 at 12:23 AM(edited)
Hi Alex, thanks. I copied the modules into /boot/modules but there were no files in that folder, it was empty.
kldstat showed this output after a reboot:
Id Refs Address Size Name
1 3 0xffffffff80200000 227ae70 kernel
2 1 0xffffffff82a12000 1a20 fdescfs.ko
... so I did a "kldload openzfs.ko" and then the output changed to:
Id Refs Address Size Name
1 11 0xffffffff80200000 227ae70 kernel
2 1 0xffffffff82a12000 1a20 fdescfs.ko
3 1 0xffffffff82a14000 45831f openzfs.ko
Does this look right? I'll run a test now anyways. Memory was lowered to 4GBs.
I'm using this particular TrueNas 12.0u3 system as a backup system. It regularly crashes whenever I export VMs to a TrueNAS iSCSI volume. The underlying ZFS pool is encrypted. This TrueNAS instance is running inside HyperV.
The stacktrace shows this:
Apr 19 02:28:30 NAS2 panic: VERIFY3(0 == spa_do_crypt_abd(B_TRUE, spa, &zio->io_bookmark, BP_GET_TYPE(bp), BP_GET_DEDUP(bp), BP_SHOULD_BYTESWAP(bp), salt, iv, mac, psize, zio->io_abd,
eabd, &no_crypt)) failed (0 == 5)
Apr 19 02:28:30 NAS2 cpuid = 0
Apr 19 02:28:30 NAS2 time = 1618763221
Apr 19 02:28:30 NAS2 KDB: stack backtrace:
Apr 19 02:28:30 NAS2 db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe007e06e990
Apr 19 02:28:30 NAS2 vpanic() at vpanic+0x17b/frame 0xfffffe007e06e9e0
Apr 19 02:28:30 NAS2 spl_panic() at spl_panic+0x3a/frame 0xfffffe007e06ea40
Apr 19 02:28:30 NAS2 zio_encrypt() at zio_encrypt+0x543/frame 0xfffffe007e06ead0
Apr 19 02:28:30 NAS2 zio_execute() at zio_execute+0x6a/frame 0xfffffe007e06eb20
Apr 19 02:28:30 NAS2 taskqueue_run_locked() at taskqueue_run_locked+0x144/frame 0xfffffe007e06eb80
Apr 19 02:28:30 NAS2 taskqueue_thread_loop() at taskqueue_thread_loop+0xb6/frame 0xfffffe007e06ebb0
Apr 19 02:28:30 NAS2 fork_exit() at fork_exit+0x7e/frame 0xfffffe007e06ebf0
Apr 19 02:28:30 NAS2 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe007e06ebf0
Apr 19 02:28:30 NAS2 — trap 0, rip = 0, rsp = 0, rbp = 0 —
Apr 19 02:28:30 NAS2 KDB: enter: panic
Apr 19 02:28:30 NAS2 ---< >---
Apr 19 02:28:30 NAS2 Copyright (c) 1992-2020 The FreeBSD Project.
Apr 19 02:28:30 NAS2 Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
Apr 19 02:28:30 NAS2 The Regents of the University of California. All rights reserved.
Apr 19 02:28:30 NAS2 FreeBSD is a registered trademark of The FreeBSD Foundation.
Apr 19 02:28:30 NAS2 FreeBSD 12.2-RELEASE-p6 f2858df162b(HEAD) TRUENAS amd64