I have TrueNAS12 Core installation joined to Active Directory and serving nfs with krb5i (auth+integrity). Clients are Linux with kernel 5.8. Directory listing works, but transfers of all except the tiniest files fail with input/output errors.
1) Transfers start with 70MB/s or so traffic, then clients hang, network traffic stops, but gssd service remains with high CPU usage on TrueNAS side for a while. Remounting on client side allows getting directory listing again, but further filetransfers hang too.
2) Switching to krb5 (auth only) fixes the transfers.
3) Switching to krb5p (auth+integrity+privacy) somewhat postpones the hang. Network traffic is present for longer.
4) Hangs also happen with SMB when mounted with krb5i.
5) I can not reproduce this with TrueNAS 12 running in KVM virtual machine on Intel i7 6850K CPU.
6) I can repoduce this running with TrueNAS12 on bare metal Intel Atom C3558 or in bhyve VM on the same CPU.
7) Tested with both aes128-cts-hmac-sha1-96 and aes256-cts-hmac-sha1-96
8) Same setup with FreeBSD 11.3 works fine on Intel Atom C3558.
I suspect this might be related to CPU. Intel Atom C3558 not only has AES-NI, but it also has SHA extensions for HW accelerated computation of SHA checksums. FreeBSD11 does not support using SHA extensions and uses software approach, but FreeBSD12 does use these extensions if CPU supports them. This might explain why FreeNAS11.3 works.
iXsystems sell TrueNAS Mini X which also has Intel Atom C3558 CPU. If you are going to try to reproduce this issue, don't forget to try on that CPU.