The following system core files were found: python3.9.core.

Thanks for the feedback - I've been able to dig that core tar out and have re-attached it if you'd like to do more inspection?

  thanks for the information and the screenshots. It's been very helpful. Unfortunately, the rar archive that you provided of the core dump seems to be corrupt. It has probably been truncated or was never a complete core dump file. Anyways, the other information you provided makes me suspect that this is fixed in 12.0-U7. I just fixed a ZFS bug where it was not handling inconsistent filesystems. (Inconsistent just means 1. the filesystem is in process of being destroyed or 2. it is in the process of being received). Please upgrade to that release one it's out and open a fresh ticket if you get any more core dumps.

Attached new memtest86 PRO tests - both machines ran 32 passes, all tests (including ECC error injections) are green. From what I understand, that should conclusively prove memory is working nominally as 32 passes guarantee exhausting the whole set of possible value combinations a given memory cell can have.

After a full week of troubleshooting, and at least 10 failed replications (of which at least 70% left the target pool in unusable state and 1 even managed to almost render the source pool unusable as well) I was able to replicate the pool to another machine - but not in a single pass. I was able to determine that the replication tasks seem to be crashing at particular points in replication, which led me to suspect a few datasets that seemed to have some (or all) of the above:

• nested datasets of different record sizes (parent dataset record size 1 MB, child dataset record size 16 kB)

• long paths (potentially exceeding 255 characters)

• "special" characters in filenames (a few examples would include Japanese chars in file names, or a character "squared" (superscripted symbol "2")

I cannot claim decidedly that any or all of these might have caused the crashes - but I do observe I was able to transfer these datasets (in isolation) through replication tasks (both local and remote) after trimming the path lengths down or removing the "special" chars from filenames (potentially from directory names as well, but I haven't been that observant as I was trimming these).

After the replication was successful, I ran scrubs on all the pools that participated in this (there is one source pool [RAID-Z2] and two target pools [one striped mirror, and one single-drive]) - no errors found on any.

One additional note - 4 passes of memtest86 PRO (v9.2) are good on each machine. I am doing a full 32 pass on each machine to fully exclude any memory issues as I write - will update the ticket as these finish.

Investigation continues - it seems some snapshots created yesterday are always tripping up the machine causing machine reboot. I have no idea what/how these are special but I was able to capture a screenshot of a recent kernel panic.

I was able to replicate 95% of the source pool and these small datasets seem to be causing some kind of issue with TrueNAS. I will continue the transfer with copying - this instability is not manageable.