Low Encryption Performance on Atom Processors

Fixed by https://github.com/truenas/zfs/pull/95

hello, we have seen a visible improvement in performance with Ryan's fix.

The green line is before the fix with encryption on and the purple line is the performance after ryans fix. This is 1m sequential reads with a 1200G ADSS.

Here is the same test with writes:

The fix, as mentioned by Ryan Moeller, is to use the GPL-only symbols for FPU enter/exit which has a very obvious impact when looking at the time doing KFPU exit in the flame graphs. :

1200G 1mSEQ write before fix:

after fix:

However performance is still not up to the relative enc/unencrypted of core. In both situations we seem to be bottlenecked by the cpu on scale during reads even with the fix. examine the disck and cpu utiliz for scale: (red, green, and purple)

This includes iowait. I have kept it in here. for a better understanding of the definition of iowait see here: https://utcc.utoronto.ca/~cks/space/blog/unix/IowaitAndMultipleCPUs

we are also investigating a potential issue with the average iosize.

After discussing with Brian Behelendorf, I will not be attempting to upstream this fix. He says we’re welcome to carry the patch downstream, but would rather we not open that can of worms with an upstream PR.

My own patch for this looks promising in the benchmarks. may report more details on this. I am now discussing with upstream to gauge how receptive they would be to receiving this patch.

The patch in the issue comments changes the licensing of GPL-only symbols for FPU enter/exit in the Linux kernel so ZFS can use them, which we cannot distribute.