nscd and rrdcached constantly write to boot-pool instead of system dataset

I originally posted this in the TrueNAS SCALE forum.

On TrueNAS SCALE 21.02 I noticed that my boot-pool was constantly being written to even after moving the system dataset off of it. This is particularly important in the unsupported USB boot-pool case.

Looking at iotop and lsof a few things seem to still be constantly writing to the boot-pool even after moving the system dataset and rebooting:

15:55:02 50648 be/4 root 0.00 K/s 398.96 K/s 0.00 % 0.00 % rrdcached -j /var/lib/rrdcached/journal/ -p /var/run/rrdcached.pid -l unix:/var/run/rrdcached.sock
15:55:12 50648 be/4 root 0.00 K/s 544.28 K/s 0.00 % 0.00 % rrdcached -j /var/lib/rrdcached/journal/ -p /var/run/rrdcached.pid -l unix:/var/run/rrdcached.sock
15:55:13 6674 be/4 root 0.00 K/s 46.14 K/s 0.00 % 0.00 % nscd
15:55:19 6678 be/4 root 5.70 K/s 22.79 K/s 0.00 % 0.00 % nscd
15:55:22 50648 be/4 root 0.00 K/s 567.15 K/s 0.00 % 0.00 % rrdcached -j /var/lib/rrdcached/journal/ -p /var/run/rrdcached.pid -l unix:/var/run/rrdcached.sock
15:55:32 50648 be/4 root 0.00 K/s 185.62 K/s 0.00 % 0.00 % rrdcached -j /var/lib/rrdcached/journal/ -p /var/run/rrdcached.pid -l unix:/var/run/rrdcached.sock
15:55:32 50648 be/4 root 0.00 K/s 360.99 K/s 0.00 % 0.00 % rrdcached -j /var/lib/rrdcached/journal/ -p /var/run/rrdcached.pid -l unix:/var/run/rrdcached.sock
15:55:42 50648 be/4 root 0.00 K/s 384.96 K/s 0.00 % 0.00 % rrdcached -j /var/lib/rrdcached/journal/ -p /var/run/rrdcached.pid -l unix:/var/run/rrdcached.sock
15:55:52 50648 be/4 root 0.00 K/s 47.85 K/s 0.00 % 0.00 % rrdcached -j /var/lib/rrdcached/journal/ -p /var/run/rrdcached.pid -l unix:/var/run/rrdcached.sock
15:55:52 50648 be/4 root 0.00 K/s 398.93 K/s 0.00 % 0.00 % rrdcached -j /var/lib/rrdcached/journal/ -p /var/run/rrdcached.pid -l unix:/var/run/rrdcached.sock
15:55:54 6674 be/4 root 0.00 K/s 56.71 K/s 0.00 % 0.00 % nscd
15:55:59 6680 be/4 root 7.47 K/s 0.00 K/s 0.00 % 0.00 % nscd

lsof shows the following files on boot-pool open for write:

asyncio_l 1274 0 23u REG 0,24 1097728 309 /data/freenas-v1.db
asyncio_l 1274 0 41w REG 0,24 0 131736 /root/syslog/failover.log

dhclient 3809 0 7w REG 0,24 1084 131859 /var/lib/dhcp/dhclient.leases.enp5s0

nscd 6667 0 6u REG 0,24 217032 131590 /var/cache/nscd/passwd
nscd 6667 0 7r REG 0,24 217032 131590 /var/cache/nscd/passwd
nscd 6667 0 8u REG 0,24 217032 131591 /var/cache/nscd/group
nscd 6667 0 9r REG 0,24 217032 131591 /var/cache/nscd/group
nscd 6667 0 10u REG 0,24 217032 131592 /var/cache/nscd/hosts
nscd 6667 0 11r REG 0,24 217032 131592 /var/cache/nscd/hosts
nscd 6667 0 12u REG 0,24 217032 131593 /var/cache/nscd/services
nscd 6667 0 13r REG 0,24 217032 131593 /var/cache/nscd/services
nscd 6667 0 14u REG 0,24 217032 131594 /var/cache/nscd/netgroup
nscd 6667 0 15r REG 0,24 217032 131594 /var/cache/nscd/netgroup

syslog-ng 6771 0 10u REG 0,24 16384 132228 /var/lib/syslog-ng/syslog-ng.persist

container 6777 0 3uW REG 0,24 524288 131596 /var/lib/containerd/io.containerd.metadata.v1.bolt/meta.db

rrdcached 7031 0 3w REG 0,24 10321920 132112 /var/lib/rrdcached/journal/rrd.journal.1618516563.583758

So the boot-pool appears to still be constantly written to by nscd and rrdcached.

There is already a /var/db/system/rrd-(uuid) directory that the rrd files probably could be moved to. nscd might need a separate system directory setup.