TrueNAS Core 12.0 Beta 1. It was installed from scratch and then had config imported. I don't think this behavior was present when it was updated from 11.3, but I am not sure. This might be an artefact of the db import.
After every boot, ssh host key changes.