Checking middlewared/zettarepl process shows file descriptor leak over time, which on our production system ended up eating all pipekva space. We got receiving the following kernel messages:

kern.ipc.maxpipekva exceeded; see tuning(7)

Restarting middlewared solves the problem temporary, but over time resources will again be exhausted.

The leak can be observer:

nas-a# ps axw|grep middlewared.*zettar
86105 - S 0:23.00 python3.7: middlewared (zettarepl) (python3.7)
91844 1 S+ 0:00.00 grep middlewared.*zettar
nas-a# procstat -f 86105 | wc -l
431
nas-a# procstat -f 86105 | wc -l
433
nas-a# procstat -f 86105 | wc -l
433
nas-a# procstat -f 86105 | wc -l
433
nas-a# procstat -f 86105 | wc -l
434
nas-a# procstat -f 86105 | wc -l
434
nas-a# procstat -f 86105 | wc -l
434
nas-a# procstat -f 86105 | wc -l
435
nas-a# procstat -f 86105 | wc -l
436
nas-a# procstat -f 86105 | wc -l
436
nas-a# procstat -f 86105 | wc -l
436
nas-a# procstat -f 86105 | wc -l
436
nas-a# procstat -f 86105 | wc -l
437
nas-a# procstat -f 86105 | wc -l
437

This has been captured during a replication task running, while sending multiple snapshots.