After a day, many zfs receives all hung
Description
Problem/Justification
Impact
SmartDraw Connector
Katalon Manual Tests (BETA)
Activity

There is nothing actionable we can do with this and your other tickets without a debug unfortunately but wait for a similar report with a debug. If we can get a debug we can reopen.

Replication system uptime: 5 minutes.
Status: zfs receive is already hung.

And I cannot reboot the receiver, it must be power cycled:
init: some processes would not die; ps axl advised
except that running "ps" is not possible at this point.

This is what it looks like on the sender:
0 3484 85221 0 20 0 0 0 - Z - 0:00.26 <defunct>
0 3525 85221 0 20 0 0 0 - Z - 0:00.29 <defunct>
0 13692 85221 0 20 0 0 0 - Z - 0:00.26 <defunct>
0 15628 85221 0 20 0 0 0 - Z - 0:00.26 <defunct>
0 16067 85221 0 20 0 0 0 - Z - 0:00.26 <defunct>
0 27077 85221 0 20 0 0 0 - Z - 0:00.26 <defunct>
0 27756 85221 0 20 0 0 0 - Z - 0:00.26 <defunct>
0 28480 85221 0 20 0 0 0 - Z - 0:00.32 <defunct>
0 40133 85221 0 20 0 0 0 - Z - 0:00.26 <defunct>
0 40559 85221 0 20 0 0 0 - Z - 0:00.26 <defunct>
0 40634 85221 0 20 0 0 0 - Z - 0:00.27 <defunct>
0 52244 85221 0 52 0 9936 4136 pipewr I - 0:00.16 zfs: sending tank/shared@auto-20200210.0900-2w (0%: 8416568/587516616176) (zfs)
0 52245 85221 0 52 0 7028 2796 wait I - 0:00.00 /bin/sh -c /usr/local/bin/lz4c | /usr/local/bin/pipewatcher $$ | /usr/local/bin/ssh -c aes128-ctr,aes192-ctr,aes256-ctr -i /data/
0 53535 85221 0 20 0 0 0 - Z - 0:00.26 <defunct>
0 54068 85221 0 20 0 0 0 - Z - 0:00.28 <defunct>
0 65769 85221 0 20 0 0 0 - Z - 0:00.26 <defunct>
0 66269 85221 0 20 0 0 0 - Z - 0:00.25 <defunct>
0 78051 85221 0 20 0 0 0 - Z - 0:00.26 <defunct>
0 78540 85221 0 20 0 0 0 - Z - 0:00.29 <defunct>
0 85221 85220 0 52 0 65656 59772 wait Is - 0:26.47 python /usr/local/www/freenasUI/tools/autorepl.py (python3.6)
0 85388 85221 0 20 0 0 0 - Z - 0:05.36 <defunct>
0 90144 85221 0 20 0 0 0 - Z - 0:00.26 <defunct>
0 90577 85221 0 20 0 0 0 - Z - 0:00.25 <defunct>
0 99066 85221 0 20 0 0 0 - Z - 0:05.27 <defunct>
0 59018 58077 0 20 0 6704 2676 piperd S+ 0 0:00.00 grep 85221
Details
Details
Assignee

Reporter

Oberserved just now (guess it is time to reboot the replication destination again):
16 - DL 0:09.97 [zfskern]
1705 - Is 0:00.03 /usr/sbin/zfsd
4415 - Is 0:00.02 csh -c /usr/bin/env lz4c -d | /sbin/zfs receive -F -d 'tank' && echo Succeeded
4418 - D 0:06.40 /sbin/zfs receive -F -d tank
5331 - Is 0:00.02 csh -c /usr/bin/env lz4c -d | /sbin/zfs receive -F -d 'tank' && echo Succeeded
5334 - D 0:00.01 /sbin/zfs receive -F -d tank
6242 - Is 0:00.02 csh -c /usr/bin/env lz4c -d | /sbin/zfs receive -F -d 'tank' && echo Succeeded
6245 - D 0:00.01 /sbin/zfs receive -F -d tank
7148 - Is 0:00.02 csh -c /usr/bin/env lz4c -d | /sbin/zfs receive -F -d 'tank' && echo Succeeded
7151 - D 0:00.01 /sbin/zfs receive -F -d tank
8046 - Is 0:00.02 csh -c /usr/bin/env lz4c -d | /sbin/zfs receive -F -d 'tank' && echo Succeeded
8049 - D 0:00.00 /sbin/zfs receive -F -d tank
8959 - Is 0:00.02 csh -c /usr/bin/env lz4c -d | /sbin/zfs receive -F -d 'tank' && echo Succeeded
8962 - D 0:00.01 /sbin/zfs receive -F -d tank
9858 - Is 0:00.02 csh -c /usr/bin/env lz4c -d | /sbin/zfs receive -F -d 'tank' && echo Succeeded
9861 - D 0:00.00 /sbin/zfs receive -F -d tank
10757 - Is 0:00.02 csh -c /usr/bin/env lz4c -d | /sbin/zfs receive -F -d 'tank' && echo Succeeded
10760 - D 0:00.01 /sbin/zfs receive -F -d tank
11662 - Is 0:00.02 csh -c /usr/bin/env lz4c -d | /sbin/zfs receive -F -d 'tank' && echo Succeeded
11665 - D 0:00.00 /sbin/zfs receive -F -d tank
12639 - Is 0:00.02 csh -c /usr/bin/env lz4c -d | /sbin/zfs receive -F -d 'tank' && echo Succeeded
12642 - D 0:00.01 /sbin/zfs receive -F -d tank
13548 - Is 0:00.02 csh -c /usr/bin/env lz4c -d | /sbin/zfs receive -F -d 'tank' && echo Succeeded
13551 - D 0:00.01 /sbin/zfs receive -F -d tank
14432 - Is 0:00.02 csh -c /usr/bin/env lz4c -d | /sbin/zfs receive -F -d 'tank' && echo Succeeded
14435 - D 0:00.00 /sbin/zfs receive -F -d tank
15376 - Is 0:00.02 csh -c /usr/bin/env lz4c -d | /sbin/zfs receive -F -d 'tank' && echo Succeeded
15379 - D 0:00.01 /sbin/zfs receive -F -d tank
16340 - Is 0:00.02 csh -c /usr/bin/env lz4c -d | /sbin/zfs receive -F -d 'tank' && echo Succeeded
16343 - D 0:00.01 /sbin/zfs receive -F -d tank
17250 - Is 0:00.02 csh -c /usr/bin/env lz4c -d | /sbin/zfs receive -F -d 'tank' && echo Succeeded
17253 - D 0:00.01 /sbin/zfs receive -F -d tank
18151 - Is 0:00.02 csh -c /usr/bin/env lz4c -d | /sbin/zfs receive -F -d 'tank' && echo Succeeded
18154 - D 0:00.01 /sbin/zfs receive -F -d tank
19075 - Is 0:00.02 csh -c /usr/bin/env lz4c -d | /sbin/zfs receive -F -d 'tank' && echo Succeeded
19078 - D 0:00.01 /sbin/zfs receive -F -d tank
19988 - Is 0:00.02 csh -c /usr/bin/env lz4c -d | /sbin/zfs receive -F -d 'tank' && echo Succeeded
19991 - D 0:00.01 /sbin/zfs receive -F -d tank
20882 - Is 0:00.02 csh -c /usr/bin/env lz4c -d | /sbin/zfs receive -F -d 'tank' && echo Succeeded
20885 - D 0:00.01 /sbin/zfs receive -F -d tank
21799 - Is 0:00.02 csh -c /usr/bin/env lz4c -d | /sbin/zfs receive -F -d 'tank' && echo Succeeded
21802 - D 0:00.01 /sbin/zfs receive -F -d tank
22709 - Is 0:00.02 csh -c /usr/bin/env lz4c -d | /sbin/zfs receive -F -d 'tank' && echo Succeeded
22712 - D 0:00.01 /sbin/zfs receive -F -d tank
23124 0 S+ 0:00.00 grep zfs