ZFS bad session hang during "ls" of PATH/.zfs/snapshot. dtrace attached.

SImple ordinary pool, a couple of levels deep, the sort of path I would "ls" without thinking twice on 11.3 to recover a lost file that's in a previous snapshot.

Except on 12-beta1 it hangs the session.

Having reported one bug (NAS-107049) related to zfs that turned out to relate to some kind of bug asserting holds during a send/recv, I thought maybe traversing to a snapshot might also involve holds, so it could be the same or similar bug. So I ran the same dtrace on it in one window, and ran "ls (snapshot dir)" in another window.

RESULT:

The moment I hit return after "ls", the session with "ls" hung, and dtrace started dumping text to the screen.

The attached screenshots show the commands used and outputs in both sessions.

The dtrace log is a bit different from the output of NAS-107049, so I don't know if it's the exact same bug. But if not, its a close cousin.  The dtrace command used was from that bug report, and a sample from the start of the dtrace log is below:

dtrace -n 'zfs-dbgmsg { print(stringof(arg0)); } set-error { stack(); }' > ~/debug2.txt

  7  74436                  none:zfs-dbgmsg string [ 'z', 'i', 'o', '.', 'c', ':', '2', '0', '0', '0', ':', 'z', 'i', 'o', '_', 'd', 'e', 'a', 'd', 'm', 'a', 'n', '_', 'i', 'm', 'p', 'l', '(', ')', ':', ' ', 's', 'l', 'o', 'w', ' ', 'z', 'i', 'o', '[', '3', ']', ':', ' ', 'z', 'i', 'o', '=', '0', 'x', 'f', 'f', 'f', 'f', 'f', '8', '3', 'd', '7', '0', '7', '4', '1', '4', 'b', '8', 'x', ' ', 't', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '=', '3', '4', '7', '6', '6', '0', '3', '0', '4', '7', '6', '8', '5', '2', ' ', 'd', 'e', 'l', 't', 'a', '=', '1', '3', '0', '7', '5', '0', '9', '3', '4', '8', '7', '2', '7', '1', ' ', 'q', 'u', 'e', 'u', 'e', 'd', '=', '0', ' ', 'i', 'o', '=', '3', '4', '7', '6', '6', '0', '3', '0', '4', '7', '8', '2', '9', '4', ' ', 'p', 'a', 't', 'h', '=', '/', 'd', 'e', 'v', '/', 'n', 'v', 'd', '1', 'p', '1', ' ', 'l', 'a', 's', 't', '=', '2', '4', '3', '6', '8', '1', '9', '4', '9', '3', '8', '3', '7', '9', ' ', 't', 'y', 'p', 'e', '=', '1', ' ', 'p', 'r', 'i', 'o', 'r', 'i', 't', 'y', '=', '0', ' ', 'f', 'l', 'a', 'g', 's', '=', '0', 'x', '1', '8', '0', '8', '8', '0', ' ', 's', 't', 'a', 'g', 'e', '=', '0', 'x', '1', '0', '0', '0', '0', '0', ' ', 'p', 'i', 'p', 'e', 'l', 'i', 'n', 'e', '=', '0', 'x', '1', 'f', '0', '0', '0', '0', '0', ' ', 'p', 'i', 'p', 'e', 'l', 'i', 'n', 'e', '-', 't', 'r' ]

  7  74436                  none:zfs-dbgmsg string "zio.c:2035:zio_deadman(): zio_wait waiting for hung I/O to pool 'Main_pool'"

  2  74396                   none:set-error
              openzfs.ko`zfsdev_ioctl_common+0x470
              openzfs.ko`zfsdev_ioctl+0x29e
              kernel`devfs_ioctl+0xb0
              kernel`VOP_IOCTL_APV+0x7b
              kernel`vn_ioctl+0x16a
              kernel`devfs_ioctl_f+0x1e
              kernel`kern_ioctl+0x2b7
              kernel`sys_ioctl+0xfa
              kernel`amd64_syscall+0x387
              kernel`0xffffffff80fa50e0

  2  74396                   none:set-error
              openzfs.ko`vdev_config_generate+0x3ab
              openzfs.ko`spa_config_generate+0x2a5
              openzfs.ko`spa_open_common+0x214
              openzfs.ko`spa_get_stats+0x4e
              openzfs.ko`zfs_ioc_pool_stats+0x22
              openzfs.ko`zfsdev_ioctl_common+0x470
              openzfs.ko`zfsdev_ioctl+0x29e
              kernel`devfs_ioctl+0xb0
              kernel`VOP_IOCTL_APV+0x7b
              kernel`vn_ioctl+0x16a
              kernel`devfs_ioctl_f+0x1e
              kernel`kern_ioctl+0x2b7
              kernel`sys_ioctl+0xfa
              kernel`amd64_syscall+0x387
              kernel`0xffffffff80fa50e0

  2  74396                   none:set-error
              openzfs.ko`vdev_config_generate+0x3d6
              openzfs.ko`spa_config_generate+0x2a5
              openzfs.ko`spa_open_common+0x214
              openzfs.ko`spa_get_stats+0x4e
              openzfs.ko`zfs_ioc_pool_stats+0x22
              openzfs.ko`zfsdev_ioctl_common+0x470
              openzfs.ko`zfsdev_ioctl+0x29e
              kernel`devfs_ioctl+0xb0
              kernel`VOP_IOCTL_APV+0x7b
              kernel`vn_ioctl+0x16a
              kernel`devfs_ioctl_f+0x1e
              kernel`kern_ioctl+0x2b7
              kernel`sys_ioctl+0xfa
              kernel`amd64_syscall+0x387
              kernel`0xffffffff80fa50e0

  2  74396                   none:set-error
              openzfs.ko`zfs_ioc_pool_stats+0x36
              openzfs.ko`zfsdev_ioctl_common+0x470
              openzfs.ko`zfsdev_ioctl+0x29e
              kernel`devfs_ioctl+0xb0
              kernel`VOP_IOCTL_APV+0x7b
              kernel`vn_ioctl+0x16a
              kernel`devfs_ioctl_f+0x1e
              kernel`kern_ioctl+0x2b7
              kernel`sys_ioctl+0xfa
              kernel`amd64_syscall+0x387
              kernel`0xffffffff80fa50e0

  2  74396                   none:set-error
              openzfs.ko`vdev_config_generate+0x3ab
              openzfs.ko`spa_config_generate+0x2a5
              openzfs.ko`spa_open_common+0x214
              openzfs.ko`spa_get_stats+0x4e
              openzfs.ko`zfs_ioc_pool_stats+0x22
              openzfs.ko`zfsdev_ioctl_common+0x470
              openzfs.ko`zfsdev_ioctl+0x29e
              kernel`devfs_ioctl+0xb0
              kernel`VOP_IOCTL_APV+0x7b
              kernel`vn_ioctl+0x16a
              kernel`devfs_ioctl_f+0x1e
              kernel`kern_ioctl+0x2b7
              kernel`sys_ioctl+0xfa
              kernel`amd64_syscall+0x387
              kernel`0xffffffff80fa50e0

  2  74396                   none:set-error
              openzfs.ko`vdev_config_generate+0x3d6
              openzfs.ko`spa_config_generate+0x2a5
              openzfs.ko`spa_open_common+0x214
              openzfs.ko`spa_get_stats+0x4e
              openzfs.ko`zfs_ioc_pool_stats+0x22
              openzfs.ko`zfsdev_ioctl_common+0x470
              openzfs.ko`zfsdev_ioctl+0x29e
              kernel`devfs_ioctl+0xb0
              kernel`VOP_IOCTL_APV+0x7b
              kernel`vn_ioctl+0x16a
              kernel`devfs_ioctl_f+0x1e
              kernel`kern_ioctl+0x2b7
              kernel`sys_ioctl+0xfa
              kernel`amd64_syscall+0x387
              kernel`0xffffffff80fa50e0