Uploaded image for project: 'TrueNAS'
  1. TrueNAS
  2. NAS-107138

ZFS bad session hang during "ls" of PATH/.zfs/snapshot. dtrace attached.

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Engineering Closed (View Workflow)
    • Priority: Low
    • Resolution: Duplicate
    • Affects Version/s: 12.0-BETA1
    • Fix Version/s: N/A
    • Component/s: None
    • Labels:

      Description

      SImple ordinary pool, a couple of levels deep, the sort of path I would "ls" without thinking twice on 11.3 to recover a lost file that's in a previous snapshot.

      Except on 12-beta1 it hangs the session.

      Having reported one bug (NAS-107049) related to zfs that turned out to relate to some kind of bug asserting holds during a send/recv, I thought maybe traversing to a snapshot might also involve holds, so it could be the same or similar bug. So I ran the same dtrace on it in one window, and ran "ls (snapshot dir)" in another window.

       

      RESULT:

      The moment I hit return after "ls", the session with "ls" hung, and dtrace started dumping text to the screen.

      The attached screenshots show the commands used and outputs in both sessions.

      The dtrace log is a bit different from the output of NAS-107049, so I don't know if it's the exact same bug. But if not, its a close cousin.  The dtrace command used was from that bug report, and a sample from the start of the dtrace log is below:

      dtrace -n 'zfs-dbgmsg { print(stringof(arg0)); } set-error { stack(); }' > ~/debug2.txt

       

        7  74436                  none:zfs-dbgmsg string [ 'z', 'i', 'o', '.', 'c', ':', '2', '0', '0', '0', ':', 'z', 'i', 'o', '_', 'd', 'e', 'a', 'd', 'm', 'a', 'n', '_', 'i', 'm', 'p', 'l', '(', ')', ':', ' ', 's', 'l', 'o', 'w', ' ', 'z', 'i', 'o', '[', '3', ']', ':', ' ', 'z', 'i', 'o', '=', '0', 'x', 'f', 'f', 'f', 'f', 'f', '8', '3', 'd', '7', '0', '7', '4', '1', '4', 'b', '8', 'x', ' ', 't', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '=', '3', '4', '7', '6', '6', '0', '3', '0', '4', '7', '6', '8', '5', '2', ' ', 'd', 'e', 'l', 't', 'a', '=', '1', '3', '0', '7', '5', '0', '9', '3', '4', '8', '7', '2', '7', '1', ' ', 'q', 'u', 'e', 'u', 'e', 'd', '=', '0', ' ', 'i', 'o', '=', '3', '4', '7', '6', '6', '0', '3', '0', '4', '7', '8', '2', '9', '4', ' ', 'p', 'a', 't', 'h', '=', '/', 'd', 'e', 'v', '/', 'n', 'v', 'd', '1', 'p', '1', ' ', 'l', 'a', 's', 't', '=', '2', '4', '3', '6', '8', '1', '9', '4', '9', '3', '8', '3', '7', '9', ' ', 't', 'y', 'p', 'e', '=', '1', ' ', 'p', 'r', 'i', 'o', 'r', 'i', 't', 'y', '=', '0', ' ', 'f', 'l', 'a', 'g', 's', '=', '0', 'x', '1', '8', '0', '8', '8', '0', ' ', 's', 't', 'a', 'g', 'e', '=', '0', 'x', '1', '0', '0', '0', '0', '0', ' ', 'p', 'i', 'p', 'e', 'l', 'i', 'n', 'e', '=', '0', 'x', '1', 'f', '0', '0', '0', '0', '0', ' ', 'p', 'i', 'p', 'e', 'l', 'i', 'n', 'e', '-', 't', 'r' ]
      
        7  74436                  none:zfs-dbgmsg string "zio.c:2035:zio_deadman(): zio_wait waiting for hung I/O to pool 'Main_pool'"
      
        2  74396                   none:set-error
                    openzfs.ko`zfsdev_ioctl_common+0x470
                    openzfs.ko`zfsdev_ioctl+0x29e
                    kernel`devfs_ioctl+0xb0
                    kernel`VOP_IOCTL_APV+0x7b
                    kernel`vn_ioctl+0x16a
                    kernel`devfs_ioctl_f+0x1e
                    kernel`kern_ioctl+0x2b7
                    kernel`sys_ioctl+0xfa
                    kernel`amd64_syscall+0x387
                    kernel`0xffffffff80fa50e0
      
        2  74396                   none:set-error
                    openzfs.ko`vdev_config_generate+0x3ab
                    openzfs.ko`spa_config_generate+0x2a5
                    openzfs.ko`spa_open_common+0x214
                    openzfs.ko`spa_get_stats+0x4e
                    openzfs.ko`zfs_ioc_pool_stats+0x22
                    openzfs.ko`zfsdev_ioctl_common+0x470
                    openzfs.ko`zfsdev_ioctl+0x29e
                    kernel`devfs_ioctl+0xb0
                    kernel`VOP_IOCTL_APV+0x7b
                    kernel`vn_ioctl+0x16a
                    kernel`devfs_ioctl_f+0x1e
                    kernel`kern_ioctl+0x2b7
                    kernel`sys_ioctl+0xfa
                    kernel`amd64_syscall+0x387
                    kernel`0xffffffff80fa50e0
      
        2  74396                   none:set-error
                    openzfs.ko`vdev_config_generate+0x3d6
                    openzfs.ko`spa_config_generate+0x2a5
                    openzfs.ko`spa_open_common+0x214
                    openzfs.ko`spa_get_stats+0x4e
                    openzfs.ko`zfs_ioc_pool_stats+0x22
                    openzfs.ko`zfsdev_ioctl_common+0x470
                    openzfs.ko`zfsdev_ioctl+0x29e
                    kernel`devfs_ioctl+0xb0
                    kernel`VOP_IOCTL_APV+0x7b
                    kernel`vn_ioctl+0x16a
                    kernel`devfs_ioctl_f+0x1e
                    kernel`kern_ioctl+0x2b7
                    kernel`sys_ioctl+0xfa
                    kernel`amd64_syscall+0x387
                    kernel`0xffffffff80fa50e0
      
        2  74396                   none:set-error
                    openzfs.ko`zfs_ioc_pool_stats+0x36
                    openzfs.ko`zfsdev_ioctl_common+0x470
                    openzfs.ko`zfsdev_ioctl+0x29e
                    kernel`devfs_ioctl+0xb0
                    kernel`VOP_IOCTL_APV+0x7b
                    kernel`vn_ioctl+0x16a
                    kernel`devfs_ioctl_f+0x1e
                    kernel`kern_ioctl+0x2b7
                    kernel`sys_ioctl+0xfa
                    kernel`amd64_syscall+0x387
                    kernel`0xffffffff80fa50e0
      
        2  74396                   none:set-error
                    openzfs.ko`vdev_config_generate+0x3ab
                    openzfs.ko`spa_config_generate+0x2a5
                    openzfs.ko`spa_open_common+0x214
                    openzfs.ko`spa_get_stats+0x4e
                    openzfs.ko`zfs_ioc_pool_stats+0x22
                    openzfs.ko`zfsdev_ioctl_common+0x470
                    openzfs.ko`zfsdev_ioctl+0x29e
                    kernel`devfs_ioctl+0xb0
                    kernel`VOP_IOCTL_APV+0x7b
                    kernel`vn_ioctl+0x16a
                    kernel`devfs_ioctl_f+0x1e
                    kernel`kern_ioctl+0x2b7
                    kernel`sys_ioctl+0xfa
                    kernel`amd64_syscall+0x387
                    kernel`0xffffffff80fa50e0
      
        2  74396                   none:set-error
                    openzfs.ko`vdev_config_generate+0x3d6
                    openzfs.ko`spa_config_generate+0x2a5
                    openzfs.ko`spa_open_common+0x214
                    openzfs.ko`spa_get_stats+0x4e
                    openzfs.ko`zfs_ioc_pool_stats+0x22
                    openzfs.ko`zfsdev_ioctl_common+0x470
                    openzfs.ko`zfsdev_ioctl+0x29e
                    kernel`devfs_ioctl+0xb0
                    kernel`VOP_IOCTL_APV+0x7b
                    kernel`vn_ioctl+0x16a
                    kernel`devfs_ioctl_f+0x1e
                    kernel`kern_ioctl+0x2b7
                    kernel`sys_ioctl+0xfa
                    kernel`amd64_syscall+0x387
                    kernel`0xffffffff80fa50e0
      
      
      
      
      
      
      

        Attachments

          Attachments

            JEditor

              Issue Links

                Activity

                  People

                  Assignee:
                  releng Triage Team
                  Reporter:
                  Stilez Stilez y
                  Votes:
                  0 Vote for this issue
                  Watchers:
                  3 Start watching this issue

                    Dates

                    Created:
                    Updated:
                    Resolved: