Uploaded image for project: 'TrueNAS'
  1. TrueNAS
  2. NAS-106928

zettarepl middlewared file descriptor leak

    XMLWordPrintable

    Details

    • Impact:
      Medium

      Description

      Checking middlewared/zettarepl process shows file descriptor leak over time, which on our production system ended up eating all pipekva space. We got receiving the following kernel messages:

      kern.ipc.maxpipekva exceeded; see tuning(7)
       

      Restarting middlewared solves the problem temporary, but over time resources will again be exhausted.

      The leak can be observer:

       

      nas-a# ps axw|grep middlewared.*zettar
      86105  -  S         0:23.00 python3.7: middlewared (zettarepl) (python3.7)
      91844  1  S+        0:00.00 grep middlewared.*zettar
      nas-a# procstat -f 86105 | wc -l
           431
      nas-a# procstat -f 86105 | wc -l
           433
      nas-a# procstat -f 86105 | wc -l
           433
      nas-a# procstat -f 86105 | wc -l
           433
      nas-a# procstat -f 86105 | wc -l
           434
      nas-a# procstat -f 86105 | wc -l
           434
      nas-a# procstat -f 86105 | wc -l
           434
      nas-a# procstat -f 86105 | wc -l
           435
      nas-a# procstat -f 86105 | wc -l
           436
      nas-a# procstat -f 86105 | wc -l
           436
      nas-a# procstat -f 86105 | wc -l
           436
      nas-a# procstat -f 86105 | wc -l
           436
      nas-a# procstat -f 86105 | wc -l
           437
      nas-a# procstat -f 86105 | wc -l
           437
       

      This has been captured during a replication task running, while sending multiple snapshots.

        Attachments

          Attachments

            JEditor

              Issue Links

                Activity

                  People

                  Assignee:
                  vladimirv Vladimir Vinogradenko
                  Reporter:
                  krichy Richard Kojedzinszky
                  Votes:
                  0 Vote for this issue
                  Watchers:
                  3 Start watching this issue

                    Dates

                    Created:
                    Updated:
                    Resolved: