Uploaded image for project: 'TrueNAS'
  1. TrueNAS
  2. NAS-107957

middlewared crashing too many files no login

    XMLWordPrintable

    Details

      Description

      Once this service crashes, unable to login but SSH works.

       

      From /var/log/middlewared.log: (this is spamming it and creating 10MB log files within 30 seconds)

      OSError: [Errno 24] Too many open files
      [2020/10/15 14:58:20] (ERROR) asyncio.default_exception_handler():1707 - socket.accept() out of system resource
      socket: <asyncio.TransportSocket fd=50, family=AddressFamily.AF_UNIX, type=SocketKind.SOCK_STREAM, proto=0, laddr=/var/run/middlewared.sock>
      Traceback (most recent call last):
        File "/usr/lib/python3.8/asyncio/selector_events.py", line 162, in _accept_connection
        File "/usr/lib/python3.8/socket.py", line 292, in accept
      OSError: [Errno 24] Too many open files
      [2020/10/15 14:58:20] (ERROR) asyncio.default_exception_handler():1707 - socket.accept() out of system resource
      socket: <asyncio.TransportSocket fd=50, family=AddressFamily.AF_UNIX, type=SocketKind.SOCK_STREAM, proto=0, laddr=/var/run/middlewared.sock>
      Traceback (most recent call last):
        File "/usr/lib/python3.8/asyncio/selector_events.py", line 162, in _accept_connection
        File "/usr/lib/python3.8/socket.py", line 292, in accept
      OSError: [Errno 24] Too many open files
      [2020/10/15 14:58:20] (ERROR) asyncio.default_exception_handler():1707 - socket.accept() out of system resource
      socket: <asyncio.TransportSocket fd=50, family=AddressFamily.AF_UNIX, type=SocketKind.SOCK_STREAM, proto=0, laddr=/var/run/middlewared.sock>
      Traceback (most recent call last):
        File "/usr/lib/python3.8/asyncio/selector_events.py", line 162, in _accept_connection
        File "/usr/lib/python3.8/socket.py", line 292, in accept
       

       

      This problem is resolved by restarting middlewared.service and then nginx.

       

      Not sure what else may be needed.

       

        Attachments

        1. debug-hpc-freenas01-20201015150010.tgz
          330 kB
        2. ls_output.txt
          70 kB
        3. netstat_output.txt
          39 kB
        4. psax_output.txt
          34 kB

          Attachments

            JEditor

              Issue Links

                Activity

                  People

                  Assignee:
                  awalker Andrew Walker
                  Reporter:
                  ismith.hpu Ian Smith
                  Votes:
                  0 Vote for this issue
                  Watchers:
                  5 Start watching this issue

                    Dates

                    Created:
                    Updated:
                    Resolved: