Skip to:

  1. Skip to Jira Navigation
  2. Skip to Side Navigation
  3. Skip to Main Content

middlewared crashing too many files no login

Description

Once this service crashes, unable to login but SSH works.

From /var/log/middlewared.log: (this is spamming it and creating 10MB log files within 30 seconds)

OSError: [Errno 24] Too many open files
[2020/10/15 14:58:20] (ERROR) asyncio.default_exception_handler():1707 - socket.accept() out of system resource
socket: <asyncio.TransportSocket fd=50, family=AddressFamily.AF_UNIX, type=SocketKind.SOCK_STREAM, proto=0, laddr=/var/run/middlewared.sock>
Traceback (most recent call last):
File "/usr/lib/python3.8/asyncio/selector_events.py", line 162, in _accept_connection
File "/usr/lib/python3.8/socket.py", line 292, in accept
OSError: [Errno 24] Too many open files
[2020/10/15 14:58:20] (ERROR) asyncio.default_exception_handler():1707 - socket.accept() out of system resource
socket: <asyncio.TransportSocket fd=50, family=AddressFamily.AF_UNIX, type=SocketKind.SOCK_STREAM, proto=0, laddr=/var/run/middlewared.sock>
Traceback (most recent call last):
File "/usr/lib/python3.8/asyncio/selector_events.py", line 162, in _accept_connection
File "/usr/lib/python3.8/socket.py", line 292, in accept
OSError: [Errno 24] Too many open files
[2020/10/15 14:58:20] (ERROR) asyncio.default_exception_handler():1707 - socket.accept() out of system resource
socket: <asyncio.TransportSocket fd=50, family=AddressFamily.AF_UNIX, type=SocketKind.SOCK_STREAM, proto=0, laddr=/var/run/middlewared.sock>
Traceback (most recent call last):
File "/usr/lib/python3.8/asyncio/selector_events.py", line 162, in _accept_connection
File "/usr/lib/python3.8/socket.py", line 292, in accept

This problem is resolved by restarting middlewared.service and then nginx.

Not sure what else may be needed.

Problem/Justification

None

Impact

None

SmartDraw Connector

Katalon Manual Tests (BETA)

Activity

Show:

Andrew Walker 
November 2, 2020 at 9:50 PM

I identified a resource leak in Samba's libnet python bindings. It should be fixed in next nightly build. I would still like to see collateral because the resource usage still seems somewhat high based on uptime and frequency of periodic tasks that would use these bindings. There may be an additional resource leak.

Bug Clerk 
November 2, 2020 at 9:23 PM

Andrew Walker 
November 2, 2020 at 5:42 PM

It looks like we're missing some SMB-related collateral. Can you upload output of "testparm -s" and also contents of /var/log/samba and /var/log/samba4/?

There was an in between period when we switched from the one log path to the other. Output of "smbd -b" would also be useful.

Ian Smith 
November 2, 2020 at 4:09 PM

Here are the outputs youve requested.

 

 

Vladimir Vinogradenko 
October 28, 2020 at 8:10 PM

when does this happen? Can you please capture the output of

  • ls -al /proc/`pidof middlewared`/fd

  • netstat -vpan

  • ps ax

When this happens the next time?

Complete

Details

Assignee

Reporter

Labels

Components

Fix versions

Affects versions

Priority

More fields

Katalon Platform

Created October 15, 2020 at 8:43 PM
Updated July 1, 2022 at 4:54 PM
Resolved November 2, 2020 at 9:23 PM