Uploaded image for project: 'FreeNAS / TrueNAS'
  1. FreeNAS / TrueNAS
  2. NAS-101787

Fix blocked event loop for alerts

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Done (View Workflow)
    • Priority: Low
    • Resolution: Done
    • Affects Version/s: None
    • Fix Version/s: 11.2-U5
    • Component/s: Middleware
    • Labels:
      None
    • Attempts to pass:
      1
    • Testing System:
      X10-HA
    • QA Status:
      Test Passes TrueNAS
    • Acceptance Criteria:
      Hide
      This takes a long time and zero action so it can probably be tested while working on other tickets. Messages like these usually appear at random time from once an hour to twice a day so just run system for 24 hours without rebooting and that'll be enough. Messages like this (starting with "middlewared._loop_monitor_thread():1191 - Task seems blocked:" and containing lines related to "alert.py") should not appear anymore.
      Show
      This takes a long time and zero action so it can probably be tested while working on other tickets. Messages like these usually appear at random time from once an hour to twice a day so just run system for 24 hours without rebooting and that'll be enough. Messages like this (starting with "middlewared._loop_monitor_thread():1191 - Task seems blocked:" and containing lines related to "alert.py") should not appear anymore.

      Description

      Running 11.2-U4-INTERNAL9, it seems like the event loop is being blocked.

      "
      [2019/05/15 13:16:06] (WARNING) middlewared._loop_monitor_thread():1191 - Task seems blocked:  File "/usr/local/bin/middlewared", line 10, in <module>
         sys.exit(main())
       File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1341, in main
         startup_seq_path=startup_seq_path,
       File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1259, in run
         self.__loop.run_forever()
       File "/usr/local/lib/python3.6/asyncio/base_events.py", line 422, in run_forever
         self._run_once()
       File "/usr/local/lib/python3.6/asyncio/base_events.py", line 1432, in _run_once
         handle._run()
       File "/usr/local/lib/python3.6/asyncio/events.py", line 145, in _run
         self._callback(*self._args)
       File "/usr/local/lib/python3.6/site-packages/middlewared/job.py", line 363, in __run_body
         rv = await self.method(*([self] + args))
       File "/usr/local/lib/python3.6/site-packages/middlewared/plugins/alert.py", line 197, in process_alerts
         await self.__run_alerts()
       File "/usr/local/lib/python3.6/site-packages/middlewared/plugins/alert.py", line 361, in __run_alerts
         alerts_a = await self.__run_source(alert_source.name)
       File "/usr/local/lib/python3.6/site-packages/middlewared/plugins/alert.py", line 459, in __run_source
         alerts = (await alert_source.check()) or []

      [2019/05/15 13:24:49] (WARNING) middlewared._loop_monitor_thread():1191 - Task seems blocked:  File "/usr/local/bin/middlewared", line 10, in <module>
         sys.exit(main())
       File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1341, in main
         startup_seq_path=startup_seq_path,
       File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1259, in run
         self.__loop.run_forever()
       File "/usr/local/lib/python3.6/asyncio/base_events.py", line 422, in run_forever
         self._run_once()
       File "/usr/local/lib/python3.6/asyncio/base_events.py", line 1432, in _run_once
         handle._run()
       File "/usr/local/lib/python3.6/asyncio/events.py", line 145, in _run
         self._callback(*self._args)
       File "/usr/local/lib/python3.6/site-packages/middlewared/job.py", line 363, in __run_body
         rv = await self.method(*([self] + args))
       File "/usr/local/lib/python3.6/site-packages/middlewared/plugins/alert.py", line 197, in process_alerts
         await self.__run_alerts()
       File "/usr/local/lib/python3.6/site-packages/middlewared/plugins/alert.py", line 374, in __run_alerts
         [alert_source.name])
       File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1101, in call
         return await self._call(name, serviceobj, methodobj, params, app=app, pipes=pipes, io_thread=True)

      "

        Attachments

          Container Issues

            Activity

              People

              • Assignee:
                themylogin Vladimir Vinogradenko
                Reporter:
                caleb Caleb St. John
                QE Engineer:
                Bonnie Follweiler
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel