Uploaded image for project: 'TrueNAS'
  1. TrueNAS
  2. NAS-108901

Email Alerts: "Failed to check for alert Quota"

    XMLWordPrintable

Details

    • Low

    Description

      Every email alert I receive includes several error messages related to "failed to check for alert Quota". Here are a couple of examples:

      TrueNAS @ truenas.myers
      
      New alerts:
      * Failed to check for alert Quota:
      concurrent.futures.process._RemoteTraceback: 
      """
      Traceback (most recent call last):
      File "/usr/local/lib/python3.8/concurrent/futures/process.py", line 239, in _process_worker
      r = call_item.fn(*call_item.args, **call_item.kwargs)
      File "/usr/local/lib/python3.8/site-packages/middlewared/worker.py", line 91, in main_worker
      res = MIDDLEWARE._run(*call_args)
      File "/usr/local/lib/python3.8/site-packages/middlewared/worker.py", line 45, in _run
      return self._call(name, serviceobj, methodobj, args, job=job)
      File "/usr/local/lib/python3.8/site-packages/middlewared/worker.py", line 33, in _call
      with Client('ws+unix:///var/run/middlewared-internal.sock', py_exceptions=True) as c:
      File "/usr/local/lib/python3.8/site-packages/middlewared/client/client.py", line 281, in __init__
      self._ws.connect()
      File "/usr/local/lib/python3.8/site-packages/middlewared/client/client.py", line 124, in connect
      rv = super(WSClient, self).connect()
      File "/usr/local/lib/python3.8/site-packages/ws4py/client/__init__.py", line 223, in connect
      bytes = self.sock.recv(128)
      socket.timeout: timed out
      """
      
      The above exception was the direct cause of the following exception:
      
      Traceback (most recent call last):
      File "/usr/local/lib/python3.8/site-packages/middlewared/plugins/alert.py", line 706, in __run_source
      alerts = (await alert_source.check()) or []
      File "/usr/local/lib/python3.8/site-packages/middlewared/alert/base.py", line 210, in check
      return await self.middleware.run_in_thread(self.check_sync)
      File "/usr/local/lib/python3.8/site-packages/middlewared/utils/run_in_thread.py", line 10, in run_in_thread
      return await self.loop.run_in_executor(self.run_in_thread_executor, functools.partial(method, *args, **kwargs))
      File "/usr/local/lib/python3.8/site-packages/middlewared/utils/io_thread_pool_executor.py", line 25, in run
      result = self.fn(*self.args, **self.kwargs)
      File "/usr/local/lib/python3.8/site-packages/middlewared/alert/source/quota.py", line 38, in check_sync
      datasets = self.middleware.call_sync("zfs.dataset.query_for_quota_alert")
      File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1258, in call_sync
      return self.run_coroutine(self._call_worker(name, *prepared_call.args))
      File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1294, in run_coroutine
      return fut.result()
      File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 432, in result
      return self.__get_result()
      File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 388, in __get_result
      raise self._exception
      File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1209, in _call_worker
      return await self.run_in_proc(main_worker, name, args, job)
      File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1136, in run_in_proc
      return await self.run_in_executor(self.__procpool, method, *args, **kwargs)
      File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1110, in run_in_executor
      return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
      socket.timeout: timed out
      
      
      Current alerts:
      * truenas.myers had an unscheduled system reboot.
      The operating system successfully came back online at Sun Dec 13 00:01:59 2020.
      
      * Space usage for pool "Time-Machine-Backups" is 90%. Optimal pool performance requires used space remain below 80%.
      * Failed to check for alert Quota:
      concurrent.futures.process._RemoteTraceback: 
      """
      Traceback (most recent call last):
      File "/usr/local/lib/python3.8/concurrent/futures/process.py", line 239, in _process_worker
      r = call_item.fn(*call_item.args, **call_item.kwargs)
      File "/usr/local/lib/python3.8/site-packages/middlewared/worker.py", line 91, in main_worker
      res = MIDDLEWARE._run(*call_args)
      File "/usr/local/lib/python3.8/site-packages/middlewared/worker.py", line 45, in _run
      return self._call(name, serviceobj, methodobj, args, job=job)
      File "/usr/local/lib/python3.8/site-packages/middlewared/worker.py", line 33, in _call
      with Client('ws+unix:///var/run/middlewared-internal.sock', py_exceptions=True) as c:
      File "/usr/local/lib/python3.8/site-packages/middlewared/client/client.py", line 281, in __init__
      self._ws.connect()
      File "/usr/local/lib/python3.8/site-packages/middlewared/client/client.py", line 124, in connect
      rv = super(WSClient, self).connect()
      File "/usr/local/lib/python3.8/site-packages/ws4py/client/__init__.py", line 223, in connect
      bytes = self.sock.recv(128)
      socket.timeout: timed out
      """
      
      The above exception was the direct cause of the following exception:
      
      Traceback (most recent call last):
      File "/usr/local/lib/python3.8/site-packages/middlewared/plugins/alert.py", line 706, in __run_source
      alerts = (await alert_source.check()) or []
      File "/usr/local/lib/python3.8/site-packages/middlewared/alert/base.py", line 210, in check
      return await self.middleware.run_in_thread(self.check_sync)
      File "/usr/local/lib/python3.8/site-packages/middlewared/utils/run_in_thread.py", line 10, in run_in_thread
      return await self.loop.run_in_executor(self.run_in_thread_executor, functools.partial(method, *args, **kwargs))
      File "/usr/local/lib/python3.8/site-packages/middlewared/utils/io_thread_pool_executor.py", line 25, in run
      result = self.fn(*self.args, **self.kwargs)
      File "/usr/local/lib/python3.8/site-packages/middlewared/alert/source/quota.py", line 38, in check_sync
      datasets = self.middleware.call_sync("zfs.dataset.query_for_quota_alert")
      File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1258, in call_sync
      return self.run_coroutine(self._call_worker(name, *prepared_call.args))
      File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1294, in run_coroutine
      return fut.result()
      File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 432, in result
      return self.__get_result()
      File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 388, in __get_result
      raise self._exception
      File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1209, in _call_worker
      return await self.run_in_proc(main_worker, name, args, job)
      File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1136, in run_in_proc
      return await self.run_in_executor(self.__procpool, method, *args, **kwargs)
      File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1110, in run_in_executor
      return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
      socket.timeout: timed out
      TrueNAS @ truenas.myers
      
      The following alert has been cleared:
      
      
      * Failed to check for alert Quota:
      concurrent.futures.process._RemoteTraceback: 
      """
      Traceback (most recent call last):
      File "/usr/local/lib/python3.8/concurrent/futures/process.py", line 239, in _process_worker
      r = call_item.fn(*call_item.args, **call_item.kwargs)
      File "/usr/local/lib/python3.8/site-packages/middlewared/worker.py", line 91, in main_worker
      res = MIDDLEWARE._run(*call_args)
      File "/usr/local/lib/python3.8/site-packages/middlewared/worker.py", line 45, in _run
      return self._call(name, serviceobj, methodobj, args, job=job)
      File "/usr/local/lib/python3.8/site-packages/middlewared/worker.py", line 33, in _call
      with Client('ws+unix:///var/run/middlewared-internal.sock', py_exceptions=True) as c:
      File "/usr/local/lib/python3.8/site-packages/middlewared/client/client.py", line 281, in __init__
      self._ws.connect()
      File "/usr/local/lib/python3.8/site-packages/middlewared/client/client.py", line 124, in connect
      rv = super(WSClient, self).connect()
      File "/usr/local/lib/python3.8/site-packages/ws4py/client/__init__.py", line 223, in connect
      bytes = self.sock.recv(128)
      socket.timeout: timed out
      """
      
      The above exception was the direct cause of the following exception:
      
      Traceback (most recent call last):
      File "/usr/local/lib/python3.8/site-packages/middlewared/plugins/alert.py", line 706, in __run_source
      alerts = (await alert_source.check()) or []
      File "/usr/local/lib/python3.8/site-packages/middlewared/alert/base.py", line 210, in check
      return await self.middleware.run_in_thread(self.check_sync)
      File "/usr/local/lib/python3.8/site-packages/middlewared/utils/run_in_thread.py", line 10, in run_in_thread
      return await self.loop.run_in_executor(self.run_in_thread_executor, functools.partial(method, *args, **kwargs))
      File "/usr/local/lib/python3.8/site-packages/middlewared/utils/io_thread_pool_executor.py", line 25, in run
      result = self.fn(*self.args, **self.kwargs)
      File "/usr/local/lib/python3.8/site-packages/middlewared/alert/source/quota.py", line 38, in check_sync
      datasets = self.middleware.call_sync("zfs.dataset.query_for_quota_alert")
      File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1258, in call_sync
      return self.run_coroutine(self._call_worker(name, *prepared_call.args))
      File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1294, in run_coroutine
      return fut.result()
      File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 432, in result
      return self.__get_result()
      File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 388, in __get_result
      raise self._exception
      File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1209, in _call_worker
      return await self.run_in_proc(main_worker, name, args, job)
      File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1136, in run_in_proc
      return await self.run_in_executor(self.__procpool, method, *args, **kwargs)
      File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1110, in run_in_executor
      return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
      socket.timeout: timed out
      
      
      Current alerts:
      * truenas.myers had an unscheduled system reboot.
      The operating system successfully came back online at Sun Dec 13 00:01:59 2020.
      
      * Space usage for pool "Time-Machine-Backups" is 90%. Optimal pool performance requires used space remain below 80%.

       

      Attachments

        Attachments

          JEditor

            Activity

              People

                releng Triage Team
                geoffmyers Geoff Myers
                Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                  Created:
                  Updated:
                  Resolved: