Uploaded image for project: 'FreeNAS / TrueNAS'
  1. FreeNAS / TrueNAS
  2. NAS-104537

race condition in sync_encrypted method

    XMLWordPrintable

    Details

    • Support Suite Ticket:
      TFE-402-53808

      JEditor

        Description

        When importing a GELI encrypted zpool, the method volume_import is called and makes alterations to the storage_encrypted table in the database.

        At the same time, a devd event is triggered which causes us to run the sync_encrypted method which also updates the storage_encrypted table in the databse. Depending on which event takes place, one of those methods will traceback with the following information:

         

        Traceback (most recent call last):
          File "/usr/local/lib/python3.6/site-packages/django/db/backends/utils.py", line 64, in execute
            return self.cursor.execute(sql, params)
          File "/usr/local/www/freenasUI/freeadmin/sqlite3_ha/base.py", line 381, in execute
            execute = super().execute(query, params)
        sqlite3.IntegrityError: UNIQUE constraint failed: storage_encrypteddisk.encrypted_provider
        
        The above exception was the direct cause of the following exception:
        
        Traceback (most recent call last):
          File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 165, in call_method
            result = await self.middleware.call_method(self, message)
          File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1107, in call_method
            return await self._call(message['method'], serviceobj, methodobj, params, app=app, io_thread=False)
          File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1069, in _call
            return await run_method(methodobj, *args)
          File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 983, in _run_in_conn_threadpool
            return await self.run_in_executor(self.__threadpool, method, *args, **kwargs)
          File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 971, in run_in_executor
            return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
          File "/usr/local/lib/python3.6/concurrent/futures/thread.py", line 56, in run
            result = self.fn(*self.args, **self.kwargs)
          File "/usr/local/www/freenasUI/middleware/notifier.py", line 2908, in sync_encrypted
            ed.save()
          File "/usr/local/lib/python3.6/site-packages/django/db/models/base.py", line 796, in save
            force_update=force_update, update_fields=update_fields)
          File "/usr/local/lib/python3.6/site-packages/django/db/models/base.py", line 824, in save_base
            updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)
          File "/usr/local/lib/python3.6/site-packages/django/db/models/base.py", line 908, in _save_table
            result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)
          File "/usr/local/lib/python3.6/site-packages/django/db/models/base.py", line 947, in _do_insert
            using=using, raw=raw)
          File "/usr/local/lib/python3.6/site-packages/django/db/models/manager.py", line 85, in manager_method
            return getattr(self.get_queryset(), name)(*args, **kwargs)
          File "/usr/local/lib/python3.6/site-packages/django/db/models/query.py", line 1045, in _insert
            return query.get_compiler(using=using).execute_sql(return_id)
          File "/usr/local/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 1054, in execute_sql
            cursor.execute(sql, params)
          File "/usr/local/lib/python3.6/site-packages/django/db/backends/utils.py", line 64, in execute
            return self.cursor.execute(sql, params)
          File "/usr/local/lib/python3.6/site-packages/django/db/utils.py", line 94, in __exit__
            six.reraise(dj_exc_type, dj_exc_value, traceback)
          File "/usr/local/lib/python3.6/site-packages/django/utils/six.py", line 685, in reraise
            raise value.with_traceback(tb)
          File "/usr/local/lib/python3.6/site-packages/django/db/backends/utils.py", line 64, in execute
            return self.cursor.execute(sql, params)
          File "/usr/local/www/freenasUI/freeadmin/sqlite3_ha/base.py", line 381, in execute
            execute = super().execute(query, params)
        django.db.utils.IntegrityError: UNIQUE constraint failed: storage_encrypteddisk.encrypted_provider


        If the volume_import method is the function that tracebacks, it actually reverts all the entries that were put in the db before and detaches GELI providers. This causes very strange behavior.

        1. database locked errors
        2. constraint violation errors in sqlite
        3. geli encrypted zpools to be imported but missing devices in the zpool

          Attachments

            Attachments

              Activity

                People

                Assignee:
                caleb Caleb St. John
                Reporter:
                caleb Caleb St. John
                Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                  Dates

                  Created:
                  Updated:
                  Resolved: