[gentoo-commits] proj/portage:master commit in: lib/portage/cache/, lib/portage/tests/dbapi/

[gentoo-commits] proj/portage:master commit in: lib/portage/cache/, lib/portage/tests/dbapi/

[Zac Medico]

commit:     255c7d5c430d445ee43ceda2f6bf716c75710f23
Author:     Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Tue Oct 31 02:01:49 2023 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Tue Oct 31 02:06:18 2023 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=255c7d5c

sqlite: multiprocessing spawn compat

Override __getstate__ to omit unpicklable attributes, and
regenerate the unpicklable attributes after unpickling.

Bug: https://bugs.gentoo.org/914876
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/portage/cache/sqlite.py           | 15 ++++++++++++++-
 lib/portage/tests/dbapi/test_auxdb.py |  2 +-
 2 files changed, 15 insertions(+), 2 deletions(-)

diff --git a/lib/portage/cache/sqlite.py b/lib/portage/cache/sqlite.py
index 21ecd7ea34..722b6b98d1 100644
--- a/lib/portage/cache/sqlite.py
+++ b/lib/portage/cache/sqlite.py
@@ -1,4 +1,4 @@
-# Copyright 1999-2020 Gentoo Authors
+# Copyright 1999-2023 Gentoo Authors
 # Distributed under the terms of the GNU General Public License v2
 
 import collections
@@ -53,6 +53,19 @@ class database(fs_template.FsBased):
         self._config = config
         self._db_connection_info = None
 
+    def __getstate__(self):
+        state = self.__dict__.copy()
+        # These attributes are not picklable, so they are automatically
+        # regenerated after unpickling.
+        state["_db_module"] = None
+        state["_db_error"] = None
+        state["_db_connection_info"] = None
+        return state
+
+    def __setstate__(self, state):
+        self.__dict__.update(state)
+        self._import_sqlite()
+
     def _import_sqlite(self):
         # sqlite3 is optional with >=python-2.5
         try:

diff --git a/lib/portage/tests/dbapi/test_auxdb.py b/lib/portage/tests/dbapi/test_auxdb.py
index 1bbf1bde35..c11eed73e8 100644
--- a/lib/portage/tests/dbapi/test_auxdb.py
+++ b/lib/portage/tests/dbapi/test_auxdb.py
@@ -31,7 +31,7 @@ class AuxdbTestCase(TestCase):
             import sqlite3
         except ImportError:
             self.skipTest("sqlite3 import failed")
-        self._test_mod("portage.cache.sqlite.database", picklable=False)
+        self._test_mod("portage.cache.sqlite.database", picklable=True)
 
     def _test_mod(self, auxdbmodule, multiproc=True, picklable=True):
         ebuilds = {

[gentoo-commits] proj/portage:master commit in: lib/portage/cache/, lib/portage/tests/dbapi/

[Zac Medico]

commit:     414234a218bc79564ab17312d5cc247a4c8091d7
Author:     Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Tue Feb 13 03:30:00 2024 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Wed Feb 21 15:27:30 2024 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=414234a2

anydbm: Pickle support for multiprocessing spawn

The egencache usage in ResolverPlayground that was used to trigger
bug 924319 triggered a pickling error for AuxdbTestCase.test_anydbm
with the multiprocessing spawn start method, so fix the anydbm
cache module to omit the unpicklable database object from pickled
state, and regenerate it after unpickling.

Bug: https://bugs.gentoo.org/924319
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/portage/cache/anydbm.py           | 17 ++++++++++++++++-
 lib/portage/tests/dbapi/test_auxdb.py |  4 +---
 2 files changed, 17 insertions(+), 4 deletions(-)

diff --git a/lib/portage/cache/anydbm.py b/lib/portage/cache/anydbm.py
index 94a270a483..ad7042ae41 100644
--- a/lib/portage/cache/anydbm.py
+++ b/lib/portage/cache/anydbm.py
@@ -1,4 +1,4 @@
-# Copyright 2005-2020 Gentoo Authors
+# Copyright 2005-2024 Gentoo Authors
 # Distributed under the terms of the GNU General Public License v2
 # Author(s): Brian Harring (ferringb@gentoo.org)
 
@@ -67,6 +67,21 @@ class database(fs_template.FsBased):
                 raise cache_errors.InitializationError(self.__class__, e)
         self._ensure_access(self._db_path)
 
+    def __getstate__(self):
+        state = self.__dict__.copy()
+        # These attributes are not picklable, so they are automatically
+        # regenerated after unpickling.
+        state["_database__db"] = None
+        return state
+
+    def __setstate__(self, state):
+        self.__dict__.update(state)
+        mode = "w"
+        if dbm.whichdb(self._db_path) in ("dbm.gnu", "gdbm"):
+            # Allow multiple concurrent writers (see bug #53607).
+            mode += "u"
+        self.__db = dbm.open(self._db_path, mode, self._perms)
+
     def iteritems(self):
         # dbm doesn't implement items()
         for k in self.__db.keys():

diff --git a/lib/portage/tests/dbapi/test_auxdb.py b/lib/portage/tests/dbapi/test_auxdb.py
index 0de0123a5f..aac6ce361c 100644
--- a/lib/portage/tests/dbapi/test_auxdb.py
+++ b/lib/portage/tests/dbapi/test_auxdb.py
@@ -16,9 +16,7 @@ class AuxdbTestCase(TestCase):
             from portage.cache.anydbm import database
         except ImportError:
             self.skipTest("dbm import failed")
-        self._test_mod(
-            "portage.cache.anydbm.database", multiproc=False, picklable=False
-        )
+        self._test_mod("portage.cache.anydbm.database", multiproc=False, picklable=True)
 
     def test_flat_hash_md5(self):
         self._test_mod("portage.cache.flat_hash.md5_database")