[gentoo-portage-dev] [PATCH 0/9] EbuildPhase/EbuildMetadataPhase: async check_locale

[gentoo-portage-dev] [PATCH 0/9] EbuildPhase/EbuildMetadataPhase: async check_locale

[Zac Medico]

Change config.environ() check_locale calls to async_check_locale
calls in the EbuildPhase/EbuildMetadataPhase _async_start method
in order to eliminate synchronous waiting for child processes in
the main event loop thread.

Note that this series of changes causes access to the portdbapi
doebuild_settings attribute to no longer be serialized via the
EbuildMetadataPhase _start_method. As a result, exclusive access
to config instances needs to be guaranteed in some other way to
avoid triggering problems (see bug 924319), such as by maintaining
a config pool or by serializing access via an asyncio.Lock instance.

This series can also be reviewed at https://github.com/gentoo/portage/pull/1267.

Bug: https://bugs.gentoo.org/923841
Bug: https://bugs.gentoo.org/924319
Signed-off-by: Zac Medico <zmedico@gentoo.org>

Zac Medico (9):
  anydbm: Pickle support for multiprocessing spawn
  EbuildMetadataPhase: Add deallocate_config future
  MetadataRegen: Use EbuildMetadataPhase deallocate_config
  ResolverPlayground: Use egencache to create manifests
  asyncio: Wrap asyncio.Lock for python 3.9 compat
  async_aux_get: Use EbuildMetadataPhase deallocate_config future
  actions: disable pytest-xdist for spawn start-method (workers crash)
  EbuildMetadataPhase: Migrate to _async_start
  EbuildPhase: async_check_locale

 .github/workflows/ci.yml                      |   5 +-
 lib/_emerge/EbuildMetadataPhase.py            |  37 ++++-
 lib/_emerge/EbuildPhase.py                    |  28 +++-
 lib/_emerge/MetadataRegen.py                  |  16 ++-
 lib/_emerge/SubProcess.py                     |   5 +-
 lib/_emerge/depgraph.py                       |  11 ++
 lib/portage/_emirrordist/FetchIterator.py     |  10 +-
 lib/portage/cache/anydbm.py                   |  17 ++-
 lib/portage/dbapi/porttree.py                 | 129 ++++++++++++------
 lib/portage/package/ebuild/config.py          |  26 ++--
 lib/portage/tests/dbapi/test_auxdb.py         |   4 +-
 lib/portage/tests/dbapi/test_portdb_cache.py  |   4 +-
 .../tests/resolver/ResolverPlayground.py      |  38 +++---
 lib/portage/tests/update/test_move_ent.py     |   3 +
 lib/portage/util/futures/_asyncio/__init__.py |  26 ++++
 lib/portage/util/locale.py                    |  28 ++--
 16 files changed, 289 insertions(+), 98 deletions(-)

-- 
2.41.0

[gentoo-portage-dev] [PATCH 1/9] anydbm: Pickle support for multiprocessing spawn

[Zac Medico]

The egencache usage in ResolverPlayground that was used to trigger
bug 924319 triggered a pickling error for AuxdbTestCase.test_anydbm
with the multiprocessing spawn start method, so fix the anydbm
cache module to omit the unpicklable database object from pickled
state, and regenerate it after unpickling.

Bug: https://bugs.gentoo.org/924319
Signed-off-by: Zac Medico <zmedico@gentoo.org>
---
 lib/portage/cache/anydbm.py           | 17 ++++++++++++++++-
 lib/portage/tests/dbapi/test_auxdb.py |  4 +---
 2 files changed, 17 insertions(+), 4 deletions(-)

diff --git a/lib/portage/cache/anydbm.py b/lib/portage/cache/anydbm.py
index 94a270a483..ad7042ae41 100644
--- a/lib/portage/cache/anydbm.py
+++ b/lib/portage/cache/anydbm.py
@@ -1,4 +1,4 @@
-# Copyright 2005-2020 Gentoo Authors
+# Copyright 2005-2024 Gentoo Authors
 # Distributed under the terms of the GNU General Public License v2
 # Author(s): Brian Harring (ferringb@gentoo.org)
 
@@ -67,6 +67,21 @@ class database(fs_template.FsBased):
                 raise cache_errors.InitializationError(self.__class__, e)
         self._ensure_access(self._db_path)
 
+    def __getstate__(self):
+        state = self.__dict__.copy()
+        # These attributes are not picklable, so they are automatically
+        # regenerated after unpickling.
+        state["_database__db"] = None
+        return state
+
+    def __setstate__(self, state):
+        self.__dict__.update(state)
+        mode = "w"
+        if dbm.whichdb(self._db_path) in ("dbm.gnu", "gdbm"):
+            # Allow multiple concurrent writers (see bug #53607).
+            mode += "u"
+        self.__db = dbm.open(self._db_path, mode, self._perms)
+
     def iteritems(self):
         # dbm doesn't implement items()
         for k in self.__db.keys():
diff --git a/lib/portage/tests/dbapi/test_auxdb.py b/lib/portage/tests/dbapi/test_auxdb.py
index 0de0123a5f..aac6ce361c 100644
--- a/lib/portage/tests/dbapi/test_auxdb.py
+++ b/lib/portage/tests/dbapi/test_auxdb.py
@@ -16,9 +16,7 @@ class AuxdbTestCase(TestCase):
             from portage.cache.anydbm import database
         except ImportError:
             self.skipTest("dbm import failed")
-        self._test_mod(
-            "portage.cache.anydbm.database", multiproc=False, picklable=False
-        )
+        self._test_mod("portage.cache.anydbm.database", multiproc=False, picklable=True)
 
     def test_flat_hash_md5(self):
         self._test_mod("portage.cache.flat_hash.md5_database")
-- 
2.41.0

[gentoo-portage-dev] [PATCH 2/9] EbuildMetadataPhase: Add deallocate_config future

[Zac Medico]

Use a deallocate_config future to release self.settings when
it is no longer needed. It's necessary to manage concurrency
since commit c95fc64abf96 because mutation of self.settings
is no longer limited to the EbuildMetadataPhase _start method,
where exclusive access was guaranteed within the main thread.

Add support to the isAlive() method to detect when the
EbuildMetadataPhase has started but the pid is not yet
available (due to async_check_locale usage from commit
c95fc64abf96). This can be used to check if an
EbuildMetadataPhase instance has been successfully started
so that it can be relied upon to set the result of the
deallocate_config future.

Bug: https://bugs.gentoo.org/924319
Signed-off-by: Zac Medico <zmedico@gentoo.org>
---
 lib/_emerge/EbuildMetadataPhase.py | 36 ++++++++++++++++++++++++++++++
 lib/_emerge/SubProcess.py          |  5 ++++-
 2 files changed, 40 insertions(+), 1 deletion(-)

diff --git a/lib/_emerge/EbuildMetadataPhase.py b/lib/_emerge/EbuildMetadataPhase.py
index f4f685e81c..784712e8cb 100644
--- a/lib/_emerge/EbuildMetadataPhase.py
+++ b/lib/_emerge/EbuildMetadataPhase.py
@@ -14,6 +14,7 @@ from portage import os
 from portage import _encodings
 from portage import _unicode_decode
 from portage import _unicode_encode
+from portage.util.futures import asyncio
 
 import fcntl
 
@@ -33,6 +34,7 @@ class EbuildMetadataPhase(SubProcess):
         "portdb",
         "repo_path",
         "settings",
+        "deallocate_config",
         "write_auxdb",
     ) + (
         "_eapi",
@@ -127,6 +129,15 @@ class EbuildMetadataPhase(SubProcess):
             returnproc=True,
         )
         settings.pop("PORTAGE_PIPE_FD", None)
+        # At this point we can return settings to the caller
+        # since we never use it for anything more than an
+        # eapi_invalid call after this, and eapi_invalid is
+        # insensitive to concurrent modifications.
+        if (
+            self.deallocate_config is not None
+            and not self.deallocate_config.cancelled()
+        ):
+            self.deallocate_config.set_result(settings)
 
         os.close(slave_fd)
         null_input.close()
@@ -139,6 +150,31 @@ class EbuildMetadataPhase(SubProcess):
 
         self._proc = retval
 
+        asyncio.ensure_future(
+            self._async_start(), loop=self.scheduler
+        ).add_done_callback(self._async_start_done)
+
+    async def _async_start(self):
+        # Call async check_locale here for bug 923841, but code
+        # also needs to migrate from _start to here, including
+        # the self.deallocate_config set_result call.
+        pass
+
+    def _async_start_done(self, future):
+        future.cancelled() or future.result()
+        if self._was_cancelled():
+            pass
+        elif future.cancelled():
+            self.cancel()
+            self._was_cancelled()
+
+        if self.deallocate_config is not None and not self.deallocate_config.done():
+            self.deallocate_config.set_result(self.settings)
+
+        if self.returncode is not None:
+            self._unregister()
+            self.wait()
+
     def _output_handler(self):
         while True:
             buf = self._read_buf(self._files.ebuild)
diff --git a/lib/_emerge/SubProcess.py b/lib/_emerge/SubProcess.py
index 029bbc3f44..057e0adc24 100644
--- a/lib/_emerge/SubProcess.py
+++ b/lib/_emerge/SubProcess.py
@@ -18,9 +18,12 @@ class SubProcess(AbstractPollTask):
     # we've sent a kill signal to our subprocess.
     _cancel_timeout = 1  # seconds
 
+    def isAlive(self):
+        return (self._registered or self.pid is not None) and self.returncode is None
+
     @property
     def pid(self):
-        return self._proc.pid
+        return None if self._proc is None else self._proc.pid
 
     def _poll(self):
         # Simply rely on _async_waitpid_cb to set the returncode.
-- 
2.41.0

[gentoo-portage-dev] [PATCH 3/9] MetadataRegen: Use EbuildMetadataPhase deallocate_config

[Zac Medico]

For EbuildMetadataPhase consumers like MetadataRegen and
depgraph, store a pool of config instances in a config_pool
list, and return instaces to the list when the deallocate_config
future is done.

Bug: https://bugs.gentoo.org/924319
Fixes: c95fc64abf96 ("EbuildPhase: async_check_locale")
Signed-off-by: Zac Medico <zmedico@gentoo.org>
---
 lib/_emerge/MetadataRegen.py | 16 ++++++++++++++--
 lib/_emerge/depgraph.py      | 11 +++++++++++
 2 files changed, 25 insertions(+), 2 deletions(-)

diff --git a/lib/_emerge/MetadataRegen.py b/lib/_emerge/MetadataRegen.py
index d29722b94c..538a94b450 100644
--- a/lib/_emerge/MetadataRegen.py
+++ b/lib/_emerge/MetadataRegen.py
@@ -1,4 +1,4 @@
-# Copyright 1999-2020 Gentoo Authors
+# Copyright 1999-2024 Gentoo Authors
 # Distributed under the terms of the GNU General Public License v2
 
 from _emerge.EbuildMetadataPhase import EbuildMetadataPhase
@@ -44,6 +44,7 @@ class MetadataRegen(AsyncScheduler):
         valid_pkgs = self._valid_pkgs
         cp_set = self._cp_set
         consumer = self._consumer
+        config_pool = []
 
         portage.writemsg_stdout("Regenerating cache entries...\n")
         for cp in self._cp_iter:
@@ -73,12 +74,23 @@ class MetadataRegen(AsyncScheduler):
                             consumer(cpv, repo_path, metadata, ebuild_hash, True)
                         continue
 
+                    if config_pool:
+                        settings = config_pool.pop()
+                    else:
+                        settings = portage.config(clone=portdb.settings)
+
+                    deallocate_config = self.scheduler.create_future()
+                    deallocate_config.add_done_callback(
+                        lambda future: config_pool.append(future.result())
+                    )
+
                     yield EbuildMetadataPhase(
                         cpv=cpv,
                         ebuild_hash=ebuild_hash,
                         portdb=portdb,
                         repo_path=repo_path,
-                        settings=portdb.doebuild_settings,
+                        settings=settings,
+                        deallocate_config=deallocate_config,
                         write_auxdb=self._write_auxdb,
                     )
 
diff --git a/lib/_emerge/depgraph.py b/lib/_emerge/depgraph.py
index 1674fa289e..70b83ee1f4 100644
--- a/lib/_emerge/depgraph.py
+++ b/lib/_emerge/depgraph.py
@@ -754,6 +754,7 @@ class depgraph:
 
     def _dynamic_deps_preload(self, fake_vartree):
         portdb = fake_vartree._portdb
+        config_pool = []
         for pkg in fake_vartree.dbapi:
             self._spinner_update()
             self._dynamic_config._package_tracker.add_installed_pkg(pkg)
@@ -768,12 +769,22 @@ class depgraph:
             if metadata is not None:
                 fake_vartree.dynamic_deps_preload(pkg, metadata)
             else:
+                if config_pool:
+                    settings = config_pool.pop()
+                else:
+                    settings = portage.config(clone=portdb.settings)
+
+                deallocate_config = portdb._event_loop.create_future()
+                deallocate_config.add_done_callback(
+                    lambda future: config_pool.append(future.result())
+                )
                 proc = EbuildMetadataPhase(
                     cpv=pkg.cpv,
                     ebuild_hash=ebuild_hash,
                     portdb=portdb,
                     repo_path=repo_path,
                     settings=portdb.doebuild_settings,
+                    deallocate_config=deallocate_config,
                 )
                 proc.addExitListener(self._dynamic_deps_proc_exit(pkg, fake_vartree))
                 yield proc
-- 
2.41.0

[gentoo-portage-dev] [PATCH 4/9] ResolverPlayground: Use egencache to create manifests

[Zac Medico]

Make the ResolverPlayground _create_ebuild_manifests method
call egencache --jobs, which reliably triggers the KeyError
from bug 924319 for multiple tests:

lib/portage/tests/bin/test_doins.py::DoIns::testDoInsFallback Exception in callback EbuildMetadataPhase._async_start_done(<Task finishe...Error('EAPI')>)
handle: <Handle EbuildMetadataPhase._async_start_done(<Task finishe...Error('EAPI')>)>
Traceback (most recent call last):
  File "/usr/lib/python3.12/asyncio/events.py", line 88, in _run
    self._context.run(self._callback, *self._args)
  File "lib/_emerge/EbuildMetadataPhase.py", line 154, in _async_start_done
    future.cancelled() or future.result()
                          ^^^^^^^^^^^^^^^
  File "lib/_emerge/EbuildMetadataPhase.py", line 130, in _async_start
    retval = portage.doebuild(
             ^^^^^^^^^^^^^^^^^
  File "lib/portage/package/ebuild/doebuild.py", line 1030, in doebuild
    doebuild_environment(
  File "lib/portage/package/ebuild/doebuild.py", line 519, in doebuild_environment
    eapi = mysettings.configdict["pkg"]["EAPI"]
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^
  File "lib/portage/util/__init__.py", line 1684, in __getitem__
    return UserDict.__getitem__(self, item_key)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "lib/portage/cache/mappings.py", line 175, in __getitem__
    return self.data[key]
           ~~~~~~~~~^^^^^
KeyError: 'EAPI'

Bug: https://bugs.gentoo.org/924319
Signed-off-by: Zac Medico <zmedico@gentoo.org>
---
 lib/portage/tests/dbapi/test_portdb_cache.py  |  4 +-
 .../tests/resolver/ResolverPlayground.py      | 38 ++++++++++---------
 2 files changed, 23 insertions(+), 19 deletions(-)

diff --git a/lib/portage/tests/dbapi/test_portdb_cache.py b/lib/portage/tests/dbapi/test_portdb_cache.py
index c7c6913b49..c24a4f2098 100644
--- a/lib/portage/tests/dbapi/test_portdb_cache.py
+++ b/lib/portage/tests/dbapi/test_portdb_cache.py
@@ -1,6 +1,7 @@
-# Copyright 2012-2023 Gentoo Authors
+# Copyright 2012-2024 Gentoo Authors
 # Distributed under the terms of the GNU General Public License v2
 
+import shutil
 import subprocess
 import sys
 import textwrap
@@ -63,6 +64,7 @@ class PortdbCacheTestCase(TestCase):
         python_cmd = (portage_python, "-b", "-Wd", "-c")
 
         test_commands = (
+            (lambda: shutil.rmtree(md5_cache_dir) or True,),
             (lambda: not os.path.exists(pms_cache_dir),),
             (lambda: not os.path.exists(md5_cache_dir),),
             python_cmd
diff --git a/lib/portage/tests/resolver/ResolverPlayground.py b/lib/portage/tests/resolver/ResolverPlayground.py
index 2d26012873..75c86b615c 100644
--- a/lib/portage/tests/resolver/ResolverPlayground.py
+++ b/lib/portage/tests/resolver/ResolverPlayground.py
@@ -3,6 +3,7 @@
 
 import bz2
 import fnmatch
+import subprocess
 import tempfile
 import portage
 
@@ -18,8 +19,6 @@ from portage.const import (
 from portage.process import find_binary
 from portage.dep import Atom, _repo_separator
 from portage.dbapi.bintree import binarytree
-from portage.package.ebuild.config import config
-from portage.package.ebuild.digestgen import digestgen
 from portage._sets import load_default_config
 from portage._sets.base import InternalPackageSet
 from portage.tests import cnf_path
@@ -323,22 +322,25 @@ class ResolverPlayground:
                     f.write(misc_content)
 
     def _create_ebuild_manifests(self, ebuilds):
-        tmpsettings = config(clone=self.settings)
-        tmpsettings["PORTAGE_QUIET"] = "1"
-        for cpv in ebuilds:
-            a = Atom("=" + cpv, allow_repo=True)
-            repo = a.repo
-            if repo is None:
-                repo = "test_repo"
-
-            repo_dir = self._get_repo_dir(repo)
-            ebuild_dir = os.path.join(repo_dir, a.cp)
-            ebuild_path = os.path.join(ebuild_dir, a.cpv.split("/")[1] + ".ebuild")
-
-            portdb = self.trees[self.eroot]["porttree"].dbapi
-            tmpsettings["O"] = ebuild_dir
-            if not digestgen(mysettings=tmpsettings, myportdb=portdb):
-                raise AssertionError(f"digest creation failed for {ebuild_path}")
+        for repo_name in self._repositories:
+            if repo_name == "DEFAULT":
+                continue
+            egencache_cmd = [
+                "egencache",
+                f"--repo={repo_name}",
+                "--update",
+                "--update-manifests",
+                "--sign-manifests=n",
+                "--strict-manifests=n",
+                f"--repositories-configuration={self.settings['PORTAGE_REPOSITORIES']}",
+                f"--jobs={portage.util.cpuinfo.get_cpu_count()}",
+            ]
+            result = subprocess.run(
+                egencache_cmd,
+                env=self.settings.environ(),
+            )
+            if result.returncode != os.EX_OK:
+                raise AssertionError(f"command failed: {egencache_cmd}")
 
     def _create_binpkgs(self, binpkgs):
         # When using BUILD_ID, there can be multiple instances for the
-- 
2.41.0

[gentoo-portage-dev] [PATCH 5/9] asyncio: Wrap asyncio.Lock for python 3.9 compat

[Zac Medico]

Wrap asyncio.Lock for compatibility with python 3.9 where the
deprecated loop parameter is required in order to avoid "got
Future <Future pending> attached to a different loop" errors.

The pordbapi async_aux_get method can use asyncio.Lock to
serialize access to its doebuild_settings attribute in order
to prevent issues like bug 924319.

Bug: https://bugs.gentoo.org/924319
Signed-off-by: Zac Medico <zmedico@gentoo.org>
---
 lib/portage/util/futures/_asyncio/__init__.py | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)

diff --git a/lib/portage/util/futures/_asyncio/__init__.py b/lib/portage/util/futures/_asyncio/__init__.py
index 8f1b8e8275..b6481c281e 100644
--- a/lib/portage/util/futures/_asyncio/__init__.py
+++ b/lib/portage/util/futures/_asyncio/__init__.py
@@ -9,6 +9,7 @@ __all__ = (
     "CancelledError",
     "Future",
     "InvalidStateError",
+    "Lock",
     "TimeoutError",
     "get_child_watcher",
     "get_event_loop",
@@ -22,6 +23,7 @@ __all__ = (
     "wait_for",
 )
 
+import sys
 import types
 import weakref
 
@@ -35,6 +37,7 @@ from asyncio import (
     FIRST_EXCEPTION,
     Future,
     InvalidStateError,
+    Lock as _Lock,
     shield,
     TimeoutError,
     wait_for,
@@ -159,6 +162,20 @@ def iscoroutinefunction(func):
     return False
 
 
+class Lock(_Lock):
+    """
+    Inject loop parameter for python3.9 or less in order to avoid
+    "got Future <Future pending> attached to a different loop" errors.
+    """
+
+    def __init__(self, **kwargs):
+        if sys.version_info >= (3, 10):
+            kwargs.pop("loop", None)
+        elif "loop" not in kwargs:
+            kwargs["loop"] = _safe_loop()._loop
+        super().__init__(**kwargs)
+
+
 class Task(Future):
     """
     Schedule the execution of a coroutine: wrap it in a future. A task
-- 
2.41.0

[gentoo-portage-dev] [PATCH 6/9] async_aux_get: Use EbuildMetadataPhase deallocate_config future

[Zac Medico]

For the portdbapi async_aux_get method, there is not a very
good place to store a config pool, so instead use asyncio.Lock
to manage access to the portdbapi doebuild_settings attribute
when using the main event loop in the main thread. For other
threads, clone a config instance since we do not have a
thread-safe config pool. This cloning is expensive, but since
portage internals do not trigger this case, it suffices for now
(an AssertionError ensures that internals do not trigger it).
For the main event loop running in the main thread, performance
with the asyncio.Lock should not be significantly different to
performance prior to commit c95fc64abf96, since check_locale
results are typically cached and before there was only a single
shared doebuild_settings instance with access serialized via
the EbuildMetadataPhase _start method.

Update async_aux_get callers to use asyncio.ensure_future on
the returned coroutine when needed, since it used to return
a future instead of a coroutine, and sometimes a future is
needed for add_done_callback usage.

In the portdbapi async_fetch_map method, fix a broken reference
to "future" which should have been "aux_get_future", an error
discovered while testing this patch.

Bug: https://bugs.gentoo.org/924319
Fixes: c95fc64abf96 ("EbuildPhase: async_check_locale")
Signed-off-by: Zac Medico <zmedico@gentoo.org>
---
 lib/portage/_emirrordist/FetchIterator.py |  10 +-
 lib/portage/dbapi/porttree.py             | 129 +++++++++++++++-------
 lib/portage/tests/update/test_move_ent.py |   3 +
 3 files changed, 97 insertions(+), 45 deletions(-)

diff --git a/lib/portage/_emirrordist/FetchIterator.py b/lib/portage/_emirrordist/FetchIterator.py
index eaf3e53596..e4fdd092af 100644
--- a/lib/portage/_emirrordist/FetchIterator.py
+++ b/lib/portage/_emirrordist/FetchIterator.py
@@ -1,4 +1,4 @@
-# Copyright 2013-2018 Gentoo Foundation
+# Copyright 2013-2024 Gentoo Authors
 # Distributed under the terms of the GNU General Public License v2
 
 import threading
@@ -14,6 +14,7 @@ from portage.exception import PortageException, PortageKeyError
 from portage.package.ebuild.fetch import DistfileName
 from portage.util._async.AsyncTaskFuture import AsyncTaskFuture
 from portage.util._async.TaskScheduler import TaskScheduler
+from portage.util.futures import asyncio
 from portage.util.futures.iter_completed import iter_gather
 from .FetchTask import FetchTask
 from _emerge.CompositeTask import CompositeTask
@@ -276,8 +277,11 @@ def _async_fetch_tasks(config, hash_filter, repo_config, digests_future, cpv, lo
         result.set_result(fetch_tasks)
 
     def future_generator():
-        yield config.portdb.async_aux_get(
-            cpv, ("RESTRICT",), myrepo=repo_config.name, loop=loop
+        yield asyncio.ensure_future(
+            config.portdb.async_aux_get(
+                cpv, ("RESTRICT",), myrepo=repo_config.name, loop=loop
+            ),
+            loop,
         )
         yield config.portdb.async_fetch_map(cpv, mytree=repo_config.location, loop=loop)
 
diff --git a/lib/portage/dbapi/porttree.py b/lib/portage/dbapi/porttree.py
index 61d431f917..4eebe1183f 100644
--- a/lib/portage/dbapi/porttree.py
+++ b/lib/portage/dbapi/porttree.py
@@ -1,4 +1,4 @@
-# Copyright 1998-2021 Gentoo Authors
+# Copyright 1998-2024 Gentoo Authors
 # Distributed under the terms of the GNU General Public License v2
 
 __all__ = ["close_portdbapi_caches", "FetchlistDict", "portagetree", "portdbapi"]
@@ -41,7 +41,9 @@ from portage.util.futures import asyncio
 from portage.util.futures.iter_completed import iter_gather
 from _emerge.EbuildMetadataPhase import EbuildMetadataPhase
 
+import contextlib
 import os as _os
+import threading
 import traceback
 import warnings
 import errno
@@ -239,6 +241,7 @@ class portdbapi(dbapi):
         # this purpose because doebuild makes many changes to the config
         # instance that is passed in.
         self.doebuild_settings = config(clone=self.settings)
+        self._doebuild_settings_lock = asyncio.Lock()
         self.depcachedir = os.path.realpath(self.settings.depcachedir)
 
         if os.environ.get("SANDBOX_ON") == "1":
@@ -356,6 +359,17 @@ class portdbapi(dbapi):
         self._better_cache = None
         self._broken_ebuilds = set()
 
+    def __getstate__(self):
+        state = self.__dict__.copy()
+        # These attributes are not picklable, so they are automatically
+        # regenerated after unpickling.
+        state["_doebuild_settings_lock"] = None
+        return state
+
+    def __setstate__(self, state):
+        self.__dict__.update(state)
+        self._doebuild_settings_lock = asyncio.Lock()
+
     def _set_porttrees(self, porttrees):
         """
         Consumers, such as emirrordist, may modify the porttrees attribute in
@@ -669,7 +683,7 @@ class portdbapi(dbapi):
             self.async_aux_get(mycpv, mylist, mytree=mytree, myrepo=myrepo, loop=loop)
         )
 
-    def async_aux_get(self, mycpv, mylist, mytree=None, myrepo=None, loop=None):
+    async def async_aux_get(self, mycpv, mylist, mytree=None, myrepo=None, loop=None):
         """
         Asynchronous form form of aux_get.
 
@@ -694,13 +708,11 @@ class portdbapi(dbapi):
         # Callers of this method certainly want the same event loop to
         # be used for all calls.
         loop = asyncio._wrap_loop(loop)
-        future = loop.create_future()
         cache_me = False
         if myrepo is not None:
             mytree = self.treemap.get(myrepo)
             if mytree is None:
-                future.set_exception(PortageKeyError(myrepo))
-                return future
+                raise PortageKeyError(myrepo)
 
         if (
             mytree is not None
@@ -719,16 +731,14 @@ class portdbapi(dbapi):
         ):
             aux_cache = self._aux_cache.get(mycpv)
             if aux_cache is not None:
-                future.set_result([aux_cache.get(x, "") for x in mylist])
-                return future
+                return [aux_cache.get(x, "") for x in mylist]
             cache_me = True
 
         try:
             cat, pkg = mycpv.split("/", 1)
         except ValueError:
             # Missing slash. Can't find ebuild so raise PortageKeyError.
-            future.set_exception(PortageKeyError(mycpv))
-            return future
+            raise PortageKeyError(mycpv)
 
         myebuild, mylocation = self.findname2(mycpv, mytree)
 
@@ -737,12 +747,12 @@ class portdbapi(dbapi):
                 "!!! aux_get(): %s\n" % _("ebuild not found for '%s'") % mycpv,
                 noiselevel=1,
             )
-            future.set_exception(PortageKeyError(mycpv))
-            return future
+            raise PortageKeyError(mycpv)
 
         mydata, ebuild_hash = self._pull_valid_cache(mycpv, myebuild, mylocation)
 
         if mydata is not None:
+            future = loop.create_future()
             self._aux_get_return(
                 future,
                 mycpv,
@@ -754,37 +764,71 @@ class portdbapi(dbapi):
                 cache_me,
                 None,
             )
-            return future
+            return future.result()
 
         if myebuild in self._broken_ebuilds:
-            future.set_exception(PortageKeyError(mycpv))
-            return future
-
-        proc = EbuildMetadataPhase(
-            cpv=mycpv,
-            ebuild_hash=ebuild_hash,
-            portdb=self,
-            repo_path=mylocation,
-            scheduler=loop,
-            settings=self.doebuild_settings,
-        )
+            raise PortageKeyError(mycpv)
 
-        proc.addExitListener(
-            functools.partial(
-                self._aux_get_return,
-                future,
-                mycpv,
-                mylist,
-                myebuild,
-                ebuild_hash,
-                mydata,
-                mylocation,
-                cache_me,
-            )
-        )
-        future.add_done_callback(functools.partial(self._aux_get_cancel, proc))
-        proc.start()
-        return future
+        proc = None
+        deallocate_config = None
+        async with contextlib.AsyncExitStack() as stack:
+            try:
+                if (
+                    threading.current_thread() is threading.main_thread()
+                    and loop is asyncio._safe_loop()
+                ):
+                    # In this case use self._doebuild_settings_lock to manage concurrency.
+                    deallocate_config = loop.create_future()
+                    await stack.enter_async_context(self._doebuild_settings_lock)
+                    settings = self.doebuild_settings
+                else:
+                    if portage._internal_caller:
+                        raise AssertionError(
+                            f"async_aux_get called from thread {threading.current_thread()} with loop {loop}"
+                        )
+                    # Clone a config instance since we do not have a thread-safe config pool.
+                    settings = portage.config(clone=self.settings)
+
+                proc = EbuildMetadataPhase(
+                    cpv=mycpv,
+                    ebuild_hash=ebuild_hash,
+                    portdb=self,
+                    repo_path=mylocation,
+                    scheduler=loop,
+                    settings=settings,
+                    deallocate_config=deallocate_config,
+                )
+
+                future = loop.create_future()
+                proc.addExitListener(
+                    functools.partial(
+                        self._aux_get_return,
+                        future,
+                        mycpv,
+                        mylist,
+                        myebuild,
+                        ebuild_hash,
+                        mydata,
+                        mylocation,
+                        cache_me,
+                    )
+                )
+                future.add_done_callback(functools.partial(self._aux_get_cancel, proc))
+                proc.start()
+
+            finally:
+                # Wait for deallocate_config before releasing
+                # self._doebuild_settings_lock if needed.
+                if deallocate_config is not None:
+                    if proc is None or not proc.isAlive():
+                        deallocate_config.done() or deallocate_config.cancel()
+                    else:
+                        await deallocate_config
+
+        # After deallocate_config is done, release self._doebuild_settings_lock
+        # by leaving the stack context, and wait for proc to finish and
+        # trigger a call to self._aux_get_return.
+        return await future
 
     @staticmethod
     def _aux_get_cancel(proc, future):
@@ -889,7 +933,7 @@ class portdbapi(dbapi):
                         )
                     )
                 else:
-                    result.set_exception(future.exception())
+                    result.set_exception(aux_get_future.exception())
                 return
 
             eapi, myuris = aux_get_future.result()
@@ -913,8 +957,9 @@ class portdbapi(dbapi):
             except Exception as e:
                 result.set_exception(e)
 
-        aux_get_future = self.async_aux_get(
-            mypkg, ["EAPI", "SRC_URI"], mytree=mytree, loop=loop
+        aux_get_future = asyncio.ensure_future(
+            self.async_aux_get(mypkg, ["EAPI", "SRC_URI"], mytree=mytree, loop=loop),
+            loop,
         )
         result.add_done_callback(
             lambda result: aux_get_future.cancel() if result.cancelled() else None
diff --git a/lib/portage/tests/update/test_move_ent.py b/lib/portage/tests/update/test_move_ent.py
index fe968f12a0..0b938dd287 100644
--- a/lib/portage/tests/update/test_move_ent.py
+++ b/lib/portage/tests/update/test_move_ent.py
@@ -231,6 +231,9 @@ class MoveEntTestCase(TestCase):
                 finally:
                     playground.cleanup()
 
+    # Ignore "The loop argument is deprecated" since this argument is conditionally
+    # added to asyncio.Lock as needed for compatibility with python 3.9.
+    @pytest.mark.filterwarnings("ignore:The loop argument is deprecated")
     @pytest.mark.filterwarnings("error")
     def testMoveEntWithCorruptIndex(self):
         """
-- 
2.41.0

[gentoo-portage-dev] [PATCH 7/9] actions: disable pytest-xdist for spawn start-method (workers crash)

[Zac Medico]

Bug: https://bugs.gentoo.org/924416
Signed-off-by: Zac Medico <zmedico@gentoo.org>
---
 .github/workflows/ci.yml | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 5bffd97206..762999b7cc 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -90,5 +90,8 @@ jobs:
       - name: Run tests for ${{ matrix.python-version }}
         run: |
           [[ "${{ matrix.start-method }}" == "spawn" ]] && export PORTAGE_MULTIPROCESSING_START_METHOD=spawn
-          export PYTEST_ADDOPTS="-vv -ra -l -o console_output_style=count -n $(nproc) --dist=worksteal"
+          # spawn start-method crashes pytest-xdist workers (bug 924416)
+          [[ "${{ matrix.start-method }}" == "spawn" ]] && \
+            export PYTEST_ADDOPTS="-vv -ra -l -o console_output_style=count" || \
+            export PYTEST_ADDOPTS="-vv -ra -l -o console_output_style=count -n $(nproc) --dist=worksteal"
           meson test -C /tmp/build --verbose
-- 
2.41.0

[gentoo-portage-dev] [PATCH 8/9] EbuildMetadataPhase: Migrate to _async_start

[Zac Medico]

Bug: https://bugs.gentoo.org/923841
Signed-off-by: Zac Medico <zmedico@gentoo.org>
---
 lib/_emerge/EbuildMetadataPhase.py | 17 ++++++-----------
 1 file changed, 6 insertions(+), 11 deletions(-)

diff --git a/lib/_emerge/EbuildMetadataPhase.py b/lib/_emerge/EbuildMetadataPhase.py
index 784712e8cb..9fcdabe840 100644
--- a/lib/_emerge/EbuildMetadataPhase.py
+++ b/lib/_emerge/EbuildMetadataPhase.py
@@ -46,6 +46,12 @@ class EbuildMetadataPhase(SubProcess):
     _files_dict = slot_dict_class(_file_names, prefix="")
 
     def _start(self):
+        asyncio.ensure_future(
+            self._async_start(), loop=self.scheduler
+        ).add_done_callback(self._async_start_done)
+        self._registered = True
+
+    async def _async_start(self):
         ebuild_path = self.ebuild_hash.location
 
         with open(
@@ -116,7 +122,6 @@ class EbuildMetadataPhase(SubProcess):
         self._raw_metadata = []
         files.ebuild = master_fd
         self.scheduler.add_reader(files.ebuild, self._output_handler)
-        self._registered = True
 
         retval = portage.doebuild(
             ebuild_path,
@@ -150,16 +155,6 @@ class EbuildMetadataPhase(SubProcess):
 
         self._proc = retval
 
-        asyncio.ensure_future(
-            self._async_start(), loop=self.scheduler
-        ).add_done_callback(self._async_start_done)
-
-    async def _async_start(self):
-        # Call async check_locale here for bug 923841, but code
-        # also needs to migrate from _start to here, including
-        # the self.deallocate_config set_result call.
-        pass
-
     def _async_start_done(self, future):
         future.cancelled() or future.result()
         if self._was_cancelled():
-- 
2.41.0

[gentoo-portage-dev] [PATCH 9/9] EbuildPhase: async_check_locale

[Zac Medico]

Change config.environ() check_locale calls to async_check_locale
calls in the EbuildPhase _async_start method in order to eliminate
synchronous waiting for child processes in the main event loop
thread.

Bug: https://bugs.gentoo.org/923841
Signed-off-by: Zac Medico <zmedico@gentoo.org>
---
 lib/_emerge/EbuildMetadataPhase.py            |  4 +++
 lib/_emerge/EbuildPhase.py                    | 28 ++++++++++++++++++-
 lib/portage/package/ebuild/config.py          | 26 ++++++++---------
 lib/portage/util/futures/_asyncio/__init__.py |  9 ++++++
 lib/portage/util/locale.py                    | 28 +++++++++++++------
 5 files changed, 70 insertions(+), 25 deletions(-)

diff --git a/lib/_emerge/EbuildMetadataPhase.py b/lib/_emerge/EbuildMetadataPhase.py
index 9fcdabe840..90a3ea05aa 100644
--- a/lib/_emerge/EbuildMetadataPhase.py
+++ b/lib/_emerge/EbuildMetadataPhase.py
@@ -8,6 +8,7 @@ import portage
 
 portage.proxy.lazyimport.lazyimport(
     globals(),
+    "_emerge.EbuildPhase:_setup_locale",
     "portage.package.ebuild._metadata_invalid:eapi_invalid",
 )
 from portage import os
@@ -83,6 +84,9 @@ class EbuildMetadataPhase(SubProcess):
         settings.setcpv(self.cpv)
         settings.configdict["pkg"]["EAPI"] = parsed_eapi
 
+        # This requires above setcpv and EAPI setup.
+        await _setup_locale(self.settings)
+
         debug = settings.get("PORTAGE_DEBUG") == "1"
         master_fd = None
         slave_fd = None
diff --git a/lib/_emerge/EbuildPhase.py b/lib/_emerge/EbuildPhase.py
index c81bf54a81..c8caf73722 100644
--- a/lib/_emerge/EbuildPhase.py
+++ b/lib/_emerge/EbuildPhase.py
@@ -1,4 +1,4 @@
-# Copyright 1999-2021 Gentoo Authors
+# Copyright 1999-2024 Gentoo Authors
 # Distributed under the terms of the GNU General Public License v2
 
 import functools
@@ -24,6 +24,7 @@ from portage.package.ebuild.prepare_build_dirs import (
     _prepare_fake_distdir,
     _prepare_fake_filesdir,
 )
+from portage.eapi import _get_eapi_attrs
 from portage.util import writemsg, ensure_dirs
 from portage.util._async.AsyncTaskFuture import AsyncTaskFuture
 from portage.util._async.BuildLogger import BuildLogger
@@ -54,12 +55,34 @@ portage.proxy.lazyimport.lazyimport(
     + "_post_src_install_write_metadata,"
     + "_preinst_bsdflags",
     "portage.util.futures.unix_events:_set_nonblocking",
+    "portage.util.locale:async_check_locale,split_LC_ALL",
 )
 from portage import os
 from portage import _encodings
 from portage import _unicode_encode
 
 
+async def _setup_locale(settings):
+    eapi_attrs = _get_eapi_attrs(settings["EAPI"])
+    if eapi_attrs.posixish_locale:
+        split_LC_ALL(settings)
+        settings["LC_COLLATE"] = "C"
+        # check_locale() returns None when check can not be executed.
+        if await async_check_locale(silent=True, env=settings.environ()) is False:
+            # try another locale
+            for l in ("C.UTF-8", "en_US.UTF-8", "en_GB.UTF-8", "C"):
+                settings["LC_CTYPE"] = l
+                if await async_check_locale(silent=True, env=settings.environ()):
+                    # TODO: output the following only once
+                    # writemsg(
+                    #     _("!!! LC_CTYPE unsupported, using %s instead\n")
+                    #     % self.settings["LC_CTYPE"]
+                    # )
+                    break
+            else:
+                raise AssertionError("C locale did not pass the test!")
+
+
 class EbuildPhase(CompositeTask):
     __slots__ = ("actionmap", "fd_pipes", "phase", "settings") + ("_ebuild_lock",)
 
@@ -94,6 +117,9 @@ class EbuildPhase(CompositeTask):
         self._start_task(AsyncTaskFuture(future=future), self._async_start_exit)
 
     async def _async_start(self):
+
+        await _setup_locale(self.settings)
+
         need_builddir = self.phase not in EbuildProcess._phases_without_builddir
 
         if need_builddir:
diff --git a/lib/portage/package/ebuild/config.py b/lib/portage/package/ebuild/config.py
index d7b0ca5676..35c77486ec 100644
--- a/lib/portage/package/ebuild/config.py
+++ b/lib/portage/package/ebuild/config.py
@@ -29,7 +29,6 @@ portage.proxy.lazyimport.lazyimport(
     "portage.dbapi.vartree:vartree",
     "portage.package.ebuild.doebuild:_phase_func_map",
     "portage.util.compression_probe:_compressors",
-    "portage.util.locale:check_locale,split_LC_ALL",
 )
 from portage import bsd_chflags, load_mod, os, selinux, _unicode_decode
 from portage.const import (
@@ -3368,20 +3367,17 @@ class config:
                 mydict["EBUILD_PHASE_FUNC"] = phase_func
 
         if eapi_attrs.posixish_locale:
-            split_LC_ALL(mydict)
-            mydict["LC_COLLATE"] = "C"
-            # check_locale() returns None when check can not be executed.
-            if check_locale(silent=True, env=mydict) is False:
-                # try another locale
-                for l in ("C.UTF-8", "en_US.UTF-8", "en_GB.UTF-8", "C"):
-                    mydict["LC_CTYPE"] = l
-                    if check_locale(silent=True, env=mydict):
-                        # TODO: output the following only once
-                        # 						writemsg(_("!!! LC_CTYPE unsupported, using %s instead\n")
-                        # 								% mydict["LC_CTYPE"])
-                        break
-                else:
-                    raise AssertionError("C locale did not pass the test!")
+            if mydict.get("LC_ALL"):
+                # Sometimes this method is called for processes
+                # that are not ebuild phases, so only raise
+                # AssertionError for actual ebuild phases.
+                if phase and phase not in ("clean", "cleanrm", "fetch"):
+                    raise AssertionError(
+                        f"LC_ALL={mydict['LC_ALL']} for posixish locale. It seems that split_LC_ALL was not called for phase {phase}?"
+                    )
+            elif "LC_ALL" in mydict:
+                # Delete placeholder from split_LC_ALL.
+                del mydict["LC_ALL"]
 
         if not eapi_attrs.exports_PORTDIR:
             mydict.pop("PORTDIR", None)
diff --git a/lib/portage/util/futures/_asyncio/__init__.py b/lib/portage/util/futures/_asyncio/__init__.py
index b6481c281e..22241f335d 100644
--- a/lib/portage/util/futures/_asyncio/__init__.py
+++ b/lib/portage/util/futures/_asyncio/__init__.py
@@ -16,6 +16,7 @@ __all__ = (
     "set_child_watcher",
     "get_event_loop_policy",
     "set_event_loop_policy",
+    "run",
     "shield",
     "sleep",
     "Task",
@@ -109,6 +110,14 @@ def set_child_watcher(watcher):
     return get_event_loop_policy().set_child_watcher(watcher)
 
 
+# Emulate run since it's the preferred python API.
+def run(coro):
+    return _safe_loop().run_until_complete(coro)
+
+
+run.__doc__ = _real_asyncio.run.__doc__
+
+
 def create_subprocess_exec(*args, **kwargs):
     """
     Create a subprocess.
diff --git a/lib/portage/util/locale.py b/lib/portage/util/locale.py
index b5da8d949b..b6a41e7655 100644
--- a/lib/portage/util/locale.py
+++ b/lib/portage/util/locale.py
@@ -17,6 +17,7 @@ import traceback
 import portage
 from portage.util import _unicode_decode, writemsg_level
 from portage.util._ctypes import find_library, LoadLibrary
+from portage.util.futures import asyncio
 
 
 locale_categories = (
@@ -121,7 +122,10 @@ def check_locale(silent=False, env=None):
     warning and returns False if it is not. Returns None if the check
     can not be executed due to platform limitations.
     """
+    return asyncio.run(async_check_locale(silent=silent, env=env))
 
+
+async def async_check_locale(silent=False, env=None):
     if env is not None:
         for v in ("LC_ALL", "LC_CTYPE", "LANG"):
             if v in env:
@@ -135,20 +139,17 @@ def check_locale(silent=False, env=None):
         except KeyError:
             pass
 
-    # TODO: Make async version of check_locale and call it from
-    # EbuildPhase instead of config.environ(), since it's bad to
-    # synchronously wait for the process in the main event loop
-    # thread where config.environ() tends to be called.
     proc = multiprocessing.Process(
         target=_set_and_check_locale,
         args=(silent, env, None if env is None else portage._native_string(mylocale)),
     )
     proc.start()
-    proc.join()
+    proc = portage.process.MultiprocessingProcess(proc)
+    await proc.wait()
 
     pyret = None
-    if proc.exitcode >= 0:
-        ret = proc.exitcode
+    if proc.returncode >= 0:
+        ret = proc.returncode
         if ret != 2:
             pyret = ret == 0
 
@@ -157,13 +158,22 @@ def check_locale(silent=False, env=None):
     return pyret
 
 
+async_check_locale.__doc__ = check_locale.__doc__
+async_check_locale.__doc__ += """
+    This function is a coroutine.
+"""
+
+
 def split_LC_ALL(env):
     """
     Replace LC_ALL with split-up LC_* variables if it is defined.
     Works on the passed environment (or settings instance).
     """
     lc_all = env.get("LC_ALL")
-    if lc_all is not None:
+    if lc_all:
         for c in locale_categories:
             env[c] = lc_all
-        del env["LC_ALL"]
+        # Set empty so that config.reset() can restore LC_ALL state,
+        # since del can permanently delete variables which are not
+        # stored in the config's backupenv.
+        env["LC_ALL"] = ""
-- 
2.41.0