[gentoo-commits] proj/portage:master commit in: lib/portage/util/futures/_asyncio/

[gentoo-commits] proj/portage:master commit in: lib/portage/util/futures/_asyncio/

[Zac Medico]

commit:     d718cea94a180042b2285698b2c19113c5d25987
Author:     Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Thu Feb 22 06:41:49 2024 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Thu Feb 22 07:28:38 2024 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=d718cea9

_get_running_loop: Support real asyncio.run

When called via the real asyncio.run implementation, wrap
the running asyncio loop. Otherwise, it's not possible to
call portage libraries via the real asyncio.run without
triggering Future "attached to a different loop" errors.

Bug: https://bugs.gentoo.org/761538
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/portage/util/futures/_asyncio/__init__.py | 28 +++++++++++++++++++++++++--
 1 file changed, 26 insertions(+), 2 deletions(-)

diff --git a/lib/portage/util/futures/_asyncio/__init__.py b/lib/portage/util/futures/_asyncio/__init__.py
index 22241f335d..4eecc46a89 100644
--- a/lib/portage/util/futures/_asyncio/__init__.py
+++ b/lib/portage/util/futures/_asyncio/__init__.py
@@ -325,13 +325,37 @@ def _safe_loop():
 
 
 def _get_running_loop():
+    """
+    This calls the real asyncio get_running_loop() and wraps that with
+    portage's internal AsyncioEventLoop wrapper. If there is no running
+    asyncio event loop but portage has a reference to another running
+    loop in this thread, then use that instead.
+
+    This behavior enables portage internals to use the real asyncio.run
+    while remaining compatible with internal code that does not use the
+    real asyncio.run.
+    """
+    try:
+        _loop = _real_asyncio.get_running_loop()
+    except RuntimeError:
+        _loop = None
+
     with _thread_weakrefs.lock:
         if _thread_weakrefs.pid == portage.getpid():
             try:
                 loop = _thread_weakrefs.loops[threading.get_ident()]
             except KeyError:
-                return None
-            return loop if loop.is_running() else None
+                pass
+            else:
+                if _loop is loop._loop:
+                    return loop
+                elif _loop is None:
+                    return loop if loop.is_running() else None
+
+    # If _loop it not None here it means it was probably a temporary
+    # loop created by asyncio.run, so we don't try to cache it, and
+    # just return a temporary wrapper.
+    return None if _loop is None else _AsyncioEventLoop(loop=_loop)
 
 
 def _thread_weakrefs_atexit():

[gentoo-commits] proj/portage:master commit in: lib/portage/util/futures/_asyncio/

[Zac Medico]

commit:     e5457915f7929db3781ded384bdb089b0760221f
Author:     Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Sat Aug 31 17:32:12 2024 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Sun Sep  1 06:59:34 2024 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=e5457915

asyncio: Use default ensure_future implementation when possible

When a loop argument is not given, use the default asyncio
ensure_future implementation and avoid unnecessary _wrap_loop
usage.

Bug: https://bugs.gentoo.org/761538
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/portage/util/futures/_asyncio/__init__.py | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/lib/portage/util/futures/_asyncio/__init__.py b/lib/portage/util/futures/_asyncio/__init__.py
index a235d87246..48d9b68104 100644
--- a/lib/portage/util/futures/_asyncio/__init__.py
+++ b/lib/portage/util/futures/_asyncio/__init__.py
@@ -186,6 +186,9 @@ def ensure_future(coro_or_future, loop=None):
     @rtype: asyncio.Future (or compatible)
     @return: an instance of Future
     """
+    if loop is None:
+        return _real_asyncio.ensure_future(coro_or_future)
+
     loop = _wrap_loop(loop)
     if isinstance(loop._asyncio_wrapper, _AsyncioEventLoop):
         # Use the real asyncio loop and ensure_future.

[gentoo-commits] proj/portage:master commit in: lib/portage/util/futures/_asyncio/

[Zac Medico]

commit:     b30ddb1913e8aa2947d20e43f455d2060aa6257f
Author:     Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Sat Aug 31 20:08:35 2024 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Sun Sep  1 06:59:44 2024 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=b30ddb19

asyncio: Avoid _wrap_loop prior to create_subprocess_exec

Bug: https://bugs.gentoo.org/761538
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/portage/util/futures/_asyncio/__init__.py | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/lib/portage/util/futures/_asyncio/__init__.py b/lib/portage/util/futures/_asyncio/__init__.py
index 48d9b68104..4cf337998c 100644
--- a/lib/portage/util/futures/_asyncio/__init__.py
+++ b/lib/portage/util/futures/_asyncio/__init__.py
@@ -119,7 +119,7 @@ def run(coro):
 run.__doc__ = _real_asyncio.run.__doc__
 
 
-def create_subprocess_exec(*args, **kwargs):
+def create_subprocess_exec(*args, loop=None, **kwargs):
     """
     Create a subprocess.
 
@@ -140,7 +140,6 @@ def create_subprocess_exec(*args, **kwargs):
     @rtype: asyncio.subprocess.Process (or compatible)
     @return: asyncio.subprocess.Process interface
     """
-    loop = _wrap_loop(kwargs.pop("loop", None))
     # Python 3.4 and later implement PEP 446, which makes newly
     # created file descriptors non-inheritable by default.
     kwargs.setdefault("close_fds", False)

[gentoo-commits] proj/portage:master commit in: lib/portage/util/futures/_asyncio/

[Zac Medico]

commit:     7f6b2b04878209130d44fc06105f521bae2b2173
Author:     Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Sat Aug 31 05:35:32 2024 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Sun Sep  1 06:58:56 2024 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=7f6b2b04

asyncio: Use default sleep implementation when possible

When a loop argument is not given, use the default asyncio sleep
implementation and avoid unnecessary _wrap_loop usage.

Bug: https://bugs.gentoo.org/761538
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/portage/util/futures/_asyncio/__init__.py | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/lib/portage/util/futures/_asyncio/__init__.py b/lib/portage/util/futures/_asyncio/__init__.py
index 23c664e763..a235d87246 100644
--- a/lib/portage/util/futures/_asyncio/__init__.py
+++ b/lib/portage/util/futures/_asyncio/__init__.py
@@ -210,9 +210,12 @@ def sleep(delay, result=None, loop=None):
     @param result: result of the future
     @type loop: asyncio.AbstractEventLoop (or compatible)
     @param loop: event loop
-    @rtype: asyncio.Future (or compatible)
-    @return: an instance of Future
+    @rtype: collections.abc.Coroutine or asyncio.Future
+    @return: an instance of Coroutine or Future
     """
+    if loop is None:
+        return _real_asyncio.sleep(delay, result=result)
+
     loop = _wrap_loop(loop)
     future = loop.create_future()
     handle = loop.call_later(delay, future.set_result, result)

[gentoo-commits] proj/portage:master commit in: lib/portage/util/futures/_asyncio/

[Zac Medico]

commit:     ee17cbd807ba976491e4c657be8aa9b9a29fe059
Author:     Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Sat Aug 31 19:06:25 2024 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Sat Aug 31 19:06:25 2024 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=ee17cbd8

_safe_loop: Discard wrapped asyncio.run loop that was closed

Since commit cb0c09d8cecb, _get_running_loop can wrap loops from
asyncio.run, so these loops need to be discarded if they've been
closed.

Fixes: cb0c09d8cecb ("Support coroutine exitfuncs for non-main loops")
Bug: https://bugs.gentoo.org/938761
Bug: https://bugs.gentoo.org/761538
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/portage/util/futures/_asyncio/__init__.py | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/lib/portage/util/futures/_asyncio/__init__.py b/lib/portage/util/futures/_asyncio/__init__.py
index bdacda59ce..23c664e763 100644
--- a/lib/portage/util/futures/_asyncio/__init__.py
+++ b/lib/portage/util/futures/_asyncio/__init__.py
@@ -311,6 +311,13 @@ def _safe_loop(create: Optional[bool] = True) -> Optional[_AsyncioEventLoop]:
             _thread_weakrefs.loops = weakref.WeakValueDictionary()
         try:
             loop = _thread_weakrefs.loops[thread_key]
+            if loop.is_closed():
+                # Discard wrapped asyncio.run loop that was closed.
+                del _thread_weakrefs.loops[thread_key]
+                if loop is _thread_weakrefs.mainloop:
+                    _thread_weakrefs.mainloop = None
+                loop = None
+                raise KeyError(thread_key)
         except KeyError:
             if not create:
                 return None

[gentoo-commits] proj/portage:master commit in: lib/portage/util/futures/_asyncio/

[Zac Medico]

commit:     a62faf99dbd0078cd58a76e6419e0a2d0d14d636
Author:     Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Sun Aug 18 15:00:07 2024 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Sun Aug 18 15:47:01 2024 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=a62faf99

Remove unused and unimplemented asyncio.Task class

This class originated from commit 142d08c0636b and it is unused
since _PortageEventLoop was removed in commit 20204fd8c29.

Fixes: 20204fd8c29 ("Remove unused _PortageEventLoop and _PortageChildWatcher")
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/portage/util/futures/_asyncio/__init__.py | 11 -----------
 1 file changed, 11 deletions(-)

diff --git a/lib/portage/util/futures/_asyncio/__init__.py b/lib/portage/util/futures/_asyncio/__init__.py
index c960d03630..bdacda59ce 100644
--- a/lib/portage/util/futures/_asyncio/__init__.py
+++ b/lib/portage/util/futures/_asyncio/__init__.py
@@ -19,7 +19,6 @@ __all__ = (
     "run",
     "shield",
     "sleep",
-    "Task",
     "wait",
     "wait_for",
 )
@@ -174,16 +173,6 @@ class Lock(_Lock):
         super().__init__(**kwargs)
 
 
-class Task(Future):
-    """
-    Schedule the execution of a coroutine: wrap it in a future. A task
-    is a subclass of Future.
-    """
-
-    def __init__(self, coro, loop=None):
-        raise NotImplementedError
-
-
 def ensure_future(coro_or_future, loop=None):
     """
     Wrap a coroutine or an awaitable in a future.

[gentoo-commits] proj/portage:master commit in: lib/portage/util/futures/_asyncio/

[Zac Medico]

commit:     e97acd3c62ff02eb41ff643e75eb5e07c27717f8
Author:     Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Sun Aug 18 14:59:46 2024 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Sun Aug 18 15:46:45 2024 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=e97acd3c

_wrap_loop: Prevent redundant AsyncioEventLoop instances

Ultimately the loop arguments that necessitate the _wrap_loop
function can be removed, because our aim since bug 761538 should
be to eliminate them. Meanwhile, we don't want _wrap_loop to return
redundant AsyncioEventLoop instances if we can easily prevent it.

Therefore, use _safe_loop(create=False) to look up the AsyncioEventLoop
instance associated with the current thread, and avoid creating
redundant instances. This serves to guard against garbage collection
of AsyncioEventLoop instances which may have _coroutine_exithandlers
added by the atexit_register function since commit cb0c09d8cecb from
bug 937740.

If _safe_loop(create=False) fails to associate a loop with the current
thread, raise an AssertionError for portage internal API consumers.
It's not known whether external API consumers will trigger this case,
so if it happens then emit a UserWarning and return a temporary
AsyncioEventLoop instance.

Fixes: cb0c09d8cecb ("Support coroutine exitfuncs for non-main loops")
Bug: https://bugs.gentoo.org/938127
Bug: https://bugs.gentoo.org/937740
Bug: https://bugs.gentoo.org/761538
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/portage/util/futures/_asyncio/__init__.py | 43 +++++++++++++++++++++++----
 1 file changed, 38 insertions(+), 5 deletions(-)

diff --git a/lib/portage/util/futures/_asyncio/__init__.py b/lib/portage/util/futures/_asyncio/__init__.py
index 8805e35756..c960d03630 100644
--- a/lib/portage/util/futures/_asyncio/__init__.py
+++ b/lib/portage/util/futures/_asyncio/__init__.py
@@ -26,6 +26,7 @@ __all__ = (
 
 import sys
 import types
+import warnings
 import weakref
 
 import asyncio as _real_asyncio
@@ -46,6 +47,7 @@ from asyncio import (
 )
 
 import threading
+from typing import Optional
 
 import portage
 
@@ -251,11 +253,35 @@ def _wrap_loop(loop=None):
     # The default loop returned by _wrap_loop should be consistent
     # with global_event_loop, in order to avoid accidental registration
     # of callbacks with a loop that is not intended to run.
-    loop = loop or _safe_loop()
-    return loop if hasattr(loop, "_asyncio_wrapper") else _AsyncioEventLoop(loop=loop)
+    if hasattr(loop, "_asyncio_wrapper"):
+        return loop
+
+    # This returns a running loop if it exists, and otherwise returns
+    # a loop associated with the current thread.
+    safe_loop = _safe_loop(create=loop is None)
+    if safe_loop is not None and (loop is None or safe_loop._loop is loop):
+        return safe_loop
+
+    if safe_loop is None:
+        msg = f"_wrap_loop argument '{loop}' not associated with thread '{threading.get_ident()}'"
+    else:
+        msg = f"_wrap_loop argument '{loop}' different frome loop '{safe_loop._loop}' already associated with thread '{threading.get_ident()}'"
+
+    if portage._internal_caller:
+        raise AssertionError(msg)
 
+    # It's not known whether external API consumers will trigger this case,
+    # so if it happens then emit a UserWarning before returning a temporary
+    # AsyncioEventLoop instance.
+    warnings.warn(msg, UserWarning, stacklevel=2)
 
-def _safe_loop():
+    # We could possibly add a weak reference in _thread_weakrefs.loops when
+    # safe_loop is None, but if safe_loop is not None, then there is a
+    # collision in _thread_weakrefs.loops that would need to be resolved.
+    return _AsyncioEventLoop(loop=loop)
+
+
+def _safe_loop(create: Optional[bool] = True) -> Optional[_AsyncioEventLoop]:
     """
     Return an event loop that's safe to use within the current context.
     For portage internal callers or external API consumers calling from
@@ -276,8 +302,13 @@ def _safe_loop():
     are added to a WeakValueDictionary, and closed via an atexit hook
     if they still exist during exit for the current pid.
 
-    @rtype: asyncio.AbstractEventLoop (or compatible)
-    @return: event loop instance
+    @type create: bool
+    @param create: Create a loop by default if a loop is not already associated
+        with the current thread. If create is False, then return None if a loop
+        is not already associated with the current thread.
+    @rtype: AsyncioEventLoop or None
+    @return: event loop instance, or None if the create parameter is False and
+        a loop is not already associated with the current thread.
     """
     loop = _get_running_loop()
     if loop is not None:
@@ -292,6 +323,8 @@ def _safe_loop():
         try:
             loop = _thread_weakrefs.loops[thread_key]
         except KeyError:
+            if not create:
+                return None
             try:
                 try:
                     _loop = _real_asyncio.get_running_loop()

[gentoo-commits] proj/portage:master commit in: lib/portage/util/futures/_asyncio/

[Zac Medico]

commit:     a42c2164ada634262ae1f791ad60298fe3468a94
Author:     Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Tue Feb 13 03:39:35 2024 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Wed Feb 21 15:27:31 2024 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=a42c2164

asyncio: Wrap asyncio.Lock for python 3.9 compat

Wrap asyncio.Lock for compatibility with python 3.9 where the
deprecated loop parameter is required in order to avoid "got
Future <Future pending> attached to a different loop" errors.

The pordbapi async_aux_get method can use asyncio.Lock to
serialize access to its doebuild_settings attribute in order
to prevent issues like bug 924319.

Bug: https://bugs.gentoo.org/924319
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/portage/util/futures/_asyncio/__init__.py | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)

diff --git a/lib/portage/util/futures/_asyncio/__init__.py b/lib/portage/util/futures/_asyncio/__init__.py
index 8f1b8e8275..b6481c281e 100644
--- a/lib/portage/util/futures/_asyncio/__init__.py
+++ b/lib/portage/util/futures/_asyncio/__init__.py
@@ -9,6 +9,7 @@ __all__ = (
     "CancelledError",
     "Future",
     "InvalidStateError",
+    "Lock",
     "TimeoutError",
     "get_child_watcher",
     "get_event_loop",
@@ -22,6 +23,7 @@ __all__ = (
     "wait_for",
 )
 
+import sys
 import types
 import weakref
 
@@ -35,6 +37,7 @@ from asyncio import (
     FIRST_EXCEPTION,
     Future,
     InvalidStateError,
+    Lock as _Lock,
     shield,
     TimeoutError,
     wait_for,
@@ -159,6 +162,20 @@ def iscoroutinefunction(func):
     return False
 
 
+class Lock(_Lock):
+    """
+    Inject loop parameter for python3.9 or less in order to avoid
+    "got Future <Future pending> attached to a different loop" errors.
+    """
+
+    def __init__(self, **kwargs):
+        if sys.version_info >= (3, 10):
+            kwargs.pop("loop", None)
+        elif "loop" not in kwargs:
+            kwargs["loop"] = _safe_loop()._loop
+        super().__init__(**kwargs)
+
+
 class Task(Future):
     """
     Schedule the execution of a coroutine: wrap it in a future. A task

[gentoo-commits] proj/portage:master commit in: lib/portage/util/futures/_asyncio/

[Zac Medico]

commit:     631bedffe29124d693de3b539fc908d9feec1420
Author:     Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Mon Sep 20 05:05:38 2021 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Mon Sep 20 05:08:47 2021 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=631bedff

_safe_loop: fix python3.10 DeprecationWarning

DeprecationWarning: There is no current event loop

Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/portage/util/futures/_asyncio/__init__.py | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)

diff --git a/lib/portage/util/futures/_asyncio/__init__.py b/lib/portage/util/futures/_asyncio/__init__.py
index c1229528a..ccf800c66 100644
--- a/lib/portage/util/futures/_asyncio/__init__.py
+++ b/lib/portage/util/futures/_asyncio/__init__.py
@@ -280,10 +280,14 @@ def _safe_loop():
             loop = _thread_weakrefs.loops[thread_key]
         except KeyError:
             try:
-                _real_asyncio.get_event_loop()
+                try:
+                    _loop = _real_asyncio.get_running_loop()
+                except AttributeError:
+                    _loop = _real_asyncio.get_event_loop()
             except RuntimeError:
-                _real_asyncio.set_event_loop(_real_asyncio.new_event_loop())
-            loop = _thread_weakrefs.loops[thread_key] = _AsyncioEventLoop()
+                _loop = _real_asyncio.new_event_loop()
+                _real_asyncio.set_event_loop(_loop)
+            loop = _thread_weakrefs.loops[thread_key] = _AsyncioEventLoop(loop=_loop)
 
     if (
         _thread_weakrefs.mainloop is None

[gentoo-commits] proj/portage:master commit in: lib/portage/util/futures/_asyncio/

[Zac Medico]

commit:     9eea2af22c9e51475a4adba57fdded3a2a88c886
Author:     Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Sun Mar  7 14:18:44 2021 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Sun Mar  7 14:37:46 2021 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=9eea2af2

_writer: Use async and await syntax

Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/portage/util/futures/_asyncio/streams.py | 13 +++++--------
 1 file changed, 5 insertions(+), 8 deletions(-)

diff --git a/lib/portage/util/futures/_asyncio/streams.py b/lib/portage/util/futures/_asyncio/streams.py
index ea5882dd3..7a8d4a3e0 100644
--- a/lib/portage/util/futures/_asyncio/streams.py
+++ b/lib/portage/util/futures/_asyncio/streams.py
@@ -1,4 +1,4 @@
-# Copyright 2018-2020 Gentoo Authors
+# Copyright 2018-2021 Gentoo Authors
 # Distributed under the terms of the GNU General Public License v2
 
 import errno
@@ -9,7 +9,6 @@ portage.proxy.lazyimport.lazyimport(globals(),
 	'_emerge.PipeReader:PipeReader',
 	'portage.util.futures:asyncio',
 )
-from portage.util.futures.compat_coroutine import coroutine
 
 
 def _reader(input_file, loop=None):
@@ -55,8 +54,7 @@ class _Reader:
 			self._pipe_reader = None
 
 
-@coroutine
-def _writer(output_file, content, loop=None):
+async def _writer(output_file, content, loop=DeprecationWarning):
 	"""
 	Asynchronously write bytes to output file. The output file is
 	assumed to be in non-blocking mode. If an EnvironmentError
@@ -68,10 +66,9 @@ def _writer(output_file, content, loop=None):
 	@type output_file: file object
 	@param content: content to write
 	@type content: bytes
-	@param loop: asyncio.AbstractEventLoop (or compatible)
-	@type loop: event loop
+	@param loop: deprecated
 	"""
-	loop = asyncio._wrap_loop(loop)
+	loop = asyncio.get_event_loop()
 	fd = output_file.fileno()
 	while content:
 		try:
@@ -82,7 +79,7 @@ def _writer(output_file, content, loop=None):
 			waiter = loop.create_future()
 			loop.add_writer(fd, lambda: waiter.done() or waiter.set_result(None))
 			try:
-				yield waiter
+				await waiter
 			finally:
 				# The loop and output file may have been closed.
 				if not loop.is_closed():

[gentoo-commits] proj/portage:master commit in: lib/portage/util/futures/_asyncio/

[Zac Medico]

commit:     15049a041909f85b02a52f5b1938c6dd4171c9e3
Author:     Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Sun Mar  7 03:03:40 2021 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Sun Mar  7 05:12:48 2021 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=15049a04

Removed unused portage.util.futures._asyncio.tasks

Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/portage/util/futures/_asyncio/__init__.py | 46 ++++++-------
 lib/portage/util/futures/_asyncio/tasks.py    | 96 ---------------------------
 2 files changed, 19 insertions(+), 123 deletions(-)

diff --git a/lib/portage/util/futures/_asyncio/__init__.py b/lib/portage/util/futures/_asyncio/__init__.py
index 207e7205d..4643697e0 100644
--- a/lib/portage/util/futures/_asyncio/__init__.py
+++ b/lib/portage/util/futures/_asyncio/__init__.py
@@ -25,7 +25,16 @@ import types
 import weakref
 
 import asyncio as _real_asyncio
-from asyncio.subprocess import Process
+# pylint: disable=redefined-builtin
+from asyncio import (
+	ALL_COMPLETED,
+	CancelledError,
+	FIRST_COMPLETED,
+	FIRST_EXCEPTION,
+	Future,
+	InvalidStateError,
+	TimeoutError,
+)
 
 try:
 	import threading
@@ -38,20 +47,6 @@ portage.proxy.lazyimport.lazyimport(globals(),
 	'portage.util.futures:compat_coroutine@_compat_coroutine',
 )
 from portage.util._eventloop.asyncio_event_loop import AsyncioEventLoop as _AsyncioEventLoop
-# pylint: disable=redefined-builtin
-from portage.util.futures.futures import (
-	CancelledError,
-	Future,
-	InvalidStateError,
-	TimeoutError,
-)
-# pylint: enable=redefined-builtin
-from portage.util.futures._asyncio.tasks import (
-	ALL_COMPLETED,
-	FIRST_COMPLETED,
-	FIRST_EXCEPTION,
-	wait,
-)
 
 
 _lock = threading.Lock()
@@ -131,20 +126,17 @@ def create_subprocess_exec(*args, **kwargs):
 	# Python 3.4 and later implement PEP 446, which makes newly
 	# created file descriptors non-inheritable by default.
 	kwargs.setdefault('close_fds', False)
-	if isinstance(loop._asyncio_wrapper, _AsyncioEventLoop):
-		# Use the real asyncio create_subprocess_exec (loop argument
-		# is deprecated since since Python 3.8).
-		return _real_asyncio.create_subprocess_exec(*args, **kwargs)
-
-	result = loop.create_future()
+	# Use the real asyncio create_subprocess_exec (loop argument
+	# is deprecated since since Python 3.8).
+	return ensure_future(_real_asyncio.create_subprocess_exec(*args, **kwargs), loop=loop)
 
-	result.set_result(Process(subprocess.Popen(
-		args,
-		stdin=kwargs.pop('stdin', None),
-		stdout=kwargs.pop('stdout', None),
-		stderr=kwargs.pop('stderr', None), **kwargs), loop))
 
-	return result
+def wait(futures, loop=None, timeout=None, return_when=ALL_COMPLETED):
+	"""
+	Wraps asyncio.wait() and omits the loop argument which is not
+	supported since python 3.10.
+	"""
+	return _real_asyncio.wait(futures, timeout=timeout, return_when=return_when)
 
 
 def iscoroutinefunction(func):

diff --git a/lib/portage/util/futures/_asyncio/tasks.py b/lib/portage/util/futures/_asyncio/tasks.py
deleted file mode 100644
index c9db3146e..000000000
--- a/lib/portage/util/futures/_asyncio/tasks.py
+++ /dev/null
@@ -1,96 +0,0 @@
-# Copyright 2018-2020 Gentoo Authors
-# Distributed under the terms of the GNU General Public License v2
-
-___all___ = (
-	'ALL_COMPLETED',
-	'FIRST_COMPLETED',
-	'FIRST_EXCEPTION',
-	'wait',
-)
-
-from asyncio import ALL_COMPLETED, FIRST_COMPLETED, FIRST_EXCEPTION
-
-import portage
-portage.proxy.lazyimport.lazyimport(globals(),
-	'portage.util.futures:asyncio',
-)
-
-def wait(futures, loop=None, timeout=None, return_when=ALL_COMPLETED):
-	"""
-	Use portage's internal EventLoop to emulate asyncio.wait:
-	https://docs.python.org/3/library/asyncio-task.html#asyncio.wait
-
-	@param futures: futures to wait for
-	@type futures: asyncio.Future (or compatible)
-	@param timeout: number of seconds to wait (wait indefinitely if
-		not specified)
-	@type timeout: int or float
-	@param return_when: indicates when this function should return, must
-		be one of the constants ALL_COMPLETED, FIRST_COMPLETED, or
-		FIRST_EXCEPTION (default is ALL_COMPLETED)
-	@type return_when: object
-	@param loop: event loop
-	@type loop: EventLoop
-	@return: tuple of (done, pending).
-	@rtype: asyncio.Future (or compatible)
-	"""
-	loop = asyncio._wrap_loop(loop)
-	result_future = loop.create_future()
-	_Waiter(futures, timeout, return_when, result_future, loop)
-	return result_future
-
-
-class _Waiter:
-	def __init__(self, futures, timeout, return_when, result_future, loop):
-		self._futures = futures
-		self._completed = set()
-		self._exceptions = set()
-		self._return_when = return_when
-		self._result_future = result_future
-		self._loop = loop
-		self._ready = False
-		self._timeout = None
-		result_future.add_done_callback(self._cancel_callback)
-		for future in self._futures:
-			future.add_done_callback(self._done_callback)
-		if timeout is not None:
-			self._timeout = loop.call_later(timeout, self._timeout_callback)
-
-	def _cancel_callback(self, future):
-		if future.cancelled():
-			self._ready_callback()
-
-	def _timeout_callback(self):
-		if not self._ready:
-			self._ready = True
-			self._ready_callback()
-
-	def _done_callback(self, future):
-		if future.cancelled() or future.exception() is None:
-			self._completed.add(id(future))
-		else:
-			self._exceptions.add(id(future))
-		if not self._ready and (
-			(self._return_when is FIRST_COMPLETED and self._completed) or
-			(self._return_when is FIRST_EXCEPTION and self._exceptions) or
-			(len(self._futures) == len(self._completed) + len(self._exceptions))):
-			self._ready = True
-			# use call_soon in case multiple callbacks complete in quick succession
-			self._loop.call_soon(self._ready_callback)
-
-	def _ready_callback(self):
-		if self._timeout is not None:
-			self._timeout.cancel()
-			self._timeout = None
-		if self._result_future.cancelled():
-			return
-		done = []
-		pending = []
-		done_ids = self._completed.union(self._exceptions)
-		for future in self._futures:
-			if id(future) in done_ids:
-				done.append(future)
-			else:
-				pending.append(future)
-				future.remove_done_callback(self._done_callback)
-		self._result_future.set_result((set(done), set(pending)))

[gentoo-commits] proj/portage:master commit in: lib/portage/util/futures/_asyncio/

[Zac Medico]

commit:     1e843f853a9afe82d599e6ab09064147ddc1d271
Author:     Matt Turner <mattst88 <AT> gentoo <DOT> org>
AuthorDate: Thu Mar  4 19:24:58 2021 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Sat Mar  6 09:06:53 2021 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=1e843f85

Use asyncio.subprocess.Process directly

With no need to support Python 2, we can remove our private
implementation.

Signed-off-by: Matt Turner <mattst88 <AT> gentoo.org>
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/portage/util/futures/_asyncio/__init__.py |   8 +-
 lib/portage/util/futures/_asyncio/process.py  | 116 --------------------------
 2 files changed, 4 insertions(+), 120 deletions(-)

diff --git a/lib/portage/util/futures/_asyncio/__init__.py b/lib/portage/util/futures/_asyncio/__init__.py
index 5590963f1..207e7205d 100644
--- a/lib/portage/util/futures/_asyncio/__init__.py
+++ b/lib/portage/util/futures/_asyncio/__init__.py
@@ -25,6 +25,7 @@ import types
 import weakref
 
 import asyncio as _real_asyncio
+from asyncio.subprocess import Process
 
 try:
 	import threading
@@ -45,7 +46,6 @@ from portage.util.futures.futures import (
 	TimeoutError,
 )
 # pylint: enable=redefined-builtin
-from portage.util.futures._asyncio.process import _Process
 from portage.util.futures._asyncio.tasks import (
 	ALL_COMPLETED,
 	FIRST_COMPLETED,
@@ -124,8 +124,8 @@ def create_subprocess_exec(*args, **kwargs):
 	@type loop: event loop
 	@type kwargs: varies
 	@param kwargs: subprocess.Popen parameters
-	@rtype: asyncio.Future (or compatible)
-	@return: subset of asyncio.subprocess.Process interface
+	@rtype: asyncio.subprocess.Process (or compatible)
+	@return: asyncio.subprocess.Process interface
 	"""
 	loop = _wrap_loop(kwargs.pop('loop', None))
 	# Python 3.4 and later implement PEP 446, which makes newly
@@ -138,7 +138,7 @@ def create_subprocess_exec(*args, **kwargs):
 
 	result = loop.create_future()
 
-	result.set_result(_Process(subprocess.Popen(
+	result.set_result(Process(subprocess.Popen(
 		args,
 		stdin=kwargs.pop('stdin', None),
 		stdout=kwargs.pop('stdout', None),

diff --git a/lib/portage/util/futures/_asyncio/process.py b/lib/portage/util/futures/_asyncio/process.py
deleted file mode 100644
index 275c9031a..000000000
--- a/lib/portage/util/futures/_asyncio/process.py
+++ /dev/null
@@ -1,116 +0,0 @@
-# Copyright 2018-2020 Gentoo Authors
-# Distributed under the terms of the GNU General Public License v2
-
-import os
-
-import portage
-portage.proxy.lazyimport.lazyimport(globals(),
-	'portage.util.futures:asyncio',
-	'portage.util.futures.unix_events:_set_nonblocking',
-)
-from portage.util.futures._asyncio.streams import _reader, _writer
-from portage.util.futures.compat_coroutine import coroutine, coroutine_return
-
-
-class _Process:
-	"""
-	Emulate a subset of the asyncio.subprocess.Process interface,
-	for python2.
-	"""
-	def __init__(self, proc, loop):
-		"""
-		@param proc: process instance
-		@type proc: subprocess.Popen
-		@param loop: asyncio.AbstractEventLoop (or compatible)
-		@type loop: event loop
-		"""
-		self._proc = proc
-		self._loop = loop
-		self.terminate = proc.terminate
-		self.kill = proc.kill
-		self.send_signal = proc.send_signal
-		self.pid = proc.pid
-		self._waiters = []
-		loop._asyncio_child_watcher.\
-			add_child_handler(self.pid, self._proc_exit)
-
-	@property
-	def returncode(self):
-		return self._proc.returncode
-
-	@coroutine
-	def communicate(self, input=None, loop=None): # pylint: disable=redefined-builtin
-		"""
-		Read data from stdout and stderr, until end-of-file is reached.
-		Wait for process to terminate.
-
-		@param input: stdin content to write
-		@type input: bytes
-		@return: tuple (stdout_data, stderr_data)
-		@rtype: asyncio.Future (or compatible)
-		"""
-		loop = asyncio._wrap_loop(loop or self._loop)
-		futures = []
-		for input_file in (self._proc.stdout, self._proc.stderr):
-			if input_file is None:
-				future = loop.create_future()
-				future.set_result(None)
-			else:
-				future = _reader(input_file, loop=loop)
-			futures.append(future)
-
-		writer = None
-		if input is not None:
-			if self._proc.stdin is None:
-				raise TypeError('communicate: expected file or int, got {}'.format(type(self._proc.stdin)))
-			stdin = self._proc.stdin
-			stdin = os.fdopen(stdin, 'wb', 0) if isinstance(stdin, int) else stdin
-			_set_nonblocking(stdin.fileno())
-			writer = asyncio.ensure_future(_writer(stdin, input, loop=loop), loop=loop)
-			writer.add_done_callback(lambda writer: stdin.close())
-
-		try:
-			yield asyncio.wait(futures + [self.wait(loop=loop)], loop=loop)
-		finally:
-			if writer is not None:
-				if writer.done():
-					# Consume expected exceptions.
-					try:
-						writer.result()
-					except EnvironmentError:
-						# This is normal if the other end of the pipe was closed.
-						pass
-				else:
-					writer.cancel()
-
-		coroutine_return(tuple(future.result() for future in futures))
-
-	def wait(self, loop=None):
-		"""
-		Wait for child process to terminate. Set and return returncode attribute.
-
-		@return: returncode
-		@rtype: asyncio.Future (or compatible)
-		"""
-		loop = asyncio._wrap_loop(loop or self._loop)
-		waiter = loop.create_future()
-		if self.returncode is None:
-			self._waiters.append(waiter)
-			waiter.add_done_callback(self._waiter_cancel)
-		else:
-			waiter.set_result(self.returncode)
-		return waiter
-
-	def _waiter_cancel(self, waiter):
-		if waiter.cancelled():
-			try:
-				self._waiters.remove(waiter)
-			except ValueError:
-				pass
-
-	def _proc_exit(self, pid, returncode):
-		self._proc.returncode = returncode
-		waiters = self._waiters
-		self._waiters = []
-		for waiter in waiters:
-			waiter.set_result(returncode)

[gentoo-commits] proj/portage:master commit in: lib/portage/util/futures/_asyncio/

[Zac Medico]

commit:     dcbcac809213537afaa6b4f9822146a2e984f773
Author:     Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Mon Dec  7 06:05:04 2020 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Mon Dec  7 07:48:10 2020 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=dcbcac80

_safe_loop: instantiate asyncio loop for API consumer thread

In order to maintain compatibility with an API consumer thread which
has not instantiated an asyncio loop for the current thread prior
to calling the portage API, instantiate a loop on its behalf. Since
a ResourceWarning will be triggered if the loop has not been closed
before the process exits, add the loop to a WeakValueDictionary,
and close it if it still exists during exit for the current pid.

Fixes: cecd2f8a259c ("Use default asyncio event loop implementation in API consumer threads")
Bug: https://bugs.gentoo.org/758755
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/portage/util/futures/_asyncio/__init__.py | 49 ++++++++++++++++++++++-----
 1 file changed, 40 insertions(+), 9 deletions(-)

diff --git a/lib/portage/util/futures/_asyncio/__init__.py b/lib/portage/util/futures/_asyncio/__init__.py
index 6f3395a91..d39f31786 100644
--- a/lib/portage/util/futures/_asyncio/__init__.py
+++ b/lib/portage/util/futures/_asyncio/__init__.py
@@ -21,7 +21,8 @@ __all__ = (
 )
 
 import subprocess
-import sys
+import types
+import weakref
 
 import asyncio as _real_asyncio
 
@@ -249,23 +250,53 @@ def _safe_loop():
 	the main thread, this returns a globally shared event loop instance.
 
 	For external API consumers calling from a non-main thread, an
-	asyncio loop must be registered for the current thread, or else an
-	error will be raised like this:
+	asyncio loop must be registered for the current thread, or else the
+	asyncio.get_event_loop() function will raise an error like this:
 
 	  RuntimeError: There is no current event loop in thread 'Thread-1'.
 
-	In order to avoid this RuntimeError, the external API consumer
-	is responsible for setting an event loop and managing its lifecycle.
-	For example, this code will set an event loop for the current thread:
+	In order to avoid this RuntimeError, a loop will be automatically
+	created like this:
 
 	  asyncio.set_event_loop(asyncio.new_event_loop())
 
-	In order to avoid a ResourceWarning, the caller should also close the
-	corresponding loop before the current thread terminates.
+	In order to avoid a ResourceWarning, automatically created loops
+	are added to a WeakValueDictionary, and closed via an atexit hook
+	if they still exist during exit for the current pid.
 
 	@rtype: asyncio.AbstractEventLoop (or compatible)
 	@return: event loop instance
 	"""
 	if portage._internal_caller or threading.current_thread() is threading.main_thread():
 		return _global_event_loop()
-	return _AsyncioEventLoop()
+
+	thread_key = threading.get_ident()
+	with _thread_weakrefs.lock:
+		if _thread_weakrefs.pid != portage.getpid():
+			_thread_weakrefs.pid = portage.getpid()
+			_thread_weakrefs.loops = weakref.WeakValueDictionary()
+		try:
+			loop = _thread_weakrefs.loops[thread_key]
+		except KeyError:
+			try:
+				_real_asyncio.get_event_loop()
+			except RuntimeError:
+				_real_asyncio.set_event_loop(_real_asyncio.new_event_loop())
+			loop = _thread_weakrefs.loops[thread_key] = _AsyncioEventLoop()
+	return loop
+
+
+def _thread_weakrefs_atexit():
+	with _thread_weakrefs.lock:
+		if _thread_weakrefs.pid == portage.getpid():
+			while True:
+				try:
+					thread_key, loop = _thread_weakrefs.loops.popitem()
+				except KeyError:
+					break
+				else:
+					loop.close()
+
+
+_thread_weakrefs = types.SimpleNamespace(lock=threading.Lock(), loops=None, pid=None)
+portage.process.atexit_register(_thread_weakrefs_atexit)

[gentoo-commits] proj/portage:master commit in: lib/portage/util/futures/_asyncio/

[Zac Medico]

commit:     d4979a4a2fc6e99a48bd417eae26ce77ae288444
Author:     Aaron Bauman <bman <AT> gentoo <DOT> org>
AuthorDate: Mon Aug  3 19:06:04 2020 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Mon Aug  3 19:22:48 2020 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=d4979a4a

lib/portage/util/futures/_asyncio/tasks.py: drop unused-import

* Drop unused import
* Update copyright

Signed-off-by: Aaron Bauman <bman <AT> gentoo.org>
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/portage/util/futures/_asyncio/tasks.py | 6 +-----
 1 file changed, 1 insertion(+), 5 deletions(-)

diff --git a/lib/portage/util/futures/_asyncio/tasks.py b/lib/portage/util/futures/_asyncio/tasks.py
index 84c6f4462..c9db3146e 100644
--- a/lib/portage/util/futures/_asyncio/tasks.py
+++ b/lib/portage/util/futures/_asyncio/tasks.py
@@ -1,4 +1,4 @@
-# Copyright 2018 Gentoo Foundation
+# Copyright 2018-2020 Gentoo Authors
 # Distributed under the terms of the GNU General Public License v2
 
 ___all___ = (
@@ -14,10 +14,6 @@ import portage
 portage.proxy.lazyimport.lazyimport(globals(),
 	'portage.util.futures:asyncio',
 )
-from portage.util._eventloop.global_event_loop import (
-	global_event_loop as _global_event_loop,
-)
-
 
 def wait(futures, loop=None, timeout=None, return_when=ALL_COMPLETED):
 	"""

[gentoo-commits] proj/portage:master commit in: lib/portage/util/futures/_asyncio/

[Zac Medico]

commit:     92be5a02e452eb0810d2974bc7fa5ee2056ef8e7
Author:     Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Thu Jun 18 05:33:26 2020 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Thu Jun 18 17:54:37 2020 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=92be5a02

_writer: fix unsafe finally clause (bug 728580)

In the coroutine finally clause, do not call remove_writer in cases
where fd has been closed and then re-allocated to a concurrent
coroutine as in bug 716636.

Also, assume that the caller will put the file in non-blocking mode
and close the file when done, so that this function is suitable for
use within a loop.

Bug: https://bugs.gentoo.org/728580
Reviewed-by: Brian Dolbec <dolsen <AT> gentoo.org>
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/portage/util/futures/_asyncio/process.py | 11 ++++--
 lib/portage/util/futures/_asyncio/streams.py | 50 +++++++++++++---------------
 2 files changed, 33 insertions(+), 28 deletions(-)

diff --git a/lib/portage/util/futures/_asyncio/process.py b/lib/portage/util/futures/_asyncio/process.py
index 020164c9b..2d3e9b0fd 100644
--- a/lib/portage/util/futures/_asyncio/process.py
+++ b/lib/portage/util/futures/_asyncio/process.py
@@ -1,9 +1,12 @@
-# Copyright 2018 Gentoo Foundation
+# Copyright 2018-2020 Gentoo Authors
 # Distributed under the terms of the GNU General Public License v2
 
+import os
+
 import portage
 portage.proxy.lazyimport.lazyimport(globals(),
 	'portage.util.futures:asyncio',
+	'portage.util.futures.unix_events:_set_nonblocking',
 )
 from portage.util.futures._asyncio.streams import _reader, _writer
 from portage.util.futures.compat_coroutine import coroutine, coroutine_return
@@ -59,7 +62,11 @@ class _Process(object):
 		if input is not None:
 			if self._proc.stdin is None:
 				raise TypeError('communicate: expected file or int, got {}'.format(type(self._proc.stdin)))
-			writer = asyncio.ensure_future(_writer(self._proc.stdin, input), loop=self._loop)
+			stdin = self._proc.stdin
+			stdin = os.fdopen(stdin, 'wb', 0) if isinstance(stdin, int) else stdin
+			_set_nonblocking(stdin.fileno())
+			writer = asyncio.ensure_future(_writer(stdin, input, loop=self._loop), loop=self._loop)
+			writer.add_done_callback(lambda writer: stdin.close())
 
 		try:
 			yield asyncio.wait(futures + [self.wait()], loop=self._loop)

diff --git a/lib/portage/util/futures/_asyncio/streams.py b/lib/portage/util/futures/_asyncio/streams.py
index 650a16491..870307e1e 100644
--- a/lib/portage/util/futures/_asyncio/streams.py
+++ b/lib/portage/util/futures/_asyncio/streams.py
@@ -1,4 +1,4 @@
-# Copyright 2018 Gentoo Foundation
+# Copyright 2018-2020 Gentoo Authors
 # Distributed under the terms of the GNU General Public License v2
 
 import errno
@@ -8,7 +8,6 @@ import portage
 portage.proxy.lazyimport.lazyimport(globals(),
 	'_emerge.PipeReader:PipeReader',
 	'portage.util.futures:asyncio',
-	'portage.util.futures.unix_events:_set_nonblocking',
 )
 from portage.util.futures.compat_coroutine import coroutine
 
@@ -59,38 +58,37 @@ class _Reader(object):
 @coroutine
 def _writer(output_file, content, loop=None):
 	"""
-	Asynchronously write bytes to output file, and close it when
-	done. If an EnvironmentError other than EAGAIN is encountered,
-	which typically indicates that the other end of the pipe has
-	close, the error is raised. This function is a coroutine.
+	Asynchronously write bytes to output file. The output file is
+	assumed to be in non-blocking mode. If an EnvironmentError
+	other than EAGAIN is encountered, which typically indicates that
+	the other end of the pipe has closed, the error is raised.
+	This function is a coroutine.
 
-	@param output_file: output file descriptor
-	@type output_file: file or int
+	@param output_file: output file
+	@type output_file: file object
 	@param content: content to write
 	@type content: bytes
 	@param loop: asyncio.AbstractEventLoop (or compatible)
 	@type loop: event loop
 	"""
-	fd = output_file if isinstance(output_file, int) else output_file.fileno()
-	_set_nonblocking(fd)
 	loop = asyncio._wrap_loop(loop)
-	try:
-		while content:
+	fd = output_file.fileno()
+	while content:
+		try:
+			content = content[os.write(fd, content):]
+		except EnvironmentError as e:
+			if e.errno != errno.EAGAIN:
+				raise
 			waiter = loop.create_future()
-			loop.add_writer(fd, lambda: waiter.set_result(None))
+			loop.add_writer(fd, lambda: waiter.done() or waiter.set_result(None))
 			try:
 				yield waiter
-				while content:
-					try:
-						content = content[os.write(fd, content):]
-					except EnvironmentError as e:
-						if e.errno == errno.EAGAIN:
-							break
-						else:
-							raise
 			finally:
-				loop.remove_writer(fd)
-	except GeneratorExit:
-		raise
-	finally:
-		os.close(output_file) if isinstance(output_file, int) else output_file.close()
+				# The loop and output file may have been closed.
+				if not loop.is_closed():
+					waiter.done() or waiter.cancel()
+					# Do not call remove_writer in cases where fd has
+					# been closed and then re-allocated to a concurrent
+					# coroutine as in bug 716636.
+					if not output_file.closed:
+						loop.remove_writer(fd)

[gentoo-commits] proj/portage:master commit in: lib/portage/util/futures/_asyncio/

[Zac Medico]

commit:     64b11fe4dbcd7f2b4c36d8c40a09425a2c624c7a
Author:     Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Sat Feb 29 04:24:19 2020 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Sat Feb 29 04:31:50 2020 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=64b11fe4

asyncio: improve _AsyncioEventLoop isinstance logic

Since _AsyncioEventLoop can be wrapped, use the _asyncio_wrapper
attributre for isinstance checks (_wrap_loop guarantees that this
attribute exists).

Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/portage/util/futures/_asyncio/__init__.py | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/lib/portage/util/futures/_asyncio/__init__.py b/lib/portage/util/futures/_asyncio/__init__.py
index 7635dbb5e..f4b03891f 100644
--- a/lib/portage/util/futures/_asyncio/__init__.py
+++ b/lib/portage/util/futures/_asyncio/__init__.py
@@ -1,4 +1,4 @@
-# Copyright 2018 Gentoo Foundation
+# Copyright 2018-2020 Gentoo Authors
 # Distributed under the terms of the GNU General Public License v2
 
 __all__ = (
@@ -139,7 +139,7 @@ def create_subprocess_exec(*args, **kwargs):
 	"""
 	loop = _wrap_loop(kwargs.pop('loop', None))
 	kwargs.setdefault('close_fds', _close_fds_default)
-	if _asyncio_enabled and isinstance(loop, _AsyncioEventLoop):
+	if _asyncio_enabled and isinstance(loop._asyncio_wrapper, _AsyncioEventLoop):
 		# Use the real asyncio create_subprocess_exec (loop argument
 		# is deprecated since since Python 3.8).
 		return _real_asyncio.create_subprocess_exec(*args, **kwargs)
@@ -191,10 +191,10 @@ def ensure_future(coro_or_future, loop=None):
 	@return: an instance of Future
 	"""
 	loop = _wrap_loop(loop)
-	if _asyncio_enabled and isinstance(loop, _AsyncioEventLoop):
+	if _asyncio_enabled and isinstance(loop._asyncio_wrapper, _AsyncioEventLoop):
 		# Use the real asyncio loop and ensure_future.
 		return _real_asyncio.ensure_future(
-			coro_or_future, loop=loop._loop)
+			coro_or_future, loop=loop._asyncio_wrapper._loop)
 
 	if isinstance(coro_or_future, Future):
 		return coro_or_future

[gentoo-commits] proj/portage:master commit in: lib/portage/util/futures/_asyncio/

[Zac Medico]

commit:     67e0bba35f1b7afd5ee5ca648154838741875d6a
Author:     Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Fri Oct 18 03:35:46 2019 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Fri Oct 18 03:37:40 2019 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=67e0bba3

asyncio.create_subprocess_exec: suppress DeprecationWarning for loop argument

See: https://bugs.python.org/issue36373
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/portage/util/futures/_asyncio/__init__.py | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/lib/portage/util/futures/_asyncio/__init__.py b/lib/portage/util/futures/_asyncio/__init__.py
index bd9f2e47c..7635dbb5e 100644
--- a/lib/portage/util/futures/_asyncio/__init__.py
+++ b/lib/portage/util/futures/_asyncio/__init__.py
@@ -140,8 +140,9 @@ def create_subprocess_exec(*args, **kwargs):
 	loop = _wrap_loop(kwargs.pop('loop', None))
 	kwargs.setdefault('close_fds', _close_fds_default)
 	if _asyncio_enabled and isinstance(loop, _AsyncioEventLoop):
-		# Use the real asyncio loop and create_subprocess_exec.
-		return _real_asyncio.create_subprocess_exec(*args, loop=loop._loop, **kwargs)
+		# Use the real asyncio create_subprocess_exec (loop argument
+		# is deprecated since since Python 3.8).
+		return _real_asyncio.create_subprocess_exec(*args, **kwargs)
 
 	result = loop.create_future()
 

[gentoo-commits] proj/portage:master commit in: lib/portage/util/futures/_asyncio/

[Zac Medico]

commit:     cef213c31ce05920fd1b8bbf2749f012944f0c2a
Author:     Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Sat May 18 22:16:13 2019 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Sat May 18 22:20:50 2019 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=cef213c3

_safe_loop: call global_event_loop() directly

Since _wrap_loop() always returns an identical result in this context,
call global_event_loop() directly.

Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/portage/util/futures/_asyncio/__init__.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/portage/util/futures/_asyncio/__init__.py b/lib/portage/util/futures/_asyncio/__init__.py
index e77c7a690..bd9f2e47c 100644
--- a/lib/portage/util/futures/_asyncio/__init__.py
+++ b/lib/portage/util/futures/_asyncio/__init__.py
@@ -265,6 +265,6 @@ def _safe_loop():
 	@return: event loop instance
 	"""
 	if portage._internal_caller:
-		return _wrap_loop()
+		return _global_event_loop()
 	else:
 		return _EventLoop(main=False)