From: "Zac Medico" <zmedico@gentoo.org>
To: gentoo-commits@lists.gentoo.org
Subject: [gentoo-commits] proj/portage:master commit in: lib/portage/
Date: Tue, 3 Oct 2023 15:07:46 +0000 (UTC) [thread overview]
Message-ID: <1696345157.7622da95fd97c07d7a51f14dfcfb07048db687c1.zmedico@gentoo> (raw)
commit: 7622da95fd97c07d7a51f14dfcfb07048db687c1
Author: Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Tue Oct 3 14:59:17 2023 +0000
Commit: Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Tue Oct 3 14:59:17 2023 +0000
URL: https://gitweb.gentoo.org/proj/portage.git/commit/?id=7622da95
_get_lock_fn: support multiprocessing spawn start method (bug 915119)
Ensure that _get_lock_fn arguments to multiprocessing.Process will
successfully pickle, as required by the spawn start method, which
is the default for macOS since Python 3.8.
Since file descriptors are not inherited unless the fork start
method is used, the subprocess should only try to close an
inherited file descriptor for the fork start method.
Bug: https://bugs.gentoo.org/915119
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>
lib/portage/locks.py | 50 ++++++++++++++++++++++++++++++++------------------
1 file changed, 32 insertions(+), 18 deletions(-)
diff --git a/lib/portage/locks.py b/lib/portage/locks.py
index 1c3e13ce4f..ee40451b12 100644
--- a/lib/portage/locks.py
+++ b/lib/portage/locks.py
@@ -79,6 +79,11 @@ class _lock_manager:
del _open_inodes[self.inode_key]
+def _lockf_test_lock_fn(path, fd, flags):
+ fcntl.lockf(fd, flags)
+ return functools.partial(unlockfile, (path, fd, flags, fcntl.lockf))
+
+
def _get_lock_fn():
"""
Returns fcntl.lockf if proven to work, and otherwise returns fcntl.flock.
@@ -88,10 +93,7 @@ def _get_lock_fn():
if _lock_fn is not None:
return _lock_fn
- if _test_lock_fn(
- lambda path, fd, flags: fcntl.lockf(fd, flags)
- and functools.partial(unlockfile, (path, fd, flags, fcntl.lockf))
- ):
+ if _test_lock_fn(_lockf_test_lock_fn):
_lock_fn = fcntl.lockf
return _lock_fn
@@ -103,19 +105,6 @@ def _get_lock_fn():
def _test_lock_fn(
lock_fn: typing.Callable[[str, int, int], typing.Callable[[], None]]
) -> bool:
- def _test_lock(fd, lock_path):
- os.close(fd)
- try:
- with open(lock_path, "a") as f:
- lock_fn(lock_path, f.fileno(), fcntl.LOCK_EX | fcntl.LOCK_NB)
- except (TryAgain, OSError) as e:
- if isinstance(e, TryAgain) or e.errno == errno.EAGAIN:
- # Parent process holds lock, as expected.
- sys.exit(0)
-
- # Something went wrong.
- sys.exit(1)
-
fd, lock_path = tempfile.mkstemp()
unlock_fn = None
try:
@@ -125,7 +114,17 @@ def _test_lock_fn(
pass
else:
_lock_manager(fd, os.fstat(fd), lock_path)
- proc = multiprocessing.Process(target=_test_lock, args=(fd, lock_path))
+ proc = multiprocessing.Process(
+ target=_subprocess_test_lock,
+ args=(
+ # Since file descriptors are not inherited unless the fork start
+ # method is used, the subprocess should only try to close an
+ # inherited file descriptor for the fork start method.
+ fd if multiprocessing.get_start_method() == "fork" else None,
+ lock_fn,
+ lock_path,
+ ),
+ )
proc.start()
proc.join()
if proc.exitcode == os.EX_OK:
@@ -141,6 +140,21 @@ def _test_lock_fn(
return False
+def _subprocess_test_lock(fd, lock_fn, lock_path):
+ if fd is not None:
+ os.close(fd)
+ try:
+ with open(lock_path, "a") as f:
+ lock_fn(lock_path, f.fileno(), fcntl.LOCK_EX | fcntl.LOCK_NB)
+ except (TryAgain, OSError) as e:
+ if isinstance(e, TryAgain) or e.errno == errno.EAGAIN:
+ # Parent process holds lock, as expected.
+ sys.exit(0)
+
+ # Something went wrong.
+ sys.exit(1)
+
+
def _close_fds():
"""
This is intended to be called after a fork, in order to close file
next reply other threads:[~2023-10-03 15:07 UTC|newest]
Thread overview: 155+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-10-03 15:07 Zac Medico [this message]
-- strict thread matches above, loose matches on Subject: below --
2025-09-23 6:05 [gentoo-commits] proj/portage:master commit in: lib/portage/ Zac Medico
2025-09-14 3:14 Sam James
2025-09-14 3:14 Sam James
2025-09-12 21:18 Sam James
2025-09-11 3:17 Sam James
2025-09-11 0:53 Sam James
2025-08-29 12:19 Sam James
2025-01-21 21:14 Sam James
2025-01-11 16:01 Mike Gilbert
2025-01-11 16:01 Mike Gilbert
2025-01-01 0:33 Zac Medico
2024-11-02 22:12 Zac Medico
2024-11-02 15:48 Zac Medico
2024-11-02 15:48 Zac Medico
2024-09-09 18:08 Ulrich Müller
2024-09-09 18:08 Ulrich Müller
2024-08-14 15:22 Zac Medico
2024-06-09 17:54 Zac Medico
2024-06-02 18:28 Zac Medico
2024-04-26 22:06 Sam James
2024-04-26 22:06 Sam James
2024-02-28 16:01 Sam James
2024-02-28 15:52 Sam James
2024-02-28 15:49 Sam James
2024-02-25 8:25 Sam James
2024-02-24 20:10 Zac Medico
2024-02-21 2:08 Sam James
2024-02-21 2:08 Sam James
2024-02-12 7:58 Zac Medico
2024-02-11 19:57 Zac Medico
2024-02-10 6:09 Zac Medico
2024-02-10 6:06 Zac Medico
2024-02-09 8:51 Sam James
2024-02-09 7:08 Sam James
2024-02-07 2:35 Zac Medico
2024-02-07 2:35 Zac Medico
2024-02-05 1:03 Zac Medico
2024-02-05 1:03 Zac Medico
2024-01-29 17:49 Zac Medico
2024-01-29 16:09 Zac Medico
2023-12-26 23:15 Zac Medico
2023-11-02 14:58 Zac Medico
2023-10-24 21:26 Zac Medico
2023-10-24 1:48 Zac Medico
2023-10-02 2:10 Zac Medico
2023-09-26 5:53 Zac Medico
2023-09-08 20:36 Sam James
2023-09-08 19:49 Sam James
2023-08-24 18:23 Mike Gilbert
2023-08-02 6:31 Sam James
2023-07-29 3:57 Sam James
2023-06-29 8:22 Sam James
2023-03-21 2:30 Sam James
2023-03-21 2:30 Sam James
2023-03-21 2:30 Sam James
2023-03-21 2:30 Sam James
2023-03-21 2:30 Sam James
2023-03-21 2:30 Sam James
2023-02-27 6:15 Sam James
2023-02-17 1:23 Sam James
2023-01-02 5:25 Sam James
2022-11-02 22:58 Sam James
2022-11-02 22:58 Sam James
2022-09-29 21:37 Sam James
2022-09-29 20:45 Sam James
2022-09-28 23:56 Sam James
2022-09-26 17:52 Zac Medico
2022-09-20 19:45 Sam James
2022-09-20 3:39 Sam James
2022-09-18 18:30 Mike Gilbert
2022-08-01 22:39 Sam James
2022-08-01 17:34 Mike Gilbert
2022-07-19 21:39 Sam James
2022-07-18 18:47 Sam James
2022-07-11 23:02 Sam James
2022-07-10 15:07 Mike Gilbert
2022-07-05 22:56 Sam James
2022-06-05 20:25 Zac Medico
2022-04-11 12:11 Mike Gilbert
2022-04-11 12:11 Mike Gilbert
2022-04-09 4:32 Sam James
2022-04-04 19:04 Sam James
2022-04-04 19:04 Sam James
2022-04-04 19:04 Sam James
2022-04-04 19:04 Sam James
2022-04-04 19:04 Sam James
2022-04-04 19:04 Sam James
2022-04-04 19:04 Sam James
2022-04-04 19:04 Sam James
2022-04-01 20:30 Matt Turner
2022-03-30 23:11 Sam James
2022-03-28 1:10 Sam James
2022-03-27 23:07 Sam James
2022-03-27 23:07 Sam James
2022-03-27 23:07 Sam James
2022-03-27 23:07 Sam James
2022-03-27 23:07 Sam James
2022-03-15 2:52 Matt Turner
2022-02-09 11:13 Sam James
2021-09-20 20:06 Zac Medico
2021-09-20 19:55 Mike Gilbert
2021-09-07 7:04 Michał Górny
2021-09-04 11:53 Michał Górny
2021-05-24 6:08 Zac Medico
2021-05-24 4:55 Zac Medico
2021-03-28 3:33 Zac Medico
2021-03-11 12:32 Zac Medico
2021-03-07 14:03 Zac Medico
2021-03-06 9:18 Zac Medico
2021-03-06 9:05 Zac Medico
2021-03-06 9:05 Zac Medico
2021-03-06 8:20 Zac Medico
2021-03-06 6:16 Zac Medico
2021-02-08 4:55 Zac Medico
2020-09-11 19:02 Zac Medico
2020-08-04 1:39 Zac Medico
2020-08-03 23:28 Zac Medico
2020-08-03 23:28 Zac Medico
2020-08-03 19:30 Zac Medico
2020-08-03 19:30 Zac Medico
2020-08-03 19:30 Zac Medico
2020-08-03 19:30 Zac Medico
2020-06-27 19:46 Zac Medico
2020-06-09 0:58 Zac Medico
2020-05-17 9:37 Michał Górny
2020-05-07 20:35 Zac Medico
2020-04-20 21:16 Mike Gilbert
2020-03-28 18:57 Michał Górny
2020-03-25 19:18 Zac Medico
2020-03-25 7:57 Zac Medico
2020-03-25 7:57 Zac Medico
2020-02-04 6:43 Zac Medico
2020-02-02 9:00 Zac Medico
2019-12-15 23:04 Zac Medico
2019-11-12 22:25 Zac Medico
2019-09-17 2:59 Zac Medico
2019-09-07 6:40 Zac Medico
2019-08-18 22:15 Zac Medico
2019-08-04 18:03 Zac Medico
2019-08-02 20:03 Mike Gilbert
2019-08-01 19:02 Mike Gilbert
2019-05-28 1:49 Zac Medico
2019-04-27 19:20 Zac Medico
2019-02-20 0:58 Zac Medico
2019-02-20 0:58 Zac Medico
2019-02-20 0:58 Zac Medico
2019-02-18 1:01 Zac Medico
2019-02-11 19:46 Zac Medico
2019-01-04 3:49 Zac Medico
2018-12-31 5:27 Zac Medico
2018-12-04 1:35 Zac Medico
2018-11-25 0:03 Zac Medico
2018-11-24 21:34 Zac Medico
2018-08-07 18:36 Zac Medico
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1696345157.7622da95fd97c07d7a51f14dfcfb07048db687c1.zmedico@gentoo \
--to=zmedico@gentoo.org \
--cc=gentoo-commits@lists.gentoo.org \
--cc=gentoo-dev@lists.gentoo.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox