[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     528bec60c4c7b32c2545d91ba4ee91e543b0d5c9
Author:     Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Thu Aug  2 18:32:58 2018 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Thu Aug  2 18:43:58 2018 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=528bec60

Scheduler: cancel job delay timeout after main loop (bug 662668)

This fixes the following error during emerge --keep-going
dependency calculation:

Exception in callback PollScheduler._schedule()
handle: <TimerHandle when=44692.1015056609 PollScheduler._schedule()>
Traceback (most recent call last):
  File "/usr/lib64/python3.6/asyncio/events.py", line 145, in _run
    self._callback(*self._args)
  File "/usr/lib64/python3.6/site-packages/_emerge/PollScheduler.py", line 154, in _schedule
    self._schedule_tasks()
  File "/usr/lib64/python3.6/site-packages/_emerge/Scheduler.py", line 1605, in _schedule_tasks
    self._keep_scheduling() or self._main_exit.done()):
AttributeError: 'NoneType' object has no attribute 'done'

Reported-by: Rick Farina <zerochaos <AT> gentoo.org>
Bug: https://bugs.gentoo.org/662668

 lib/_emerge/Scheduler.py | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/lib/_emerge/Scheduler.py b/lib/_emerge/Scheduler.py
index 422308184..49b376281 100644
--- a/lib/_emerge/Scheduler.py
+++ b/lib/_emerge/Scheduler.py
@@ -1436,6 +1436,9 @@ class Scheduler(PollScheduler):
 		if self._main_loadavg_handle is not None:
 			self._main_loadavg_handle.cancel()
 			self._main_loadavg_handle = None
+		if self._job_delay_timeout_id is not None:
+			self._job_delay_timeout_id.cancel()
+			self._job_delay_timeout_id = None
 
 	def _choose_pkg(self):
 		"""

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     2e87bfaabd0550f0b34556470b28906d30bd5e48
Author:     Andrew Jeddeloh <andrew.jeddeloh <AT> coreos <DOT> com>
AuthorDate: Fri Oct  5 22:35:04 2018 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Fri Oct  5 23:08:47 2018 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=2e87bfaa

emerge: don't ignore binpkgs with buildpkgonly

Do not ignore binpkgs when using --buildpkgonly. If binpackages exist
that can be used, they should be. This prevents rebuilding binpkgs when
using --emptytree with --buildpkgonly.

Signed-off-by: Andrew Jeddeloh <andrew.jeddeloh <AT> coreos.com>
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/actions.py | 7 -------
 1 file changed, 7 deletions(-)

diff --git a/lib/_emerge/actions.py b/lib/_emerge/actions.py
index 46cb02791..e5200d73d 100644
--- a/lib/_emerge/actions.py
+++ b/lib/_emerge/actions.py
@@ -2890,13 +2890,6 @@ def run_action(emerge_config):
 	if "--usepkgonly" in emerge_config.opts:
 		emerge_config.opts["--usepkg"] = True
 
-	if "--buildpkgonly" in emerge_config.opts:
-		# --buildpkgonly will not merge anything, so
-		# it cancels all binary package options.
-		for opt in ("--getbinpkg", "--getbinpkgonly",
-			"--usepkg", "--usepkgonly"):
-			emerge_config.opts.pop(opt, None)
-
 	# Populate the bintree with current --getbinpkg setting.
 	# This needs to happen before:
 	# * expand_set_arguments, in case any sets use the bintree

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     e30523f041b010d7b37e83ac385a2eb45f6ee500
Author:     Greg Kubaryk <kurly <AT> vt <DOT> edu>
AuthorDate: Sun Nov 25 08:22:39 2018 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Sun Nov 25 08:24:12 2018 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=e30523f0

emerge --help: add -U option short for --changed-use

Fixes: d55690aa3a1a ("emerge: add -U option short for --changed-use")
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/help.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/_emerge/help.py b/lib/_emerge/help.py
index dd3a3475c..2ccd323aa 100644
--- a/lib/_emerge/help.py
+++ b/lib/_emerge/help.py
@@ -13,7 +13,7 @@ def help():
 	print("   "+turquoise("emerge")+" < "+turquoise("--sync")+" | "+turquoise("--metadata")+" | "+turquoise("--info")+" >")
 	print("   "+turquoise("emerge")+" "+turquoise("--resume")+" [ "+green("--pretend")+" | "+green("--ask")+" | "+green("--skipfirst")+" ]")
 	print("   "+turquoise("emerge")+" "+turquoise("--help"))
-	print(bold("Options:")+" "+green("-")+"["+green("abBcCdDefgGhjkKlnNoOpPqrsStuvVw")+"]")
+	print(bold("Options:")+" "+green("-")+"["+green("abBcCdDefgGhjkKlnNoOpPqrsStuUvVw")+"]")
 	print("          [ " + green("--color")+" < " + turquoise("y") + " | "+ turquoise("n")+" >            ] [ "+green("--columns")+"    ]")
 	print("          [ "+green("--complete-graph")+"             ] [ "+green("--deep")+"       ]")
 	print("          [ "+green("--jobs") + " " + turquoise("JOBS")+" ] [ "+green("--keep-going")+" ] [ " + green("--load-average")+" " + turquoise("LOAD") + "            ]")

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     0331a8ca059b09df411e9681dcd598b3abb3f85f
Author:     Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Mon Jan 21 21:50:36 2019 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Mon Jan 21 21:59:11 2019 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=0331a8ca

SpawnProcess: allow unshare_mount and unshare_pid

Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/SpawnProcess.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/_emerge/SpawnProcess.py b/lib/_emerge/SpawnProcess.py
index cd535d143..395d66bb9 100644
--- a/lib/_emerge/SpawnProcess.py
+++ b/lib/_emerge/SpawnProcess.py
@@ -32,7 +32,7 @@ class SpawnProcess(SubProcess):
 	_spawn_kwarg_names = ("env", "opt_name", "fd_pipes",
 		"uid", "gid", "groups", "umask", "logfile",
 		"path_lookup", "pre_exec", "close_fds", "cgroup",
-		"unshare_ipc", "unshare_net")
+		"unshare_ipc", "unshare_mount", "unshare_pid", "unshare_net")
 
 	__slots__ = ("args",) + \
 		_spawn_kwarg_names + ("_pipe_logger", "_selinux_type",)

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     58fe19ffa3971e6767240392a8f6951899c412bb
Author:     Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Sun Apr 21 01:01:05 2019 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Sun Apr 21 01:02:01 2019 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=58fe19ff

emerge --info: show per-package FEATURES (bug 553224)

Bug: https://bugs.gentoo.org/553224
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/actions.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/_emerge/actions.py b/lib/_emerge/actions.py
index e5200d73d..3e32bf8bc 100644
--- a/lib/_emerge/actions.py
+++ b/lib/_emerge/actions.py
@@ -1895,7 +1895,7 @@ def action_info(settings, trees, myopts, myfiles):
 	if mypkgs:
 		# Get our global settings (we only print stuff if it varies from
 		# the current config)
-		mydesiredvars = [ 'CHOST', 'CFLAGS', 'CXXFLAGS', 'LDFLAGS' ]
+		mydesiredvars = ['CHOST', 'CFLAGS', 'CXXFLAGS', 'FEATURES', 'LDFLAGS']
 		auxkeys = mydesiredvars + list(vardb._aux_cache_keys)
 		auxkeys.append('DEFINED_PHASES')
 		pkgsettings = portage.config(clone=settings)

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     71f6a23c497980f4424fca51ac3e2864608ab08d
Author:     Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Wed Apr 24 18:50:42 2019 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Wed Apr 24 18:52:57 2019 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=71f6a23c

emergelog: handle /dev/null symlink (bug 684232)

If the log path is a symlink to /dev/null, os.path.isfile returns
False, so use os.path.exists in order to avoid chown/chmod in
this case.

Reported-by: Harald Becker <hb-xxl <AT> web.de>
Bug: https://bugs.gentoo.org/684232
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/emergelog.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/_emerge/emergelog.py b/lib/_emerge/emergelog.py
index aea94f74e..cba98f395 100644
--- a/lib/_emerge/emergelog.py
+++ b/lib/_emerge/emergelog.py
@@ -36,7 +36,7 @@ def emergelog(xterm_titles, mystr, short_msg=None):
 		xtermTitle(short_msg)
 	try:
 		file_path = os.path.join(_emerge_log_dir, 'emerge.log')
-		existing_log = os.path.isfile(file_path)
+		existing_log = os.path.exists(file_path)
 		mylogfile = io.open(_unicode_encode(file_path,
 			encoding=_encodings['fs'], errors='strict'),
 			mode='a', encoding=_encodings['content'],

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     cfc3ca6647235d8a3fbe719b9d447863a75deeb6
Author:     James Le Cuirot <chewi <AT> gentoo <DOT> org>
AuthorDate: Sun Jul  7 13:18:54 2019 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Mon Jul  8 06:37:15 2019 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=cfc3ca66

emerge: Don't treat empty EPREFIX or PORTAGE_CONFIGROOT as unset

If a prefix user wanted to build within a ROOT but without a prefix,
they previously had to set EPREFIX=/ rather than EPREFIX= as the
latter was simply treated as unset.

Also applies to ROOT and SYSROOT but probably makes no difference to
these as they are blank by default. This should be safe to do as all
these variables get normalised anyway.

Bug: https://bugs.gentoo.org/642604
Closes: https://github.com/gentoo/portage/pull/434
Signed-off-by: James Le Cuirot <chewi <AT> gentoo.org>
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/actions.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/_emerge/actions.py b/lib/_emerge/actions.py
index 3e32bf8bc..6ca323501 100644
--- a/lib/_emerge/actions.py
+++ b/lib/_emerge/actions.py
@@ -2455,7 +2455,7 @@ def load_emerge_config(emerge_config=None, env=None, **kargs):
 	for k, envvar in (("config_root", "PORTAGE_CONFIGROOT"), ("target_root", "ROOT"),
 			("sysroot", "SYSROOT"), ("eprefix", "EPREFIX")):
 		v = env.get(envvar)
-		if v and v.strip():
+		if v is not None:
 			kwargs[k] = v
 	emerge_config.trees = portage.create_trees(trees=emerge_config.trees,
 				**kwargs)

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     8ffc88d4b6269ad6f6b7ee5480508556cc352878
Author:     Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Tue Aug  6 03:13:32 2019 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Tue Aug  6 03:14:00 2019 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=8ffc88d4

EbuildExecuter: remove unused _live_eclasses attribute

Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/EbuildExecuter.py | 2 --
 1 file changed, 2 deletions(-)

diff --git a/lib/_emerge/EbuildExecuter.py b/lib/_emerge/EbuildExecuter.py
index ca9859437..acc57c1d0 100644
--- a/lib/_emerge/EbuildExecuter.py
+++ b/lib/_emerge/EbuildExecuter.py
@@ -15,8 +15,6 @@ class EbuildExecuter(CompositeTask):
 
 	_phases = ("prepare", "configure", "compile", "test", "install")
 
-	_live_eclasses = portage.const.LIVE_ECLASSES
-
 	def _start(self):
 		pkg = self.pkg
 		scheduler = self.scheduler

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     bb0bb4104b0d6cab16abce9cd6e1683c0b6d5c8d
Author:     Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Sun Sep  1 00:24:43 2019 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Sun Sep  1 00:30:04 2019 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=bb0bb410

_backtrack_depgraph: fix premature backtracking termination (bug 693242)

Make backtracking continue as long as the backtracker has remaining
nodes to explore. This fixes a case where it would terminate prematurely
when the depgraph.need_restart() method returned False, even though the
backtracker had remaining nodes to explore.

Bug: https://bugs.gentoo.org/693242
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/depgraph.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/lib/_emerge/depgraph.py b/lib/_emerge/depgraph.py
index 3e99ac077..08240af67 100644
--- a/lib/_emerge/depgraph.py
+++ b/lib/_emerge/depgraph.py
@@ -9794,8 +9794,8 @@ def _backtrack_depgraph(settings, trees, myopts, myparams, myaction, myfiles, sp
 		elif mydepgraph.need_restart():
 			backtracked += 1
 			backtracker.feedback(mydepgraph.get_backtrack_infos())
-		else:
-			break
+		elif backtracker:
+			backtracked += 1
 
 	if not (success or mydepgraph.need_config_change()) and backtracked:
 

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     498961b2786d4bd3a042e96d7c6c5bd45d260c2c
Author:     Justin Turner Arthur <justinarthur <AT> gmail <DOT> com>
AuthorDate: Tue Oct 22 03:12:14 2019 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Wed Oct 23 17:01:52 2019 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=498961b2

Improve runtime performance of depgraph module functions.

Use set literals, compile-time str concats, reduce function calls,
especially those on built-in types that can be replaced by operators.

Signed-off-by: Justin Turner Arthur <justinarthur <AT> gmail.com>
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/depgraph.py | 325 +++++++++++++++++++++++++-----------------------
 1 file changed, 169 insertions(+), 156 deletions(-)

diff --git a/lib/_emerge/depgraph.py b/lib/_emerge/depgraph.py
index a641bfc21..68b5bdb2e 100644
--- a/lib/_emerge/depgraph.py
+++ b/lib/_emerge/depgraph.py
@@ -167,7 +167,7 @@ class _frozen_depgraph_config(object):
 		if params.get("ignore_world", False):
 			self._required_set_names = set()
 		else:
-			self._required_set_names = set(["world"])
+			self._required_set_names = {"world"}
 
 		atoms = ' '.join(myopts.get("--exclude", [])).split()
 		self.excluded_pkgs = _wildcard_set(atoms)
@@ -1095,11 +1095,11 @@ class depgraph(object):
 
 		self._show_merge_list()
 
-		if ignored_binaries.get("respect_use"):
+		if "respect_use" in ignored_binaries:
 			self._show_ignored_binaries_respect_use(
 				ignored_binaries["respect_use"])
 
-		if ignored_binaries.get("changed_deps"):
+		if "changed_deps" in ignored_binaries:
 			self._show_ignored_binaries_changed_deps(
 				ignored_binaries["changed_deps"])
 
@@ -1271,9 +1271,10 @@ class depgraph(object):
 			return
 
 		self._show_merge_list()
-		msg = []
-		msg.append("\nWARNING: One or more updates/rebuilds have been " + \
-			"skipped due to a dependency conflict:\n\n")
+		msg = [
+			"\nWARNING: One or more updates/rebuilds have been "
+			"skipped due to a dependency conflict:\n\n"
+		]
 
 		indent = "  "
 		for pkg, parent_atoms in missed_updates:
@@ -1337,33 +1338,37 @@ class depgraph(object):
 		if "--quiet" in self._frozen_config.myopts:
 			return
 
-		msg = []
-		msg.append("It may be possible to solve this problem ")
-		msg.append("by using package.mask to prevent one of ")
-		msg.append("those packages from being selected. ")
-		msg.append("However, it is also possible that conflicting ")
-		msg.append("dependencies exist such that they are impossible to ")
-		msg.append("satisfy simultaneously.  If such a conflict exists in ")
-		msg.append("the dependencies of two different packages, then those ")
-		msg.append("packages can not be installed simultaneously.")
+		msg = [
+			"It may be possible to solve this problem "
+			"by using package.mask to prevent one of "
+			"those packages from being selected. "
+			"However, it is also possible that conflicting "
+			"dependencies exist such that they are impossible to "
+			"satisfy simultaneously.  If such a conflict exists in "
+			"the dependencies of two different packages, then those "
+			"packages can not be installed simultaneously."
+		]
 		backtrack_opt = self._frozen_config.myopts.get('--backtrack')
 		if not self._dynamic_config._allow_backtracking and \
 			(backtrack_opt is None or \
 			(backtrack_opt > 0 and backtrack_opt < 30)):
-			msg.append(" You may want to try a larger value of the ")
-			msg.append("--backtrack option, such as --backtrack=30, ")
-			msg.append("in order to see if that will solve this conflict ")
-			msg.append("automatically.")
+			msg.append(
+				" You may want to try a larger value of the "
+				"--backtrack option, such as --backtrack=30, "
+				"in order to see if that will solve this conflict "
+				"automatically."
+			)
 
 		for line in textwrap.wrap(''.join(msg), 70):
 			writemsg(line + '\n', noiselevel=-1)
 		writemsg('\n', noiselevel=-1)
 
-		msg = []
-		msg.append("For more information, see MASKED PACKAGES ")
-		msg.append("section in the emerge man page or refer ")
-		msg.append("to the Gentoo Handbook.")
-		for line in textwrap.wrap(''.join(msg), 70):
+		msg = (
+			"For more information, see MASKED PACKAGES "
+			"section in the emerge man page or refer "
+			"to the Gentoo Handbook."
+		)
+		for line in textwrap.wrap(msg, 70):
 			writemsg(line + '\n', noiselevel=-1)
 		writemsg('\n', noiselevel=-1)
 
@@ -1404,7 +1409,7 @@ class depgraph(object):
 		indirect_conflict_candidates = set()
 		for pkg in conflict_pkgs:
 			indirect_conflict_candidates.update(self._dynamic_config.digraph.child_nodes(pkg))
-		indirect_conflict_candidates.difference_update(conflict_pkgs)
+		indirect_conflict_candidates -= conflict_pkgs
 
 		indirect_conflict_pkgs = set()
 		while indirect_conflict_candidates:
@@ -1538,9 +1543,9 @@ class depgraph(object):
 
 		# Now select required packages. Collect them in the
 		# 'forced' set.
-		forced = set([non_conflict_node])
-		forced.update(non_matching_forced)
-		unexplored = set([non_conflict_node])
+		forced = {non_conflict_node}
+		forced |= non_matching_forced
+		unexplored = {non_conflict_node}
 		# or_tuples get special handling. We first explore
 		# all packages in the hope of having forced one of
 		# the packages in the tuple. This way we don't have
@@ -1590,7 +1595,7 @@ class depgraph(object):
 				break
 
 		# Remove 'non_conflict_node' and or_tuples from 'forced'.
-		forced = set(pkg for pkg in forced if isinstance(pkg, Package))
+		forced = {pkg for pkg in forced if isinstance(pkg, Package)}
 
 		# Add dependendencies of forced packages.
 		stack = list(forced)
@@ -1604,7 +1609,7 @@ class depgraph(object):
 					forced.add(child)
 					stack.append(child)
 
-		non_forced = set(pkg for pkg in conflict_pkgs if pkg not in forced)
+		non_forced = {pkg for pkg in conflict_pkgs if pkg not in forced}
 
 		if debug:
 			writemsg_level(
@@ -1635,12 +1640,16 @@ class depgraph(object):
 
 		# Process the dependencies of choosen conflict packages
 		# again to  properly account for blockers.
-		broken_packages.update(forced)
+		broken_packages |= forced
 
 		# Filter out broken packages which have been removed during
 		# recursive removal in self._remove_pkg.
-		broken_packages = list(pkg for pkg in broken_packages if pkg in broken_packages \
-			if self._dynamic_config._package_tracker.contains(pkg, installed=False))
+		broken_packages = [
+			pkg
+			for pkg in broken_packages
+			if pkg in broken_packages
+			and self._dynamic_config._package_tracker.contains(pkg, installed=False)
+		]
 
 		self._dynamic_config._dep_stack.extend(broken_packages)
 
@@ -1651,8 +1660,6 @@ class depgraph(object):
 
 		# Record missed updates.
 		for conflict in conflicts:
-			if not any(pkg in non_forced for pkg in conflict):
-				continue
 			for pkg in conflict:
 				if pkg not in non_forced:
 					continue
@@ -1668,7 +1675,6 @@ class depgraph(object):
 							self._dynamic_config._conflict_missed_update[pkg]["slot conflict"].add(
 								(parent, atom))
 
-
 	def _process_slot_conflicts(self):
 		"""
 		If there are any slot conflicts and backtracking is enabled,
@@ -1794,16 +1800,18 @@ class depgraph(object):
 			"slot conflict", []).append(backtrack_data)
 		self._dynamic_config._need_restart = True
 		if debug:
-			msg = []
-			msg.append("")
-			msg.append("")
-			msg.append("backtracking due to slot conflict:")
-			msg.append("   first package:  %s" % existing_node)
-			msg.append("  package to mask: %s" % to_be_masked)
-			msg.append("      slot: %s" % slot_atom)
-			msg.append("   parents: %s" % ", ".join( \
-				"(%s, '%s')" % (ppkg, atom) for ppkg, atom in all_parents))
-			msg.append("")
+			msg = [
+				"",
+				"",
+				"backtracking due to slot conflict:",
+				"   first package:  %s" % existing_node,
+				"  package to mask: %s" % to_be_masked,
+				"      slot: %s" % slot_atom,
+				"   parents: %s" % ", ".join(
+					"(%s, '%s')" % (ppkg, atom) for ppkg, atom in all_parents
+				),
+				""
+			]
 			writemsg_level("".join("%s\n" % l for l in msg),
 				noiselevel=-1, level=logging.DEBUG)
 
@@ -1886,19 +1894,19 @@ class depgraph(object):
 	def _slot_change_backtrack(self, dep, new_child_slot):
 		child = dep.child
 		if "--debug" in self._frozen_config.myopts:
-			msg = []
-			msg.append("")
-			msg.append("")
-			msg.append("backtracking due to slot/sub-slot change:")
-			msg.append("   child package:  %s" % child)
-			msg.append("      child slot:  %s/%s" %
-				(child.slot, child.sub_slot))
-			msg.append("       new child:  %s" % new_child_slot)
-			msg.append("  new child slot:  %s/%s" %
-				(new_child_slot.slot, new_child_slot.sub_slot))
-			msg.append("   parent package: %s" % dep.parent)
-			msg.append("   atom: %s" % dep.atom)
-			msg.append("")
+			msg = [
+				"",
+				"",
+				"backtracking due to slot/sub-slot change:",
+				"   child package:  %s" % child,
+				"      child slot:  %s/%s" % (child.slot, child.sub_slot),
+				"       new child:  %s" % new_child_slot,
+				"  new child slot:  %s/%s" %
+					(new_child_slot.slot, new_child_slot.sub_slot),
+				"   parent package: %s" % dep.parent,
+				"   atom: %s" % dep.atom,
+				""
+			]
 			writemsg_level("\n".join(msg),
 				noiselevel=-1, level=logging.DEBUG)
 		backtrack_infos = self._dynamic_config._backtrack_infos
@@ -1930,11 +1938,12 @@ class depgraph(object):
 		else:
 			child = new_child_slot
 		if "--debug" in self._frozen_config.myopts:
-			msg = []
-			msg.append("")
-			msg.append("")
-			msg.append("backtracking due to missed slot abi update:")
-			msg.append("   child package:  %s" % child)
+			msg = [
+				"",
+				"",
+				"backtracking due to missed slot abi update:",
+				"   child package:  %s" % child
+			]
 			if new_child_slot is not None:
 				msg.append("   new child slot package:  %s" % new_child_slot)
 			msg.append("   parent package: %s" % dep.parent)
@@ -2259,15 +2268,16 @@ class depgraph(object):
 						selected = (pkg, atom, i)
 
 			if debug:
-				msg = []
-				msg.append("")
-				msg.append("")
-				msg.append("slot_operator_update_probe:")
-				msg.append("   existing child package:  %s" % dep.child)
-				msg.append("   existing parent package: %s" % dep.parent)
-				msg.append("   new child package:  %s" % selected[0])
-				msg.append("   new parent package: %s" % replacement_parent)
-				msg.append("")
+				msg = (
+					"",
+					"",
+					"slot_operator_update_probe:",
+					"   existing child package:  %s" % dep.child,
+					"   existing parent package: %s" % dep.parent,
+					"   new child package:  %s" % selected[0],
+					"   new parent package: %s" % replacement_parent,
+					""
+				)
 				writemsg_level("\n".join(msg),
 					noiselevel=-1, level=logging.DEBUG)
 
@@ -2275,15 +2285,16 @@ class depgraph(object):
 				child=selected[0], atom=selected[1])
 
 		if debug:
-			msg = []
-			msg.append("")
-			msg.append("")
-			msg.append("slot_operator_update_probe:")
-			msg.append("   existing child package:  %s" % dep.child)
-			msg.append("   existing parent package: %s" % dep.parent)
-			msg.append("   new child package:  %s" % None)
-			msg.append("   new parent package: %s" % None)
-			msg.append("")
+			msg = (
+				"",
+				"",
+				"slot_operator_update_probe:",
+				"   existing child package:  %s" % dep.child,
+				"   existing parent package: %s" % dep.parent,
+				"   new child package:  %s" % None,
+				"   new parent package: %s" % None,
+				"",
+			)
 			writemsg_level("\n".join(msg),
 				noiselevel=-1, level=logging.DEBUG)
 
@@ -2318,30 +2329,32 @@ class depgraph(object):
 				if pkg is not None:
 
 					if debug:
-						msg = []
-						msg.append("")
-						msg.append("")
-						msg.append("slot_operator_unsatisfied_probe:")
-						msg.append("   existing parent package: %s" % dep.parent)
-						msg.append("   existing parent atom: %s" % dep.atom)
-						msg.append("   new parent package: %s" % replacement_parent)
-						msg.append("   new child package:  %s" % pkg)
-						msg.append("")
+						msg = (
+							"",
+							"",
+							"slot_operator_unsatisfied_probe:",
+							"   existing parent package: %s" % dep.parent,
+							"   existing parent atom: %s" % dep.atom,
+							"   new parent package: %s" % replacement_parent,
+							"   new child package:  %s" % pkg,
+							"",
+						)
 						writemsg_level("\n".join(msg),
 							noiselevel=-1, level=logging.DEBUG)
 
 					return True
 
 		if debug:
-			msg = []
-			msg.append("")
-			msg.append("")
-			msg.append("slot_operator_unsatisfied_probe:")
-			msg.append("   existing parent package: %s" % dep.parent)
-			msg.append("   existing parent atom: %s" % dep.atom)
-			msg.append("   new parent package: %s" % None)
-			msg.append("   new child package:  %s" % None)
-			msg.append("")
+			msg = (
+				"",
+				"",
+				"slot_operator_unsatisfied_probe:",
+				"   existing parent package: %s" % dep.parent,
+				"   existing parent atom: %s" % dep.atom,
+				"   new parent package: %s" % None,
+				"   new child package:  %s" % None,
+				""
+			)
 			writemsg_level("\n".join(msg),
 				noiselevel=-1, level=logging.DEBUG)
 
@@ -2352,14 +2365,14 @@ class depgraph(object):
 		parent = dep.parent
 
 		if "--debug" in self._frozen_config.myopts:
-			msg = []
-			msg.append("")
-			msg.append("")
-			msg.append("backtracking due to unsatisfied "
-				"built slot-operator dep:")
-			msg.append("   parent package: %s" % parent)
-			msg.append("   atom: %s" % dep.atom)
-			msg.append("")
+			msg = (
+				"",
+				"",
+				"backtracking due to unsatisfied built slot-operator dep:",
+				"   parent package: %s" % parent,
+				"   atom: %s" % dep.atom,
+				""
+			)
 			writemsg_level("\n".join(msg),
 				noiselevel=-1, level=logging.DEBUG)
 
@@ -2612,18 +2625,21 @@ class depgraph(object):
 			_get_eapi_attrs(pkg.eapi))
 
 		if newuse or (binpkg_respect_use and not changed_use):
-			flags = set(orig_iuse.symmetric_difference(
-				cur_iuse).difference(forced_flags))
-			flags.update(orig_iuse.intersection(orig_use).symmetric_difference(
-				cur_iuse.intersection(cur_use)))
-			flags.difference_update(feature_flags)
+			flags = set(orig_iuse)
+			flags ^= cur_iuse
+			flags -= forced_flags
+			flags |= (
+				orig_iuse.intersection(orig_use)
+				^ cur_iuse.intersection(cur_use)
+			)
+			flags -= feature_flags
 			if flags:
 				return flags
-
 		elif changed_use or binpkg_respect_use:
-			flags = set(orig_iuse.intersection(orig_use).symmetric_difference(
-				cur_iuse.intersection(cur_use)))
-			flags.difference_update(feature_flags)
+			flags = set(orig_iuse)
+			flags.intersection_update(orig_use)
+			flags ^= cur_iuse.intersection(cur_use)
+			flags -= feature_flags
 			if flags:
 				return flags
 		return None
@@ -5311,8 +5327,8 @@ class depgraph(object):
 				mreasons.append("Missing IUSE: %s" % " ".join(missing_iuse))
 				missing_iuse_reasons.append((pkg, mreasons))
 			else:
-				need_enable = sorted(atom.use.enabled.difference(use).intersection(pkg.iuse.all))
-				need_disable = sorted(atom.use.disabled.intersection(use).intersection(pkg.iuse.all))
+				need_enable = sorted((atom.use.enabled - use) & pkg.iuse.all)
+				need_disable = sorted((atom.use.disabled & use) & pkg.iuse.all)
 
 				untouchable_flags = \
 					frozenset(chain(pkg.use.mask, pkg.use.force))
@@ -6133,7 +6149,7 @@ class depgraph(object):
 					if new_changes.get(real_flag) == True:
 						return old_use
 					new_changes[real_flag] = False
-		new_use.update(old_use.difference(target_use))
+		new_use |= old_use.difference(target_use)
 
 		def want_restart_for_use_change(pkg, new_use):
 			if pkg not in self._dynamic_config.digraph.nodes:
@@ -6409,9 +6425,9 @@ class depgraph(object):
 							if any(x in atom.use.enabled for x in missing_disabled):
 								use_match = False
 								can_adjust_use = False
-							need_enabled = atom.use.enabled.difference(use)
+							need_enabled = atom.use.enabled - use
 							if need_enabled:
-								need_enabled = need_enabled.difference(missing_enabled)
+								need_enabled -= missing_enabled
 								if need_enabled:
 									use_match = False
 									if can_adjust_use:
@@ -6422,9 +6438,9 @@ class depgraph(object):
 							if any(x in atom.use.disabled for x in missing_enabled):
 								use_match = False
 								can_adjust_use = False
-							need_disabled = atom.use.disabled.intersection(use)
+							need_disabled = atom.use.disabled & use
 							if need_disabled:
-								need_disabled = need_disabled.difference(missing_disabled)
+								need_disabled -= missing_disabled
 								if need_disabled:
 									use_match = False
 									if can_adjust_use:
@@ -6835,7 +6851,7 @@ class depgraph(object):
 						# to use.mask/force changes in the profile.
 						if complete_if_new_use and \
 							(node.iuse.all != inst_pkg.iuse.all or
-							self._pkg_use_enabled(node).intersection(node.iuse.all) !=
+							(self._pkg_use_enabled(node) & node.iuse.all) !=
 							self._pkg_use_enabled(inst_pkg).intersection(inst_pkg.iuse.all)):
 							use_change = True
 							break
@@ -7796,10 +7812,11 @@ class depgraph(object):
 						# do the uninstalls first.
 						good_uninstalls = None
 						if len(nodes) > 1:
-							good_uninstalls = []
-							for node in nodes:
-								if node.operation == "uninstall":
-									good_uninstalls.append(node)
+							good_uninstalls = [
+								node
+								for node in nodes
+								if node.operation == "uninstall"
+							]
 
 							if good_uninstalls:
 								nodes = good_uninstalls
@@ -7918,7 +7935,7 @@ class depgraph(object):
 						mygraph.child_nodes(node,
 							ignore_priority = \
 							DepPrioritySatisfiedRange.ignore_medium_soft))
-					medium_soft.difference_update(soft)
+					medium_soft -= soft
 					for child in medium_soft:
 						if child in selected_nodes:
 							continue
@@ -8070,8 +8087,7 @@ class depgraph(object):
 					# common cases.
 					self._spinner_update()
 					mergeable_parent = False
-					parent_deps = set()
-					parent_deps.add(task)
+					parent_deps = {task}
 					for parent in mygraph.parent_nodes(task):
 						parent_deps.update(mygraph.child_nodes(parent,
 							ignore_priority=priority_range.ignore_medium_soft))
@@ -8237,12 +8253,10 @@ class depgraph(object):
 					# since the user might be interested and also
 					# it serves as an indicator that blocking packages
 					# will be temporarily installed simultaneously.
-					for blocker in solved_blockers:
-						retlist.append(blocker)
+					retlist.extend(solved_blockers)
 
 		unsolvable_blockers = set(self._dynamic_config._unsolvable_blockers.leaf_nodes())
-		for node in myblocker_uninstalls.root_nodes():
-			unsolvable_blockers.add(node)
+		unsolvable_blockers.update(myblocker_uninstalls.root_nodes())
 
 		# If any Uninstall tasks need to be executed in order
 		# to avoid a conflict, complete the graph with any
@@ -8254,10 +8268,11 @@ class depgraph(object):
 			not unsolvable_blockers:
 			self._dynamic_config.myparams["complete"] = True
 			if '--debug' in self._frozen_config.myopts:
-				msg = []
-				msg.append("enabling 'complete' depgraph mode " + \
-					"due to uninstall task(s):")
-				msg.append("")
+				msg = [
+					"enabling 'complete' depgraph mode "
+					"due to uninstall task(s):",
+					""
+				]
 				for node in retlist:
 					if isinstance(node, Package) and \
 						node.operation == 'uninstall':
@@ -8274,9 +8289,7 @@ class depgraph(object):
 			if isinstance(node, Blocker):
 				node.satisfied = True
 
-		for blocker in unsolvable_blockers:
-			retlist.append(blocker)
-
+		retlist.extend(unsolvable_blockers)
 		retlist = tuple(retlist)
 
 		if unsolvable_blockers and \
@@ -8385,7 +8398,7 @@ class depgraph(object):
 				if not parent_atoms:
 					atom = self._dynamic_config._blocked_world_pkgs.get(pkg)
 					if atom is not None:
-						parent_atoms = set([("@selected", atom)])
+						parent_atoms = {("@selected", atom)}
 				if parent_atoms:
 					conflict_pkgs[pkg] = parent_atoms
 
@@ -8405,8 +8418,7 @@ class depgraph(object):
 				del conflict_pkgs[pkg]
 
 		if conflict_pkgs:
-			msg = []
-			msg.append("\n")
+			msg = ["\n"]
 			indent = "  "
 			for pkg, parent_atoms in conflict_pkgs.items():
 
@@ -8973,8 +8985,7 @@ class depgraph(object):
 				refs = arg_refs.setdefault(arg_atom, [])
 				if parent not in refs:
 					refs.append(parent)
-			msg = []
-			msg.append(bad("\nWARNING: "))
+			msg = [bad("\nWARNING: ")]
 			if len(self._dynamic_config._pprovided_args) > 1:
 				msg.append("Requested packages will not be " + \
 					"merged because they are listed in\n")
@@ -8993,12 +9004,14 @@ class depgraph(object):
 				msg.append("  %s%s\n" % (colorize("INFORM", str(arg)), ref_string))
 			msg.append("\n")
 			if "selected" in problems_sets or "world" in problems_sets:
-				msg.append("This problem can be solved in one of the following ways:\n\n")
-				msg.append("  A) Use emaint to clean offending packages from world (if not installed).\n")
-				msg.append("  B) Uninstall offending packages (cleans them from world).\n")
-				msg.append("  C) Remove offending entries from package.provided.\n\n")
-				msg.append("The best course of action depends on the reason that an offending\n")
-				msg.append("package.provided entry exists.\n\n")
+				msg.append(
+					"This problem can be solved in one of the following ways:\n\n"
+					"  A) Use emaint to clean offending packages from world (if not installed).\n"
+					"  B) Uninstall offending packages (cleans them from world).\n"
+					"  C) Remove offending entries from package.provided.\n\n"
+					"The best course of action depends on the reason that an offending\n"
+					"package.provided entry exists.\n\n"
+				)
 			writemsg("".join(msg), noiselevel=-1)
 
 		masked_packages = []
@@ -9097,10 +9110,10 @@ class depgraph(object):
 			s = SETPREFIX + k
 			if s in world_set:
 				continue
-			all_added.append(SETPREFIX + k)
+			all_added.append(s)
 		all_added.extend(added_favorites)
-		all_added.sort()
 		if all_added:
+			all_added.sort()
 			skip = False
 			if "--ask" in self._frozen_config.myopts:
 				writemsg_stdout("\n", noiselevel=-1)

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     6d519a589a9656a511b369c223e27e2f8d8437de
Author:     Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Sun Oct 27 18:58:37 2019 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Sun Oct 27 19:11:40 2019 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=6d519a58

emerge: fix error message for unknown options (bug 673400)

Do not use parse_known_args to parse positional arguments, since that
causes unknown options to be handled like unknown positional arguments.

Bug: https://bugs.gentoo.org/673400
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/main.py | 9 ++++-----
 1 file changed, 4 insertions(+), 5 deletions(-)

diff --git a/lib/_emerge/main.py b/lib/_emerge/main.py
index 486664c84..0d2c45a4f 100644
--- a/lib/_emerge/main.py
+++ b/lib/_emerge/main.py
@@ -299,7 +299,6 @@ def _find_bad_atoms(atoms, less_strict=False):
 def parse_opts(tmpcmdline, silent=False):
 	myaction=None
 	myopts = {}
-	myfiles=[]
 
 	actions = frozenset([
 		"clean", "check-news", "config", "depclean", "help",
@@ -810,9 +809,11 @@ def parse_opts(tmpcmdline, silent=False):
 		parser.add_argument(dest=myopt.lstrip("--").replace("-", "_"),
 			*args, **kwargs)
 
+	parser.add_argument('positional_args', nargs='*')
+
 	tmpcmdline = insert_optional_args(tmpcmdline)
 
-	myoptions, myargs = parser.parse_known_args(args=tmpcmdline)
+	myoptions = parser.parse_args(args=tmpcmdline)
 
 	if myoptions.alert in true_y:
 		myoptions.alert = True
@@ -1165,9 +1166,7 @@ def parse_opts(tmpcmdline, silent=False):
 	if myaction is None and myoptions.deselect is True:
 		myaction = 'deselect'
 
-	myfiles += myargs
-
-	return myaction, myopts, myfiles
+	return myaction, myopts, myoptions.positional_args
 
 def profile_check(trees, myaction):
 	if myaction in ("help", "info", "search", "sync", "version"):

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     bdb9be18b287c3c2d35e3ce577c2aeab1fcc86b7
Author:     Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Sat Nov 16 09:21:12 2019 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Sat Nov 16 09:22:26 2019 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=bdb9be18

Binpkg: use aux_get for check_missing_metadata

Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/Binpkg.py | 18 +++++++-----------
 1 file changed, 7 insertions(+), 11 deletions(-)

diff --git a/lib/_emerge/Binpkg.py b/lib/_emerge/Binpkg.py
index 7791ec236..e7dad32fa 100644
--- a/lib/_emerge/Binpkg.py
+++ b/lib/_emerge/Binpkg.py
@@ -1,4 +1,4 @@
-# Copyright 1999-2018 Gentoo Foundation
+# Copyright 1999-2019 Gentoo Authors
 # Distributed under the terms of the GNU General Public License v2
 
 import functools
@@ -261,17 +261,13 @@ class Binpkg(CompositeTask):
 		self._writemsg_level(">>> Extracting info\n")
 
 		pkg_xpak = portage.xpak.tbz2(self._pkg_path)
-		check_missing_metadata = ("CATEGORY", "PF")
-		missing_metadata = set()
-		for k in check_missing_metadata:
-			v = pkg_xpak.getfile(_unicode_encode(k,
-				encoding=_encodings['repo.content']))
-			if not v:
-				missing_metadata.add(k)
-
 		pkg_xpak.unpackinfo(infloc)
-		for k in missing_metadata:
-			if k == "CATEGORY":
+		check_missing_metadata = ("CATEGORY", "PF")
+		for k, v in zip(check_missing_metadata,
+			self._bintree.dbapi.aux_get(self.pkg.cpv, check_missing_metadata)):
+			if v:
+				continue
+			elif k == "CATEGORY":
 				v = pkg.category
 			elif k == "PF":
 				v = pkg.pf

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     0fddb27ef3fd005d236738b6c696152bbb877419
Author:     Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Sun Nov 17 20:52:56 2019 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Sun Nov 17 21:00:59 2019 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=0fddb27e

Binpkg: use cached BINPKGMD5 if available

Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/Binpkg.py | 16 ++++++++--------
 1 file changed, 8 insertions(+), 8 deletions(-)

diff --git a/lib/_emerge/Binpkg.py b/lib/_emerge/Binpkg.py
index e7dad32fa..542ef8fb0 100644
--- a/lib/_emerge/Binpkg.py
+++ b/lib/_emerge/Binpkg.py
@@ -284,14 +284,14 @@ class Binpkg(CompositeTask):
 				f.close()
 
 		# Store the md5sum in the vdb.
-		f = io.open(_unicode_encode(os.path.join(infloc, 'BINPKGMD5'),
-			encoding=_encodings['fs'], errors='strict'),
-			mode='w', encoding=_encodings['content'], errors='strict')
-		try:
-			f.write(_unicode_decode(
-				str(portage.checksum.perform_md5(pkg_path)) + "\n"))
-		finally:
-			f.close()
+		if pkg_path is not None:
+			md5sum, = self._bintree.dbapi.aux_get(self.pkg.cpv, ['MD5'])
+			if not md5sum:
+				md5sum = portage.checksum.perform_md5(pkg_path)
+			with io.open(_unicode_encode(os.path.join(infloc, 'BINPKGMD5'),
+				encoding=_encodings['fs'], errors='strict'),
+				mode='w', encoding=_encodings['content'], errors='strict') as f:
+				f.write(_unicode_decode('{}\n'.format(md5sum)))
 
 		env_extractor = BinpkgEnvExtractor(background=self.background,
 			scheduler=self.scheduler, settings=self.settings)

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     2d8c7fe07d7a503faff2712c6f18f5ff0b23a45c
Author:     Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Mon Nov 18 02:49:51 2019 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Mon Nov 18 02:55:23 2019 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=2d8c7fe0

Binpkg: fix _chpathtool_exit rmtree ENOENT after rename

Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/Binpkg.py | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/lib/_emerge/Binpkg.py b/lib/_emerge/Binpkg.py
index 542ef8fb0..f9cffa26d 100644
--- a/lib/_emerge/Binpkg.py
+++ b/lib/_emerge/Binpkg.py
@@ -378,14 +378,15 @@ class Binpkg(CompositeTask):
 		image_tmp_dir = os.path.join(
 			self.settings["PORTAGE_BUILDDIR"], "image_tmp")
 		build_d = os.path.join(self.settings["D"],
-			self._build_prefix.lstrip(os.sep))
+			self._build_prefix.lstrip(os.sep)).rstrip(os.sep)
 		if not os.path.isdir(build_d):
 			# Assume this is a virtual package or something.
 			shutil.rmtree(self._image_dir)
 			ensure_dirs(self.settings["ED"])
 		else:
 			os.rename(build_d, image_tmp_dir)
-			shutil.rmtree(self._image_dir)
+			if build_d != self._image_dir:
+				shutil.rmtree(self._image_dir)
 			ensure_dirs(os.path.dirname(self.settings["ED"].rstrip(os.sep)))
 			os.rename(image_tmp_dir, self.settings["ED"])
 

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     8d32e06295226e3914c24cd73de013748f3f9bec
Author:     Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Mon Nov 25 06:33:18 2019 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Mon Nov 25 06:37:11 2019 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=8d32e062

Scheduler._run_pkg_pretend: don't use local event loop

Relevant hooks should be added/removed in the _merge method.

Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/Scheduler.py | 6 +-----
 1 file changed, 1 insertion(+), 5 deletions(-)

diff --git a/lib/_emerge/Scheduler.py b/lib/_emerge/Scheduler.py
index 49b376281..af43a2e24 100644
--- a/lib/_emerge/Scheduler.py
+++ b/lib/_emerge/Scheduler.py
@@ -775,11 +775,7 @@ class Scheduler(PollScheduler):
 		"""
 
 		failures = 0
-
-		# Use a local EventLoop instance here, since we don't
-		# want tasks here to trigger the usual Scheduler callbacks
-		# that handle job scheduling and status display.
-		sched_iface = SchedulerInterface(EventLoop(main=False))
+		sched_iface = self._sched_iface
 
 		for x in self._mergelist:
 			if not isinstance(x, Package):

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     a107f5bad841e5fad65298881e5a1feb9ffdbed1
Author:     Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Thu Nov 28 01:40:50 2019 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Thu Nov 28 01:42:26 2019 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=a107f5ba

depgraph: fix buildtime_blockers logic

Fixes: 1b3131db0e22 ("emerge --buildpkgonly: respect buildtime hard blockers")
Bug: https://bugs.gentoo.org/689226
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/depgraph.py | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)

diff --git a/lib/_emerge/depgraph.py b/lib/_emerge/depgraph.py
index 6d8e73172..f80b077bc 100644
--- a/lib/_emerge/depgraph.py
+++ b/lib/_emerge/depgraph.py
@@ -8309,9 +8309,7 @@ class depgraph(object):
 				if blocker.priority.buildtime and blocker.atom.blocker.overlap.forbid:
 					buildtime_blockers.append(blocker)
 
-		if unsolvable_blockers and \
-			not buildtime_blockers and \
-			not self._accept_blocker_conflicts():
+		if unsolvable_blockers and (buildtime_blockers or not self._accept_blocker_conflicts()):
 			self._dynamic_config._unsatisfied_blockers_for_display = (tuple(buildtime_blockers)
 				if buildtime_blockers else unsolvable_blockers)
 			self._dynamic_config._serialized_tasks_cache = retlist

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     f68752b7020c66e501b49acc580d023654a96949
Author:     Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Mon Dec 23 23:15:45 2019 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Mon Dec 23 23:28:19 2019 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=f68752b7

_queue_disjunctive_deps: group disjunctions recursively (bug 701996)

When disjunctive dependencies are queued, recursively group together
disjunctions from the same dependency string so that any overlap between
them will trigger expansion to DNF.

Fixes: fa7b6ea6ecdc ("_queue_disjunctive_deps: group disjunctions (bug 701996)")
Bug: https://bugs.gentoo.org/701996
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/depgraph.py | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/lib/_emerge/depgraph.py b/lib/_emerge/depgraph.py
index 83631fe70..2ab1bf4ac 100644
--- a/lib/_emerge/depgraph.py
+++ b/lib/_emerge/depgraph.py
@@ -3865,20 +3865,20 @@ class depgraph(object):
 					child_pkgs.sort()
 				yield (atom, child_pkgs[-1])
 
-	def _queue_disjunctive_deps(self, pkg, dep_root, dep_priority, dep_struct):
+	def _queue_disjunctive_deps(self, pkg, dep_root, dep_priority, dep_struct, _disjunctions_recursive=None):
 		"""
 		Queue disjunctive (virtual and ||) deps in self._dynamic_config._dep_disjunctive_stack.
 		Yields non-disjunctive deps. Raises InvalidDependString when
 		necessary.
 		"""
-		disjunctions = []
+		disjunctions = [] if _disjunctions_recursive is None else _disjunctions_recursive
 		for x in dep_struct:
 			if isinstance(x, list):
 				if x and x[0] == "||":
 					disjunctions.append(x)
 				else:
 					for y in self._queue_disjunctive_deps(
-						pkg, dep_root, dep_priority, x):
+						pkg, dep_root, dep_priority, x, _disjunctions_recursive=disjunctions):
 						yield y
 			else:
 				# Note: Eventually this will check for PROPERTIES=virtual
@@ -3889,7 +3889,7 @@ class depgraph(object):
 				else:
 					yield x
 
-		if disjunctions:
+		if _disjunctions_recursive is None and disjunctions:
 			self._queue_disjunction(pkg, dep_root, dep_priority, disjunctions)
 
 	def _queue_disjunction(self, pkg, dep_root, dep_priority, dep_struct):

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     677e19cc20322485475c771687baf4a917065b9c
Author:     Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Thu Dec 26 21:01:35 2019 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Thu Dec 26 21:21:12 2019 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=677e19cc

emerge --with-test-deps: use _queue_disjunctive_deps

This delays evaluation of || deps, avoiding possible redundant
packages and conflicts.

Fixes: 85f0dd173ab7 ("emerge --with-test-deps: allow circular deps")
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/depgraph.py | 14 +++++++++-----
 1 file changed, 9 insertions(+), 5 deletions(-)

diff --git a/lib/_emerge/depgraph.py b/lib/_emerge/depgraph.py
index ed7aeccad..0ee50d5de 100644
--- a/lib/_emerge/depgraph.py
+++ b/lib/_emerge/depgraph.py
@@ -3426,11 +3426,15 @@ class depgraph(object):
 							eapi=pkg.eapi,
 							subset={'test'})
 
-						if test_deps and not self._add_pkg_dep_string(
-							pkg, dep_root, self._priority(runtime_post=True),
-							test_deps,
-							allow_unsatisfied):
-							return 0
+						if test_deps:
+							test_deps = list(self._queue_disjunctive_deps(pkg,
+								dep_root, self._priority(runtime_post=True),
+								test_deps))
+
+							if test_deps and not self._add_pkg_dep_string(pkg,
+								dep_root, self._priority(runtime_post=True),
+								test_deps, allow_unsatisfied):
+								return 0
 
 					dep_string = portage.dep.use_reduce(dep_string,
 						uselist=use_enabled,

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     7c6fda0e063ecc1f2411eaad20ea369dffd5bfe5
Author:     Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Mon Feb  3 20:10:01 2020 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Mon Feb  3 20:29:16 2020 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=7c6fda0e

create_world_atom: fix outdated comments regarding greedy atoms

Fixes: 1342b456c460 ("Remove --update greedy SLOTs code from depgraph.select_files()")
Reported-by: Arfrever Frehtes Taifersar Arahesis <Arfrever <AT> Apache.Org>
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/create_world_atom.py | 11 ++++++-----
 1 file changed, 6 insertions(+), 5 deletions(-)

diff --git a/lib/_emerge/create_world_atom.py b/lib/_emerge/create_world_atom.py
index c5e1f58be..277bd0777 100644
--- a/lib/_emerge/create_world_atom.py
+++ b/lib/_emerge/create_world_atom.py
@@ -1,4 +1,4 @@
-# Copyright 1999-2012 Gentoo Foundation
+# Copyright 1999-2020 Gentoo Authors
 # Distributed under the terms of the GNU General Public License v2
 
 import sys
@@ -15,8 +15,8 @@ def create_world_atom(pkg, args_set, root_config, before_install=False):
 	"""Create a new atom for the world file if one does not exist.  If the
 	argument atom is precise enough to identify a specific slot then a slot
 	atom will be returned. Atoms that are in the system set may also be stored
-	in world since system atoms can only match one slot while world atoms can
-	be greedy with respect to slots.  Unslotted system packages will not be
+	in world since a user might want to select muliple slots of a slotted
+	package like gcc for example. Unslotted system packages will not be
 	stored in world."""
 
 	arg_atom = args_set.findAtomForPackage(pkg)
@@ -111,8 +111,9 @@ def create_world_atom(pkg, args_set, root_config, before_install=False):
 		# Both atoms would be identical, so there's nothing to add.
 		return None
 	if not slotted and not arg_atom.repo:
-		# Unlike world atoms, system atoms are not greedy for slots, so they
-		# can't be safely excluded from world if they are slotted.
+		# Don't exclude slotted atoms for system packages from world, since
+		# a user might want to select muliple slots of a slotted package like
+		# gcc for example.
 		system_atom = sets["system"].findAtomForPackage(pkg)
 		if system_atom:
 			if not system_atom.cp.startswith("virtual/"):

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     3102f41a9dcee0f1753723737c293e8517a5937b
Author:     Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Mon Feb  3 20:33:50 2020 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Mon Feb  3 20:34:22 2020 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=3102f41a

create_world_atom: fix spelling of multiple in comments

Reported-by: Arfrever Frehtes Taifersar Arahesis <Arfrever <AT> Apache.Org>
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/create_world_atom.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/lib/_emerge/create_world_atom.py b/lib/_emerge/create_world_atom.py
index 277bd0777..a7f3e30bf 100644
--- a/lib/_emerge/create_world_atom.py
+++ b/lib/_emerge/create_world_atom.py
@@ -15,7 +15,7 @@ def create_world_atom(pkg, args_set, root_config, before_install=False):
 	"""Create a new atom for the world file if one does not exist.  If the
 	argument atom is precise enough to identify a specific slot then a slot
 	atom will be returned. Atoms that are in the system set may also be stored
-	in world since a user might want to select muliple slots of a slotted
+	in world since a user might want to select multiple slots of a slotted
 	package like gcc for example. Unslotted system packages will not be
 	stored in world."""
 
@@ -112,7 +112,7 @@ def create_world_atom(pkg, args_set, root_config, before_install=False):
 		return None
 	if not slotted and not arg_atom.repo:
 		# Don't exclude slotted atoms for system packages from world, since
-		# a user might want to select muliple slots of a slotted package like
+		# a user might want to select multiple slots of a slotted package like
 		# gcc for example.
 		system_atom = sets["system"].findAtomForPackage(pkg)
 		if system_atom:

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     ab472882b13ebe7a8a649c723928250dc8aeb740
Author:     Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Mon Feb 10 05:04:20 2020 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Mon Feb 10 05:05:23 2020 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=ab472882

depgraph: sort package set atoms for consistent results

Traverse the unordered set returned from PackageSet.getAtoms()
in sorted order, for more consistent results.

Bug: https://bugs.gentoo.org/649622
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/depgraph.py | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/lib/_emerge/depgraph.py b/lib/_emerge/depgraph.py
index 1eaf20035..dd29b607c 100644
--- a/lib/_emerge/depgraph.py
+++ b/lib/_emerge/depgraph.py
@@ -4383,7 +4383,7 @@ class depgraph(object):
 		args = self._dynamic_config._initial_arg_list[:]
 
 		for arg in self._expand_set_args(args, add_to_digraph=True):
-			for atom in arg.pset.getAtoms():
+			for atom in sorted(arg.pset.getAtoms()):
 				self._spinner_update()
 				dep = Dependency(atom=atom, onlydeps=onlydeps,
 					root=myroot, parent=arg)
@@ -6975,7 +6975,7 @@ class depgraph(object):
 
 		self._set_args(args)
 		for arg in self._expand_set_args(args, add_to_digraph=True):
-			for atom in arg.pset.getAtoms():
+			for atom in sorted(arg.pset.getAtoms(), reverse=True):
 				self._dynamic_config._dep_stack.append(
 					Dependency(atom=atom, root=arg.root_config.root,
 						parent=arg, depth=self._UNREACHABLE_DEPTH))
@@ -9366,7 +9366,7 @@ class depgraph(object):
 			# added via _add_pkg() so that they are included in the
 			# digraph (needed at least for --tree display).
 			for arg in self._expand_set_args(args, add_to_digraph=True):
-				for atom in arg.pset.getAtoms():
+				for atom in sorted(arg.pset.getAtoms()):
 					pkg, existing_node = self._select_package(
 						arg.root_config.root, atom)
 					if existing_node is None and \

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     350b8f486b96871386d721e48e768a77a1c4b1a6
Author:     Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Tue Feb 11 20:39:37 2020 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Tue Feb 11 20:48:56 2020 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=350b8f48

depgraph: sort nested package set names for consistent results

Traverse the unordered set returned from PackageSet.getNonAtoms()
in sorted order, for more consistent results. Use a reverse sort
since we pop these sets from a stack.

Bug: https://bugs.gentoo.org/649622
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/depgraph.py | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/lib/_emerge/depgraph.py b/lib/_emerge/depgraph.py
index dd29b607c..8e0d79e29 100644
--- a/lib/_emerge/depgraph.py
+++ b/lib/_emerge/depgraph.py
@@ -2788,8 +2788,10 @@ class depgraph(object):
 
 				# Traverse nested sets and add them to the stack
 				# if they're not already in the graph. Also, graph
-				# edges between parent and nested sets.
-				for token in arg.pset.getNonAtoms():
+				# edges between parent and nested sets. Traverse in
+				# sorted order for consistent results, using reverse
+				# sort since we pop these sets from a stack.
+				for token in sorted(arg.pset.getNonAtoms(), reverse=True):
 					if not token.startswith(SETPREFIX):
 						continue
 					s = token[len(SETPREFIX):]

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     a287c49f84ad3af7c8e20bebd116ea972f318e04
Author:     Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Mon Feb 17 21:39:21 2020 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Mon Feb 17 22:49:32 2020 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=a287c49f

AbstractEbuildProcess: add _async_start coroutine

Convert the _start method to an _async_start coroutine, since
eventually this method will need to be a coroutine in order to write
messages to the build log as discussed in bug 709746.

Bug: https://bugs.gentoo.org/709746
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/AbstractEbuildProcess.py | 33 +++++++++------------------------
 lib/_emerge/MiscFunctionsProcess.py  |  8 +++++---
 2 files changed, 14 insertions(+), 27 deletions(-)

diff --git a/lib/_emerge/AbstractEbuildProcess.py b/lib/_emerge/AbstractEbuildProcess.py
index ddf04e9b3..7eb5dfd1b 100644
--- a/lib/_emerge/AbstractEbuildProcess.py
+++ b/lib/_emerge/AbstractEbuildProcess.py
@@ -1,4 +1,4 @@
-# Copyright 1999-2019 Gentoo Foundation
+# Copyright 1999-2020 Gentoo Authors
 # Distributed under the terms of the GNU General Public License v2
 
 import errno
@@ -19,6 +19,7 @@ from portage.package.ebuild._ipc.ExitCommand import ExitCommand
 from portage.package.ebuild._ipc.QueryCommand import QueryCommand
 from portage import shutil, os
 from portage.util.futures import asyncio
+from portage.util.futures.compat_coroutine import coroutine, coroutine_return
 from portage.util._pty import _create_pty_or_pipe
 from portage.util import apply_secpass_permissions
 
@@ -30,7 +31,7 @@ class AbstractEbuildProcess(SpawnProcess):
 
 	__slots__ = ('phase', 'settings',) + \
 		('_build_dir', '_build_dir_unlock', '_ipc_daemon',
-		'_exit_command', '_exit_timeout_id', '_start_future')
+		'_exit_command', '_exit_timeout_id')
 
 	_phases_without_builddir = ('clean', 'cleanrm', 'depend', 'help',)
 	_phases_interactive_whitelist = ('config',)
@@ -55,6 +56,10 @@ class AbstractEbuildProcess(SpawnProcess):
 			self.phase = phase
 
 	def _start(self):
+		self.scheduler.run_until_complete(self._async_start())
+
+	@coroutine
+	def _async_start(self):
 
 		need_builddir = self.phase not in self._phases_without_builddir
 
@@ -69,7 +74,7 @@ class AbstractEbuildProcess(SpawnProcess):
 			self._eerror(textwrap.wrap(msg, 72))
 			self.returncode = 1
 			self._async_wait()
-			return
+			coroutine_return()
 
 		# Check if the cgroup hierarchy is in place. If it's not, mount it.
 		if (os.geteuid() == 0 and platform.system() == 'Linux'
@@ -142,11 +147,7 @@ class AbstractEbuildProcess(SpawnProcess):
 				if 'PORTAGE_BUILDDIR_LOCKED' not in self.settings:
 					self._build_dir = EbuildBuildDir(
 						scheduler=self.scheduler, settings=self.settings)
-					self._start_future = self._build_dir.async_lock()
-					self._start_future.add_done_callback(
-						functools.partial(self._start_post_builddir_lock,
-						start_ipc_daemon=start_ipc_daemon))
-					return
+					yield self._build_dir.async_lock()
 			else:
 				self.settings.pop('PORTAGE_IPC_DAEMON', None)
 		else:
@@ -167,22 +168,6 @@ class AbstractEbuildProcess(SpawnProcess):
 			else:
 				self.settings.pop('PORTAGE_EBUILD_EXIT_FILE', None)
 
-		self._start_post_builddir_lock(start_ipc_daemon=start_ipc_daemon)
-
-	def _start_post_builddir_lock(self, lock_future=None, start_ipc_daemon=False):
-		if lock_future is not None:
-			if lock_future is not self._start_future:
-				raise AssertionError('lock_future is not self._start_future')
-			self._start_future = None
-			if lock_future.cancelled():
-				self._build_dir = None
-				self.cancelled = True
-				self._was_cancelled()
-				self._async_wait()
-				return
-
-			lock_future.result()
-
 		if start_ipc_daemon:
 			self.settings['PORTAGE_IPC_DAEMON'] = "1"
 			self._start_ipc_daemon()

diff --git a/lib/_emerge/MiscFunctionsProcess.py b/lib/_emerge/MiscFunctionsProcess.py
index 89fd22635..cb0363820 100644
--- a/lib/_emerge/MiscFunctionsProcess.py
+++ b/lib/_emerge/MiscFunctionsProcess.py
@@ -1,8 +1,9 @@
-# Copyright 1999-2013 Gentoo Foundation
+# Copyright 1999-2020 Gentoo Authors
 # Distributed under the terms of the GNU General Public License v2
 
 from _emerge.AbstractEbuildProcess import AbstractEbuildProcess
 import portage
+from portage.util.futures.compat_coroutine import coroutine
 portage.proxy.lazyimport.lazyimport(globals(),
 	'portage.package.ebuild.doebuild:spawn'
 )
@@ -15,7 +16,8 @@ class MiscFunctionsProcess(AbstractEbuildProcess):
 
 	__slots__ = ('commands', 'ld_preload_sandbox')
 
-	def _start(self):
+	@coroutine
+	def _async_start(self):
 		settings = self.settings
 		portage_bin_path = settings["PORTAGE_BIN_PATH"]
 		misc_sh_binary = os.path.join(portage_bin_path,
@@ -26,7 +28,7 @@ class MiscFunctionsProcess(AbstractEbuildProcess):
 			self.settings.get("PORTAGE_BACKGROUND") != "subprocess":
 			self.logfile = settings.get("PORTAGE_LOG_FILE")
 
-		AbstractEbuildProcess._start(self)
+		yield AbstractEbuildProcess._async_start(self)
 
 	def _spawn(self, args, **kwargs):
 		# If self.ld_preload_sandbox is None, default to free=False,

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     c7e52d0466211907d20cdbc04f1e90e7da626694
Author:     Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Mon Feb 17 23:39:12 2020 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Tue Feb 18 00:14:00 2020 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=c7e52d04

EbuildPhase: add _async_start coroutine

Convert the _start method to an _async_start coroutine, since
eventually this method will need to be a coroutine in order to write
messages to the build log as discussed in bug 709746.

Also convert SequentialTaskQueue to use the async_start method,
which is now required in order to start EbuildPhase instances.

Bug: https://bugs.gentoo.org/709746
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/EbuildPhase.py         | 21 +++++++++++----------
 lib/_emerge/SequentialTaskQueue.py | 20 ++++++++++++++++----
 2 files changed, 27 insertions(+), 14 deletions(-)

diff --git a/lib/_emerge/EbuildPhase.py b/lib/_emerge/EbuildPhase.py
index 234a273a0..63d0746da 100644
--- a/lib/_emerge/EbuildPhase.py
+++ b/lib/_emerge/EbuildPhase.py
@@ -23,7 +23,7 @@ from portage.util._dyn_libs.soname_deps_qa import (
 )
 from portage.package.ebuild.prepare_build_dirs import (_prepare_workdir,
 		_prepare_fake_distdir, _prepare_fake_filesdir)
-from portage.util.futures.compat_coroutine import coroutine
+from portage.util.futures.compat_coroutine import coroutine, coroutine_return
 from portage.util import writemsg
 from portage.util._async.AsyncTaskFuture import AsyncTaskFuture
 from portage.util.futures.executor.fork import ForkExecutor
@@ -69,6 +69,10 @@ class EbuildPhase(CompositeTask):
 	_locked_phases = ("setup", "preinst", "postinst", "prerm", "postrm")
 
 	def _start(self):
+		self.scheduler.run_until_complete(self._async_start())
+
+	@coroutine
+	def _async_start(self):
 
 		need_builddir = self.phase not in EbuildProcess._phases_without_builddir
 
@@ -138,20 +142,17 @@ class EbuildPhase(CompositeTask):
 			env_extractor = BinpkgEnvExtractor(background=self.background,
 				scheduler=self.scheduler, settings=self.settings)
 			if env_extractor.saved_env_exists():
-				self._start_task(env_extractor, self._env_extractor_exit)
-				return
+				self._current_task = env_extractor
+				yield env_extractor.async_start()
+				yield env_extractor.async_wait()
+				if self._default_exit(env_extractor) != os.EX_OK:
+					self._async_wait()
+					coroutine_return()
 			# If the environment.bz2 doesn't exist, then ebuild.sh will
 			# source the ebuild as a fallback.
 
 		self._start_lock()
 
-	def _env_extractor_exit(self, env_extractor):
-		if self._default_exit(env_extractor) != os.EX_OK:
-			self.wait()
-			return
-
-		self._start_lock()
-
 	def _start_lock(self):
 		if (self.phase in self._locked_phases and
 			"ebuild-locks" in self.settings.features):

diff --git a/lib/_emerge/SequentialTaskQueue.py b/lib/_emerge/SequentialTaskQueue.py
index 80908936c..a4555275f 100644
--- a/lib/_emerge/SequentialTaskQueue.py
+++ b/lib/_emerge/SequentialTaskQueue.py
@@ -1,9 +1,12 @@
-# Copyright 1999-2012 Gentoo Foundation
+# Copyright 1999-2020 Gentoo Authors
 # Distributed under the terms of the GNU General Public License v2
 
 from collections import deque
+import functools
 import sys
 
+from portage.util.futures import asyncio
+from portage.util.futures.compat_coroutine import coroutine
 from portage.util.SlotObject import SlotObject
 
 class SequentialTaskQueue(SlotObject):
@@ -41,18 +44,27 @@ class SequentialTaskQueue(SlotObject):
 				cancelled = getattr(task, "cancelled", None)
 				if not cancelled:
 					self.running_tasks.add(task)
-					task.addExitListener(self._task_exit)
-					task.start()
+					future = asyncio.ensure_future(self._task_coroutine(task), loop=task.scheduler)
+					future.add_done_callback(functools.partial(self._task_exit, task))
 		finally:
 			self._scheduling = False
 
-	def _task_exit(self, task):
+	@coroutine
+	def _task_coroutine(self, task):
+		yield task.async_start()
+		yield task.async_wait()
+
+	def _task_exit(self, task, future):
 		"""
 		Since we can always rely on exit listeners being called, the set of
  		running tasks is always pruned automatically and there is never any need
 		to actively prune it.
 		"""
 		self.running_tasks.remove(task)
+		try:
+			future.result()
+		except asyncio.CancelledError:
+			self.clear()
 		if self._task_queue:
 			self.schedule()
 

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     036c644712c36b2d67e818e6f60aa753ebd6acaf
Author:     Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Tue Feb 18 06:40:57 2020 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Tue Feb 18 06:44:43 2020 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=036c6447

AsynchronousTask: Call _start_hook after _start_async

The start listeners need to run after _start_async has
completed in case they rely on any state that _start_async
has created (same goes for _start).

Fixes: d66e9ec0b105 ("AsynchronousTask: add coroutine async_start method")
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/AsynchronousTask.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/lib/_emerge/AsynchronousTask.py b/lib/_emerge/AsynchronousTask.py
index 280ed16da..d1e23cdf1 100644
--- a/lib/_emerge/AsynchronousTask.py
+++ b/lib/_emerge/AsynchronousTask.py
@@ -25,8 +25,8 @@ class AsynchronousTask(SlotObject):
 
 	@coroutine
 	def async_start(self):
-		self._start_hook()
 		yield self._async_start()
+		self._start_hook()
 
 	@coroutine
 	def _async_start(self):
@@ -38,8 +38,8 @@ class AsynchronousTask(SlotObject):
 		"""
 		Start an asynchronous task and then return as soon as possible.
 		"""
-		self._start_hook()
 		self._start()
+		self._start_hook()
 
 	def _start(self):
 		self.returncode = os.EX_OK

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     d11bf32950bcf01773a38b3afbf6bb35ece051bc
Author:     Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Sat Feb 22 00:04:37 2020 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Sat Feb 22 00:06:07 2020 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=d11bf329

action_sync: display error message with --quiet

Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/actions.py | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/lib/_emerge/actions.py b/lib/_emerge/actions.py
index 4bf9ce425..7a39d5ec7 100644
--- a/lib/_emerge/actions.py
+++ b/lib/_emerge/actions.py
@@ -2082,6 +2082,9 @@ def action_sync(emerge_config, trees=DeprecationWarning,
 		success, msgs = syncer.auto_sync(options=options)
 	if return_messages:
 		print_results(msgs)
+	elif msgs and not success:
+		writemsg_level("".join("%s\n" % (line,) for line in msgs),
+			level=logging.ERROR, noiselevel=-1)
 
 	return os.EX_OK if success else 1
 

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     2c596f49bab63c6c81dd4d68789823d45341264d
Author:     Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Sat Feb 22 22:20:00 2020 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Sun Feb 23 22:01:03 2020 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=2c596f49

EbuildBuildDir: use async_start method

Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/EbuildBuildDir.py | 122 +++++++++++++++---------------------------
 1 file changed, 44 insertions(+), 78 deletions(-)

diff --git a/lib/_emerge/EbuildBuildDir.py b/lib/_emerge/EbuildBuildDir.py
index 477113db8..77dbff1fb 100644
--- a/lib/_emerge/EbuildBuildDir.py
+++ b/lib/_emerge/EbuildBuildDir.py
@@ -1,13 +1,12 @@
-# Copyright 1999-2012 Gentoo Foundation
+# Copyright 1999-2020 Gentoo Authors
 # Distributed under the terms of the GNU General Public License v2
 
-import functools
-
 from _emerge.AsynchronousLock import AsynchronousLock
 
 import portage
 from portage import os
 from portage.exception import PortageException
+from portage.util.futures.compat_coroutine import coroutine
 from portage.util.SlotObject import SlotObject
 
 class EbuildBuildDir(SlotObject):
@@ -39,6 +38,7 @@ class EbuildBuildDir(SlotObject):
 			except OSError:
 				pass
 
+	@coroutine
 	def async_lock(self):
 		"""
 		Acquire the lock asynchronously. Notification is available
@@ -59,47 +59,6 @@ class EbuildBuildDir(SlotObject):
 			raise AssertionError('PORTAGE_BUILDDIR is unset')
 		catdir = os.path.dirname(dir_path)
 		self._catdir = catdir
-		catdir_lock = AsynchronousLock(path=catdir, scheduler=self.scheduler)
-		builddir_lock = AsynchronousLock(path=dir_path, scheduler=self.scheduler)
-		result = self.scheduler.create_future()
-
-		def catdir_locked(catdir_lock):
-			try:
-				self._assert_lock(catdir_lock)
-			except AssertionError as e:
-				result.set_exception(e)
-				return
-
-			try:
-				portage.util.ensure_dirs(catdir,
-					gid=portage.portage_gid,
-					mode=0o70, mask=0)
-			except PortageException as e:
-				if not os.path.isdir(catdir):
-					result.set_exception(e)
-					return
-
-			builddir_lock.addExitListener(builddir_locked)
-			builddir_lock.start()
-
-		def builddir_locked(builddir_lock):
-			try:
-				self._assert_lock(builddir_lock)
-			except AssertionError as e:
-				catdir_lock.async_unlock.add_done_callback(
-					functools.partial(catdir_unlocked, exception=e))
-				return
-
-			self._lock_obj = builddir_lock
-			self.locked = True
-			self.settings['PORTAGE_BUILDDIR_LOCKED'] = '1'
-			catdir_lock.async_unlock().add_done_callback(catdir_unlocked)
-
-		def catdir_unlocked(future, exception=None):
-			if not (exception is None and future.exception() is None):
-				result.set_exception(exception or future.exception())
-			else:
-				result.set_result(None)
 
 		try:
 			portage.util.ensure_dirs(os.path.dirname(catdir),
@@ -109,10 +68,36 @@ class EbuildBuildDir(SlotObject):
 			if not os.path.isdir(os.path.dirname(catdir)):
 				raise
 
-		catdir_lock.addExitListener(catdir_locked)
-		catdir_lock.start()
-		return result
+		catdir_lock = AsynchronousLock(path=catdir, scheduler=self.scheduler)
+		yield catdir_lock.async_start()
+		yield catdir_lock.async_wait()
+
+		self._assert_lock(catdir_lock)
+
+		try:
+			portage.util.ensure_dirs(catdir,
+				gid=portage.portage_gid,
+				mode=0o70, mask=0)
+		except PortageException:
+			if not os.path.isdir(catdir):
+				raise
+
+		builddir_lock = AsynchronousLock(path=dir_path, scheduler=self.scheduler)
+		yield builddir_lock.async_start()
+		yield builddir_lock.async_wait()
+
+		try:
+			self._assert_lock(builddir_lock)
+		except AssertionError:
+			yield catdir_lock.async_unlock()
+			raise
 
+		self._lock_obj = builddir_lock
+		self.locked = True
+		self.settings['PORTAGE_BUILDDIR_LOCKED'] = '1'
+		yield catdir_lock.async_unlock()
+
+	@coroutine
 	def async_unlock(self):
 		"""
 		Release the lock asynchronously. Release notification is available
@@ -120,41 +105,22 @@ class EbuildBuildDir(SlotObject):
 
 		@returns: Future, result is None
 		"""
-		result = self.scheduler.create_future()
-
-		def builddir_unlocked(future):
-			if future.exception() is not None:
-				result.set_exception(future.exception())
-			else:
-				self._lock_obj = None
-				self.locked = False
-				self.settings.pop('PORTAGE_BUILDDIR_LOCKED', None)
-				catdir_lock = AsynchronousLock(
-					path=self._catdir, scheduler=self.scheduler)
-				catdir_lock.addExitListener(catdir_locked)
-				catdir_lock.start()
-
-		def catdir_locked(catdir_lock):
-			if catdir_lock.wait() != os.EX_OK:
-				result.set_result(None)
-			else:
+		if self._lock_obj is not None:
+			yield self._lock_obj.async_unlock()
+
+			self._lock_obj = None
+			self.locked = False
+			self.settings.pop('PORTAGE_BUILDDIR_LOCKED', None)
+			catdir_lock = AsynchronousLock(
+				path=self._catdir, scheduler=self.scheduler)
+			yield catdir_lock.async_start()
+			yield catdir_lock.async_wait()
+			if catdir_lock.returncode == os.EX_OK:
 				try:
 					os.rmdir(self._catdir)
 				except OSError:
 					pass
-				catdir_lock.async_unlock().add_done_callback(catdir_unlocked)
-
-		def catdir_unlocked(future):
-			if future.exception() is None:
-				result.set_result(None)
-			else:
-				result.set_exception(future.exception())
-
-		if self._lock_obj is None:
-			self.scheduler.call_soon(result.set_result, None)
-		else:
-			self._lock_obj.async_unlock().add_done_callback(builddir_unlocked)
-		return result
+				yield catdir_lock.async_unlock()
 
 	class AlreadyLocked(portage.exception.PortageException):
 		pass

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     8074127bbc213fde75d51309c8fb4ee33ad278aa
Author:     Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Sat Feb 29 08:33:46 2020 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Sat Feb 29 08:34:34 2020 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=8074127b

SpawnProcess: add _main coroutine

Add a _main coroutine method and make the PipeLogger instance
a local variable.

Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/SpawnProcess.py | 33 ++++++++++++++++++++++-----------
 1 file changed, 22 insertions(+), 11 deletions(-)

diff --git a/lib/_emerge/SpawnProcess.py b/lib/_emerge/SpawnProcess.py
index ba58d9d0e..cda615ded 100644
--- a/lib/_emerge/SpawnProcess.py
+++ b/lib/_emerge/SpawnProcess.py
@@ -20,6 +20,7 @@ from portage.localization import _
 from portage.output import EOutput
 from portage.util import writemsg_level
 from portage.util._async.PipeLogger import PipeLogger
+from portage.util.futures import asyncio
 from portage.util.futures.compat_coroutine import coroutine
 
 class SpawnProcess(SubProcess):
@@ -36,7 +37,7 @@ class SpawnProcess(SubProcess):
 		"unshare_ipc", "unshare_mount", "unshare_pid", "unshare_net")
 
 	__slots__ = ("args",) + \
-		_spawn_kwarg_names + ("_pipe_logger", "_selinux_type",)
+		_spawn_kwarg_names + ("_main_task", "_selinux_type",)
 
 	# Max number of attempts to kill the processes listed in cgroup.procs,
 	# given that processes may fork before they can be killed.
@@ -141,13 +142,28 @@ class SpawnProcess(SubProcess):
 						fcntl.fcntl(stdout_fd,
 						fcntl.F_GETFD) | fcntl.FD_CLOEXEC)
 
-		self._pipe_logger = PipeLogger(background=self.background,
+		pipe_logger = PipeLogger(background=self.background,
 			scheduler=self.scheduler, input_fd=master_fd,
 			log_file_path=log_file_path,
 			stdout_fd=stdout_fd)
-		self._pipe_logger.addExitListener(self._pipe_logger_exit)
 		self._registered = True
-		yield self._pipe_logger.async_start()
+		yield pipe_logger.async_start()
+
+		self._main_task = asyncio.ensure_future(
+			self._main(pipe_logger), loop=self.scheduler)
+		self._main_task.add_done_callback(self._main_exit)
+
+	@coroutine
+	def _main(self, pipe_logger):
+		if pipe_logger.poll() is None:
+			yield pipe_logger.async_wait()
+
+	def _main_exit(self, main_task):
+		try:
+			main_task.result()
+		except asyncio.CancelledError:
+			self.cancel()
+		self._async_waitpid()
 
 	def _can_log(self, slave_fd):
 		return True
@@ -171,21 +187,16 @@ class SpawnProcess(SubProcess):
 
 		return spawn_func(args, **kwargs)
 
-	def _pipe_logger_exit(self, pipe_logger):
-		self._pipe_logger = None
-		self._async_waitpid()
-
 	def _unregister(self):
 		SubProcess._unregister(self)
 		if self.cgroup is not None:
 			self._cgroup_cleanup()
 			self.cgroup = None
-		if self._pipe_logger is not None:
-			self._pipe_logger.cancel()
-			self._pipe_logger = None
 
 	def _cancel(self):
 		SubProcess._cancel(self)
+		if self._main_task is not None:
+			self._main_task.cancel()
 		self._cgroup_cleanup()
 
 	def _cgroup_cleanup(self):

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     10cc6febb701cde8f7c099f5ecdf177486e94b3c
Author:     Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Sat Feb 29 18:49:48 2020 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Sat Feb 29 18:50:28 2020 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=10cc6feb

SpawnProcess: only cancel self._main_task if it's not done

Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/SpawnProcess.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/_emerge/SpawnProcess.py b/lib/_emerge/SpawnProcess.py
index cda615ded..aef000e22 100644
--- a/lib/_emerge/SpawnProcess.py
+++ b/lib/_emerge/SpawnProcess.py
@@ -196,7 +196,7 @@ class SpawnProcess(SubProcess):
 	def _cancel(self):
 		SubProcess._cancel(self)
 		if self._main_task is not None:
-			self._main_task.cancel()
+			self._main_task.done() or self._main_task.cancel()
 		self._cgroup_cleanup()
 
 	def _cgroup_cleanup(self):

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     6eb6500435c80837f106d3f9f9f6bb246ff00c16
Author:     Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Sat Feb 29 21:43:29 2020 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Sat Feb 29 21:46:49 2020 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=6eb65004

SpawnProcess: make _main cancel pipe_logger when appropriate

Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/SpawnProcess.py | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

diff --git a/lib/_emerge/SpawnProcess.py b/lib/_emerge/SpawnProcess.py
index aef000e22..941b56683 100644
--- a/lib/_emerge/SpawnProcess.py
+++ b/lib/_emerge/SpawnProcess.py
@@ -155,8 +155,13 @@ class SpawnProcess(SubProcess):
 
 	@coroutine
 	def _main(self, pipe_logger):
-		if pipe_logger.poll() is None:
-			yield pipe_logger.async_wait()
+		try:
+			if pipe_logger.poll() is None:
+				yield pipe_logger.async_wait()
+		except asyncio.CancelledError:
+			if pipe_logger.poll() is None:
+				pipe_logger.cancel()
+			raise
 
 	def _main_exit(self, main_task):
 		try:

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     d331f03ab25422e9afc2e752ea8e39c93a4b0bec
Author:     Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Sat Feb 29 22:38:35 2020 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Sat Feb 29 22:43:51 2020 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=d331f03a

SpawnProcess: cancel pipe_logger after async_start CancelledError

Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/SpawnProcess.py | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/lib/_emerge/SpawnProcess.py b/lib/_emerge/SpawnProcess.py
index 941b56683..ab7971ca8 100644
--- a/lib/_emerge/SpawnProcess.py
+++ b/lib/_emerge/SpawnProcess.py
@@ -147,7 +147,12 @@ class SpawnProcess(SubProcess):
 			log_file_path=log_file_path,
 			stdout_fd=stdout_fd)
 		self._registered = True
-		yield pipe_logger.async_start()
+		try:
+			yield pipe_logger.async_start()
+		except asyncio.CancelledError:
+			if pipe_logger.poll() is None:
+				pipe_logger.cancel()
+			raise
 
 		self._main_task = asyncio.ensure_future(
 			self._main(pipe_logger), loop=self.scheduler)

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     1c681e0cc783ca241b1f2e1c782fccd8f1aac769
Author:     Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Sun Mar  1 00:50:37 2020 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Sun Mar  1 00:51:35 2020 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=1c681e0c

CompositeTask: call self.cancel() for _start_task CancelledError

Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/CompositeTask.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/_emerge/CompositeTask.py b/lib/_emerge/CompositeTask.py
index ce7136c3d..319f9f995 100644
--- a/lib/_emerge/CompositeTask.py
+++ b/lib/_emerge/CompositeTask.py
@@ -118,7 +118,7 @@ class CompositeTask(AsynchronousTask):
 		try:
 			future.result()
 		except asyncio.CancelledError:
-			self.cancelled = True
+			self.cancel()
 			self._was_cancelled()
 			self._async_wait()
 

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     899d5e61010a85702e7c83506d2214ede179c964
Author:     Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Sun Mar  1 01:34:47 2020 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Sun Mar  1 01:41:03 2020 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=899d5e61

EbuildBuildDir: cancel current tasks for CancelledError

Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/EbuildBuildDir.py | 46 ++++++++++++++++++++++++++++---------------
 1 file changed, 30 insertions(+), 16 deletions(-)

diff --git a/lib/_emerge/EbuildBuildDir.py b/lib/_emerge/EbuildBuildDir.py
index 77dbff1fb..f1c49f041 100644
--- a/lib/_emerge/EbuildBuildDir.py
+++ b/lib/_emerge/EbuildBuildDir.py
@@ -6,6 +6,7 @@ from _emerge.AsynchronousLock import AsynchronousLock
 import portage
 from portage import os
 from portage.exception import PortageException
+from portage.util.futures import asyncio
 from portage.util.futures.compat_coroutine import coroutine
 from portage.util.SlotObject import SlotObject
 
@@ -69,22 +70,29 @@ class EbuildBuildDir(SlotObject):
 				raise
 
 		catdir_lock = AsynchronousLock(path=catdir, scheduler=self.scheduler)
-		yield catdir_lock.async_start()
-		yield catdir_lock.async_wait()
-
-		self._assert_lock(catdir_lock)
-
+		builddir_lock = AsynchronousLock(path=dir_path, scheduler=self.scheduler)
 		try:
-			portage.util.ensure_dirs(catdir,
-				gid=portage.portage_gid,
-				mode=0o70, mask=0)
-		except PortageException:
-			if not os.path.isdir(catdir):
-				raise
+			yield catdir_lock.async_start()
+			yield catdir_lock.async_wait()
 
-		builddir_lock = AsynchronousLock(path=dir_path, scheduler=self.scheduler)
-		yield builddir_lock.async_start()
-		yield builddir_lock.async_wait()
+			self._assert_lock(catdir_lock)
+
+			try:
+				portage.util.ensure_dirs(catdir,
+					gid=portage.portage_gid,
+					mode=0o70, mask=0)
+			except PortageException:
+				if not os.path.isdir(catdir):
+					raise
+
+			yield builddir_lock.async_start()
+			yield builddir_lock.async_wait()
+		except asyncio.CancelledError:
+			if catdir_lock.poll() is None:
+				catdir_lock.cancel()
+			if builddir_lock.poll() is None:
+				builddir_lock.cancel()
+			raise
 
 		try:
 			self._assert_lock(builddir_lock)
@@ -113,8 +121,14 @@ class EbuildBuildDir(SlotObject):
 			self.settings.pop('PORTAGE_BUILDDIR_LOCKED', None)
 			catdir_lock = AsynchronousLock(
 				path=self._catdir, scheduler=self.scheduler)
-			yield catdir_lock.async_start()
-			yield catdir_lock.async_wait()
+			try:
+				yield catdir_lock.async_start()
+				yield catdir_lock.async_wait()
+			except asyncio.CancelledError:
+				if catdir_lock.poll() is None:
+					catdir_lock.cancel()
+				raise
+
 			if catdir_lock.returncode == os.EX_OK:
 				try:
 					os.rmdir(self._catdir)

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     69f13236cd4f69451a911a8a0073f9bfe2981f40
Author:     Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Sun Mar  1 02:12:49 2020 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Sun Mar  1 02:15:48 2020 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=69f13236

_PostPhaseCommands: avoid CancelledError in _soname_deps_qa done callback

Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/EbuildPhase.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/_emerge/EbuildPhase.py b/lib/_emerge/EbuildPhase.py
index 63d0746da..f6b380e05 100644
--- a/lib/_emerge/EbuildPhase.py
+++ b/lib/_emerge/EbuildPhase.py
@@ -458,7 +458,7 @@ class _PostPhaseCommands(CompositeTask):
 				# This operates on REQUIRES metadata generated by the above function call.
 				future = self._soname_deps_qa()
 				# If an unexpected exception occurs, then this will raise it.
-				future.add_done_callback(lambda future: future.result())
+				future.add_done_callback(lambda future: future.cancelled() or future.result())
 				self._start_task(AsyncTaskFuture(future=future), self._default_final_exit)
 			else:
 				self._default_final_exit(task)

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     1681309f252a4e91d7256b895a9af26ef82a9b30
Author:     Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Sun Mar  1 18:28:21 2020 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Sun Mar  1 18:36:04 2020 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=1681309f

_BinpkgFetcherProcess: fix async_lock event loop recursion (bug 711178)

Make the async_lock method use the AsynchronousLock async_start
method in order to avoid event loop recursion.

Fixes: 5c40c3e7e ("SpawnProcess: use async_start method (bug 709746)")
Bug: https://bugs.gentoo.org/711178
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/BinpkgFetcher.py | 32 +++++++++++++++++---------------
 1 file changed, 17 insertions(+), 15 deletions(-)

diff --git a/lib/_emerge/BinpkgFetcher.py b/lib/_emerge/BinpkgFetcher.py
index 640eead91..e788cb05d 100644
--- a/lib/_emerge/BinpkgFetcher.py
+++ b/lib/_emerge/BinpkgFetcher.py
@@ -16,6 +16,7 @@ import portage
 from portage import os
 from portage.util._async.AsyncTaskFuture import AsyncTaskFuture
 from portage.util._pty import _create_pty_or_pipe
+from portage.util.futures import asyncio
 from portage.util.futures.compat_coroutine import coroutine
 
 if sys.hexversion >= 0x3000000:
@@ -205,6 +206,7 @@ class _BinpkgFetcherProcess(SpawnProcess):
 								except OSError:
 									pass
 
+	@coroutine
 	def async_lock(self):
 		"""
 		This raises an AlreadyLocked exception if lock() is called
@@ -215,22 +217,22 @@ class _BinpkgFetcherProcess(SpawnProcess):
 		if self._lock_obj is not None:
 			raise self.AlreadyLocked((self._lock_obj,))
 
-		result = self.scheduler.create_future()
-
-		def acquired_lock(async_lock):
-			if async_lock.wait() == os.EX_OK:
-				self.locked = True
-				result.set_result(None)
-			else:
-				result.set_exception(AssertionError(
-					"AsynchronousLock failed with returncode %s"
-					% (async_lock.returncode,)))
-
-		self._lock_obj = AsynchronousLock(path=self.pkg_path,
+		async_lock = self._lock_obj = AsynchronousLock(path=self.pkg_path,
 			scheduler=self.scheduler)
-		self._lock_obj.addExitListener(acquired_lock)
-		self._lock_obj.start()
-		return result
+		try:
+			yield async_lock.async_start()
+			yield async_lock.async_wait()
+		except asyncio.CancelledError:
+			if async_lock.poll() is None:
+				async_lock.cancel()
+			raise
+
+		if async_lock.returncode != os.EX_OK:
+			raise AssertionError(
+				"AsynchronousLock failed with returncode %s"
+				% (async_lock.returncode,))
+
+		self.locked = True
 
 	class AlreadyLocked(portage.exception.PortageException):
 		pass

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     97587a47fcd00d070c081ad3933e4036cf0f8e81
Author:     Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Sun Mar  1 19:28:33 2020 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Sun Mar  1 20:30:42 2020 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=97587a47

SpawnProcess: cancel _main_task in _unregister (bug 711174)

Cancel _main_task in _unregister, in order to ensure that the build
log is properly closed. Note that is analogous to the fix for bug
658806 in commit a9a62e57194c, but behavior related to coroutine
cancellation now plays a role since the PipeLogger is now cancelled
indirectly via _main_task. We still need to verify that coroutine
cancellation will raise an appropriate exception in the coroutine
(asyncio.CancelledError, GeneratorExit, or StopIteration) and also
that the specific coroutine will react appropriately to the
specific exception that will be raised.

Fixes: 8074127bbc21 ("SpawnProcess: add _main coroutine")
Bug: https://bugs.gentoo.org/711174
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/SpawnProcess.py | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/lib/_emerge/SpawnProcess.py b/lib/_emerge/SpawnProcess.py
index 34668b287..bf5c45189 100644
--- a/lib/_emerge/SpawnProcess.py
+++ b/lib/_emerge/SpawnProcess.py
@@ -218,6 +218,8 @@ class SpawnProcess(SubProcess):
 		if self.cgroup is not None:
 			self._cgroup_cleanup()
 			self.cgroup = None
+		if self._main_task is not None:
+			self._main_task.done() or self._main_task.cancel()
 
 	def _cancel(self):
 		SubProcess._cancel(self)

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     05c2708f45c951978a76cc897ca58b7a6f79c533
Author:     Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Mon Mar  2 01:43:23 2020 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Mon Mar  2 03:29:57 2020 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=05c2708f

Subprocess: delay unregister in _async_wait (bug 711174)

Since the super class AbstractPollTask _async_wait method
calls _unregister, it can trigger premature _unregister before
pid status is available. Therefore, only call the super class
_async_wait method after pid status is available. This should
help prevent premature _unregister events that trigger reading
of build logs before they're closed as in bug 658806 and
bug 711174.

Bug: https://bugs.gentoo.org/711174
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/SubProcess.py | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)

diff --git a/lib/_emerge/SubProcess.py b/lib/_emerge/SubProcess.py
index 7d6b03272..1ddfe57fd 100644
--- a/lib/_emerge/SubProcess.py
+++ b/lib/_emerge/SubProcess.py
@@ -1,10 +1,11 @@
-# Copyright 1999-2018 Gentoo Foundation
+# Copyright 1999-2020 Gentoo Authors
 # Distributed under the terms of the GNU General Public License v2
 
 import logging
 
 from portage import os
 from portage.util import writemsg_level
+from portage.util.futures import asyncio
 from _emerge.AbstractPollTask import AbstractPollTask
 import signal
 import errno
@@ -40,6 +41,14 @@ class SubProcess(AbstractPollTask):
 		return self.pid is not None and \
 			self.returncode is None
 
+	def _async_wait(self):
+		if self.returncode is None:
+			raise asyncio.InvalidStateError('Result is not ready for %s' % (self,))
+		else:
+			# This calls _unregister, so don't call it until pid status
+			# is available.
+			super(SubProcess, self)._async_wait()
+
 	def _async_waitpid(self):
 		"""
 		Wait for exit status of self.pid asynchronously, and then

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     0b78b4a90657ef582a68258f27eaf521dfb42038
Author:     Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Tue Mar  3 05:39:44 2020 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Tue Mar  3 05:46:36 2020 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=0b78b4a9

EbuildMetadataPhase: make _unregister idempotent (bug 711362)

Make the _unregister method check if self._files is None, as
necessary for idempotency. This will prevent the AttributeError
reported in bug 711362, which was triggered by recent changes
in callback scheduling order.

Closes: https://bugs.gentoo.org/711362
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/EbuildMetadataPhase.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/lib/_emerge/EbuildMetadataPhase.py b/lib/_emerge/EbuildMetadataPhase.py
index 4940d40b6..efe71892c 100644
--- a/lib/_emerge/EbuildMetadataPhase.py
+++ b/lib/_emerge/EbuildMetadataPhase.py
@@ -144,7 +144,8 @@ class EbuildMetadataPhase(SubProcess):
 						break
 
 	def _unregister(self):
-		self.scheduler.remove_reader(self._files.ebuild)
+		if self._files is not None:
+			self.scheduler.remove_reader(self._files.ebuild)
 		SubProcess._unregister(self)
 
 	def _async_waitpid_cb(self, *args, **kwargs):

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     459b3535baa416888b546cd1635ae28324259a70
Author:     Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Wed Mar  4 08:17:28 2020 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Wed Mar  4 09:53:27 2020 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=459b3535

SequentialTaskQueue: update bool(self) sooner (bug 711322)

Use addExitListener to add a _task_exit callback that will be invoked
as soon as the task exits (before the future's done callback is called).
This is required in order for bool(self) to have an updated value for
Scheduler._schedule to base assumptions upon. Delayed updates to
bool(self) is what caused Scheduler to hang as in bug 711322.

This reverts changes in SequentialTaskQueue task queue exit listener
behavior from commit c7e52d046621, so that only the changes necessary
to support async_start remain.

Fixes: c7e52d046621 ("EbuildPhase: add _async_start coroutine")
Bug: https://bugs.gentoo.org/711322
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/SequentialTaskQueue.py | 16 +++++++++-------
 1 file changed, 9 insertions(+), 7 deletions(-)

diff --git a/lib/_emerge/SequentialTaskQueue.py b/lib/_emerge/SequentialTaskQueue.py
index a4555275f..318bd6c55 100644
--- a/lib/_emerge/SequentialTaskQueue.py
+++ b/lib/_emerge/SequentialTaskQueue.py
@@ -2,7 +2,6 @@
 # Distributed under the terms of the GNU General Public License v2
 
 from collections import deque
-import functools
 import sys
 
 from portage.util.futures import asyncio
@@ -45,7 +44,14 @@ class SequentialTaskQueue(SlotObject):
 				if not cancelled:
 					self.running_tasks.add(task)
 					future = asyncio.ensure_future(self._task_coroutine(task), loop=task.scheduler)
-					future.add_done_callback(functools.partial(self._task_exit, task))
+					future.add_done_callback(lambda future: future.cancelled() or future.result())
+					# This callback will be invoked as soon as the task
+					# exits (before the future's done callback is called),
+					# and this is required in order for bool(self) to have
+					# an updated value for Scheduler._schedule to base
+					# assumptions upon. Delayed updates to bool(self) is
+					# what caused Scheduler to hang as in bug 709746.
+					task.addExitListener(self._task_exit)
 		finally:
 			self._scheduling = False
 
@@ -54,17 +60,13 @@ class SequentialTaskQueue(SlotObject):
 		yield task.async_start()
 		yield task.async_wait()
 
-	def _task_exit(self, task, future):
+	def _task_exit(self, task):
 		"""
 		Since we can always rely on exit listeners being called, the set of
  		running tasks is always pruned automatically and there is never any need
 		to actively prune it.
 		"""
 		self.running_tasks.remove(task)
-		try:
-			future.result()
-		except asyncio.CancelledError:
-			self.clear()
 		if self._task_queue:
 			self.schedule()
 

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     d389b3b378d88b8c41dfaba2a90bc9643a9ba99e
Author:     Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Thu Mar  5 06:46:26 2020 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Thu Mar  5 08:06:19 2020 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=d389b3b3

Scheduler: use add_done_callback (bug 711322)

Use add_done_callback instead of addExistListener, in order to avoid
callback races like the SequentialTaskQueue exit listener race that
triggered bug 711322. The addExistListener method is prone to races
because its listeners are executed in quick succession. In contrast,
callbacks scheduled via add_done_callback are placed in a fifo
queue, ensuring that they execute in an order that is unsurprising
relative to other callbacks.

Bug: https://bugs.gentoo.org/711322
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/Scheduler.py | 36 ++++++++++++++++++++++--------------
 1 file changed, 22 insertions(+), 14 deletions(-)

diff --git a/lib/_emerge/Scheduler.py b/lib/_emerge/Scheduler.py
index 98eaf3bcc..98bc789ff 100644
--- a/lib/_emerge/Scheduler.py
+++ b/lib/_emerge/Scheduler.py
@@ -1,9 +1,10 @@
-# Copyright 1999-2019 Gentoo Authors
+# Copyright 1999-2020 Gentoo Authors
 # Distributed under the terms of the GNU General Public License v2
 
 from __future__ import division, print_function, unicode_literals
 
 from collections import deque
+import functools
 import gc
 import gzip
 import logging
@@ -1259,11 +1260,13 @@ class Scheduler(PollScheduler):
 				child not in completed_tasks:
 				unsatisfied.add(child)
 
-	def _merge_wait_exit_handler(self, task):
+	def _merge_wait_exit_handler(self, task, future):
+		future.cancelled() or future.result()
 		self._merge_wait_scheduled.remove(task)
-		self._merge_exit(task)
+		self._merge_exit(task, future)
 
-	def _merge_exit(self, merge):
+	def _merge_exit(self, merge, future):
+		future.cancelled() or future.result()
 		self._running_tasks.pop(id(merge), None)
 		self._do_merge_exit(merge)
 		self._deallocate_config(merge.merge.settings)
@@ -1327,7 +1330,8 @@ class Scheduler(PollScheduler):
 			del mtimedb["resume"]
 		mtimedb.commit()
 
-	def _build_exit(self, build):
+	def _build_exit(self, build, future):
+		future.cancelled() or future.result()
 		self._running_tasks.pop(id(build), None)
 		if build.returncode == os.EX_OK and self._terminated_tasks:
 			# We've been interrupted, so we won't
@@ -1336,7 +1340,7 @@ class Scheduler(PollScheduler):
 			self._deallocate_config(build.settings)
 		elif build.returncode == os.EX_OK:
 			self.curval += 1
-			merge = PackageMerge(merge=build)
+			merge = PackageMerge(merge=build, scheduler=self._sched_iface)
 			self._running_tasks[id(merge)] = merge
 			if not build.build_opts.buildpkgonly and \
 				build.pkg in self._deep_system_deps:
@@ -1345,8 +1349,8 @@ class Scheduler(PollScheduler):
 				self._merge_wait_queue.append(merge)
 				merge.addStartListener(self._system_merge_started)
 			else:
-				merge.addExitListener(self._merge_exit)
 				self._task_queues.merge.add(merge)
+				merge.async_wait().add_done_callback(functools.partial(self._merge_exit, merge))
 				self._status_display.merges = len(self._task_queues.merge)
 		else:
 			settings = build.settings
@@ -1365,8 +1369,9 @@ class Scheduler(PollScheduler):
 		self._status_display.running = self._jobs
 		self._schedule()
 
-	def _extract_exit(self, build):
-		self._build_exit(build)
+	def _extract_exit(self, build, future):
+		future.cancelled() or future.result()
+		self._build_exit(build, future)
 
 	def _task_complete(self, pkg):
 		self._completed_tasks.add(pkg)
@@ -1580,9 +1585,10 @@ class Scheduler(PollScheduler):
 			if (self._merge_wait_queue and not self._jobs and
 				not self._task_queues.merge):
 				task = self._merge_wait_queue.popleft()
-				task.addExitListener(self._merge_wait_exit_handler)
+				task.scheduler = self._sched_iface
 				self._merge_wait_scheduled.append(task)
 				self._task_queues.merge.add(task)
+				task.async_wait().add_done_callback(functools.partial(self._merge_wait_exit_handler, task))
 				self._status_display.merges = len(self._task_queues.merge)
 				state_change += 1
 
@@ -1699,26 +1705,28 @@ class Scheduler(PollScheduler):
 			task = self._task(pkg)
 
 			if pkg.installed:
-				merge = PackageMerge(merge=task)
+				merge = PackageMerge(merge=task, scheduler=self._sched_iface)
 				self._running_tasks[id(merge)] = merge
-				merge.addExitListener(self._merge_exit)
 				self._task_queues.merge.addFront(merge)
+				merge.async_wait().add_done_callback(functools.partial(self._merge_exit, merge))
 
 			elif pkg.built:
 				self._jobs += 1
 				self._previous_job_start_time = time.time()
 				self._status_display.running = self._jobs
 				self._running_tasks[id(task)] = task
-				task.addExitListener(self._extract_exit)
+				task.scheduler = self._sched_iface
 				self._task_queues.jobs.add(task)
+				task.async_wait().add_done_callback(functools.partial(self._extract_exit, task))
 
 			else:
 				self._jobs += 1
 				self._previous_job_start_time = time.time()
 				self._status_display.running = self._jobs
 				self._running_tasks[id(task)] = task
-				task.addExitListener(self._build_exit)
+				task.scheduler = self._sched_iface
 				self._task_queues.jobs.add(task)
+				task.async_wait().add_done_callback(functools.partial(self._build_exit, task))
 
 		return bool(state_change)
 

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     62b0dc613d7d8eb099231bc4cb7303a0abdaf480
Author:     Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Thu Mar  5 16:25:45 2020 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Thu Mar  5 16:31:04 2020 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=62b0dc61

AsynchronousTask: schedule start listeners via call_soon (bug 711322)

Schedule start listeners via call_soon, in order to avoid callback races
like the SequentialTaskQueue exit listener race that triggered bug
711322. Callbacks scheduled via call_soon are placed in a fifo queue,
ensuring that they execute in an order that is unsurprising relative to
other callbacks.

Bug: https://bugs.gentoo.org/711322
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/AsynchronousTask.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/_emerge/AsynchronousTask.py b/lib/_emerge/AsynchronousTask.py
index 580eef050..1b450e3b0 100644
--- a/lib/_emerge/AsynchronousTask.py
+++ b/lib/_emerge/AsynchronousTask.py
@@ -167,7 +167,7 @@ class AsynchronousTask(SlotObject):
 			self._start_listeners = None
 
 			for f in start_listeners:
-				f(self)
+				self.scheduler.call_soon(f, self)
 
 	def addExitListener(self, f):
 		"""

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     46903f3e5622bc479d4687c76c0e9fada8eb53db
Author:     Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Thu Mar  5 16:45:25 2020 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Thu Mar  5 23:30:03 2020 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=46903f3e

AsynchronousTask: schedule exit listeners via call_soon (bug 711322)

Schedule exit listeners via call_soon, in order to avoid callback races
like the SequentialTaskQueue exit listener race that triggered bug
711322. Callbacks scheduled via call_soon are placed in a fifo queue,
ensuring that they execute in an order that is unsurprising relative to
other callbacks.

Bug: https://bugs.gentoo.org/711322
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/AsynchronousTask.py | 53 ++++++++++++++++-------------------------
 1 file changed, 21 insertions(+), 32 deletions(-)

diff --git a/lib/_emerge/AsynchronousTask.py b/lib/_emerge/AsynchronousTask.py
index 1b450e3b0..799e66a4a 100644
--- a/lib/_emerge/AsynchronousTask.py
+++ b/lib/_emerge/AsynchronousTask.py
@@ -19,7 +19,7 @@ class AsynchronousTask(SlotObject):
 	"""
 
 	__slots__ = ("background", "cancelled", "returncode", "scheduler") + \
-		("_exit_listeners", "_exit_listener_stack", "_start_listeners")
+		("_exit_listener_handles", "_exit_listeners", "_start_listeners")
 
 	_cancelled_returncode = - signal.SIGINT
 
@@ -178,17 +178,16 @@ class AsynchronousTask(SlotObject):
 		self._exit_listeners.append(f)
 
 	def removeExitListener(self, f):
-		if self._exit_listeners is None:
-			if self._exit_listener_stack is not None:
-				try:
-					self._exit_listener_stack.remove(f)
-				except ValueError:
-					pass
-			return
-		try:
-			self._exit_listeners.remove(f)
-		except ValueError:
-			pass
+		if self._exit_listeners is not None:
+			try:
+				self._exit_listeners.remove(f)
+			except ValueError:
+				pass
+
+		if self._exit_listener_handles is not None:
+			handle = self._exit_listener_handles.pop(f, None)
+			if handle is not None:
+				handle.cancel()
 
 	def _wait_hook(self):
 		"""
@@ -200,26 +199,16 @@ class AsynchronousTask(SlotObject):
 		if self.returncode is not None and \
 			self._exit_listeners is not None:
 
-			# This prevents recursion, in case one of the
-			# exit handlers triggers this method again by
-			# calling wait(). Use a stack that gives
-			# removeExitListener() an opportunity to consume
-			# listeners from the stack, before they can get
-			# called below. This is necessary because a call
-			# to one exit listener may result in a call to
-			# removeExitListener() for another listener on
-			# the stack. That listener needs to be removed
-			# from the stack since it would be inconsistent
-			# to call it after it has been been passed into
-			# removeExitListener().
-			self._exit_listener_stack = self._exit_listeners
+			listeners = self._exit_listeners
 			self._exit_listeners = None
+			if self._exit_listener_handles is None:
+				self._exit_listener_handles = {}
 
-			# Execute exit listeners in reverse order, so that
-			# the last added listener is executed first. This
-			# allows SequentialTaskQueue to decrement its running
-			# task count as soon as one of its tasks exits, so that
-			# the value is accurate when other listeners execute.
-			while self._exit_listener_stack:
-				self._exit_listener_stack.pop()(self)
+			for listener in listeners:
+				if listener not in self._exit_listener_handles:
+					self._exit_listener_handles[listener] = \
+						self.scheduler.call_soon(self._exit_listener_cb, listener)
 
+	def _exit_listener_cb(self, listener):
+		del self._exit_listener_handles[listener]
+		listener(self)

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     ca053dec87ea593596f83e8d20c63b40678bf03a
Author:     Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Fri Mar  6 03:15:40 2020 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Fri Mar  6 03:29:07 2020 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=ca053dec

Scheduler: replace add_done_callback with addExitListener

For simplicity, use addExitListener instead of add_done_callback, since
addExitListener has been fixed to use call_soon in commit 46903f3e5622.
Note that each addExitListener call occurs *after* a call to the
SequentialTaskQueue add method, since the SequentialTaskQueue needs to
be notified of task exit *first* (see commit 459b3535baa4).

Bug: https://bugs.gentoo.org/711322
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/Scheduler.py | 27 +++++++++++----------------
 1 file changed, 11 insertions(+), 16 deletions(-)

diff --git a/lib/_emerge/Scheduler.py b/lib/_emerge/Scheduler.py
index 98bc789ff..ee8f3dd5e 100644
--- a/lib/_emerge/Scheduler.py
+++ b/lib/_emerge/Scheduler.py
@@ -4,7 +4,6 @@
 from __future__ import division, print_function, unicode_literals
 
 from collections import deque
-import functools
 import gc
 import gzip
 import logging
@@ -1260,13 +1259,11 @@ class Scheduler(PollScheduler):
 				child not in completed_tasks:
 				unsatisfied.add(child)
 
-	def _merge_wait_exit_handler(self, task, future):
-		future.cancelled() or future.result()
+	def _merge_wait_exit_handler(self, task):
 		self._merge_wait_scheduled.remove(task)
-		self._merge_exit(task, future)
+		self._merge_exit(task)
 
-	def _merge_exit(self, merge, future):
-		future.cancelled() or future.result()
+	def _merge_exit(self, merge):
 		self._running_tasks.pop(id(merge), None)
 		self._do_merge_exit(merge)
 		self._deallocate_config(merge.merge.settings)
@@ -1330,8 +1327,7 @@ class Scheduler(PollScheduler):
 			del mtimedb["resume"]
 		mtimedb.commit()
 
-	def _build_exit(self, build, future):
-		future.cancelled() or future.result()
+	def _build_exit(self, build):
 		self._running_tasks.pop(id(build), None)
 		if build.returncode == os.EX_OK and self._terminated_tasks:
 			# We've been interrupted, so we won't
@@ -1350,7 +1346,7 @@ class Scheduler(PollScheduler):
 				merge.addStartListener(self._system_merge_started)
 			else:
 				self._task_queues.merge.add(merge)
-				merge.async_wait().add_done_callback(functools.partial(self._merge_exit, merge))
+				merge.addExitListener(self._merge_exit)
 				self._status_display.merges = len(self._task_queues.merge)
 		else:
 			settings = build.settings
@@ -1369,9 +1365,8 @@ class Scheduler(PollScheduler):
 		self._status_display.running = self._jobs
 		self._schedule()
 
-	def _extract_exit(self, build, future):
-		future.cancelled() or future.result()
-		self._build_exit(build, future)
+	def _extract_exit(self, build):
+		self._build_exit(build)
 
 	def _task_complete(self, pkg):
 		self._completed_tasks.add(pkg)
@@ -1588,7 +1583,7 @@ class Scheduler(PollScheduler):
 				task.scheduler = self._sched_iface
 				self._merge_wait_scheduled.append(task)
 				self._task_queues.merge.add(task)
-				task.async_wait().add_done_callback(functools.partial(self._merge_wait_exit_handler, task))
+				task.addExitListener(self._merge_wait_exit_handler)
 				self._status_display.merges = len(self._task_queues.merge)
 				state_change += 1
 
@@ -1708,7 +1703,7 @@ class Scheduler(PollScheduler):
 				merge = PackageMerge(merge=task, scheduler=self._sched_iface)
 				self._running_tasks[id(merge)] = merge
 				self._task_queues.merge.addFront(merge)
-				merge.async_wait().add_done_callback(functools.partial(self._merge_exit, merge))
+				merge.addExitListener(self._merge_exit)
 
 			elif pkg.built:
 				self._jobs += 1
@@ -1717,7 +1712,7 @@ class Scheduler(PollScheduler):
 				self._running_tasks[id(task)] = task
 				task.scheduler = self._sched_iface
 				self._task_queues.jobs.add(task)
-				task.async_wait().add_done_callback(functools.partial(self._extract_exit, task))
+				task.addExitListener(self._extract_exit)
 
 			else:
 				self._jobs += 1
@@ -1726,7 +1721,7 @@ class Scheduler(PollScheduler):
 				self._running_tasks[id(task)] = task
 				task.scheduler = self._sched_iface
 				self._task_queues.jobs.add(task)
-				task.async_wait().add_done_callback(functools.partial(self._build_exit, task))
+				task.addExitListener(self._build_exit)
 
 		return bool(state_change)
 

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     50da2e16599202b9ecb3d4494f214a0d30b073d7
Author:     Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Sat Mar  7 19:41:49 2020 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Sat Mar  7 19:54:06 2020 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=50da2e16

AsynchronousTask: simplify isAlive (bug 711688)

Simplify all AsynchronousTask subclasses to use the default
isAlive implementation, which returns True if self.returncode
is not None. This fixes cases where the method would erroneously
return False, leading to issues like bug 711688, where the
CompositeTask isAlive implementation returned False for a
BinpkgPrefetcher instance that was still in the process of
starting via its async_start method.

Fixes: d66e9ec0b105 ("AsynchronousTask: add coroutine async_start method")
Bug: https://bugs.gentoo.org/711688
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/AbstractPollTask.py | 3 ---
 lib/_emerge/CompositeTask.py    | 3 ---
 lib/_emerge/FifoIpcDaemon.py    | 3 ---
 lib/_emerge/SubProcess.py       | 6 +-----
 4 files changed, 1 insertion(+), 14 deletions(-)

diff --git a/lib/_emerge/AbstractPollTask.py b/lib/_emerge/AbstractPollTask.py
index 4157794c6..7e9f2b536 100644
--- a/lib/_emerge/AbstractPollTask.py
+++ b/lib/_emerge/AbstractPollTask.py
@@ -16,9 +16,6 @@ class AbstractPollTask(AsynchronousTask):
 
 	_bufsize = 4096
 
-	def isAlive(self):
-		return bool(self._registered)
-
 	def _read_array(self, f):
 		"""
 		NOTE: array.fromfile() is used here only for testing purposes,

diff --git a/lib/_emerge/CompositeTask.py b/lib/_emerge/CompositeTask.py
index 319f9f995..1d199d19b 100644
--- a/lib/_emerge/CompositeTask.py
+++ b/lib/_emerge/CompositeTask.py
@@ -12,9 +12,6 @@ class CompositeTask(AsynchronousTask):
 
 	_TASK_QUEUED = -1
 
-	def isAlive(self):
-		return self._current_task is not None
-
 	def _cancel(self):
 		if self._current_task is not None:
 			if self._current_task is self._TASK_QUEUED:

diff --git a/lib/_emerge/FifoIpcDaemon.py b/lib/_emerge/FifoIpcDaemon.py
index 0cbaa13c7..2ec69d1cb 100644
--- a/lib/_emerge/FifoIpcDaemon.py
+++ b/lib/_emerge/FifoIpcDaemon.py
@@ -70,9 +70,6 @@ class FifoIpcDaemon(AbstractPollTask):
 			self._files.pipe_in,
 			self._input_handler)
 
-	def isAlive(self):
-		return self._registered
-
 	def _cancel(self):
 		if self.returncode is None:
 			self.returncode = 1

diff --git a/lib/_emerge/SubProcess.py b/lib/_emerge/SubProcess.py
index 1ddfe57fd..e834cb7d3 100644
--- a/lib/_emerge/SubProcess.py
+++ b/lib/_emerge/SubProcess.py
@@ -24,7 +24,7 @@ class SubProcess(AbstractPollTask):
 		return self.returncode
 
 	def _cancel(self):
-		if self.isAlive():
+		if self.isAlive() and self.pid is not None:
 			try:
 				os.kill(self.pid, signal.SIGTERM)
 			except OSError as e:
@@ -37,10 +37,6 @@ class SubProcess(AbstractPollTask):
 				elif e.errno != errno.ESRCH:
 					raise
 
-	def isAlive(self):
-		return self.pid is not None and \
-			self.returncode is None
-
 	def _async_wait(self):
 		if self.returncode is None:
 			raise asyncio.InvalidStateError('Result is not ready for %s' % (self,))

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     8cc84cea654238676f7edc04b9c75c001535c0b4
Author:     Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Sat Mar  7 21:52:53 2020 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Sat Mar  7 22:01:22 2020 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=8cc84cea

SequentialTaskQueue: cancel unstarted tasks when appropriate (bug 711322)

When the clear method is called, cancel any tasks which have not
started yet, in order to ensure that their start/exit listeners are
called. This fixes a case where emerge would hang after SIGINT.

Also fix the CompositeTask _cancel method to react appropriately to
the cancel event when the task has not started yet.

Bug: https://bugs.gentoo.org/711322
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/CompositeTask.py       | 4 ++++
 lib/_emerge/SequentialTaskQueue.py | 3 +++
 2 files changed, 7 insertions(+)

diff --git a/lib/_emerge/CompositeTask.py b/lib/_emerge/CompositeTask.py
index 1d199d19b..2ad1d783d 100644
--- a/lib/_emerge/CompositeTask.py
+++ b/lib/_emerge/CompositeTask.py
@@ -20,6 +20,10 @@ class CompositeTask(AsynchronousTask):
 				self._async_wait()
 			else:
 				self._current_task.cancel()
+		elif self.returncode is None:
+			# Assume that the task has not started yet.
+			self._was_cancelled()
+			self._async_wait()
 
 	def _poll(self):
 		"""

diff --git a/lib/_emerge/SequentialTaskQueue.py b/lib/_emerge/SequentialTaskQueue.py
index 318bd6c55..38ebb98d8 100644
--- a/lib/_emerge/SequentialTaskQueue.py
+++ b/lib/_emerge/SequentialTaskQueue.py
@@ -74,7 +74,10 @@ class SequentialTaskQueue(SlotObject):
 		"""
 		Clear the task queue and asynchronously terminate any running tasks.
 		"""
+		for task in self._task_queue:
+			task.cancel()
 		self._task_queue.clear()
+
 		for task in list(self.running_tasks):
 			task.cancel()
 

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     79b2cda96ea63cb9e4dca2df92a1f980f03cfc1d
Author:     Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Sun Mar 22 20:45:14 2020 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Sun Mar 22 20:46:21 2020 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=79b2cda9

AsynchronousTask: include self in InvalidStateError message

Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/AbstractEbuildProcess.py | 2 +-
 lib/_emerge/AsynchronousTask.py      | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/lib/_emerge/AbstractEbuildProcess.py b/lib/_emerge/AbstractEbuildProcess.py
index 45545ae15..cc1ea0f1a 100644
--- a/lib/_emerge/AbstractEbuildProcess.py
+++ b/lib/_emerge/AbstractEbuildProcess.py
@@ -388,7 +388,7 @@ class AbstractEbuildProcess(SpawnProcess):
 			SpawnProcess._async_wait(self)
 		elif self._build_dir_unlock is None:
 			if self.returncode is None:
-				raise asyncio.InvalidStateError('Result is not ready.')
+				raise asyncio.InvalidStateError('Result is not ready for %s' % (self,))
 			self._async_unlock_builddir(returncode=self.returncode)
 
 	def _async_unlock_builddir(self, returncode=None):

diff --git a/lib/_emerge/AsynchronousTask.py b/lib/_emerge/AsynchronousTask.py
index ec5497b69..aaab7433a 100644
--- a/lib/_emerge/AsynchronousTask.py
+++ b/lib/_emerge/AsynchronousTask.py
@@ -102,7 +102,7 @@ class AsynchronousTask(SlotObject):
 		"""
 		if self.returncode is None:
 			if self.scheduler.is_running():
-				raise asyncio.InvalidStateError('Result is not ready.')
+				raise asyncio.InvalidStateError('Result is not ready for %s' % (self,))
 			self.scheduler.run_until_complete(self.async_wait())
 		self._wait_hook()
 		return self.returncode

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     21aea3df73497184c5c94fd6c69ebe23f3d214e9
Author:     Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Mon Mar 23 00:19:27 2020 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Mon Mar 23 00:41:36 2020 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=21aea3df

AsynchronousTask: in _async_wait, call self.wait() immediately

In _async_wait, call self.wait() immediately so that the stack
trace for an InvalidStateError will contain useful debugging
information. Since commit 46903f3e5622, exit listeners are
invoked via call_soon, so it's no longer unecessary for
_async_wait to invoke self.wait() via call_soon.

Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/AsynchronousTask.py | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git a/lib/_emerge/AsynchronousTask.py b/lib/_emerge/AsynchronousTask.py
index aaab7433a..c4197d468 100644
--- a/lib/_emerge/AsynchronousTask.py
+++ b/lib/_emerge/AsynchronousTask.py
@@ -109,13 +109,13 @@ class AsynchronousTask(SlotObject):
 
 	def _async_wait(self):
 		"""
-		For cases where _start exits synchronously, this method is a
-		convenient way to trigger an asynchronous call to self.wait()
-		(in order to notify exit listeners), avoiding excessive event
-		loop recursion (or stack overflow) that synchronous calling of
-		exit listeners can cause. This method is thread-safe.
+		Subclasses call this method in order to invoke exit listeners when
+		self.returncode is set. Subclasses may override this method in order
+		to perform cleanup. The default implementation for this method simply
+		calls self.wait(), which will immediately raise an InvalidStateError
+		if the event loop is running and self.returncode is None.
 		"""
-		self.scheduler.call_soon(self.wait)
+		self.wait()
 
 	def cancel(self):
 		"""

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     ce66387db2ecad4c9b11b94a5aac6cf0181902c0
Author:     Sam James (sam_c) <sam <AT> cmpct <DOT> info>
AuthorDate: Sun Mar 22 02:23:54 2020 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Mon Mar 23 01:17:46 2020 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=ce66387d

emerge --info: Filter variables for credentials

Bug: https://bugs.gentoo.org/713726
Closes: https://github.com/gentoo/portage/pull/536
Signed-off-by: Sam James (sam_c) <sam <AT> cmpct.info>
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/actions.py | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/lib/_emerge/actions.py b/lib/_emerge/actions.py
index 7a39d5ec7..392f98d4d 100644
--- a/lib/_emerge/actions.py
+++ b/lib/_emerge/actions.py
@@ -56,6 +56,7 @@ bad = create_color_func("BAD")
 warn = create_color_func("WARN")
 from portage.package.ebuild._ipc.QueryCommand import QueryCommand
 from portage.package.ebuild.doebuild import _check_temp_dir
+from portage.package.ebuild.fetch import _hide_url_passwd
 from portage._sets import load_default_config, SETPREFIX
 from portage._sets.base import InternalPackageSet
 from portage.util import cmp_sort_key, writemsg, varexpand, \
@@ -1899,6 +1900,9 @@ def action_info(settings, trees, myopts, myfiles):
 				if default is not None and \
 					default == v:
 					continue
+
+				v = _hide_url_passwd(v)
+
 				append('%s="%s"' % (k, v))
 			else:
 				use = set(v.split())

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     79a46f6ebdc9dc2b9bec611c8b88037b2ffe43d6
Author:     Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Wed Apr  8 04:50:21 2020 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Wed Apr  8 05:29:46 2020 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=79a46f6e

Revert "EbuildBuildDir: cancel current tasks for CancelledError"

This reverts commit 899d5e61010a85702e7c83506d2214ede179c964.

Bug: https://bugs.gentoo.org/716636
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/EbuildBuildDir.py | 46 +++++++++++++++----------------------------
 1 file changed, 16 insertions(+), 30 deletions(-)

diff --git a/lib/_emerge/EbuildBuildDir.py b/lib/_emerge/EbuildBuildDir.py
index f1c49f041..77dbff1fb 100644
--- a/lib/_emerge/EbuildBuildDir.py
+++ b/lib/_emerge/EbuildBuildDir.py
@@ -6,7 +6,6 @@ from _emerge.AsynchronousLock import AsynchronousLock
 import portage
 from portage import os
 from portage.exception import PortageException
-from portage.util.futures import asyncio
 from portage.util.futures.compat_coroutine import coroutine
 from portage.util.SlotObject import SlotObject
 
@@ -70,29 +69,22 @@ class EbuildBuildDir(SlotObject):
 				raise
 
 		catdir_lock = AsynchronousLock(path=catdir, scheduler=self.scheduler)
-		builddir_lock = AsynchronousLock(path=dir_path, scheduler=self.scheduler)
-		try:
-			yield catdir_lock.async_start()
-			yield catdir_lock.async_wait()
+		yield catdir_lock.async_start()
+		yield catdir_lock.async_wait()
 
-			self._assert_lock(catdir_lock)
+		self._assert_lock(catdir_lock)
 
-			try:
-				portage.util.ensure_dirs(catdir,
-					gid=portage.portage_gid,
-					mode=0o70, mask=0)
-			except PortageException:
-				if not os.path.isdir(catdir):
-					raise
-
-			yield builddir_lock.async_start()
-			yield builddir_lock.async_wait()
-		except asyncio.CancelledError:
-			if catdir_lock.poll() is None:
-				catdir_lock.cancel()
-			if builddir_lock.poll() is None:
-				builddir_lock.cancel()
-			raise
+		try:
+			portage.util.ensure_dirs(catdir,
+				gid=portage.portage_gid,
+				mode=0o70, mask=0)
+		except PortageException:
+			if not os.path.isdir(catdir):
+				raise
+
+		builddir_lock = AsynchronousLock(path=dir_path, scheduler=self.scheduler)
+		yield builddir_lock.async_start()
+		yield builddir_lock.async_wait()
 
 		try:
 			self._assert_lock(builddir_lock)
@@ -121,14 +113,8 @@ class EbuildBuildDir(SlotObject):
 			self.settings.pop('PORTAGE_BUILDDIR_LOCKED', None)
 			catdir_lock = AsynchronousLock(
 				path=self._catdir, scheduler=self.scheduler)
-			try:
-				yield catdir_lock.async_start()
-				yield catdir_lock.async_wait()
-			except asyncio.CancelledError:
-				if catdir_lock.poll() is None:
-					catdir_lock.cancel()
-				raise
-
+			yield catdir_lock.async_start()
+			yield catdir_lock.async_wait()
 			if catdir_lock.returncode == os.EX_OK:
 				try:
 					os.rmdir(self._catdir)

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     c2ffa9413bbad9d56cbed6e1d779c204fafafc69
Author:     Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Wed Apr  8 04:48:05 2020 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Wed Apr  8 05:29:45 2020 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=c2ffa941

Revert "_BinpkgFetcherProcess: fix async_lock event loop recursion (bug 711178)"

This reverts commit 1681309f252a4e91d7256b895a9af26ef82a9b30.

Bug: https://bugs.gentoo.org/716636
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/BinpkgFetcher.py | 32 +++++++++++++++-----------------
 1 file changed, 15 insertions(+), 17 deletions(-)

diff --git a/lib/_emerge/BinpkgFetcher.py b/lib/_emerge/BinpkgFetcher.py
index e788cb05d..640eead91 100644
--- a/lib/_emerge/BinpkgFetcher.py
+++ b/lib/_emerge/BinpkgFetcher.py
@@ -16,7 +16,6 @@ import portage
 from portage import os
 from portage.util._async.AsyncTaskFuture import AsyncTaskFuture
 from portage.util._pty import _create_pty_or_pipe
-from portage.util.futures import asyncio
 from portage.util.futures.compat_coroutine import coroutine
 
 if sys.hexversion >= 0x3000000:
@@ -206,7 +205,6 @@ class _BinpkgFetcherProcess(SpawnProcess):
 								except OSError:
 									pass
 
-	@coroutine
 	def async_lock(self):
 		"""
 		This raises an AlreadyLocked exception if lock() is called
@@ -217,22 +215,22 @@ class _BinpkgFetcherProcess(SpawnProcess):
 		if self._lock_obj is not None:
 			raise self.AlreadyLocked((self._lock_obj,))
 
-		async_lock = self._lock_obj = AsynchronousLock(path=self.pkg_path,
+		result = self.scheduler.create_future()
+
+		def acquired_lock(async_lock):
+			if async_lock.wait() == os.EX_OK:
+				self.locked = True
+				result.set_result(None)
+			else:
+				result.set_exception(AssertionError(
+					"AsynchronousLock failed with returncode %s"
+					% (async_lock.returncode,)))
+
+		self._lock_obj = AsynchronousLock(path=self.pkg_path,
 			scheduler=self.scheduler)
-		try:
-			yield async_lock.async_start()
-			yield async_lock.async_wait()
-		except asyncio.CancelledError:
-			if async_lock.poll() is None:
-				async_lock.cancel()
-			raise
-
-		if async_lock.returncode != os.EX_OK:
-			raise AssertionError(
-				"AsynchronousLock failed with returncode %s"
-				% (async_lock.returncode,))
-
-		self.locked = True
+		self._lock_obj.addExitListener(acquired_lock)
+		self._lock_obj.start()
+		return result
 
 	class AlreadyLocked(portage.exception.PortageException):
 		pass

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     66c66190724b6d58961827c5d642c59db10fcdb1
Author:     Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Wed Apr  8 04:47:12 2020 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Wed Apr  8 05:29:45 2020 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=66c66190

Revert "SpawnProcess: cancel _main_task in _unregister (bug 711174)"

This reverts commit 97587a47fcd00d070c081ad3933e4036cf0f8e81.

Bug: https://bugs.gentoo.org/716636
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/SpawnProcess.py | 2 --
 1 file changed, 2 deletions(-)

diff --git a/lib/_emerge/SpawnProcess.py b/lib/_emerge/SpawnProcess.py
index bf5c45189..34668b287 100644
--- a/lib/_emerge/SpawnProcess.py
+++ b/lib/_emerge/SpawnProcess.py
@@ -218,8 +218,6 @@ class SpawnProcess(SubProcess):
 		if self.cgroup is not None:
 			self._cgroup_cleanup()
 			self.cgroup = None
-		if self._main_task is not None:
-			self._main_task.done() or self._main_task.cancel()
 
 	def _cancel(self):
 		SubProcess._cancel(self)

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     d2443903643b8662ee5ac0bd03d3bbad964a9633
Author:     Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Wed Apr  8 04:55:37 2020 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Wed Apr  8 05:29:47 2020 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=d2443903

Revert "SpawnProcess: add _main coroutine"

This reverts commit 8074127bbc213fde75d51309c8fb4ee33ad278aa.

Bug: https://bugs.gentoo.org/716636
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/SpawnProcess.py | 33 +++++++++++----------------------
 1 file changed, 11 insertions(+), 22 deletions(-)

diff --git a/lib/_emerge/SpawnProcess.py b/lib/_emerge/SpawnProcess.py
index cda615ded..ba58d9d0e 100644
--- a/lib/_emerge/SpawnProcess.py
+++ b/lib/_emerge/SpawnProcess.py
@@ -20,7 +20,6 @@ from portage.localization import _
 from portage.output import EOutput
 from portage.util import writemsg_level
 from portage.util._async.PipeLogger import PipeLogger
-from portage.util.futures import asyncio
 from portage.util.futures.compat_coroutine import coroutine
 
 class SpawnProcess(SubProcess):
@@ -37,7 +36,7 @@ class SpawnProcess(SubProcess):
 		"unshare_ipc", "unshare_mount", "unshare_pid", "unshare_net")
 
 	__slots__ = ("args",) + \
-		_spawn_kwarg_names + ("_main_task", "_selinux_type",)
+		_spawn_kwarg_names + ("_pipe_logger", "_selinux_type",)
 
 	# Max number of attempts to kill the processes listed in cgroup.procs,
 	# given that processes may fork before they can be killed.
@@ -142,28 +141,13 @@ class SpawnProcess(SubProcess):
 						fcntl.fcntl(stdout_fd,
 						fcntl.F_GETFD) | fcntl.FD_CLOEXEC)
 
-		pipe_logger = PipeLogger(background=self.background,
+		self._pipe_logger = PipeLogger(background=self.background,
 			scheduler=self.scheduler, input_fd=master_fd,
 			log_file_path=log_file_path,
 			stdout_fd=stdout_fd)
+		self._pipe_logger.addExitListener(self._pipe_logger_exit)
 		self._registered = True
-		yield pipe_logger.async_start()
-
-		self._main_task = asyncio.ensure_future(
-			self._main(pipe_logger), loop=self.scheduler)
-		self._main_task.add_done_callback(self._main_exit)
-
-	@coroutine
-	def _main(self, pipe_logger):
-		if pipe_logger.poll() is None:
-			yield pipe_logger.async_wait()
-
-	def _main_exit(self, main_task):
-		try:
-			main_task.result()
-		except asyncio.CancelledError:
-			self.cancel()
-		self._async_waitpid()
+		yield self._pipe_logger.async_start()
 
 	def _can_log(self, slave_fd):
 		return True
@@ -187,16 +171,21 @@ class SpawnProcess(SubProcess):
 
 		return spawn_func(args, **kwargs)
 
+	def _pipe_logger_exit(self, pipe_logger):
+		self._pipe_logger = None
+		self._async_waitpid()
+
 	def _unregister(self):
 		SubProcess._unregister(self)
 		if self.cgroup is not None:
 			self._cgroup_cleanup()
 			self.cgroup = None
+		if self._pipe_logger is not None:
+			self._pipe_logger.cancel()
+			self._pipe_logger = None
 
 	def _cancel(self):
 		SubProcess._cancel(self)
-		if self._main_task is not None:
-			self._main_task.cancel()
 		self._cgroup_cleanup()
 
 	def _cgroup_cleanup(self):

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     a0700108913e85b359ea7c579147f8f1d7825b9e
Author:     Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Wed Apr  8 05:07:25 2020 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Wed Apr  8 05:29:49 2020 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=a0700108

Revert "EbuildPhase: add _async_start coroutine"

This reverts commit c7e52d0466211907d20cdbc04f1e90e7da626694.

Bug: https://bugs.gentoo.org/716636
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/EbuildPhase.py         | 21 ++++++++++-----------
 lib/_emerge/SequentialTaskQueue.py | 14 +++-----------
 2 files changed, 13 insertions(+), 22 deletions(-)

diff --git a/lib/_emerge/EbuildPhase.py b/lib/_emerge/EbuildPhase.py
index f6b380e05..477e0ba97 100644
--- a/lib/_emerge/EbuildPhase.py
+++ b/lib/_emerge/EbuildPhase.py
@@ -23,7 +23,7 @@ from portage.util._dyn_libs.soname_deps_qa import (
 )
 from portage.package.ebuild.prepare_build_dirs import (_prepare_workdir,
 		_prepare_fake_distdir, _prepare_fake_filesdir)
-from portage.util.futures.compat_coroutine import coroutine, coroutine_return
+from portage.util.futures.compat_coroutine import coroutine
 from portage.util import writemsg
 from portage.util._async.AsyncTaskFuture import AsyncTaskFuture
 from portage.util.futures.executor.fork import ForkExecutor
@@ -69,10 +69,6 @@ class EbuildPhase(CompositeTask):
 	_locked_phases = ("setup", "preinst", "postinst", "prerm", "postrm")
 
 	def _start(self):
-		self.scheduler.run_until_complete(self._async_start())
-
-	@coroutine
-	def _async_start(self):
 
 		need_builddir = self.phase not in EbuildProcess._phases_without_builddir
 
@@ -142,17 +138,20 @@ class EbuildPhase(CompositeTask):
 			env_extractor = BinpkgEnvExtractor(background=self.background,
 				scheduler=self.scheduler, settings=self.settings)
 			if env_extractor.saved_env_exists():
-				self._current_task = env_extractor
-				yield env_extractor.async_start()
-				yield env_extractor.async_wait()
-				if self._default_exit(env_extractor) != os.EX_OK:
-					self._async_wait()
-					coroutine_return()
+				self._start_task(env_extractor, self._env_extractor_exit)
+				return
 			# If the environment.bz2 doesn't exist, then ebuild.sh will
 			# source the ebuild as a fallback.
 
 		self._start_lock()
 
+	def _env_extractor_exit(self, env_extractor):
+		if self._default_exit(env_extractor) != os.EX_OK:
+			self.wait()
+			return
+
+		self._start_lock()
+
 	def _start_lock(self):
 		if (self.phase in self._locked_phases and
 			"ebuild-locks" in self.settings.features):

diff --git a/lib/_emerge/SequentialTaskQueue.py b/lib/_emerge/SequentialTaskQueue.py
index 38ebb98d8..656e5cf7c 100644
--- a/lib/_emerge/SequentialTaskQueue.py
+++ b/lib/_emerge/SequentialTaskQueue.py
@@ -1,11 +1,9 @@
-# Copyright 1999-2020 Gentoo Authors
+# Copyright 1999-2012 Gentoo Foundation
 # Distributed under the terms of the GNU General Public License v2
 
 from collections import deque
 import sys
 
-from portage.util.futures import asyncio
-from portage.util.futures.compat_coroutine import coroutine
 from portage.util.SlotObject import SlotObject
 
 class SequentialTaskQueue(SlotObject):
@@ -43,23 +41,17 @@ class SequentialTaskQueue(SlotObject):
 				cancelled = getattr(task, "cancelled", None)
 				if not cancelled:
 					self.running_tasks.add(task)
-					future = asyncio.ensure_future(self._task_coroutine(task), loop=task.scheduler)
-					future.add_done_callback(lambda future: future.cancelled() or future.result())
 					# This callback will be invoked as soon as the task
 					# exits (before the future's done callback is called),
 					# and this is required in order for bool(self) to have
 					# an updated value for Scheduler._schedule to base
 					# assumptions upon. Delayed updates to bool(self) is
-					# what caused Scheduler to hang as in bug 709746.
+					# what caused Scheduler to hang as in bug 711322.
 					task.addExitListener(self._task_exit)
+					task.start()
 		finally:
 			self._scheduling = False
 
-	@coroutine
-	def _task_coroutine(self, task):
-		yield task.async_start()
-		yield task.async_wait()
-
 	def _task_exit(self, task):
 		"""
 		Since we can always rely on exit listeners being called, the set of

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     acd8f3a3841f3be50d4d649bb7180e3e0c77acfc
Author:     Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Wed Apr  8 05:03:56 2020 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Wed Apr  8 05:29:49 2020 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=acd8f3a3

Revert "AsynchronousTask: Call _start_hook after _start_async"

This reverts commit 036c644712c36b2d67e818e6f60aa753ebd6acaf.

Bug: https://bugs.gentoo.org/716636
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/AsynchronousTask.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/lib/_emerge/AsynchronousTask.py b/lib/_emerge/AsynchronousTask.py
index 962c68547..b25a146c7 100644
--- a/lib/_emerge/AsynchronousTask.py
+++ b/lib/_emerge/AsynchronousTask.py
@@ -25,8 +25,8 @@ class AsynchronousTask(SlotObject):
 
 	@coroutine
 	def async_start(self):
-		yield self._async_start()
 		self._start_hook()
+		yield self._async_start()
 
 	@coroutine
 	def _async_start(self):
@@ -38,8 +38,8 @@ class AsynchronousTask(SlotObject):
 		"""
 		Start an asynchronous task and then return as soon as possible.
 		"""
-		self._start()
 		self._start_hook()
+		self._start()
 
 	def _start(self):
 		self.returncode = os.EX_OK

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     044012ffdbe8e946712fa63da9c03d31d8e1047d
Author:     Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Wed Apr  8 04:53:06 2020 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Wed Apr  8 05:29:46 2020 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=044012ff

Revert "CompositeTask: call self.cancel() for _start_task CancelledError"

This reverts commit 1c681e0cc783ca241b1f2e1c782fccd8f1aac769.

Bug: https://bugs.gentoo.org/716636
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/CompositeTask.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/_emerge/CompositeTask.py b/lib/_emerge/CompositeTask.py
index 2ad1d783d..21129758a 100644
--- a/lib/_emerge/CompositeTask.py
+++ b/lib/_emerge/CompositeTask.py
@@ -119,7 +119,7 @@ class CompositeTask(AsynchronousTask):
 		try:
 			future.result()
 		except asyncio.CancelledError:
-			self.cancel()
+			self.cancelled = True
 			self._was_cancelled()
 			self._async_wait()
 

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     5f0f1daa25e4eef751660909610ab247aff50f39
Author:     Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Wed Apr  8 05:02:13 2020 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Wed Apr  8 05:29:48 2020 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=5f0f1daa

Revert "EbuildBuildDir: use async_start method"

This reverts commit 2c596f49bab63c6c81dd4d68789823d45341264d.

Bug: https://bugs.gentoo.org/716636
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/EbuildBuildDir.py | 122 +++++++++++++++++++++++++++---------------
 1 file changed, 78 insertions(+), 44 deletions(-)

diff --git a/lib/_emerge/EbuildBuildDir.py b/lib/_emerge/EbuildBuildDir.py
index 77dbff1fb..477113db8 100644
--- a/lib/_emerge/EbuildBuildDir.py
+++ b/lib/_emerge/EbuildBuildDir.py
@@ -1,12 +1,13 @@
-# Copyright 1999-2020 Gentoo Authors
+# Copyright 1999-2012 Gentoo Foundation
 # Distributed under the terms of the GNU General Public License v2
 
+import functools
+
 from _emerge.AsynchronousLock import AsynchronousLock
 
 import portage
 from portage import os
 from portage.exception import PortageException
-from portage.util.futures.compat_coroutine import coroutine
 from portage.util.SlotObject import SlotObject
 
 class EbuildBuildDir(SlotObject):
@@ -38,7 +39,6 @@ class EbuildBuildDir(SlotObject):
 			except OSError:
 				pass
 
-	@coroutine
 	def async_lock(self):
 		"""
 		Acquire the lock asynchronously. Notification is available
@@ -59,45 +59,60 @@ class EbuildBuildDir(SlotObject):
 			raise AssertionError('PORTAGE_BUILDDIR is unset')
 		catdir = os.path.dirname(dir_path)
 		self._catdir = catdir
-
-		try:
-			portage.util.ensure_dirs(os.path.dirname(catdir),
-				gid=portage.portage_gid,
-				mode=0o70, mask=0)
-		except PortageException:
-			if not os.path.isdir(os.path.dirname(catdir)):
-				raise
-
 		catdir_lock = AsynchronousLock(path=catdir, scheduler=self.scheduler)
-		yield catdir_lock.async_start()
-		yield catdir_lock.async_wait()
+		builddir_lock = AsynchronousLock(path=dir_path, scheduler=self.scheduler)
+		result = self.scheduler.create_future()
 
-		self._assert_lock(catdir_lock)
+		def catdir_locked(catdir_lock):
+			try:
+				self._assert_lock(catdir_lock)
+			except AssertionError as e:
+				result.set_exception(e)
+				return
+
+			try:
+				portage.util.ensure_dirs(catdir,
+					gid=portage.portage_gid,
+					mode=0o70, mask=0)
+			except PortageException as e:
+				if not os.path.isdir(catdir):
+					result.set_exception(e)
+					return
+
+			builddir_lock.addExitListener(builddir_locked)
+			builddir_lock.start()
+
+		def builddir_locked(builddir_lock):
+			try:
+				self._assert_lock(builddir_lock)
+			except AssertionError as e:
+				catdir_lock.async_unlock.add_done_callback(
+					functools.partial(catdir_unlocked, exception=e))
+				return
+
+			self._lock_obj = builddir_lock
+			self.locked = True
+			self.settings['PORTAGE_BUILDDIR_LOCKED'] = '1'
+			catdir_lock.async_unlock().add_done_callback(catdir_unlocked)
+
+		def catdir_unlocked(future, exception=None):
+			if not (exception is None and future.exception() is None):
+				result.set_exception(exception or future.exception())
+			else:
+				result.set_result(None)
 
 		try:
-			portage.util.ensure_dirs(catdir,
+			portage.util.ensure_dirs(os.path.dirname(catdir),
 				gid=portage.portage_gid,
 				mode=0o70, mask=0)
 		except PortageException:
-			if not os.path.isdir(catdir):
+			if not os.path.isdir(os.path.dirname(catdir)):
 				raise
 
-		builddir_lock = AsynchronousLock(path=dir_path, scheduler=self.scheduler)
-		yield builddir_lock.async_start()
-		yield builddir_lock.async_wait()
-
-		try:
-			self._assert_lock(builddir_lock)
-		except AssertionError:
-			yield catdir_lock.async_unlock()
-			raise
+		catdir_lock.addExitListener(catdir_locked)
+		catdir_lock.start()
+		return result
 
-		self._lock_obj = builddir_lock
-		self.locked = True
-		self.settings['PORTAGE_BUILDDIR_LOCKED'] = '1'
-		yield catdir_lock.async_unlock()
-
-	@coroutine
 	def async_unlock(self):
 		"""
 		Release the lock asynchronously. Release notification is available
@@ -105,22 +120,41 @@ class EbuildBuildDir(SlotObject):
 
 		@returns: Future, result is None
 		"""
-		if self._lock_obj is not None:
-			yield self._lock_obj.async_unlock()
-
-			self._lock_obj = None
-			self.locked = False
-			self.settings.pop('PORTAGE_BUILDDIR_LOCKED', None)
-			catdir_lock = AsynchronousLock(
-				path=self._catdir, scheduler=self.scheduler)
-			yield catdir_lock.async_start()
-			yield catdir_lock.async_wait()
-			if catdir_lock.returncode == os.EX_OK:
+		result = self.scheduler.create_future()
+
+		def builddir_unlocked(future):
+			if future.exception() is not None:
+				result.set_exception(future.exception())
+			else:
+				self._lock_obj = None
+				self.locked = False
+				self.settings.pop('PORTAGE_BUILDDIR_LOCKED', None)
+				catdir_lock = AsynchronousLock(
+					path=self._catdir, scheduler=self.scheduler)
+				catdir_lock.addExitListener(catdir_locked)
+				catdir_lock.start()
+
+		def catdir_locked(catdir_lock):
+			if catdir_lock.wait() != os.EX_OK:
+				result.set_result(None)
+			else:
 				try:
 					os.rmdir(self._catdir)
 				except OSError:
 					pass
-				yield catdir_lock.async_unlock()
+				catdir_lock.async_unlock().add_done_callback(catdir_unlocked)
+
+		def catdir_unlocked(future):
+			if future.exception() is None:
+				result.set_result(None)
+			else:
+				result.set_exception(future.exception())
+
+		if self._lock_obj is None:
+			self.scheduler.call_soon(result.set_result, None)
+		else:
+			self._lock_obj.async_unlock().add_done_callback(builddir_unlocked)
+		return result
 
 	class AlreadyLocked(portage.exception.PortageException):
 		pass

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     7e2918896d8dd84e07082bba734d3ce8429ba36d
Author:     Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Wed Apr  8 04:54:27 2020 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Wed Apr  8 05:29:47 2020 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=7e291889

Revert "SpawnProcess: make _main cancel pipe_logger when appropriate"

This reverts commit 6eb6500435c80837f106d3f9f9f6bb246ff00c16.

Bug: https://bugs.gentoo.org/716636
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/SpawnProcess.py | 9 ++-------
 1 file changed, 2 insertions(+), 7 deletions(-)

diff --git a/lib/_emerge/SpawnProcess.py b/lib/_emerge/SpawnProcess.py
index 941b56683..aef000e22 100644
--- a/lib/_emerge/SpawnProcess.py
+++ b/lib/_emerge/SpawnProcess.py
@@ -155,13 +155,8 @@ class SpawnProcess(SubProcess):
 
 	@coroutine
 	def _main(self, pipe_logger):
-		try:
-			if pipe_logger.poll() is None:
-				yield pipe_logger.async_wait()
-		except asyncio.CancelledError:
-			if pipe_logger.poll() is None:
-				pipe_logger.cancel()
-			raise
+		if pipe_logger.poll() is None:
+			yield pipe_logger.async_wait()
 
 	def _main_exit(self, main_task):
 		try:

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     f7e36d1e9d468b5a7f502e477ba1971f8334a5d9
Author:     Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Wed Apr  8 04:53:39 2020 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Wed Apr  8 05:29:47 2020 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=f7e36d1e

Revert "SpawnProcess: cancel pipe_logger after async_start CancelledError"

This reverts commit d331f03ab25422e9afc2e752ea8e39c93a4b0bec.

Bug: https://bugs.gentoo.org/716636
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/SpawnProcess.py | 7 +------
 1 file changed, 1 insertion(+), 6 deletions(-)

diff --git a/lib/_emerge/SpawnProcess.py b/lib/_emerge/SpawnProcess.py
index ab7971ca8..941b56683 100644
--- a/lib/_emerge/SpawnProcess.py
+++ b/lib/_emerge/SpawnProcess.py
@@ -147,12 +147,7 @@ class SpawnProcess(SubProcess):
 			log_file_path=log_file_path,
 			stdout_fd=stdout_fd)
 		self._registered = True
-		try:
-			yield pipe_logger.async_start()
-		except asyncio.CancelledError:
-			if pipe_logger.poll() is None:
-				pipe_logger.cancel()
-			raise
+		yield pipe_logger.async_start()
 
 		self._main_task = asyncio.ensure_future(
 			self._main(pipe_logger), loop=self.scheduler)

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     971124a4cf7b0a578b7c8b87134743f8135481b6
Author:     Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Wed Apr  8 04:55:13 2020 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Wed Apr  8 05:29:47 2020 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=971124a4

Revert "SpawnProcess: only cancel self._main_task if it's not done"

This reverts commit 10cc6febb701cde8f7c099f5ecdf177486e94b3c.

Bug: https://bugs.gentoo.org/716636
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/SpawnProcess.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/_emerge/SpawnProcess.py b/lib/_emerge/SpawnProcess.py
index aef000e22..cda615ded 100644
--- a/lib/_emerge/SpawnProcess.py
+++ b/lib/_emerge/SpawnProcess.py
@@ -196,7 +196,7 @@ class SpawnProcess(SubProcess):
 	def _cancel(self):
 		SubProcess._cancel(self)
 		if self._main_task is not None:
-			self._main_task.done() or self._main_task.cancel()
+			self._main_task.cancel()
 		self._cgroup_cleanup()
 
 	def _cgroup_cleanup(self):

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     62ee9ec33821768388cf16a3a1e7b603c845f0c9
Author:     Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Wed Apr  8 05:09:49 2020 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Wed Apr  8 05:29:49 2020 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=62ee9ec3

Revert "AbstractEbuildProcess: add _async_start coroutine"

This reverts commit a287c49f84ad3af7c8e20bebd116ea972f318e04.

Bug: https://bugs.gentoo.org/716636
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/AbstractEbuildProcess.py | 33 ++++++++++++++++++++++++---------
 lib/_emerge/MiscFunctionsProcess.py  |  8 +++-----
 2 files changed, 27 insertions(+), 14 deletions(-)

diff --git a/lib/_emerge/AbstractEbuildProcess.py b/lib/_emerge/AbstractEbuildProcess.py
index fd8a40cc1..1c1955cfe 100644
--- a/lib/_emerge/AbstractEbuildProcess.py
+++ b/lib/_emerge/AbstractEbuildProcess.py
@@ -1,4 +1,4 @@
-# Copyright 1999-2020 Gentoo Authors
+# Copyright 1999-2019 Gentoo Foundation
 # Distributed under the terms of the GNU General Public License v2
 
 import errno
@@ -19,7 +19,6 @@ from portage.package.ebuild._ipc.ExitCommand import ExitCommand
 from portage.package.ebuild._ipc.QueryCommand import QueryCommand
 from portage import shutil, os
 from portage.util.futures import asyncio
-from portage.util.futures.compat_coroutine import coroutine, coroutine_return
 from portage.util._pty import _create_pty_or_pipe
 from portage.util import apply_secpass_permissions
 
@@ -31,7 +30,7 @@ class AbstractEbuildProcess(SpawnProcess):
 
 	__slots__ = ('phase', 'settings',) + \
 		('_build_dir', '_build_dir_unlock', '_ipc_daemon',
-		'_exit_command', '_exit_timeout_id')
+		'_exit_command', '_exit_timeout_id', '_start_future')
 
 	_phases_without_builddir = ('clean', 'cleanrm', 'depend', 'help',)
 	_phases_interactive_whitelist = ('config',)
@@ -56,10 +55,6 @@ class AbstractEbuildProcess(SpawnProcess):
 			self.phase = phase
 
 	def _start(self):
-		self.scheduler.run_until_complete(self._async_start())
-
-	@coroutine
-	def _async_start(self):
 
 		need_builddir = self.phase not in self._phases_without_builddir
 
@@ -74,7 +69,7 @@ class AbstractEbuildProcess(SpawnProcess):
 			self._eerror(textwrap.wrap(msg, 72))
 			self.returncode = 1
 			self._async_wait()
-			coroutine_return()
+			return
 
 		# Check if the cgroup hierarchy is in place. If it's not, mount it.
 		if (os.geteuid() == 0 and platform.system() == 'Linux'
@@ -147,7 +142,11 @@ class AbstractEbuildProcess(SpawnProcess):
 				if 'PORTAGE_BUILDDIR_LOCKED' not in self.settings:
 					self._build_dir = EbuildBuildDir(
 						scheduler=self.scheduler, settings=self.settings)
-					yield self._build_dir.async_lock()
+					self._start_future = self._build_dir.async_lock()
+					self._start_future.add_done_callback(
+						functools.partial(self._start_post_builddir_lock,
+						start_ipc_daemon=start_ipc_daemon))
+					return
 			else:
 				self.settings.pop('PORTAGE_IPC_DAEMON', None)
 		else:
@@ -168,6 +167,22 @@ class AbstractEbuildProcess(SpawnProcess):
 			else:
 				self.settings.pop('PORTAGE_EBUILD_EXIT_FILE', None)
 
+		self._start_post_builddir_lock(start_ipc_daemon=start_ipc_daemon)
+
+	def _start_post_builddir_lock(self, lock_future=None, start_ipc_daemon=False):
+		if lock_future is not None:
+			if lock_future is not self._start_future:
+				raise AssertionError('lock_future is not self._start_future')
+			self._start_future = None
+			if lock_future.cancelled():
+				self._build_dir = None
+				self.cancelled = True
+				self._was_cancelled()
+				self._async_wait()
+				return
+
+			lock_future.result()
+
 		if start_ipc_daemon:
 			self.settings['PORTAGE_IPC_DAEMON'] = "1"
 			self._start_ipc_daemon()

diff --git a/lib/_emerge/MiscFunctionsProcess.py b/lib/_emerge/MiscFunctionsProcess.py
index cb0363820..89fd22635 100644
--- a/lib/_emerge/MiscFunctionsProcess.py
+++ b/lib/_emerge/MiscFunctionsProcess.py
@@ -1,9 +1,8 @@
-# Copyright 1999-2020 Gentoo Authors
+# Copyright 1999-2013 Gentoo Foundation
 # Distributed under the terms of the GNU General Public License v2
 
 from _emerge.AbstractEbuildProcess import AbstractEbuildProcess
 import portage
-from portage.util.futures.compat_coroutine import coroutine
 portage.proxy.lazyimport.lazyimport(globals(),
 	'portage.package.ebuild.doebuild:spawn'
 )
@@ -16,8 +15,7 @@ class MiscFunctionsProcess(AbstractEbuildProcess):
 
 	__slots__ = ('commands', 'ld_preload_sandbox')
 
-	@coroutine
-	def _async_start(self):
+	def _start(self):
 		settings = self.settings
 		portage_bin_path = settings["PORTAGE_BIN_PATH"]
 		misc_sh_binary = os.path.join(portage_bin_path,
@@ -28,7 +26,7 @@ class MiscFunctionsProcess(AbstractEbuildProcess):
 			self.settings.get("PORTAGE_BACKGROUND") != "subprocess":
 			self.logfile = settings.get("PORTAGE_LOG_FILE")
 
-		yield AbstractEbuildProcess._async_start(self)
+		AbstractEbuildProcess._start(self)
 
 	def _spawn(self, args, **kwargs):
 		# If self.ld_preload_sandbox is None, default to free=False,

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     389429d798a186bdbeb11354d7f1299f628851fd
Author:     Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Thu Apr  9 04:45:16 2020 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Thu Apr  9 06:01:27 2020 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=389429d7

Scheduler: wakeup for empty merge queue (bug 711322)

Add a wakeup callback to schedule a new merge when the merge queue
becomes empty. This prevents the scheduler from hanging in cases
where the order of _merge_exit callback invocation may cause the
the merge queue to appear non-empty when it is about to become
empty.

Bug: https://bugs.gentoo.org/711322
Bug: https://bugs.gentoo.org/716636
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/Scheduler.py           | 23 +++++++++++++++++++++++
 lib/_emerge/SequentialTaskQueue.py | 22 ++++++++++++----------
 2 files changed, 35 insertions(+), 10 deletions(-)

diff --git a/lib/_emerge/Scheduler.py b/lib/_emerge/Scheduler.py
index ee8f3dd5e..2c0483230 100644
--- a/lib/_emerge/Scheduler.py
+++ b/lib/_emerge/Scheduler.py
@@ -27,6 +27,7 @@ bad = create_color_func("BAD")
 from portage._sets import SETPREFIX
 from portage._sets.base import InternalPackageSet
 from portage.util import ensure_dirs, writemsg, writemsg_level
+from portage.util.futures import asyncio
 from portage.util.SlotObject import SlotObject
 from portage.util._async.SchedulerInterface import SchedulerInterface
 from portage.util._eventloop.EventLoop import EventLoop
@@ -241,6 +242,7 @@ class Scheduler(PollScheduler):
 		self._completed_tasks = set()
 		self._main_exit = None
 		self._main_loadavg_handle = None
+		self._schedule_merge_wakeup_task = None
 
 		self._failed_pkgs = []
 		self._failed_pkgs_all = []
@@ -1440,6 +1442,9 @@ class Scheduler(PollScheduler):
 		if self._job_delay_timeout_id is not None:
 			self._job_delay_timeout_id.cancel()
 			self._job_delay_timeout_id = None
+		if self._schedule_merge_wakeup_task is not None:
+			self._schedule_merge_wakeup_task.cancel()
+			self._schedule_merge_wakeup_task = None
 
 	def _choose_pkg(self):
 		"""
@@ -1614,6 +1619,24 @@ class Scheduler(PollScheduler):
 			self._main_loadavg_handle = self._event_loop.call_later(
 				self._loadavg_latency, self._schedule)
 
+		# Failure to schedule *after* self._task_queues.merge becomes
+		# empty will cause the scheduler to hang as in bug 711322.
+		# Do not rely on scheduling which occurs via the _merge_exit
+		# method, since the order of callback invocation may cause
+		# self._task_queues.merge to appear non-empty when it is
+		# about to become empty.
+		if (self._task_queues.merge and (self._schedule_merge_wakeup_task is None
+			or self._schedule_merge_wakeup_task.done())):
+			self._schedule_merge_wakeup_task = asyncio.ensure_future(
+				self._task_queues.merge.wait(), loop=self._event_loop)
+			self._schedule_merge_wakeup_task.add_done_callback(
+				self._schedule_merge_wakeup)
+
+	def _schedule_merge_wakeup(self, future):
+		if not future.cancelled():
+			future.result()
+			self._schedule()
+
 	def _sigcont_handler(self, signum, frame):
 		self._sigcont_time = time.time()
 

diff --git a/lib/_emerge/SequentialTaskQueue.py b/lib/_emerge/SequentialTaskQueue.py
index 656e5cf7c..d2551b1c6 100644
--- a/lib/_emerge/SequentialTaskQueue.py
+++ b/lib/_emerge/SequentialTaskQueue.py
@@ -1,9 +1,11 @@
-# Copyright 1999-2012 Gentoo Foundation
+# Copyright 1999-2020 Gentoo Authors
 # Distributed under the terms of the GNU General Public License v2
 
 from collections import deque
 import sys
 
+from portage.util.futures import asyncio
+from portage.util.futures.compat_coroutine import coroutine
 from portage.util.SlotObject import SlotObject
 
 class SequentialTaskQueue(SlotObject):
@@ -41,12 +43,6 @@ class SequentialTaskQueue(SlotObject):
 				cancelled = getattr(task, "cancelled", None)
 				if not cancelled:
 					self.running_tasks.add(task)
-					# This callback will be invoked as soon as the task
-					# exits (before the future's done callback is called),
-					# and this is required in order for bool(self) to have
-					# an updated value for Scheduler._schedule to base
-					# assumptions upon. Delayed updates to bool(self) is
-					# what caused Scheduler to hang as in bug 711322.
 					task.addExitListener(self._task_exit)
 					task.start()
 		finally:
@@ -73,12 +69,18 @@ class SequentialTaskQueue(SlotObject):
 		for task in list(self.running_tasks):
 			task.cancel()
 
+	@coroutine
 	def wait(self):
 		"""
-		Synchronously wait for all running tasks to exit.
+		Wait for the queue to become empty. This method is a coroutine.
 		"""
-		while self.running_tasks:
-			next(iter(self.running_tasks)).wait()
+		while self:
+			task = next(iter(self.running_tasks), None)
+			if task is None:
+				# Wait for self.running_tasks to populate.
+				yield asyncio.sleep(0)
+			else:
+				yield task.async_wait()
 
 	def __bool__(self):
 		return bool(self._task_queue or self.running_tasks)

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     ad325eb10bc6e8ec2a8248f8e9173911f957c0da
Author:     Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Thu Apr  9 20:27:36 2020 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Thu Apr  9 20:47:08 2020 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=ad325eb1

_schedule_merge_wakeup: handle main loop exit

Detect main loop exit and do not attempt to schedule in this case.

Fixes: 389429d798a1 ("Scheduler: wakeup for empty merge queue (bug 711322)")
Reported-by: Rick Farina <zerochaos <AT> gentoo.org>
Bug: https://bugs.gentoo.org/711322
Bug: https://bugs.gentoo.org/716636
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/Scheduler.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/lib/_emerge/Scheduler.py b/lib/_emerge/Scheduler.py
index 2c0483230..6f4564000 100644
--- a/lib/_emerge/Scheduler.py
+++ b/lib/_emerge/Scheduler.py
@@ -1635,7 +1635,8 @@ class Scheduler(PollScheduler):
 	def _schedule_merge_wakeup(self, future):
 		if not future.cancelled():
 			future.result()
-			self._schedule()
+			if self._main_exit is not None and not self._main_exit.done():
+				self._schedule()
 
 	def _sigcont_handler(self, signum, frame):
 		self._sigcont_time = time.time()

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     0418f10fdfc25c27d0b15ebca52a0bbe8119b55b
Author:     Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Tue Jun 23 17:55:53 2020 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Tue Jun 23 18:00:24 2020 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=0418f10f

EbuildPhase._elog: _set_nonblocking(build_logger.stdin.fileno())

Fixes: dd69ce742c62 ("Support PORTAGE_LOG_FILTER_FILE_CMD (bug 709746)")
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/EbuildPhase.py | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/lib/_emerge/EbuildPhase.py b/lib/_emerge/EbuildPhase.py
index ddb3dc719..92335b6c0 100644
--- a/lib/_emerge/EbuildPhase.py
+++ b/lib/_emerge/EbuildPhase.py
@@ -47,7 +47,8 @@ portage.proxy.lazyimport.lazyimport(globals(),
 		'_post_src_install_soname_symlinks,' + \
 		'_post_src_install_uid_fix,_postinst_bsdflags,' + \
 		'_post_src_install_write_metadata,' + \
-		'_preinst_bsdflags'
+		'_preinst_bsdflags',
+	'portage.util.futures.unix_events:_set_nonblocking',
 )
 from portage import os
 from portage import _encodings
@@ -433,6 +434,7 @@ class EbuildPhase(CompositeTask):
 						log_filter_file=self.settings.get('PORTAGE_LOG_FILTER_FILE_CMD'),
 						scheduler=self.scheduler)
 					build_logger.start()
+					_set_nonblocking(build_logger.stdin.fileno())
 					log_file = build_logger.stdin
 
 				yield self.scheduler.async_output(msg, log_file=log_file,

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     ac9184e0b0445a9294ae47f71abe7117799a4577
Author:     Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Wed Jun 24 05:11:53 2020 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Wed Jun 24 05:30:54 2020 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=ac9184e0

SpawnProcess._async_wait: allow _main_task to exit normally

Unnecessary cancellation of _main_task triggered this ResourceWarning
for the ebuild(1) fetch command, since it prevented cleanup of
BuildLogger's internal PipeLogger instance:

sys:1: ResourceWarning: unclosed file <_io.FileIO name=10 mode='rb' closefd=True>

Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/SpawnProcess.py | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/lib/_emerge/SpawnProcess.py b/lib/_emerge/SpawnProcess.py
index f96911571..60239a65a 100644
--- a/lib/_emerge/SpawnProcess.py
+++ b/lib/_emerge/SpawnProcess.py
@@ -172,12 +172,23 @@ class SpawnProcess(SubProcess):
 			raise
 
 	def _main_exit(self, main_task):
+		self._main_task = None
 		try:
 			main_task.result()
 		except asyncio.CancelledError:
 			self.cancel()
 		self._async_waitpid()
 
+	def _async_wait(self):
+		# Allow _main_task to exit normally rather than via cancellation.
+		if self._main_task is None:
+			super(SpawnProcess, self)._async_wait()
+
+	def _async_waitpid(self):
+		# Allow _main_task to exit normally rather than via cancellation.
+		if self._main_task is None:
+			super(SpawnProcess, self)._async_waitpid()
+
 	def _can_log(self, slave_fd):
 		return True
 

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     f0a96005d3da49fa52624d7c3e4b9d1d12b831ad
Author:     Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Sat Jul 18 23:51:12 2020 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Sat Jul 18 23:52:45 2020 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=f0a96005

SubProcess._async_waitpid: document override usage

Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/SubProcess.py | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)

diff --git a/lib/_emerge/SubProcess.py b/lib/_emerge/SubProcess.py
index e834cb7d3..f7353926c 100644
--- a/lib/_emerge/SubProcess.py
+++ b/lib/_emerge/SubProcess.py
@@ -48,9 +48,12 @@ class SubProcess(AbstractPollTask):
 	def _async_waitpid(self):
 		"""
 		Wait for exit status of self.pid asynchronously, and then
-		set the returncode and notify exit listeners. This is
-		prefered over _waitpid_loop, since the synchronous nature
-		of _waitpid_loop can cause event loop recursion.
+		set the returncode, and finally notify exit listeners via the
+		_async_wait method. Subclasses may override this method in order
+		to implement an alternative means to retrieve pid exit status,
+		or as a means to delay action until some pending task(s) have
+		completed (such as reading data that the subprocess is supposed
+		to have written to a pipe).
 		"""
 		if self.returncode is not None:
 			self._async_wait()

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     c14ac733a4e05990973d99e13f19aaf9bde57bcb
Author:     Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Wed Jul 22 20:16:42 2020 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Wed Jul 22 20:20:11 2020 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=c14ac733

_EbuildFetcherProcess: emit eerror for fetch failure in  _proc_join_done

Since ForkProcess now receives process exit status in the
_proc_join_done method instead of the _async_waitpid_cb method,
_EbuildFetcherProcess needs to emit eerror messages there
instead.

Fixes: bde44b75407d ("ForkProcess: replace os.fork with multiprocessing.Process (bug 730192)")
Bug: https://bugs.gentoo.org/730192
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/EbuildFetcher.py | 12 ++++--------
 1 file changed, 4 insertions(+), 8 deletions(-)

diff --git a/lib/_emerge/EbuildFetcher.py b/lib/_emerge/EbuildFetcher.py
index 55349c33c..107a6d590 100644
--- a/lib/_emerge/EbuildFetcher.py
+++ b/lib/_emerge/EbuildFetcher.py
@@ -364,16 +364,11 @@ class _EbuildFetcherProcess(ForkProcess):
 		if msg:
 			self.scheduler.output(msg, log_path=self.logfile)
 
-	def _async_waitpid_cb(self, *args, **kwargs):
+	def _proc_join_done(self, proc, future):
 		"""
-		Override _async_waitpid_cb to perform cleanup that is
-		not necessarily idempotent.
+		Extend _proc_join_done to emit an eerror message for fetch failure.
 		"""
-		ForkProcess._async_waitpid_cb(self, *args, **kwargs)
-		# Collect elog messages that might have been
-		# created by the pkg_nofetch phase.
-		# Skip elog messages for prefetch, in order to avoid duplicates.
-		if not self.prefetch and self.returncode != os.EX_OK:
+		if not self.prefetch and not future.cancelled() and proc.exitcode != os.EX_OK:
 			msg_lines = []
 			msg = "Fetch failed for '%s'" % (self.pkg.cpv,)
 			if self.logfile is not None:
@@ -382,3 +377,4 @@ class _EbuildFetcherProcess(ForkProcess):
 			if self.logfile is not None:
 				msg_lines.append(" '%s'" % (self.logfile,))
 			self._eerror(msg_lines)
+		super(_EbuildFetcherProcess, self)._proc_join_done(proc, future)

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     15b039973c37dc99eae96cdd142d55203ffb18dd
Author:     Aaron Bauman <bman <AT> gentoo <DOT> org>
AuthorDate: Mon Aug  3 19:05:43 2020 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Mon Aug  3 19:21:22 2020 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=15b03997

lib/_emerge/DependencyArg.py: fix unused-import

Signed-off-by: Aaron Bauman <bman <AT> gentoo.org>
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/DependencyArg.py | 3 ---
 1 file changed, 3 deletions(-)

diff --git a/lib/_emerge/DependencyArg.py b/lib/_emerge/DependencyArg.py
index f7ac879ec..1b66fbc2a 100644
--- a/lib/_emerge/DependencyArg.py
+++ b/lib/_emerge/DependencyArg.py
@@ -1,9 +1,6 @@
 # Copyright 1999-2020 Gentoo Authors
 # Distributed under the terms of the GNU General Public License v2
 
-
-from portage import _encodings, _unicode_encode
-
 class DependencyArg:
 
 	__slots__ = ('arg', 'force_reinstall', 'internal', 'reset_depth', 'root_config')

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     9abc11ac02de3bea492e40a8f26e7d8a002e3f55
Author:     Aaron Bauman <bman <AT> gentoo <DOT> org>
AuthorDate: Mon Aug  3 19:05:42 2020 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Mon Aug  3 19:21:18 2020 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=9abc11ac

lib/_emerge/Scheduler.py: fix unused-import

Signed-off-by: Aaron Bauman <bman <AT> gentoo.org>
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/Scheduler.py | 2 --
 1 file changed, 2 deletions(-)

diff --git a/lib/_emerge/Scheduler.py b/lib/_emerge/Scheduler.py
index 7906c30c8..761ba31e0 100644
--- a/lib/_emerge/Scheduler.py
+++ b/lib/_emerge/Scheduler.py
@@ -21,7 +21,6 @@ from portage import _encodings
 from portage import _unicode_encode
 from portage.cache.mappings import slot_dict_class
 from portage.elog.messages import eerror
-from portage.localization import _
 from portage.output import colorize, create_color_func, red
 bad = create_color_func("BAD")
 from portage._sets import SETPREFIX
@@ -30,7 +29,6 @@ from portage.util import ensure_dirs, writemsg, writemsg_level
 from portage.util.futures import asyncio
 from portage.util.SlotObject import SlotObject
 from portage.util._async.SchedulerInterface import SchedulerInterface
-from portage.util._eventloop.EventLoop import EventLoop
 from portage.package.ebuild.digestcheck import digestcheck
 from portage.package.ebuild.digestgen import digestgen
 from portage.package.ebuild.doebuild import (_check_temp_dir,

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     dfd7f88609765812911815fdd655baca2e8b293e
Author:     Aaron Bauman <bman <AT> gentoo <DOT> org>
AuthorDate: Mon Aug  3 19:05:47 2020 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Mon Aug  3 19:21:39 2020 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=dfd7f886

lib/_emerge/Package.py: drop unused-import

Signed-off-by: Aaron Bauman <bman <AT> gentoo.org>
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/Package.py | 1 -
 1 file changed, 1 deletion(-)

diff --git a/lib/_emerge/Package.py b/lib/_emerge/Package.py
index 00d250125..995af6311 100644
--- a/lib/_emerge/Package.py
+++ b/lib/_emerge/Package.py
@@ -5,7 +5,6 @@ from itertools import chain
 import warnings
 
 import portage
-from portage import _encodings, _unicode_decode, _unicode_encode
 from portage.cache.mappings import slot_dict_class
 from portage.const import EBUILD_PHASES
 from portage.dep import Atom, check_required_use, use_reduce, \

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     1e67bcfff2c2c10e200fca496bf1fb1ffd510892
Author:     Aaron Bauman <bman <AT> gentoo <DOT> org>
AuthorDate: Mon Aug  3 19:05:48 2020 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Mon Aug  3 19:21:42 2020 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=1e67bcff

lib/_emerge/AbstractPollTask.py: drop unused-import

* Drop unused-import
* Update copyright

Signed-off-by: Aaron Bauman <bman <AT> gentoo.org>
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/AbstractPollTask.py | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/lib/_emerge/AbstractPollTask.py b/lib/_emerge/AbstractPollTask.py
index 661b81616..5ad24bb84 100644
--- a/lib/_emerge/AbstractPollTask.py
+++ b/lib/_emerge/AbstractPollTask.py
@@ -1,11 +1,10 @@
-# Copyright 1999-2018 Gentoo Foundation
+# Copyright 1999-2020 Gentoo Authors
 # Distributed under the terms of the GNU General Public License v2
 
 import array
 import errno
 import os
 
-from portage.util import writemsg_level
 from portage.util.futures import asyncio
 from _emerge.AsynchronousTask import AsynchronousTask
 

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     e26fad15bd71a22e46f091508628a5cbccb73472
Author:     Aaron Bauman <bman <AT> gentoo <DOT> org>
AuthorDate: Mon Aug  3 19:06:03 2020 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Mon Aug  3 19:22:39 2020 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=e26fad15

lib/_emerge/AbstractEbuildProcess.py: drop unused-import

Signed-off-by: Aaron Bauman <bman <AT> gentoo.org>
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/AbstractEbuildProcess.py | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/lib/_emerge/AbstractEbuildProcess.py b/lib/_emerge/AbstractEbuildProcess.py
index ae1aae55f..dc7426f2b 100644
--- a/lib/_emerge/AbstractEbuildProcess.py
+++ b/lib/_emerge/AbstractEbuildProcess.py
@@ -1,7 +1,6 @@
 # Copyright 1999-2020 Gentoo Authors
 # Distributed under the terms of the GNU General Public License v2
 
-import errno
 import functools
 import io
 import platform
@@ -17,7 +16,7 @@ from portage.elog import messages as elog_messages
 from portage.localization import _
 from portage.package.ebuild._ipc.ExitCommand import ExitCommand
 from portage.package.ebuild._ipc.QueryCommand import QueryCommand
-from portage import shutil, os
+from portage import os
 from portage.util.futures import asyncio
 from portage.util._pty import _create_pty_or_pipe
 from portage.util import apply_secpass_permissions

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     353d189616e6dd016f835259950f78fa5d60adca
Author:     Aaron Bauman <bman <AT> gentoo <DOT> org>
AuthorDate: Mon Aug  3 19:05:58 2020 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Mon Aug  3 19:22:20 2020 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=353d1896

lib/_emerge/UseFlagDisplay.py: drop unused-import

Signed-off-by: Aaron Bauman <bman <AT> gentoo.org>
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/UseFlagDisplay.py | 1 -
 1 file changed, 1 deletion(-)

diff --git a/lib/_emerge/UseFlagDisplay.py b/lib/_emerge/UseFlagDisplay.py
index 9496693c8..5e3ba400d 100644
--- a/lib/_emerge/UseFlagDisplay.py
+++ b/lib/_emerge/UseFlagDisplay.py
@@ -4,7 +4,6 @@
 import collections
 from itertools import chain
 
-from portage import _encodings, _unicode_encode
 from portage.output import red
 from portage.util import cmp_sort_key
 from portage.output import blue

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     d1bbd61290d372868d6ef9047958ef717ca2a711
Author:     Aaron Bauman <bman <AT> gentoo <DOT> org>
AuthorDate: Mon Aug  3 19:05:57 2020 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Mon Aug  3 19:22:16 2020 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=d1bbd612

lib/_emerge/depgraph.py: drop unused-import

Signed-off-by: Aaron Bauman <bman <AT> gentoo.org>
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/depgraph.py | 2 --
 1 file changed, 2 deletions(-)

diff --git a/lib/_emerge/depgraph.py b/lib/_emerge/depgraph.py
index 04b824ab9..0810cdaf1 100644
--- a/lib/_emerge/depgraph.py
+++ b/lib/_emerge/depgraph.py
@@ -48,8 +48,6 @@ from portage.util import writemsg_level, write_atomic
 from portage.util.digraph import digraph
 from portage.util.futures import asyncio
 from portage.util._async.TaskScheduler import TaskScheduler
-from portage.util._eventloop.EventLoop import EventLoop
-from portage.util._eventloop.global_event_loop import global_event_loop
 from portage.versions import _pkg_str, catpkgsplit
 
 from _emerge.AtomArg import AtomArg

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     bab5acb1b85420089c655b5c7e29429ad8a0bfc0
Author:     Aaron Bauman <bman <AT> gentoo <DOT> org>
AuthorDate: Mon Aug  3 20:20:16 2020 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Mon Aug  3 21:14:49 2020 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=bab5acb1

lib/_emerge/actions.py: drop unused-import

Signed-off-by: Aaron Bauman <bman <AT> gentoo.org>
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/actions.py | 16 ++++------------
 1 file changed, 4 insertions(+), 12 deletions(-)

diff --git a/lib/_emerge/actions.py b/lib/_emerge/actions.py
index ab614d641..43e9de43a 100644
--- a/lib/_emerge/actions.py
+++ b/lib/_emerge/actions.py
@@ -4,7 +4,6 @@
 from __future__ import division, print_function
 
 import collections
-import errno
 import logging
 import operator
 import platform
@@ -34,24 +33,20 @@ portage.proxy.lazyimport.lazyimport(globals(),
 
 from portage import os
 from portage import shutil
-from portage import eapi_is_supported, _encodings, _unicode_decode
-from portage.cache.cache_errors import CacheError
-from portage.const import GLOBAL_CONFIG_PATH, VCS_DIRS, _DEPCLEAN_LIB_CHECK_DEFAULT
-from portage.const import SUPPORTED_BINPKG_FORMATS, TIMESTAMP_FORMAT
+from portage import _encodings, _unicode_decode
+from portage.const import _DEPCLEAN_LIB_CHECK_DEFAULT
 from portage.dbapi.dep_expand import dep_expand
 from portage.dbapi._expand_new_virt import expand_new_virt
 from portage.dbapi.IndexedPortdb import IndexedPortdb
 from portage.dbapi.IndexedVardb import IndexedVardb
 from portage.dep import Atom, _repo_separator, _slot_separator
-from portage.eclass_cache import hashed_path
 from portage.exception import InvalidAtom, InvalidData, ParseError
-from portage.output import blue, colorize, create_color_func, darkgreen, \
-	red, xtermTitle, xtermTitleReset, yellow
+from portage.output import colorize, create_color_func, darkgreen, \
+	red, xtermTitle, xtermTitleReset
 good = create_color_func("GOOD")
 bad = create_color_func("BAD")
 warn = create_color_func("WARN")
 from portage.package.ebuild._ipc.QueryCommand import QueryCommand
-from portage.package.ebuild.doebuild import _check_temp_dir
 from portage.package.ebuild.fetch import _hide_url_passwd
 from portage._sets import load_default_config, SETPREFIX
 from portage._sets.base import InternalPackageSet
@@ -69,17 +64,14 @@ from portage.metadata import action_metadata
 from portage.emaint.main import print_results
 
 from _emerge.clear_caches import clear_caches
-from _emerge.countdown import countdown
 from _emerge.create_depgraph_params import create_depgraph_params
 from _emerge.Dependency import Dependency
 from _emerge.depgraph import backtrack_depgraph, depgraph, resume_depgraph
-from _emerge.DepPrioritySatisfiedRange import DepPrioritySatisfiedRange
 from _emerge.emergelog import emergelog
 from _emerge.is_valid_package_atom import is_valid_package_atom
 from _emerge.main import profile_check
 from _emerge.MetadataRegen import MetadataRegen
 from _emerge.Package import Package
-from _emerge.ProgressHandler import ProgressHandler
 from _emerge.RootConfig import RootConfig
 from _emerge.Scheduler import Scheduler
 from _emerge.search import search

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     f23ee4d4f7edea6cea81b4462c5d18b2a1cd19e5
Author:     Aaron Bauman <bman <AT> gentoo <DOT> org>
AuthorDate: Mon Aug  3 20:20:23 2020 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Mon Aug  3 21:25:52 2020 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=f23ee4d4

lib/_emerge/EbuildBuild.py: drop unused-import

* Drop unused-import
* Update copyright

Signed-off-by: Aaron Bauman <bman <AT> gentoo.org>
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/EbuildBuild.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/lib/_emerge/EbuildBuild.py b/lib/_emerge/EbuildBuild.py
index 4da815988..aeb8a252b 100644
--- a/lib/_emerge/EbuildBuild.py
+++ b/lib/_emerge/EbuildBuild.py
@@ -1,4 +1,4 @@
-# Copyright 1999-2018 Gentoo Foundation
+# Copyright 1999-2020 Gentoo Authors
 # Distributed under the terms of the GNU General Public License v2
 
 import functools
@@ -18,7 +18,7 @@ from _emerge.MiscFunctionsProcess import MiscFunctionsProcess
 from _emerge.TaskSequence import TaskSequence
 
 import portage
-from portage import _encodings, _unicode_decode, _unicode_encode, os
+from portage import _encodings, _unicode_encode, os
 from portage.package.ebuild.digestcheck import digestcheck
 from portage.package.ebuild.doebuild import _check_temp_dir
 from portage.package.ebuild._spawn_nofetch import SpawnNofetchWithoutBuilddir

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     639b415ff50c07a89cbf10ac0a2cff250f079cc8
Author:     Aaron Bauman <bman <AT> gentoo <DOT> org>
AuthorDate: Mon Aug  3 20:20:20 2020 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Mon Aug  3 21:15:47 2020 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=639b415f

lib/_emerge/PollScheduler.py: drop unused-import

* Drop unused-import
* Update copyright

Signed-off-by: Aaron Bauman <bman <AT> gentoo.org>
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/PollScheduler.py | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)

diff --git a/lib/_emerge/PollScheduler.py b/lib/_emerge/PollScheduler.py
index f51c5b005..89d940fed 100644
--- a/lib/_emerge/PollScheduler.py
+++ b/lib/_emerge/PollScheduler.py
@@ -1,4 +1,4 @@
-# Copyright 1999-2019 Gentoo Authors
+# Copyright 1999-2020 Gentoo Authors
 # Distributed under the terms of the GNU General Public License v2
 
 try:
@@ -6,10 +6,8 @@ try:
 except ImportError:
 	import dummy_threading as threading
 
-import portage
 from portage.util.futures import asyncio
 from portage.util._async.SchedulerInterface import SchedulerInterface
-from portage.util._eventloop.EventLoop import EventLoop
 from portage.util._eventloop.global_event_loop import global_event_loop
 
 from _emerge.getloadavg import getloadavg

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     67e680336e5acb2309dec166a6cd91ea5a562f89
Author:     Aaron Bauman <bman <AT> gentoo <DOT> org>
AuthorDate: Mon Aug  3 22:43:19 2020 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Mon Aug  3 23:28:03 2020 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=67e68033

lib/_emerge/BinpkgEnvExtractor.py: fix whitespace

Signed-off-by: Aaron Bauman <bman <AT> gentoo.org>
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/BinpkgEnvExtractor.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/lib/_emerge/BinpkgEnvExtractor.py b/lib/_emerge/BinpkgEnvExtractor.py
index 5ba14955d..b19586e83 100644
--- a/lib/_emerge/BinpkgEnvExtractor.py
+++ b/lib/_emerge/BinpkgEnvExtractor.py
@@ -1,4 +1,4 @@
-# Copyright 1999-2011 Gentoo Foundation
+# Copyright 1999-2020 Gentoo Authors
 # Distributed under the terms of the GNU General Public License v2
 
 import errno
@@ -36,7 +36,7 @@ class BinpkgEnvExtractor(CompositeTask):
 		extractor_proc = SpawnProcess(
 			args=[BASH_BINARY, "-c", shell_cmd],
 			background=self.background,
-			env=self.settings.environ(), 
+			env=self.settings.environ(),
 			scheduler=self.scheduler,
 			logfile=self.settings.get('PORTAGE_LOG_FILE'))
 

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     2550eecf9d712ae60e1ce91ed61a08d6d7e58d04
Author:     Aaron Bauman <bman <AT> gentoo <DOT> org>
AuthorDate: Mon Aug  3 22:43:23 2020 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Mon Aug  3 23:28:04 2020 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=2550eecf

lib/_emerge/search.py: fix whitespace

Signed-off-by: Aaron Bauman <bman <AT> gentoo.org>
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/search.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/lib/_emerge/search.py b/lib/_emerge/search.py
index 000c427e0..a4ad74311 100644
--- a/lib/_emerge/search.py
+++ b/lib/_emerge/search.py
@@ -1,4 +1,4 @@
-# Copyright 1999-2016 Gentoo Foundation
+# Copyright 1999-2020 Gentoo Authors
 # Distributed under the terms of the GNU General Public License v2
 
 import difflib
@@ -333,7 +333,7 @@ class search:
 				match_string = setname
 			else:
 				match_string = setname.split("/")[-1]
-			
+
 			if self.searchre.search(match_string):
 				yield ("set", setname)
 			elif self.searchdesc:

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     a4922d0607c8f3799bdcf70909b451d4d11a3dfa
Author:     Aaron Bauman <bman <AT> gentoo <DOT> org>
AuthorDate: Tue Aug  4 02:44:57 2020 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Tue Aug  4 03:10:32 2020 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=a4922d06

lib/_emerge/search.py: fix reimported

* The bold module is imported twice. Once as 'bold' and again as 'white.
  Remove the duplication and fix the one reference to 'white'

Signed-off-by: Aaron Bauman <bman <AT> gentoo.org>
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/search.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/lib/_emerge/search.py b/lib/_emerge/search.py
index a4ad74311..a59191c1a 100644
--- a/lib/_emerge/search.py
+++ b/lib/_emerge/search.py
@@ -9,7 +9,7 @@ from portage.dbapi.porttree import _parse_uri_map
 from portage.dbapi.IndexedPortdb import IndexedPortdb
 from portage.dbapi.IndexedVardb import IndexedVardb
 from portage.localization import localized_size
-from portage.output import  bold, bold as white, darkgreen, green, red
+from portage.output import bold, darkgreen, green, red
 from portage.util import writemsg_stdout
 from portage.util.iterators.MultiIterGroupBy import MultiIterGroupBy
 
@@ -413,7 +413,7 @@ class search:
 
 					if masked:
 						msg.append(green("*") + "  " + \
-							white(match) + " " + red("[ Masked ]") + "\n")
+							bold(match) + " " + red("[ Masked ]") + "\n")
 					else:
 						msg.append(green("*") + "  " + bold(match) + "\n")
 					myversion = self.getVersion(full_package, search.VERSION_RELEASE)

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     0b4b5eae5555fafcfa314465ae1094da00687779
Author:     Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Sat Aug  8 22:16:34 2020 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Sat Aug  8 23:25:03 2020 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=0b4b5eae

BinpkgEnvExtractor: fix subprocess logfile usage (bug 711174)

Do not write to log file when in a MergeProcess subprocess,
since stdout and stderr are already redirected to the log
file by MergeProcess. This fixes log corruption when
BinpkgEnvExtractor is use to extract environment.bz2 prior
to pkg_prerm.

Bug: https://bugs.gentoo.org/711174
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/BinpkgEnvExtractor.py | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/lib/_emerge/BinpkgEnvExtractor.py b/lib/_emerge/BinpkgEnvExtractor.py
index b19586e83..9398b493c 100644
--- a/lib/_emerge/BinpkgEnvExtractor.py
+++ b/lib/_emerge/BinpkgEnvExtractor.py
@@ -33,12 +33,17 @@ class BinpkgEnvExtractor(CompositeTask):
 		shell_cmd = "${PORTAGE_BUNZIP2_COMMAND:-${PORTAGE_BZIP2_COMMAND} -d} -c -- %s > %s" % \
 			(_shell_quote(saved_env_path),
 			_shell_quote(dest_env_path))
+
+		logfile = None
+		if self.settings.get("PORTAGE_BACKGROUND") != "subprocess":
+			logfile = self.settings.get("PORTAGE_LOG_FILE")
+
 		extractor_proc = SpawnProcess(
 			args=[BASH_BINARY, "-c", shell_cmd],
 			background=self.background,
 			env=self.settings.environ(),
 			scheduler=self.scheduler,
-			logfile=self.settings.get('PORTAGE_LOG_FILE'))
+			logfile=logfile)
 
 		self._start_task(extractor_proc, self._extractor_exit)
 

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     a163adec05afba127dfb18529761557bd30655ae
Author:     Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Sun Aug  9 07:38:18 2020 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Sun Aug  9 07:39:10 2020 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=a163adec

SpawnProcess: fix _cancel to cleanup PipeLogger quickly

Cleanup PipeLogger as quickly as possible, in order to prevent
access to unclosed logs.

Bug: https://bugs.gentoo.org/711174
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/SpawnProcess.py | 22 ++++++++++++++++------
 1 file changed, 16 insertions(+), 6 deletions(-)

diff --git a/lib/_emerge/SpawnProcess.py b/lib/_emerge/SpawnProcess.py
index 91f5ed1a8..cea16df27 100644
--- a/lib/_emerge/SpawnProcess.py
+++ b/lib/_emerge/SpawnProcess.py
@@ -2,6 +2,7 @@
 # Distributed under the terms of the GNU General Public License v2
 
 import errno
+import functools
 import logging
 import signal
 import sys
@@ -32,7 +33,7 @@ class SpawnProcess(SubProcess):
 		"unshare_ipc", "unshare_mount", "unshare_pid", "unshare_net")
 
 	__slots__ = ("args", "log_filter_file") + \
-		_spawn_kwarg_names + ("_main_task", "_selinux_type",)
+		_spawn_kwarg_names + ("_main_task", "_main_task_cancel", "_selinux_type",)
 
 	# Max number of attempts to kill the processes listed in cgroup.procs,
 	# given that processes may fork before they can be killed.
@@ -138,6 +139,7 @@ class SpawnProcess(SubProcess):
 		pipe_logger.start()
 
 		self._registered = True
+		self._main_task_cancel = functools.partial(self._main_cancel, build_logger, pipe_logger)
 		self._main_task = asyncio.ensure_future(self._main(build_logger, pipe_logger), loop=self.scheduler)
 		self._main_task.add_done_callback(self._main_exit)
 
@@ -149,14 +151,18 @@ class SpawnProcess(SubProcess):
 			if build_logger.poll() is None:
 				yield build_logger.async_wait()
 		except asyncio.CancelledError:
-			if pipe_logger.poll() is None:
-				pipe_logger.cancel()
-			if build_logger.poll() is None:
-				build_logger.cancel()
+			self._main_cancel(build_logger, pipe_logger)
 			raise
 
+	def _main_cancel(self, build_logger, pipe_logger):
+		if pipe_logger.poll() is None:
+			pipe_logger.cancel()
+		if build_logger.poll() is None:
+			build_logger.cancel()
+
 	def _main_exit(self, main_task):
 		self._main_task = None
+		self._main_task_cancel = None
 		try:
 			main_task.result()
 		except asyncio.CancelledError:
@@ -205,7 +211,11 @@ class SpawnProcess(SubProcess):
 
 	def _cancel(self):
 		if self._main_task is not None:
-			self._main_task.done() or self._main_task.cancel()
+			if not self._main_task.done():
+				if self._main_task_cancel is not None:
+					self._main_task_cancel()
+					self._main_task_cancel = None
+				self._main_task.cancel()
 		SubProcess._cancel(self)
 		self._cgroup_cleanup()
 

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     c7fa3f1eb1ce1ebc0d1219dacba555e1a29d5f22
Author:     Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Sun Sep 20 00:32:57 2020 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Mon Sep 21 05:17:24 2020 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=c7fa3f1e

emerge: enable parallel-fetch during pkg_pretend (bug 710432)

Execute pkg_pretend phases in a coroutine while parallel-fetch
is running concurrently. When it's time to execute the pkg_pretend
phase for a remote binary package, use a Scheduler _get_prefetcher
method to get a running prefetcher if available, and otherwise
start a new fetcher.

Since pkg_pretend phases now run inside of the --keep-going retry
loop, --keep-going is now able to recover from pkg_pretend
failures, which fixes bug 404157.

Bug: https://bugs.gentoo.org/404157
Bug: https://bugs.gentoo.org/710432
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/Scheduler.py | 142 +++++++++++++++++++++++++++++++++--------------
 1 file changed, 99 insertions(+), 43 deletions(-)

diff --git a/lib/_emerge/Scheduler.py b/lib/_emerge/Scheduler.py
index a69421288..465f928a0 100644
--- a/lib/_emerge/Scheduler.py
+++ b/lib/_emerge/Scheduler.py
@@ -25,6 +25,7 @@ from portage._sets import SETPREFIX
 from portage._sets.base import InternalPackageSet
 from portage.util import ensure_dirs, writemsg, writemsg_level
 from portage.util.futures import asyncio
+from portage.util.futures.compat_coroutine import coroutine, coroutine_return
 from portage.util.SlotObject import SlotObject
 from portage.util._async.SchedulerInterface import SchedulerInterface
 from portage.package.ebuild.digestcheck import digestcheck
@@ -766,7 +767,8 @@ class Scheduler(PollScheduler):
 
 		return prefetcher
 
-	def _run_pkg_pretend(self):
+	@coroutine
+	def _run_pkg_pretend(self, loop=None):
 		"""
 		Since pkg_pretend output may be important, this method sends all
 		output directly to stdout (regardless of options like --quiet or
@@ -774,7 +776,7 @@ class Scheduler(PollScheduler):
 		"""
 
 		failures = 0
-		sched_iface = self._sched_iface
+		sched_iface = loop = asyncio._wrap_loop(loop or self._sched_iface)
 
 		for x in self._mergelist:
 			if not isinstance(x, Package):
@@ -789,18 +791,28 @@ class Scheduler(PollScheduler):
 			if "pretend" not in x.defined_phases:
 				continue
 
-			out_str =">>> Running pre-merge checks for " + colorize("INFORM", x.cpv) + "\n"
-			portage.util.writemsg_stdout(out_str, noiselevel=-1)
+			out_str = "Running pre-merge checks for " + colorize("INFORM", x.cpv)
+			self._status_msg(out_str)
 
 			root_config = x.root_config
-			settings = self.pkgsettings[root_config.root]
+			settings = self._allocate_config(root_config.root)
 			settings.setcpv(x)
+			if not x.built:
+				# Get required SRC_URI metadata (it's not cached in x.metadata
+				# because some packages have an extremely large SRC_URI value).
+				portdb = root_config.trees["porttree"].dbapi
+				(settings.configdict["pkg"]["SRC_URI"],) = yield portdb.async_aux_get(
+					x.cpv, ["SRC_URI"], myrepo=x.repo, loop=loop
+				)
 
 			# setcpv/package.env allows for per-package PORTAGE_TMPDIR so we
 			# have to validate it for each package
 			rval = _check_temp_dir(settings)
 			if rval != os.EX_OK:
-				return rval
+				failures += 1
+				self._record_pkg_failure(x, settings, FAILURE)
+				self._deallocate_config(settings)
+				continue
 
 			build_dir_path = os.path.join(
 				os.path.realpath(settings["PORTAGE_TMPDIR"]),
@@ -809,7 +821,7 @@ class Scheduler(PollScheduler):
 			settings["PORTAGE_BUILDDIR"] = build_dir_path
 			build_dir = EbuildBuildDir(scheduler=sched_iface,
 				settings=settings)
-			sched_iface.run_until_complete(build_dir.async_lock())
+			yield build_dir.async_lock()
 			current_task = None
 
 			try:
@@ -835,7 +847,7 @@ class Scheduler(PollScheduler):
 						phase='clean', scheduler=sched_iface, settings=settings)
 					current_task = clean_phase
 					clean_phase.start()
-					clean_phase.wait()
+					yield clean_phase.async_wait()
 
 				if x.built:
 					tree = "bintree"
@@ -845,13 +857,19 @@ class Scheduler(PollScheduler):
 					# Display fetch on stdout, so that it's always clear what
 					# is consuming time here.
 					if bintree.isremote(x.cpv):
-						fetcher = BinpkgFetcher(pkg=x,
-							scheduler=sched_iface)
-						fetcher.start()
-						if fetcher.wait() != os.EX_OK:
+						fetcher = self._get_prefetcher(x)
+						if fetcher is None:
+							fetcher = BinpkgFetcher(pkg=x, scheduler=loop)
+							fetcher.start()
+							# We only set the fetched value when fetcher
+							# is a BinpkgFetcher, since BinpkgPrefetcher
+							# handles fetch, verification, and the
+							# bintree.inject call which moves the file.
+							fetched = fetcher.pkg_path
+						if (yield fetcher.async_wait()) != os.EX_OK:
 							failures += 1
+							self._record_pkg_failure(x, settings, fetcher.returncode)
 							continue
-						fetched = fetcher.pkg_path
 
 					if fetched is False:
 						filename = bintree.getname(x.cpv)
@@ -861,8 +879,9 @@ class Scheduler(PollScheduler):
 						scheduler=sched_iface, _pkg_path=filename)
 					current_task = verifier
 					verifier.start()
-					if verifier.wait() != os.EX_OK:
+					if (yield verifier.async_wait()) != os.EX_OK:
 						failures += 1
+						self._record_pkg_failure(x, settings, verifier.returncode)
 						continue
 
 					if fetched:
@@ -870,8 +889,7 @@ class Scheduler(PollScheduler):
 
 					infloc = os.path.join(build_dir_path, "build-info")
 					ensure_dirs(infloc)
-					self._sched_iface.run_until_complete(
-						bintree.dbapi.unpack_metadata(settings, infloc, loop=self._sched_iface))
+					yield bintree.dbapi.unpack_metadata(settings, infloc, loop=loop)
 					ebuild_path = os.path.join(infloc, x.pf + ".ebuild")
 					settings.configdict["pkg"]["EMERGE_FROM"] = "binary"
 					settings.configdict["pkg"]["MERGE_TYPE"] = "binary"
@@ -905,28 +923,45 @@ class Scheduler(PollScheduler):
 
 				current_task = pretend_phase
 				pretend_phase.start()
-				ret = pretend_phase.wait()
+				ret = yield pretend_phase.async_wait()
 				if ret != os.EX_OK:
 					failures += 1
+					self._record_pkg_failure(x, settings, ret)
 				portage.elog.elog_process(x.cpv, settings)
 			finally:
 
 				if current_task is not None:
 					if current_task.isAlive():
 						current_task.cancel()
-						current_task.wait()
 					if current_task.returncode == os.EX_OK:
 						clean_phase = EbuildPhase(background=False,
 							phase='clean', scheduler=sched_iface,
 							settings=settings)
 						clean_phase.start()
-						clean_phase.wait()
+						yield clean_phase.async_wait()
 
-				sched_iface.run_until_complete(build_dir.async_unlock())
+				yield build_dir.async_unlock()
+				self._deallocate_config(settings)
 
 		if failures:
-			return FAILURE
-		return os.EX_OK
+			return coroutine_return(FAILURE)
+		coroutine_return(os.EX_OK)
+
+	def _record_pkg_failure(self, pkg, settings, ret):
+		"""Record a package failure. This eliminates the package
+		from the --keep-going merge list, and immediately calls
+		_failed_pkg_msg if we have not been terminated."""
+		self._failed_pkgs.append(
+			self._failed_pkg(
+				build_dir=settings.get("PORTAGE_BUILDDIR"),
+				build_log=settings.get("PORTAGE_LOG_FILE"),
+				pkg=pkg,
+				returncode=ret,
+			)
+		)
+		if not self._terminated_tasks:
+			self._failed_pkg_msg(self._failed_pkgs[-1], "emerge", "for")
+			self._status_display.failed = len(self._failed_pkgs)
 
 	def merge(self):
 		if "--resume" in self.myopts:
@@ -988,11 +1023,6 @@ class Scheduler(PollScheduler):
 		if rval != os.EX_OK and not keep_going:
 			return rval
 
-		if not fetchonly:
-			rval = self._run_pkg_pretend()
-			if rval != os.EX_OK:
-				return rval
-
 		while True:
 
 			received_signal = []
@@ -1389,8 +1419,6 @@ class Scheduler(PollScheduler):
 		if self._opts_no_background.intersection(self.myopts):
 			self._set_max_jobs(1)
 
-		self._add_prefetchers()
-		self._add_packages()
 		failed_pkgs = self._failed_pkgs
 		portage.locks._quiet = self._background
 		portage.elog.add_listener(self._elog_listener)
@@ -1406,6 +1434,30 @@ class Scheduler(PollScheduler):
 		rval = os.EX_OK
 
 		try:
+			self._add_prefetchers()
+			if not self._build_opts.fetchonly:
+				# Run pkg_pretend concurrently with parallel-fetch, and be careful
+				# to respond appropriately to termination, so that we don't start
+				# any new tasks after we've been terminated. Temporarily make the
+				# status display quiet so that its output is not interleaved with
+				# pkg_pretend output.
+				status_quiet = self._status_display.quiet
+				self._status_display.quiet = True
+				try:
+					rval = self._sched_iface.run_until_complete(
+						self._run_pkg_pretend(loop=self._sched_iface)
+					)
+				except asyncio.CancelledError:
+					self.terminate()
+				finally:
+					self._status_display.quiet = status_quiet
+				self._termination_check()
+				if self._terminated_tasks:
+					rval = 128 + signal.SIGINT
+				if rval != os.EX_OK:
+					return rval
+
+			self._add_packages()
 			self._main_loop()
 		finally:
 			self._main_loop_cleanup()
@@ -1742,6 +1794,23 @@ class Scheduler(PollScheduler):
 
 		return bool(state_change)
 
+	def _get_prefetcher(self, pkg):
+		try:
+			prefetcher = self._prefetchers.pop(pkg, None)
+		except KeyError:
+			# KeyError observed with PyPy 1.8, despite None given as default.
+			# Note that PyPy 1.8 has the same WeakValueDictionary code as
+			# CPython 2.7, so it may be possible for CPython to raise KeyError
+			# here as well.
+			prefetcher = None
+		if prefetcher is not None and not prefetcher.isAlive():
+			try:
+				self._task_queues.fetch._task_queue.remove(prefetcher)
+			except ValueError:
+				pass
+			prefetcher = None
+		return prefetcher
+
 	def _task(self, pkg):
 
 		pkg_to_replace = None
@@ -1758,20 +1827,7 @@ class Scheduler(PollScheduler):
 					"installed", pkg.root_config, installed=True,
 					operation="uninstall")
 
-		try:
-			prefetcher = self._prefetchers.pop(pkg, None)
-		except KeyError:
-			# KeyError observed with PyPy 1.8, despite None given as default.
-			# Note that PyPy 1.8 has the same WeakValueDictionary code as
-			# CPython 2.7, so it may be possible for CPython to raise KeyError
-			# here as well.
-			prefetcher = None
-		if prefetcher is not None and not prefetcher.isAlive():
-			try:
-				self._task_queues.fetch._task_queue.remove(prefetcher)
-			except ValueError:
-				pass
-			prefetcher = None
+		prefetcher = self._get_prefetcher(pkg)
 
 		task = MergeListItem(args_set=self._args_set,
 			background=self._background, binpkg_opts=self._binpkg_opts,

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     ff270ae58eca457a1e2bbf9507d8438123082ca0
Author:     Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Thu Dec  3 23:01:30 2020 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Thu Dec  3 23:07:06 2020 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=ff270ae5

depgraph._add_pkg: fix AttributeError when myparent is not a Package

Fixes: b991f23ad915 ("Allow a package to replace its own buildtime dependency")
Bug: https://bugs.gentoo.org/757741
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/depgraph.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/_emerge/depgraph.py b/lib/_emerge/depgraph.py
index 0450291d4..f3e834a60 100644
--- a/lib/_emerge/depgraph.py
+++ b/lib/_emerge/depgraph.py
@@ -3107,7 +3107,7 @@ class depgraph:
 								self._frozen_config.myopts,
 								modified_use=self._pkg_use_enabled(pkg))),
 								level=logging.DEBUG, noiselevel=-1)
-				elif (pkg.installed and myparent and
+				elif (pkg.installed and isinstance(myparent, Package) and
 					pkg.root == myparent.root and
 					pkg.slot_atom == myparent.slot_atom):
 					# If the parent package is replacing the child package then

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     bfe334d0293e12952f61c3131339f9dcd2bb6bd0
Author:     Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Sat May  1 22:35:55 2021 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Sat May  1 22:37:39 2021 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=bfe334d0

_run_pkg_pretend: Use async and await syntax

Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/Scheduler.py | 26 ++++++++++++--------------
 1 file changed, 12 insertions(+), 14 deletions(-)

diff --git a/lib/_emerge/Scheduler.py b/lib/_emerge/Scheduler.py
index 656d382dd..d3528ee68 100644
--- a/lib/_emerge/Scheduler.py
+++ b/lib/_emerge/Scheduler.py
@@ -25,7 +25,6 @@ from portage._sets import SETPREFIX
 from portage._sets.base import InternalPackageSet
 from portage.util import ensure_dirs, writemsg, writemsg_level
 from portage.util.futures import asyncio
-from portage.util.futures.compat_coroutine import coroutine, coroutine_return
 from portage.util.SlotObject import SlotObject
 from portage.util._async.SchedulerInterface import SchedulerInterface
 from portage.package.ebuild.digestcheck import digestcheck
@@ -767,8 +766,7 @@ class Scheduler(PollScheduler):
 
 		return prefetcher
 
-	@coroutine
-	def _run_pkg_pretend(self, loop=None):
+	async def _run_pkg_pretend(self, loop=None):
 		"""
 		Since pkg_pretend output may be important, this method sends all
 		output directly to stdout (regardless of options like --quiet or
@@ -805,7 +803,7 @@ class Scheduler(PollScheduler):
 				# Get required SRC_URI metadata (it's not cached in x.metadata
 				# because some packages have an extremely large SRC_URI value).
 				portdb = root_config.trees["porttree"].dbapi
-				(settings.configdict["pkg"]["SRC_URI"],) = yield portdb.async_aux_get(
+				(settings.configdict["pkg"]["SRC_URI"],) = await portdb.async_aux_get(
 					x.cpv, ["SRC_URI"], myrepo=x.repo, loop=loop
 				)
 
@@ -825,7 +823,7 @@ class Scheduler(PollScheduler):
 			settings["PORTAGE_BUILDDIR"] = build_dir_path
 			build_dir = EbuildBuildDir(scheduler=sched_iface,
 				settings=settings)
-			yield build_dir.async_lock()
+			await build_dir.async_lock()
 			current_task = None
 
 			try:
@@ -851,7 +849,7 @@ class Scheduler(PollScheduler):
 						phase='clean', scheduler=sched_iface, settings=settings)
 					current_task = clean_phase
 					clean_phase.start()
-					yield clean_phase.async_wait()
+					await clean_phase.async_wait()
 
 				if x.built:
 					tree = "bintree"
@@ -870,7 +868,7 @@ class Scheduler(PollScheduler):
 							# handles fetch, verification, and the
 							# bintree.inject call which moves the file.
 							fetched = fetcher.pkg_path
-						if (yield fetcher.async_wait()) != os.EX_OK:
+						if await fetcher.async_wait() != os.EX_OK:
 							failures += 1
 							self._record_pkg_failure(x, settings, fetcher.returncode)
 							continue
@@ -883,7 +881,7 @@ class Scheduler(PollScheduler):
 						scheduler=sched_iface, _pkg_path=filename)
 					current_task = verifier
 					verifier.start()
-					if (yield verifier.async_wait()) != os.EX_OK:
+					if await verifier.async_wait() != os.EX_OK:
 						failures += 1
 						self._record_pkg_failure(x, settings, verifier.returncode)
 						continue
@@ -893,7 +891,7 @@ class Scheduler(PollScheduler):
 
 					infloc = os.path.join(build_dir_path, "build-info")
 					ensure_dirs(infloc)
-					yield bintree.dbapi.unpack_metadata(settings, infloc, loop=loop)
+					await bintree.dbapi.unpack_metadata(settings, infloc, loop=loop)
 					ebuild_path = os.path.join(infloc, x.pf + ".ebuild")
 					settings.configdict["pkg"]["EMERGE_FROM"] = "binary"
 					settings.configdict["pkg"]["MERGE_TYPE"] = "binary"
@@ -927,7 +925,7 @@ class Scheduler(PollScheduler):
 
 				current_task = pretend_phase
 				pretend_phase.start()
-				ret = yield pretend_phase.async_wait()
+				ret = await pretend_phase.async_wait()
 				if ret != os.EX_OK:
 					failures += 1
 					self._record_pkg_failure(x, settings, ret)
@@ -942,14 +940,14 @@ class Scheduler(PollScheduler):
 							phase='clean', scheduler=sched_iface,
 							settings=settings)
 						clean_phase.start()
-						yield clean_phase.async_wait()
+						await clean_phase.async_wait()
 
-				yield build_dir.async_unlock()
+				await build_dir.async_unlock()
 				self._deallocate_config(settings)
 
 		if failures:
-			return coroutine_return(FAILURE)
-		coroutine_return(os.EX_OK)
+			return FAILURE
+		return os.EX_OK
 
 	def _record_pkg_failure(self, pkg, settings, ret):
 		"""Record a package failure. This eliminates the package

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     1eda52d380b39095c6471db8025ab2202fedafc9
Author:     Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Sat May  1 22:23:39 2021 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Sat May  1 22:27:41 2021 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=1eda52d3

_run_pkg_pretend: raise asyncio.CancelledError when terminated

Bug: https://bugs.gentoo.org/787545
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/Scheduler.py | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/lib/_emerge/Scheduler.py b/lib/_emerge/Scheduler.py
index 0ed2ee530..656d382dd 100644
--- a/lib/_emerge/Scheduler.py
+++ b/lib/_emerge/Scheduler.py
@@ -1,4 +1,4 @@
-# Copyright 1999-2020 Gentoo Authors
+# Copyright 1999-2021 Gentoo Authors
 # Distributed under the terms of the GNU General Public License v2
 
 from collections import deque
@@ -791,6 +791,10 @@ class Scheduler(PollScheduler):
 			if "pretend" not in x.defined_phases:
 				continue
 
+			self._termination_check()
+			if self._terminated_tasks:
+				raise asyncio.CancelledError
+
 			out_str = "Running pre-merge checks for " + colorize("INFORM", x.cpv)
 			self._status_msg(out_str)
 

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     4e3c8ffebe0f377b2c39c5213983d641396aec34
Author:     Marco Sirabella <marco <AT> sirabella <DOT> org>
AuthorDate: Sat May  8 15:44:07 2021 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Sat May  8 17:53:50 2021 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=4e3c8ffe

emerge: Fail with helpful message when --jobs=""

Bug: https://bugs.gentoo.org/788967
Closes: https://github.com/gentoo/portage/pull/706
Reported-by: Christophe Lermytte <gentoo <AT> lermytte.be>
Signed-off-by: Marco Sirabella <marco <AT> sirabella.org>
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/main.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/_emerge/main.py b/lib/_emerge/main.py
index 191be6479..e8a038d83 100644
--- a/lib/_emerge/main.py
+++ b/lib/_emerge/main.py
@@ -1072,7 +1072,7 @@ def parse_opts(tmpcmdline, silent=False):
 
 		myoptions.deep = deep
 
-	if myoptions.jobs:
+	if myoptions.jobs is not None:
 		jobs = None
 		if myoptions.jobs == "True":
 			jobs = True

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     e26a08896f098ff93530efcd2d77d60d2eff0257
Author:     James Stevenson <james.al.stevenson <AT> gmail <DOT> com>
AuthorDate: Sun May 16 10:58:13 2021 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Sun May 16 22:27:15 2021 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=e26a0889

emerge --fetchonly --quiet: use emerge-fetch.log when writable (bug 285614)

Closes: https://github.com/gentoo/portage/pull/707
Bug: https://bugs.gentoo.org/285614
Signed-off-by: James Stevenson <james.al.stevenson <AT> gmail.com>
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/EbuildBuild.py | 25 +++++++++++++++++++++----
 lib/_emerge/Scheduler.py   |  4 ++--
 2 files changed, 23 insertions(+), 6 deletions(-)

diff --git a/lib/_emerge/EbuildBuild.py b/lib/_emerge/EbuildBuild.py
index aeb8a252b..db85b20b2 100644
--- a/lib/_emerge/EbuildBuild.py
+++ b/lib/_emerge/EbuildBuild.py
@@ -23,6 +23,7 @@ from portage.package.ebuild.digestcheck import digestcheck
 from portage.package.ebuild.doebuild import _check_temp_dir
 from portage.package.ebuild._spawn_nofetch import SpawnNofetchWithoutBuilddir
 from portage.util._async.AsyncTaskFuture import AsyncTaskFuture
+from portage.util.path import first_existing
 
 
 class EbuildBuild(CompositeTask):
@@ -160,16 +161,32 @@ class EbuildBuild(CompositeTask):
 						self._default_final_exit)
 				return
 
+			quiet_setting = settings.get("PORTAGE_QUIET", False)
+			fetch_log = None
+			logwrite_access = False
+			if quiet_setting:
+				fetch_log = os.path.join(
+					_emerge.emergelog._emerge_log_dir, "emerge-fetch.log"
+				)
+				logwrite_access = os.access(first_existing(fetch_log), os.W_OK)
+
 			fetcher = EbuildFetcher(
 				config_pool=self.config_pool,
 				ebuild_path=self._ebuild_path,
 				fetchall=self.opts.fetch_all_uri,
 				fetchonly=self.opts.fetchonly,
-				background=False,
-				logfile=None,
+				background=quiet_setting if logwrite_access else False,
+				logfile=fetch_log if logwrite_access else None,
 				pkg=self.pkg,
-				scheduler=self.scheduler)
-			self._start_task(fetcher, self._fetchonly_exit)
+				scheduler=self.scheduler,
+			)
+
+			if fetch_log and logwrite_access:
+				fetcher.addExitListener(self._fetchonly_exit)
+				self._task_queued(fetcher)
+				self.scheduler.fetch.schedule(fetcher, force_queue=True)
+			else:
+				self._start_task(fetcher, self._fetchonly_exit)
 			return
 
 		self._build_dir = EbuildBuildDir(

diff --git a/lib/_emerge/Scheduler.py b/lib/_emerge/Scheduler.py
index d3528ee68..5146863ab 100644
--- a/lib/_emerge/Scheduler.py
+++ b/lib/_emerge/Scheduler.py
@@ -560,7 +560,7 @@ class Scheduler(PollScheduler):
 			if pargs:
 				self.status = pargs[0]
 
-	def _schedule_fetch(self, fetcher):
+	def _schedule_fetch(self, fetcher, force_queue=False):
 		"""
 		Schedule a fetcher, in order to control the number of concurrent
 		fetchers. If self._max_jobs is greater than 1 then the fetch
@@ -571,7 +571,7 @@ class Scheduler(PollScheduler):
 		would be required before it would be possible to enable
 		concurrent fetching within the parallel-fetch queue.
 		"""
-		if self._max_jobs > 1:
+		if self._max_jobs > 1 and not force_queue:
 			fetcher.start()
 		else:
 			self._task_queues.fetch.addFront(fetcher)

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     9dcfad7bcd940337635234e315281701a87a9d83
Author:     Felix Bier <flx.bier <AT> gmail <DOT> com>
AuthorDate: Sun May  9 21:53:46 2021 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Mon May 24 06:33:17 2021 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=9dcfad7b

unmerge: Remove vartree alias

This commit removes the variable localtree, which is assigned once
to point to vartree and then never modified.

Closes: https://github.com/gentoo/portage/pull/705
Signed-off-by: Felix Bier <felix.bier <AT> rohde-schwarz.com>
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/unmerge.py | 9 ++++-----
 1 file changed, 4 insertions(+), 5 deletions(-)

diff --git a/lib/_emerge/unmerge.py b/lib/_emerge/unmerge.py
index b0067f8e6..5cbb6f825 100644
--- a/lib/_emerge/unmerge.py
+++ b/lib/_emerge/unmerge.py
@@ -99,7 +99,6 @@ def _unmerge_display(root_config, myopts, unmerge_action,
 
 			global_unmerge = 1
 
-		localtree = vartree
 		# process all arguments and add all
 		# valid db entries to candidate_catpkgs
 		if global_unmerge:
@@ -199,7 +198,7 @@ def _unmerge_display(root_config, myopts, unmerge_action,
 				sys.exit(1)
 
 			if not mymatch and x[0] not in "<>=~":
-				mymatch = localtree.dep_match(x)
+				mymatch = vartree.dep_match(x)
 			if not mymatch:
 				portage.writemsg("\n--- Couldn't find '%s' to %s.\n" % \
 					(x.replace("null/", ""), unmerge_action), noiselevel=-1)
@@ -242,14 +241,14 @@ def _unmerge_display(root_config, myopts, unmerge_action,
 				slotmap={}
 				for mypkg in mymatch:
 					if unmerge_action == "clean":
-						myslot = localtree.getslot(mypkg)
+						myslot = vartree.getslot(mypkg)
 					else:
 						# since we're pruning, we don't care about slots
 						# and put all the pkgs in together
 						myslot = 0
 					if myslot not in slotmap:
 						slotmap[myslot] = {}
-					slotmap[myslot][localtree.dbapi.cpv_counter(mypkg)] = mypkg
+					slotmap[myslot][vartree.dbapi.cpv_counter(mypkg)] = mypkg
 
 				for mypkg in vartree.dbapi.cp_list(
 					portage.cpv_getkey(mymatch[0])):
@@ -445,7 +444,7 @@ def _unmerge_display(root_config, myopts, unmerge_action,
 				continue
 			mylist.difference_update(all_selected)
 		cp = portage.cpv_getkey(next(iter(selected)))
-		for y in localtree.dep_match(cp):
+		for y in vartree.dep_match(cp):
 			if y not in pkgmap[x]["omitted"] and \
 				y not in pkgmap[x]["selected"] and \
 				y not in pkgmap[x]["protected"] and \

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     646c400e018422be4ff165d73a2f18d1a9e987a5
Author:     Felix Bier <flx.bier <AT> gmail <DOT> com>
AuthorDate: Sun May  9 21:51:37 2021 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Mon May 24 06:30:26 2021 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=646c400e

unmerge: Reuse helper variable quiet

This commit reuses the already existing helper variable quiet
to avoid rechecking if --quiet is contained in myopts.

Signed-off-by: Felix Bier <felix.bier <AT> rohde-schwarz.com>
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/unmerge.py | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)

diff --git a/lib/_emerge/unmerge.py b/lib/_emerge/unmerge.py
index 09e155de3..b0067f8e6 100644
--- a/lib/_emerge/unmerge.py
+++ b/lib/_emerge/unmerge.py
@@ -166,15 +166,14 @@ def _unmerge_display(root_config, myopts, unmerge_action,
 						"="+"/".join(sp_absx[sp_vdb_len:]))
 
 		newline=""
-		if not "--quiet" in myopts:
+		if not quiet:
 			newline="\n"
 		if settings["ROOT"] != "/":
 			writemsg_level(darkgreen(newline+ \
 				">>> Using system located in ROOT tree %s\n" % \
 				settings["ROOT"]))
 
-		if (("--pretend" in myopts) or ("--ask" in myopts)) and \
-			not "--quiet" in myopts:
+		if ("--pretend" in myopts or "--ask" in myopts) and not quiet:
 			writemsg_level(darkgreen(newline+\
 				">>> These are the packages that would be unmerged:\n"))
 

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     9946517f1e0ee35b2a6eed1ff6c1fd10acf8fd45
Author:     Felix Bier <flx.bier <AT> gmail <DOT> com>
AuthorDate: Sun May  9 23:27:19 2021 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Mon May 24 06:22:30 2021 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=9946517f

Sort emerge --unmerge order for determinism

This commit changes the order in which packages are uninstalled
with --unmerge to be deterministic.

Before this commit, when an atom specified with --unmerge matches
multiple versions, these versions were uninstalled in a random order.
For example, when some-pkg-1.0.0 and some-pkg-2.0.0 are installed,
then running emerge --unmerge some-pkg will unmerge 1.0.0 and 2.0.0
in a random order.

With this commit, when an atom matches multiple versions,
they are uninstalled in a sorted order. So in the above example,
1.0.0 will be unmerged before 2.0.0.

This does not affect the order specified on the command-line,
for example when running:
emerge --unmerge =some-pkg-2.0.0 =some-pkg-1.0.0
that order is respected, as before this commit.

Similarly, when uninstalling multiple different packages, e.g.:
emerge --unmerge some-pkg some-other-pkg
the sorting only applies to the versions matching each atom,
so the versions matching some-pkg are sorted separately from
the versions matching some-other-pkg, and since some-pkg
is specified before some-other-pkg, its versions will be
uninstalled first.

Motivation: When a package has a custom pkg_postrm hook, uninstalling
multiple versions in a random order can leave the filesystem in a
different state depending on which version is uninstalled last.
When running emerge as part of a larger build system such as catalyst,
this is an obstacle towards reproducible builds.

Bug: https://bugs.gentoo.org/782724
Signed-off-by: Felix Bier <felix.bier <AT> rohde-schwarz.com>
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/PackageUninstall.py | 2 +-
 lib/_emerge/unmerge.py          | 5 +++++
 2 files changed, 6 insertions(+), 1 deletion(-)

diff --git a/lib/_emerge/PackageUninstall.py b/lib/_emerge/PackageUninstall.py
index 43210b4bc..8786c2cd1 100644
--- a/lib/_emerge/PackageUninstall.py
+++ b/lib/_emerge/PackageUninstall.py
@@ -71,7 +71,7 @@ class PackageUninstall(CompositeTask):
 
 		# Output only gets logged if it comes after prepare_build_dirs()
 		# which initializes PORTAGE_LOG_FILE.
-		retval, pkgmap = _unmerge_display(self.pkg.root_config,
+		retval, _ = _unmerge_display(self.pkg.root_config,
 			self.opts, "unmerge", [self.pkg.cpv], clean_delay=0,
 			writemsg_level=self._writemsg_level)
 

diff --git a/lib/_emerge/unmerge.py b/lib/_emerge/unmerge.py
index e8b7c9aaa..09e155de3 100644
--- a/lib/_emerge/unmerge.py
+++ b/lib/_emerge/unmerge.py
@@ -432,6 +432,11 @@ def _unmerge_display(root_config, myopts, unmerge_action,
 				cp_dict[k].update(v)
 		pkgmap = [unordered[cp] for cp in sorted(unordered)]
 
+	# Sort each set of selected packages
+	if ordered:
+		for pkg in pkgmap:
+			pkg["selected"] = sorted(pkg["selected"], key=cpv_sort_key())
+
 	for x in range(len(pkgmap)):
 		selected = pkgmap[x]["selected"]
 		if not selected:

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     482079c7835c769843544a4920ccb4c49d363351
Author:     Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Sun Jun 13 22:38:40 2021 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Sun Jun 13 22:41:06 2021 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=482079c7

PORTAGE_NICENESS: fix atexit ResourceWarning: unclosed file

# emerge -V
Portage 3.0.20 (python 3.8.5-final-0, default/linux/amd64/17.1/desktop, gcc-9.3.0, glibc-2.31-r6, 5.4.x x86_64)
/usr/lib/python3.8/site-packages/_emerge/actions.py:2663: ResourceWarning: unclosed file <_io.TextIOWrapper name='/proc/self/autogroup' mode='w' encoding='UTF-8'>
  lambda value: autogroup_file.open("w").write(value).close(),
ResourceWarning: Enable tracemalloc to get the object allocation traceback

Fixes: a4d882964ee1 ("PORTAGE_NICENESS: Consider autogroup scheduling")
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/actions.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/_emerge/actions.py b/lib/_emerge/actions.py
index 18f8da200..bfb08ed6b 100644
--- a/lib/_emerge/actions.py
+++ b/lib/_emerge/actions.py
@@ -2660,7 +2660,7 @@ def nice(settings):
 		# terminal where portage was executed in, would
 		# continue running with that value.
 		portage.atexit_register(
-			lambda value: autogroup_file.open("w").write(value),
+			lambda value: autogroup_file.write_text(value),
 			original_autogroup_nice_value,
 		)
 

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     371049edb5ba6ffc472b7f21770ca7b48d9ffa3b
Author:     Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Tue Sep 21 04:57:50 2021 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Tue Sep 21 05:33:50 2021 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=371049ed

SequentialTaskQueue: convert compat coroutine to async

Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/SequentialTaskQueue.py | 10 ++++------
 1 file changed, 4 insertions(+), 6 deletions(-)

diff --git a/lib/_emerge/SequentialTaskQueue.py b/lib/_emerge/SequentialTaskQueue.py
index 5bf803ec1..829f44ba6 100644
--- a/lib/_emerge/SequentialTaskQueue.py
+++ b/lib/_emerge/SequentialTaskQueue.py
@@ -1,10 +1,9 @@
-# Copyright 1999-2020 Gentoo Authors
+# Copyright 1999-2021 Gentoo Authors
 # Distributed under the terms of the GNU General Public License v2
 
 from collections import deque
 
 from portage.util.futures import asyncio
-from portage.util.futures.compat_coroutine import coroutine
 from portage.util.SlotObject import SlotObject
 
 
@@ -69,8 +68,7 @@ class SequentialTaskQueue(SlotObject):
         for task in list(self.running_tasks):
             task.cancel()
 
-    @coroutine
-    def wait(self, loop=None):
+    async def wait(self, loop=None):
         """
         Wait for the queue to become empty. This method is a coroutine.
         """
@@ -78,9 +76,9 @@ class SequentialTaskQueue(SlotObject):
             task = next(iter(self.running_tasks), None)
             if task is None:
                 # Wait for self.running_tasks to populate.
-                yield asyncio.sleep(0, loop=loop)
+                await asyncio.sleep(0, loop=loop)
             else:
-                yield task.async_wait()
+                await task.async_wait()
 
     def __bool__(self):
         return bool(self._task_queue or self.running_tasks)

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     18b3cb69095a30f52d74f953c186df184e047741
Author:     Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Tue Sep 21 05:04:15 2021 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Tue Sep 21 05:33:50 2021 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=18b3cb69

SpawnProcess: convert compat coroutine to async

Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/SpawnProcess.py | 10 ++++------
 1 file changed, 4 insertions(+), 6 deletions(-)

diff --git a/lib/_emerge/SpawnProcess.py b/lib/_emerge/SpawnProcess.py
index eacc83451..c43b17c12 100644
--- a/lib/_emerge/SpawnProcess.py
+++ b/lib/_emerge/SpawnProcess.py
@@ -1,4 +1,4 @@
-# Copyright 2008-2020 Gentoo Authors
+# Copyright 2008-2021 Gentoo Authors
 # Distributed under the terms of the GNU General Public License v2
 
 import errno
@@ -17,7 +17,6 @@ from portage.util import writemsg_level
 from portage.util._async.BuildLogger import BuildLogger
 from portage.util._async.PipeLogger import PipeLogger
 from portage.util.futures import asyncio
-from portage.util.futures.compat_coroutine import coroutine
 
 
 class SpawnProcess(SubProcess):
@@ -175,13 +174,12 @@ class SpawnProcess(SubProcess):
         )
         self._main_task.add_done_callback(self._main_exit)
 
-    @coroutine
-    def _main(self, build_logger, pipe_logger, loop=None):
+    async def _main(self, build_logger, pipe_logger, loop=None):
         try:
             if pipe_logger.poll() is None:
-                yield pipe_logger.async_wait()
+                await pipe_logger.async_wait()
             if build_logger.poll() is None:
-                yield build_logger.async_wait()
+                await build_logger.async_wait()
         except asyncio.CancelledError:
             self._main_cancel(build_logger, pipe_logger)
             raise

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Sam James]

commit:     0e9e12aadb889766d61c0561b9723e71542d43e6
Author:     Sam James <sam <AT> gentoo <DOT> org>
AuthorDate: Thu Oct  7 03:15:09 2021 +0000
Commit:     Sam James <sam <AT> gentoo <DOT> org>
CommitDate: Thu Oct 28 04:07:33 2021 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=0e9e12aa

lib/_emerge/actions.py: warn on missing /run

Newer versions of build-docbook-catalog use
/run/lock. This exposed that we weren't
asking users to mount /run in the handbook.

Check if it exists and warn if it doesn't.

This should primarily (exclusively?) be a
problem in chroots given an init system
should be creating this.

Bug: https://bugs.gentoo.org/816303
Closes: https://github.com/gentoo/portage/pull/762
Reviewed-by: Alec Warner <antarus <AT> gentoo.org>
Reviewed-by: Mike Gilbert <floppym <AT> gentoo.org>
Thanks-to: Duncan
Signed-off-by: Sam James <sam <AT> gentoo.org>

 lib/_emerge/actions.py | 33 +++++++++++++++++++++------------
 1 file changed, 21 insertions(+), 12 deletions(-)

diff --git a/lib/_emerge/actions.py b/lib/_emerge/actions.py
index 05a115250..515b22b66 100644
--- a/lib/_emerge/actions.py
+++ b/lib/_emerge/actions.py
@@ -2986,17 +2986,25 @@ def validate_ebuild_environment(trees):
     check_locale()
 
 
-def check_procfs():
-    procfs_path = "/proc"
-    if platform.system() not in ("Linux",) or os.path.ismount(procfs_path):
-        return os.EX_OK
-    msg = "It seems that %s is not mounted. You have been warned." % procfs_path
-    writemsg_level(
-        "".join("!!! %s\n" % l for l in textwrap.wrap(msg, 70)),
-        level=logging.ERROR,
-        noiselevel=-1,
-    )
-    return 1
+def check_mounted_fs():
+    """We need /proc for finding CPU counts and finding other system information.
+    We need /run for e.g. lock files in ebuilds."""
+    paths = {"/proc": False, "/run": False}
+
+    for path in paths.keys():
+        if platform.system() not in ("Linux",) or os.path.ismount(path):
+            paths[path] = True
+            continue
+
+        msg = "It seems %s is not mounted. Process management may malfunction." % path
+        writemsg_level(
+            "".join("!!! %s\n" % l for l in textwrap.wrap(msg, 70)),
+            level=logging.ERROR,
+            noiselevel=-1,
+        )
+
+    # Were all of the mounts we were looking for available?
+    return all(paths.values())
 
 
 def config_protect_check(trees):
@@ -3474,7 +3482,8 @@ def run_action(emerge_config):
         repo_name_check(emerge_config.trees)
         repo_name_duplicate_check(emerge_config.trees)
         config_protect_check(emerge_config.trees)
-    check_procfs()
+
+    check_mounted_fs()
 
     for mytrees in emerge_config.trees.values():
         mydb = mytrees["porttree"].dbapi

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Sam James]

commit:     216e962ae494e8291b667665007d045788b3e391
Author:     Sam James <sam <AT> gentoo <DOT> org>
AuthorDate: Thu Oct 28 04:40:09 2021 +0000
Commit:     Sam James <sam <AT> gentoo <DOT> org>
CommitDate: Thu Oct 28 04:52:32 2021 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=216e962a

Binpkg.py: check for inconsistent PROVIDES/image when unpacking binpkg

This is part of a series of fixes for the linked bug (failure
to preserve libraries in some situations).

When unpacking a binpkg to be installed, we should check
for the existence of PROVIDES if we're installing any
dynamic libraries. If PROVIDES does not exist in that case,
this suggests that e.g. scanelf malfunctioned or some corruption occurred.

Bug: https://bugs.gentoo.org/811462
Signed-off-by: Sam James <sam <AT> gentoo.org>

 lib/_emerge/Binpkg.py | 15 ++++++++++++++-
 1 file changed, 14 insertions(+), 1 deletion(-)

diff --git a/lib/_emerge/Binpkg.py b/lib/_emerge/Binpkg.py
index c7dde69bd..001283611 100644
--- a/lib/_emerge/Binpkg.py
+++ b/lib/_emerge/Binpkg.py
@@ -2,7 +2,6 @@
 # Distributed under the terms of the GNU General Public License v2
 
 import functools
-
 import _emerge.emergelog
 from _emerge.EbuildPhase import EbuildPhase
 from _emerge.BinpkgFetcher import BinpkgFetcher
@@ -13,8 +12,10 @@ from _emerge.EbuildMerge import EbuildMerge
 from _emerge.EbuildBuildDir import EbuildBuildDir
 from _emerge.SpawnProcess import SpawnProcess
 from portage.eapi import eapi_exports_replace_vars
+from portage.output import colorize
 from portage.util import ensure_dirs
 from portage.util._async.AsyncTaskFuture import AsyncTaskFuture
+from portage.util._dyn_libs.dyn_libs import check_dyn_libs_inconsistent
 import portage
 from portage import os
 from portage import shutil
@@ -425,6 +426,18 @@ class Binpkg(CompositeTask):
             self._async_unlock_builddir(returncode=self.returncode)
             return
 
+        # Before anything else, let's do an integrity check.
+        (provides,) = self._bintree.dbapi.aux_get(self.pkg.cpv, ["PROVIDES"])
+        if check_dyn_libs_inconsistent(self.settings["D"], provides):
+            self._writemsg_level(
+                colorize(
+                    "BAD",
+                    "!!! Error! Installing dynamic libraries (.so) with blank PROVIDES!",
+                ),
+                noiselevel=-1,
+                level=logging.ERROR,
+            )
+
         try:
             with io.open(
                 _unicode_encode(

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Michał Górny]

commit:     f85f0991a90b17009e02d342c58fb5b066d7b0c9
Author:     Michał Górny <mgorny <AT> gentoo <DOT> org>
AuthorDate: Fri Dec  3 07:27:18 2021 +0000
Commit:     Michał Górny <mgorny <AT> gentoo <DOT> org>
CommitDate: Sat Dec  4 04:56:20 2021 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=f85f0991

emerge: Do not resolve ambiguous package names via installed pkgs

When dealing with ambiguous package names on command-line, do not use
installed packages to resolve them.  This is counterintuitive
in the best case, and probably the wrong answer most of the time.

After all, if a user calls emerge with a specific package name, it is
quite likely that he intends to install a missing package rather than
rebuild or upgrade an individual package (though the latter can also
happen).  Rather than making assumptions that can go wrong, just display
the ambiguity message as usual.

Bug: https://bugs.gentoo.org/828059
Closes: https://github.com/gentoo/portage/pull/775
Signed-off-by: Michał Górny <mgorny <AT> gentoo.org>

 lib/_emerge/depgraph.py | 15 ---------------
 1 file changed, 15 deletions(-)

diff --git a/lib/_emerge/depgraph.py b/lib/_emerge/depgraph.py
index 07431c8ee..f6549eba6 100644
--- a/lib/_emerge/depgraph.py
+++ b/lib/_emerge/depgraph.py
@@ -4806,21 +4806,6 @@ class depgraph:
                     if len(non_virtual_cps) == 1:
                         installed_cp_set = non_virtual_cps
 
-                if len(expanded_atoms) > 1 and len(installed_cp_set) == 1:
-                    installed_cp = next(iter(installed_cp_set))
-                    for atom in expanded_atoms:
-                        if atom.cp == installed_cp:
-                            available = False
-                            for pkg in self._iter_match_pkgs_any(
-                                root_config, atom.without_use, onlydeps=onlydeps
-                            ):
-                                if not pkg.installed:
-                                    available = True
-                                    break
-                            if available:
-                                expanded_atoms = [atom]
-                                break
-
                 # If a non-virtual package and one or more virtual packages
                 # are in expanded_atoms, use the non-virtual package.
                 if len(expanded_atoms) > 1:

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Sam James]

commit:     1327fa9f829e8670c65ff35b9b0bda446991f7ed
Author:     Mike Gilbert <floppym <AT> gentoo <DOT> org>
AuthorDate: Mon Feb 14 20:30:18 2022 +0000
Commit:     Sam James <sam <AT> gentoo <DOT> org>
CommitDate: Mon Feb 14 21:50:55 2022 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=1327fa9f

Revert "lib/_emerge/actions.py: warn on missing /run"

Portage itself does not need /run to be mounted.
build-docbook-catalog was adjusted to create locks in /etc/xml instead.

Bug: https://bugs.gentoo.org/832853
Reverts: 0e9e12aadb889766d61c0561b9723e71542d43e6
Signed-off-by: Mike Gilbert <floppym <AT> gentoo.org>
Closes: https://github.com/gentoo/portage/pull/788
Signed-off-by: Sam James <sam <AT> gentoo.org>

 lib/_emerge/actions.py | 33 ++++++++++++---------------------
 1 file changed, 12 insertions(+), 21 deletions(-)

diff --git a/lib/_emerge/actions.py b/lib/_emerge/actions.py
index 0255e3e97..f82069f9a 100644
--- a/lib/_emerge/actions.py
+++ b/lib/_emerge/actions.py
@@ -3020,25 +3020,17 @@ def validate_ebuild_environment(trees):
     check_locale()
 
 
-def check_mounted_fs():
-    """We need /proc for finding CPU counts and finding other system information.
-    We need /run for e.g. lock files in ebuilds."""
-    paths = {"/proc": False, "/run": False}
-
-    for path in paths.keys():
-        if platform.system() not in ("Linux",) or os.path.ismount(path):
-            paths[path] = True
-            continue
-
-        msg = "It seems %s is not mounted. Process management may malfunction." % path
-        writemsg_level(
-            "".join("!!! %s\n" % l for l in textwrap.wrap(msg, 70)),
-            level=logging.ERROR,
-            noiselevel=-1,
-        )
-
-    # Were all of the mounts we were looking for available?
-    return all(paths.values())
+def check_procfs():
+    procfs_path = "/proc"
+    if platform.system() not in ("Linux",) or os.path.ismount(procfs_path):
+        return os.EX_OK
+    msg = "It seems that %s is not mounted. You have been warned." % procfs_path
+    writemsg_level(
+        "".join("!!! %s\n" % l for l in textwrap.wrap(msg, 70)),
+        level=logging.ERROR,
+        noiselevel=-1,
+    )
+    return 1
 
 
 def config_protect_check(trees):
@@ -3516,8 +3508,7 @@ def run_action(emerge_config):
         repo_name_check(emerge_config.trees)
         repo_name_duplicate_check(emerge_config.trees)
         config_protect_check(emerge_config.trees)
-
-    check_mounted_fs()
+    check_procfs()
 
     for mytrees in emerge_config.trees.values():
         mydb = mytrees["porttree"].dbapi

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     00a07dcc4842a1ecde11577f29946e8d8ed05731
Author:     John Helmert III <ajak <AT> gentoo <DOT> org>
AuthorDate: Sun Mar  6 17:13:50 2022 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Sun Mar  6 19:24:24 2022 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=00a07dcc

emerge --keep-going: deduplicate atoms in 'dropped' message

The --keep-going 'dropped' message sometimes contains duplicate atoms:

 * emerge --keep-going: app-containers/podman-4.0.2 dropped because it
 * requires app-containers/catatonit, app-containers/catatonit

This patch deduplicates the atoms printed in that message:

 * emerge --keep-going: app-containers/podman-4.0.2 dropped because it
 * requires app-containers/catatonit

Closes: https://github.com/gentoo/portage/pull/792
Signed-off-by: John Helmert III <ajak <AT> gentoo.org>
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/Scheduler.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/_emerge/Scheduler.py b/lib/_emerge/Scheduler.py
index 15d646c4f..bc3531627 100644
--- a/lib/_emerge/Scheduler.py
+++ b/lib/_emerge/Scheduler.py
@@ -2164,7 +2164,7 @@ class Scheduler(PollScheduler):
             if not atoms:
                 msg += " dropped because it is masked or unavailable"
             else:
-                msg += " dropped because it requires %s" % ", ".join(atoms)
+                msg += " dropped because it requires %s" % ", ".join(set(atoms))
             for line in textwrap.wrap(msg, msg_width):
                 eerror(line, phase="other", key=pkg.cpv)
             settings = self.pkgsettings[pkg.root]

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Sam James]

commit:     195b8029ddd510188e8a2510dad5cbce67c44ecc
Author:     Kenneth Raplee <kenrap <AT> kennethraplee <DOT> com>
AuthorDate: Fri Mar 25 12:41:54 2022 +0000
Commit:     Sam James <sam <AT> gentoo <DOT> org>
CommitDate: Sun Mar 27 23:06:46 2022 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=195b8029

Consolidate checks for options with bad atoms

Signed-off-by: Kenneth Raplee <kenrap <AT> kennethraplee.com>
Signed-off-by: Sam James <sam <AT> gentoo.org>

 lib/_emerge/main.py | 57 +++++++++++++----------------------------------------
 1 file changed, 14 insertions(+), 43 deletions(-)

diff --git a/lib/_emerge/main.py b/lib/_emerge/main.py
index f3ff02404..01dc1b419 100644
--- a/lib/_emerge/main.py
+++ b/lib/_emerge/main.py
@@ -876,52 +876,23 @@ def parse_opts(tmpcmdline, silent=False):
     if myoptions.depclean_lib_check in true_y:
         myoptions.depclean_lib_check = True
 
-    if myoptions.exclude:
-        bad_atoms = _find_bad_atoms(myoptions.exclude)
-        if bad_atoms and not silent:
-            parser.error(
-                "Invalid Atom(s) in --exclude parameter: '%s' (only package names and slot atoms (with wildcards) allowed)\n"
-                % (",".join(bad_atoms),)
-            )
-
-    if myoptions.reinstall_atoms:
-        bad_atoms = _find_bad_atoms(myoptions.reinstall_atoms)
-        if bad_atoms and not silent:
-            parser.error(
-                "Invalid Atom(s) in --reinstall-atoms parameter: '%s' (only package names and slot atoms (with wildcards) allowed)\n"
-                % (",".join(bad_atoms),)
-            )
-
-    if myoptions.rebuild_exclude:
-        bad_atoms = _find_bad_atoms(myoptions.rebuild_exclude)
-        if bad_atoms and not silent:
-            parser.error(
-                "Invalid Atom(s) in --rebuild-exclude parameter: '%s' (only package names and slot atoms (with wildcards) allowed)\n"
-                % (",".join(bad_atoms),)
-            )
-
-    if myoptions.rebuild_ignore:
-        bad_atoms = _find_bad_atoms(myoptions.rebuild_ignore)
-        if bad_atoms and not silent:
-            parser.error(
-                "Invalid Atom(s) in --rebuild-ignore parameter: '%s' (only package names and slot atoms (with wildcards) allowed)\n"
-                % (",".join(bad_atoms),)
-            )
-
-    if myoptions.usepkg_exclude:
-        bad_atoms = _find_bad_atoms(myoptions.usepkg_exclude)
-        if bad_atoms and not silent:
-            parser.error(
-                "Invalid Atom(s) in --usepkg-exclude parameter: '%s' (only package names and slot atoms (with wildcards) allowed)\n"
-                % (",".join(bad_atoms),)
-            )
+    candidate_bad_options = (
+        (myoptions.exclude, "exclude"),
+        (myoptions.reinstall_atoms, "reinstall-atoms"),
+        (myoptions.rebuild_exclude, "rebuild-exclude"),
+        (myoptions.rebuild_ignore, "rebuild-ignore"),
+        (myoptions.usepkg_exclude, "usepkg-exclude"),
+        (myoptions.useoldpkg_atoms, "useoldpkg-atoms"),
+    )
+    bad_options = (
+        (_find_bad_atoms(atoms), flag) for atoms, flag in candidate_bad_options if atoms
+    )
 
-    if myoptions.useoldpkg_atoms:
-        bad_atoms = _find_bad_atoms(myoptions.useoldpkg_atoms)
+    for bad_atoms, flag in bad_options:
         if bad_atoms and not silent:
+            invalid_atoms = ",".join(bad_atoms)
             parser.error(
-                "Invalid Atom(s) in --useoldpkg-atoms parameter: '%s' (only package names and slot atoms (with wildcards) allowed)\n"
-                % (",".join(bad_atoms),)
+                f"Invalid Atom(s) in --{flag} parameter: '{invalid_atoms}' (only package names and slot atoms (with wildcards) allowed)\n"
             )
 
     if myoptions.fail_clean in true_y:

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Mike Gilbert]

commit:     ff27e63569297dcd8b459e7ee255d4335e9f9105
Author:     Mike Gilbert <floppym <AT> gentoo <DOT> org>
AuthorDate: Fri Apr 22 21:13:11 2022 +0000
Commit:     Mike Gilbert <floppym <AT> gentoo <DOT> org>
CommitDate: Fri Apr 22 21:20:32 2022 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=ff27e635

MiscFunctionsProcess: use a separate SANDBOX_LOG

This avoids clobbering the main sandbox.log file when
"misc-functions.sh die_hooks" is invoked after an ebuild phase.

Signed-off-by: Mike Gilbert <floppym <AT> gentoo.org>

 lib/_emerge/MiscFunctionsProcess.py | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/lib/_emerge/MiscFunctionsProcess.py b/lib/_emerge/MiscFunctionsProcess.py
index 7a7b1f5ac..16fd08222 100644
--- a/lib/_emerge/MiscFunctionsProcess.py
+++ b/lib/_emerge/MiscFunctionsProcess.py
@@ -48,9 +48,20 @@ class MiscFunctionsProcess(AbstractEbuildProcess):
         # Temporarily unset EBUILD_PHASE so that bashrc code doesn't
         # think this is a real phase.
         phase_backup = self.settings.pop("EBUILD_PHASE", None)
+
+        # Use a separate sandbox log file to avoid clobbering the real one.
+        sandbox_log = self.settings.pop("SANDBOX_LOG", None)
+        if sandbox_log is not None:
+            self.settings["SANDBOX_LOG"] = os.path.join(
+                self.settings["T"], "sandbox-misc.log"
+            )
+
         try:
             return spawn(" ".join(args), self.settings, **kwargs)
         finally:
             if phase_backup is not None:
                 self.settings["EBUILD_PHASE"] = phase_backup
             self.settings.pop("PORTAGE_PIPE_FD", None)
+            self.settings.pop("SANDBOX_LOG", None)
+            if sandbox_log is not None:
+                self.settings["SANDBOX_LOG"] = sandbox_log

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Mike Gilbert]

commit:     7fa4fe1b2fddff06b15156c915e7ed6bb29a469e
Author:     David Palao <david.palao <AT> gmail <DOT> com>
AuthorDate: Tue May 31 15:44:54 2022 +0000
Commit:     Mike Gilbert <floppym <AT> gentoo <DOT> org>
CommitDate: Tue Jun  7 23:47:55 2022 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=7fa4fe1b

improvement(actions): using new MtimeDB's API to make it read only

Signed-off-by: David Palao <david.palao <AT> gmail.com>
Signed-off-by: Mike Gilbert <floppym <AT> gentoo.org>

 lib/_emerge/actions.py | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/lib/_emerge/actions.py b/lib/_emerge/actions.py
index e697459c8..0bf2d88b1 100644
--- a/lib/_emerge/actions.py
+++ b/lib/_emerge/actions.py
@@ -263,8 +263,7 @@ def action_build(
     mergelist_shown = False
 
     if pretend or fetchonly:
-        # make the mtimedb readonly
-        mtimedb.filename = None
+        mtimedb.make_readonly()
     if "--digest" in myopts or "digest" in settings.features:
         if "--digest" in myopts:
             msg = "The --digest option"

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Mike Gilbert]

commit:     9ab9368c3324d615f1234d1ccb03eae939c393cf
Author:     John Helmert III <ajak <AT> gentoo <DOT> org>
AuthorDate: Wed Jun 15 22:11:17 2022 +0000
Commit:     Mike Gilbert <floppym <AT> gentoo <DOT> org>
CommitDate: Fri Jun 17 17:01:49 2022 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=9ab9368c

depclean: return failure when requested packages aren't depcleaned

Closes: https://bugs.gentoo.org/851750
Closes: https://github.com/gentoo/portage/pull/840
Signed-off-by: John Helmert III <ajak <AT> gentoo.org>
Signed-off-by: Mike Gilbert <floppym <AT> gentoo.org>

 lib/_emerge/actions.py | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/lib/_emerge/actions.py b/lib/_emerge/actions.py
index 0bf2d88b1..e2f3f2ccf 100644
--- a/lib/_emerge/actions.py
+++ b/lib/_emerge/actions.py
@@ -1697,6 +1697,11 @@ def _calc_depclean(settings, trees, ldpath_mtimes, myopts, action, args_set, spi
                     cleanlist.append(node.cpv)
 
         return _depclean_result(0, cleanlist, ordered, required_pkgs_total, resolver)
+    if args_set and "--pretend" not in myopts:
+        # If the cleanlist is empty but we were given packages to clean,
+        # we aren't successfully depcleaning. Return failure unless
+        # we're pretending.
+        return _depclean_result(1, [], False, required_pkgs_total, resolver)
     return _depclean_result(0, [], False, required_pkgs_total, resolver)
 
 

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Sam James]

commit:     bb820e4a38cefdb552d02b1d6bcb97eccab38982
Author:     Thomas Bracht Laumann Jespersen <t <AT> laumann <DOT> xyz>
AuthorDate: Mon Aug  8 11:24:48 2022 +0000
Commit:     Sam James <sam <AT> gentoo <DOT> org>
CommitDate: Sat Aug 13 17:56:10 2022 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=bb820e4a

EbuildPhase: add SLOT to package version line

Requested as a useful part in the output from tinderbox runs.

Bug: https://bugs.gentoo.org/864382
Signed-off-by: Thomas Bracht Laumann Jespersen <t <AT> laumann.xyz>
Closes: https://github.com/gentoo/portage/pull/885
Signed-off-by: Sam James <sam <AT> gentoo.org>

 lib/_emerge/EbuildPhase.py | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/lib/_emerge/EbuildPhase.py b/lib/_emerge/EbuildPhase.py
index 9a04f9c1f..a1eaf86c1 100644
--- a/lib/_emerge/EbuildPhase.py
+++ b/lib/_emerge/EbuildPhase.py
@@ -135,7 +135,9 @@ class EbuildPhase(CompositeTask):
                     maint_str = "<invalid metadata.xml>"
 
             msg = []
-            msg.append("Package:    %s" % self.settings.mycpv)
+            msg.append(
+                "Package:    %s:%s" % (self.settings.mycpv, self.settings["SLOT"])
+            )
             if self.settings.get("PORTAGE_REPO_NAME"):
                 msg.append("Repository: %s" % self.settings["PORTAGE_REPO_NAME"])
             if maint_str:

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Mike Gilbert]

commit:     47ac5aff464d414bcaa2d04e9eeff06aceec868c
Author:     Mike Gilbert <floppym <AT> gentoo <DOT> org>
AuthorDate: Sat Sep 17 23:37:46 2022 +0000
Commit:     Mike Gilbert <floppym <AT> gentoo <DOT> org>
CommitDate: Sun Sep 18 18:33:19 2022 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=47ac5aff

Move pty setup from AbstractEbuildProcess to SpawnProcess

This ensures that bash is always executed with a controlling terminal,
which prevents it from triggering sandbox failures on startup.

When merging a package, the 'instprep' phase is executed by a
MergeProcess task. The MergeProcess class derives from SpawnProcess
directly (not through AbstractEbuildProcess).

The SpawnProcess class calls portage.process.spawn() to execute a task.
When pid-sandbox is enabled, this creates a new PID namespaces, and
starts pid-ns-init to act as PID 1 in the new namespace.

pid-ns-init calls setsid(), which creates a new session and
disconnects the process from its controlling terminal.

Later, it calls ioctl(sys.stdout, termios.TIOCSCTTY) to associate the
process with the terminal attached to stdout. This only works if stdout
is a tty (not a pipe).

If pid-ns-init fails to associate the process with a controlling
terminal, bash will fail to open /dev/tty on startup. As a fallback,
bash will attempt to dereference /proc/self/fd/0 (stdin), and opens the
resulting path with O_RDWR.

If the ebuild sets PROPERTIES="interactive", stdin will be inherited
from the parent emerge process, and may be attached to a terminal device
(/dev/tty1, etc). Attempting to open this device is likely to trigger a
sandbox failure.

Bug: https://bugs.gentoo.org/870310
Signed-off-by: Mike Gilbert <floppym <AT> gentoo.org>

 lib/_emerge/AbstractEbuildProcess.py | 8 --------
 lib/_emerge/SpawnProcess.py          | 7 ++++++-
 2 files changed, 6 insertions(+), 9 deletions(-)

diff --git a/lib/_emerge/AbstractEbuildProcess.py b/lib/_emerge/AbstractEbuildProcess.py
index 6d89d40f0..8712b8ea1 100644
--- a/lib/_emerge/AbstractEbuildProcess.py
+++ b/lib/_emerge/AbstractEbuildProcess.py
@@ -18,7 +18,6 @@ from portage.package.ebuild._ipc.ExitCommand import ExitCommand
 from portage.package.ebuild._ipc.QueryCommand import QueryCommand
 from portage import os
 from portage.util.futures import asyncio
-from portage.util._pty import _create_pty_or_pipe
 from portage.util import apply_secpass_permissions
 
 portage.proxy.lazyimport.lazyimport(
@@ -336,13 +335,6 @@ class AbstractEbuildProcess(SpawnProcess):
 
         self._eerror(textwrap.wrap(msg, 72))
 
-    def _pipe(self, fd_pipes):
-        stdout_pipe = None
-        if not self.background:
-            stdout_pipe = fd_pipes.get(1)
-        got_pty, master_fd, slave_fd = _create_pty_or_pipe(copy_term_size=stdout_pipe)
-        return (master_fd, slave_fd)
-
     def _can_log(self, slave_fd):
         # With sesandbox, logging works through a pty but not through a
         # normal pipe. So, disable logging if ptys are broken.

diff --git a/lib/_emerge/SpawnProcess.py b/lib/_emerge/SpawnProcess.py
index c43b17c12..ed5724c91 100644
--- a/lib/_emerge/SpawnProcess.py
+++ b/lib/_emerge/SpawnProcess.py
@@ -16,6 +16,7 @@ from portage.output import EOutput
 from portage.util import writemsg_level
 from portage.util._async.BuildLogger import BuildLogger
 from portage.util._async.PipeLogger import PipeLogger
+from portage.util._pty import _create_pty_or_pipe
 from portage.util.futures import asyncio
 
 
@@ -217,7 +218,11 @@ class SpawnProcess(SubProcess):
         @type fd_pipes: dict
         @param fd_pipes: pipes from which to copy terminal size if desired.
         """
-        return os.pipe()
+        stdout_pipe = None
+        if not self.background:
+            stdout_pipe = fd_pipes.get(1)
+        got_pty, master_fd, slave_fd = _create_pty_or_pipe(copy_term_size=stdout_pipe)
+        return (master_fd, slave_fd)
 
     def _spawn(self, args, **kwargs):
         spawn_func = portage.process.spawn

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Sam James]

commit:     50854d7a8d7b83e0a598f4e1258d3e063a3fe9e0
Author:     Sheng Yu <syu.os <AT> protonmail <DOT> com>
AuthorDate: Fri Sep 16 12:19:19 2022 +0000
Commit:     Sam James <sam <AT> gentoo <DOT> org>
CommitDate: Tue Sep 20 03:39:22 2022 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=50854d7a

Fix wrong creation of empty binpkg from binhost

Bug: https://bugs.gentoo.org/870283
Signed-off-by: Sheng Yu <syu.os <AT> protonmail.com>
Closes: https://github.com/gentoo/portage/pull/901
Signed-off-by: Sam James <sam <AT> gentoo.org>

 lib/_emerge/Binpkg.py | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/lib/_emerge/Binpkg.py b/lib/_emerge/Binpkg.py
index 949ac8ee7..7ce0da15f 100644
--- a/lib/_emerge/Binpkg.py
+++ b/lib/_emerge/Binpkg.py
@@ -50,6 +50,7 @@ class Binpkg(CompositeTask):
         "_pkg_path",
         "_tree",
         "_verify",
+        "_pkg_allocated_path",
     )
 
     def _writemsg_level(self, msg, level=0, noiselevel=0):
@@ -68,6 +69,7 @@ class Binpkg(CompositeTask):
         self._tree = "bintree"
         self._bintree = self.pkg.root_config.trees[self._tree]
         self._verify = not self.opts.pretend
+        self._pkg_allocated_path = None
 
         # Use realpath like doebuild_environment() does, since we assert
         # that this path is literally identical to PORTAGE_BUILDDIR.
@@ -206,6 +208,7 @@ class Binpkg(CompositeTask):
         # --getbinpkg is enabled.
         if fetcher is not None:
             self._fetched_pkg = fetcher.pkg_path
+            self._pkg_allocated_path = fetcher.pkg_allocated_path
             if self._default_exit(fetcher) != os.EX_OK:
                 self._async_unlock_builddir(returncode=self.returncode)
                 return
@@ -246,7 +249,11 @@ class Binpkg(CompositeTask):
 
         if self._fetched_pkg:
             pkg_path = self._bintree.getname(
-                self._bintree.inject(pkg.cpv, current_pkg_path=self._fetched_pkg),
+                self._bintree.inject(
+                    pkg.cpv,
+                    current_pkg_path=self._fetched_pkg,
+                    allocated_pkg_path=self._pkg_allocated_path,
+                ),
                 allocate_new=False,
             )
         else:

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Sam James]

commit:     7dc66f4f5635ce4abb97b3c6698cda3396cce820
Author:     Sheng Yu <syu.os <AT> protonmail <DOT> com>
AuthorDate: Fri Sep 23 20:32:01 2022 +0000
Commit:     Sam James <sam <AT> gentoo <DOT> org>
CommitDate: Sun Sep 25 01:36:11 2022 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=7dc66f4f

_emerge: Scheduler: Use pkg_allocated_path in Scheduler

We previously forgot to plumb in allocated_pkg_path to
use pkg_allocated_path which meant that Scheduler allocated a new one rather
than the existing one.

Bug: https://bugs.gentoo.org/872392
Signed-off-by: Sheng Yu <syu.os <AT> protonmail.com>
Closes: https://github.com/gentoo/portage/pull/906
Signed-off-by: Sam James <sam <AT> gentoo.org>

 lib/_emerge/Scheduler.py | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/lib/_emerge/Scheduler.py b/lib/_emerge/Scheduler.py
index 9e210f182..356d6ce1f 100644
--- a/lib/_emerge/Scheduler.py
+++ b/lib/_emerge/Scheduler.py
@@ -975,7 +975,11 @@ class Scheduler(PollScheduler):
                         continue
 
                     if fetched:
-                        bintree.inject(x.cpv, current_pkg_path=fetched)
+                        bintree.inject(
+                            x.cpv,
+                            current_pkg_path=fetched,
+                            allocated_pkg_path=fetcher.pkg_allocated_path,
+                        )
 
                     infloc = os.path.join(build_dir_path, "build-info")
                     ensure_dirs(infloc)

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Mike Gilbert]

commit:     6313f4e79868cf8e5aaf49751ab5dd0d5f776761
Author:     David Palao <david.palao <AT> gmail <DOT> com>
AuthorDate: Fri Sep 23 13:47:17 2022 +0000
Commit:     Mike Gilbert <floppym <AT> gentoo <DOT> org>
CommitDate: Sun Sep 25 19:11:55 2022 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=6313f4e7

improvement(style): black issue fixed

Closes: https://github.com/gentoo/portage/pull/897
Signed-off-by: David Palao <david.palao <AT> gmail.com>
Signed-off-by: Mike Gilbert <floppym <AT> gentoo.org>

 lib/_emerge/actions.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/_emerge/actions.py b/lib/_emerge/actions.py
index fbfc561ec..b45a6361a 100644
--- a/lib/_emerge/actions.py
+++ b/lib/_emerge/actions.py
@@ -3512,7 +3512,7 @@ def run_action(emerge_config):
                 mytrees["bintree"].populate(
                     getbinpkgs="--getbinpkg" in emerge_config.opts,
                     getbinpkg_refresh=True,
-                    **kwargs
+                    **kwargs,
                 )
             except ParseError as e:
                 writemsg(

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     0c42cc962e1926ecbdc83d903a2804f9e037f2a9
Author:     Pin-yen Lin <treapking <AT> chromium <DOT> org>
AuthorDate: Fri Nov 25 03:36:26 2022 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Mon Nov 28 00:37:41 2022 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=0c42cc96

Move and rename check_reverse_dependencies

Move the function outside the original scope and rename the function to
_slot_operator_check_reverse_dependencies to accommodate the coding
style.

Signed-off-by: Pin-yen Lin <treapking <AT> chromium.org>
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/depgraph.py | 226 ++++++++++++++++++++++++------------------------
 1 file changed, 113 insertions(+), 113 deletions(-)

diff --git a/lib/_emerge/depgraph.py b/lib/_emerge/depgraph.py
index e556d6616..bb0824324 100644
--- a/lib/_emerge/depgraph.py
+++ b/lib/_emerge/depgraph.py
@@ -2240,6 +2240,111 @@ class depgraph:
 
         return None
 
+    def _slot_operator_check_reverse_dependencies(
+        self, existing_pkg, candidate_pkg, replacement_parent=None
+    ):
+        """
+        Check if candidate_pkg satisfies all of existing_pkg's non-
+        slot operator parents.
+        """
+        debug = "--debug" in self._frozen_config.myopts
+        built_slot_operator_parents = set()
+        for parent, atom in self._dynamic_config._parent_atoms.get(existing_pkg, []):
+            if atom.soname or atom.slot_operator_built:
+                built_slot_operator_parents.add(parent)
+
+        for parent, atom in self._dynamic_config._parent_atoms.get(existing_pkg, []):
+            if isinstance(parent, Package):
+                if parent in built_slot_operator_parents:
+                    if hasattr(atom, "_orig_atom"):
+                        # If atom is the result of virtual expansion, then
+                        # derefrence it to _orig_atom so that it will be correctly
+                        # handled as a built slot operator dependency when
+                        # appropriate (see bug 764764).
+                        atom = atom._orig_atom
+                    # This parent may need to be rebuilt, therefore
+                    # discard its soname and built slot operator
+                    # dependency components which are not necessarily
+                    # relevant.
+                    if atom.soname:
+                        continue
+                    elif atom.package and atom.slot_operator_built:
+                        # This discards the slot/subslot component.
+                        atom = atom.with_slot("=")
+
+                if replacement_parent is not None and (
+                    replacement_parent.slot_atom == parent.slot_atom
+                    or replacement_parent.cpv == parent.cpv
+                ):
+                    # This parent is irrelevant because we intend to
+                    # replace it with replacement_parent.
+                    continue
+
+                if any(
+                    pkg is not parent
+                    and (pkg.slot_atom == parent.slot_atom or pkg.cpv == parent.cpv)
+                    for pkg in self._dynamic_config._package_tracker.match(
+                        parent.root, Atom(parent.cp)
+                    )
+                ):
+                    # This parent may need to be eliminated due to a
+                    # slot conflict,  so its dependencies aren't
+                    # necessarily relevant.
+                    continue
+
+                if not self._too_deep(
+                    parent.depth
+                ) and not self._frozen_config.excluded_pkgs.findAtomForPackage(
+                    parent, modified_use=self._pkg_use_enabled(parent)
+                ):
+                    # Check for common reasons that the parent's
+                    # dependency might be irrelevant.
+                    if self._upgrade_available(parent):
+                        # This parent could be replaced by
+                        # an upgrade (bug 584626).
+                        continue
+                    if parent.installed and self._in_blocker_conflict(parent):
+                        # This parent could be uninstalled in order
+                        # to solve a blocker conflict (bug 612772).
+                        continue
+                    if self._dynamic_config.digraph.has_edge(parent, existing_pkg):
+                        # There is a direct circular dependency between
+                        # parent and existing_pkg. This type of
+                        # relationship tends to prevent updates
+                        # of packages (bug 612874). Since candidate_pkg
+                        # is available, we risk a missed update if we
+                        # don't try to eliminate this parent from the
+                        # graph. Therefore, we give candidate_pkg a
+                        # chance, and assume that it will be masked
+                        # by backtracking if necessary.
+                        continue
+
+            atom_set = InternalPackageSet(initial_atoms=(atom,), allow_repo=True)
+            if not atom_set.findAtomForPackage(
+                candidate_pkg, modified_use=self._pkg_use_enabled(candidate_pkg)
+            ):
+                if debug:
+                    parent_atoms = []
+                    for (
+                        other_parent,
+                        other_atom,
+                    ) in self._dynamic_config._parent_atoms.get(existing_pkg, []):
+                        if other_parent is parent:
+                            parent_atoms.append(other_atom)
+                    msg = (
+                        "",
+                        "",
+                        "_slot_operator_check_reverse_dependencies:",
+                        "   candidate package does not match atom '%s': %s"
+                        % (atom, candidate_pkg),
+                        "   parent: %s" % parent,
+                        "   parent atoms: %s" % " ".join(parent_atoms),
+                        "",
+                    )
+                    writemsg_level("\n".join(msg), noiselevel=-1, level=logging.DEBUG)
+                return False
+        return True
+
     def _slot_operator_update_probe(
         self, dep, new_child_slot=False, slot_conflict=False, autounmask_level=None
     ):
@@ -2274,116 +2379,6 @@ class depgraph:
         want_downgrade = None
         want_downgrade_parent = None
 
-        def check_reverse_dependencies(
-            existing_pkg, candidate_pkg, replacement_parent=None
-        ):
-            """
-            Check if candidate_pkg satisfies all of existing_pkg's non-
-            slot operator parents.
-            """
-            built_slot_operator_parents = set()
-            for parent, atom in self._dynamic_config._parent_atoms.get(
-                existing_pkg, []
-            ):
-                if atom.soname or atom.slot_operator_built:
-                    built_slot_operator_parents.add(parent)
-
-            for parent, atom in self._dynamic_config._parent_atoms.get(
-                existing_pkg, []
-            ):
-                if isinstance(parent, Package):
-                    if parent in built_slot_operator_parents:
-                        if hasattr(atom, "_orig_atom"):
-                            # If atom is the result of virtual expansion, then
-                            # derefrence it to _orig_atom so that it will be correctly
-                            # handled as a built slot operator dependency when
-                            # appropriate (see bug 764764).
-                            atom = atom._orig_atom
-                        # This parent may need to be rebuilt, therefore
-                        # discard its soname and built slot operator
-                        # dependency components which are not necessarily
-                        # relevant.
-                        if atom.soname:
-                            continue
-                        elif atom.package and atom.slot_operator_built:
-                            # This discards the slot/subslot component.
-                            atom = atom.with_slot("=")
-
-                    if replacement_parent is not None and (
-                        replacement_parent.slot_atom == parent.slot_atom
-                        or replacement_parent.cpv == parent.cpv
-                    ):
-                        # This parent is irrelevant because we intend to
-                        # replace it with replacement_parent.
-                        continue
-
-                    if any(
-                        pkg is not parent
-                        and (pkg.slot_atom == parent.slot_atom or pkg.cpv == parent.cpv)
-                        for pkg in self._dynamic_config._package_tracker.match(
-                            parent.root, Atom(parent.cp)
-                        )
-                    ):
-                        # This parent may need to be eliminated due to a
-                        # slot conflict,  so its dependencies aren't
-                        # necessarily relevant.
-                        continue
-
-                    if not self._too_deep(
-                        parent.depth
-                    ) and not self._frozen_config.excluded_pkgs.findAtomForPackage(
-                        parent, modified_use=self._pkg_use_enabled(parent)
-                    ):
-                        # Check for common reasons that the parent's
-                        # dependency might be irrelevant.
-                        if self._upgrade_available(parent):
-                            # This parent could be replaced by
-                            # an upgrade (bug 584626).
-                            continue
-                        if parent.installed and self._in_blocker_conflict(parent):
-                            # This parent could be uninstalled in order
-                            # to solve a blocker conflict (bug 612772).
-                            continue
-                        if self._dynamic_config.digraph.has_edge(parent, existing_pkg):
-                            # There is a direct circular dependency between
-                            # parent and existing_pkg. This type of
-                            # relationship tends to prevent updates
-                            # of packages (bug 612874). Since candidate_pkg
-                            # is available, we risk a missed update if we
-                            # don't try to eliminate this parent from the
-                            # graph. Therefore, we give candidate_pkg a
-                            # chance, and assume that it will be masked
-                            # by backtracking if necessary.
-                            continue
-
-                atom_set = InternalPackageSet(initial_atoms=(atom,), allow_repo=True)
-                if not atom_set.findAtomForPackage(
-                    candidate_pkg, modified_use=self._pkg_use_enabled(candidate_pkg)
-                ):
-                    if debug:
-                        parent_atoms = []
-                        for (
-                            other_parent,
-                            other_atom,
-                        ) in self._dynamic_config._parent_atoms.get(existing_pkg, []):
-                            if other_parent is parent:
-                                parent_atoms.append(other_atom)
-                        msg = (
-                            "",
-                            "",
-                            "check_reverse_dependencies:",
-                            "   candidate package does not match atom '%s': %s"
-                            % (atom, candidate_pkg),
-                            "   parent: %s" % parent,
-                            "   parent atoms: %s" % " ".join(parent_atoms),
-                            "",
-                        )
-                        writemsg_level(
-                            "\n".join(msg), noiselevel=-1, level=logging.DEBUG
-                        )
-                    return False
-            return True
-
         for replacement_parent in self._iter_similar_available(
             dep.parent, dep.parent.slot_atom, autounmask_level=autounmask_level
         ):
@@ -2397,7 +2392,9 @@ class depgraph:
                 if not want_downgrade_parent:
                     continue
 
-            if not check_reverse_dependencies(dep.parent, replacement_parent):
+            if not self._slot_operator_check_reverse_dependencies(
+                dep.parent, replacement_parent
+            ):
                 continue
 
             selected_atoms = None
@@ -2510,8 +2507,11 @@ class depgraph:
                         if atom_not_selected:
                             break
 
-                    if not insignificant and check_reverse_dependencies(
-                        dep.child, pkg, replacement_parent=replacement_parent
+                    if (
+                        not insignificant
+                        and self._slot_operator_check_reverse_dependencies(
+                            dep.child, pkg, replacement_parent=replacement_parent
+                        )
                     ):
 
                         candidate_pkg_atoms.append((pkg, unevaluated_atom or atom))

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     839ab46be1777e5886da28b98b53a462b992c5bf
Author:     Pin-yen Lin <treapking <AT> chromium <DOT> org>
AuthorDate: Fri Nov 25 03:36:27 2022 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Mon Nov 28 00:37:41 2022 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=839ab46b

Add caching to _slot_operator_check_reverse_dependencies

Add lru_cache to speed up the running time of "Calculating
dependencies".

In a ChromeOS use case, this patch decreases the running time from
311s to 197s with almost no memory usage increase.

Bug: https://bugs.gentoo.org/883071
Signed-off-by: Pin-yen Lin <treapking <AT> chromium.org>
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/depgraph.py | 1 +
 1 file changed, 1 insertion(+)

diff --git a/lib/_emerge/depgraph.py b/lib/_emerge/depgraph.py
index bb0824324..d0ea92ad9 100644
--- a/lib/_emerge/depgraph.py
+++ b/lib/_emerge/depgraph.py
@@ -2240,6 +2240,7 @@ class depgraph:
 
         return None
 
+    @functools.lru_cache(maxsize=100)
     def _slot_operator_check_reverse_dependencies(
         self, existing_pkg, candidate_pkg, replacement_parent=None
     ):

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Sam James]

commit:     5b8ae94c1bf5f9d3bd7e7970f9cdb0eb1f11a078
Author:     Sam James <sam <AT> gentoo <DOT> org>
AuthorDate: Tue Jan 10 13:28:45 2023 +0000
Commit:     Sam James <sam <AT> gentoo <DOT> org>
CommitDate: Tue Jan 10 15:12:39 2023 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=5b8ae94c

*/*: run Black again (post-pyupgrade)

Signed-off-by: Sam James <sam <AT> gentoo.org>

 lib/_emerge/actions.py  | 4 +---
 lib/_emerge/depgraph.py | 4 +---
 2 files changed, 2 insertions(+), 6 deletions(-)

diff --git a/lib/_emerge/actions.py b/lib/_emerge/actions.py
index ba6d99d25..874c0312c 100644
--- a/lib/_emerge/actions.py
+++ b/lib/_emerge/actions.py
@@ -1643,9 +1643,7 @@ def _calc_depclean(settings, trees, ldpath_mtimes, myopts, action, args_set, spi
 
                 if debug:
                     writemsg_level(
-                        "Candidates: [{}]\n".format(
-                            ", ".join(f"'{x}'" for x in atoms)
-                        ),
+                        "Candidates: [{}]\n".format(", ".join(f"'{x}'" for x in atoms)),
                         noiselevel=-1,
                         level=logging.DEBUG,
                     )

diff --git a/lib/_emerge/depgraph.py b/lib/_emerge/depgraph.py
index 76d0950ae..a88c7a1fa 100644
--- a/lib/_emerge/depgraph.py
+++ b/lib/_emerge/depgraph.py
@@ -2037,9 +2037,7 @@ class depgraph:
                 f"  package(s) to mask: {str(to_be_masked)}",
                 f"      slot: {slot_atom}",
                 "   parents: {}".format(
-                    ", ".join(
-                        f"({ppkg}, '{atom}')" for ppkg, atom in all_parents
-                    )
+                    ", ".join(f"({ppkg}, '{atom}')" for ppkg, atom in all_parents)
                 ),
                 "",
             ]

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Sam James]

commit:     8bda7308b3f481ea05f0b0c671dfec4024384dfa
Author:     Sheng Yu <syu.os <AT> protonmail <DOT> com>
AuthorDate: Sat Feb 11 03:10:23 2023 +0000
Commit:     Sam James <sam <AT> gentoo <DOT> org>
CommitDate: Fri Feb 17 01:23:14 2023 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=8bda7308

Add warning if missing PATH in binpkg index

Outdated portage does not have this metadata by default.

Bug: https://bugs.gentoo.org/884869
Signed-off-by: Sheng Yu <syu.os <AT> protonmail.com>
Signed-off-by: Sam James <sam <AT> gentoo.org>

 lib/_emerge/BinpkgFetcher.py | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/lib/_emerge/BinpkgFetcher.py b/lib/_emerge/BinpkgFetcher.py
index df99486a0..a18bed759 100644
--- a/lib/_emerge/BinpkgFetcher.py
+++ b/lib/_emerge/BinpkgFetcher.py
@@ -28,6 +28,10 @@ class BinpkgFetcher(CompositeTask):
         instance_key = bintree.dbapi._instance_key(pkg.cpv)
 
         binpkg_path = bintree._remotepkgs[instance_key].get("PATH")
+        if not binpkg_path:
+            raise FileNotFound(
+                f"PATH not found in the binpkg index, the binhost's portage is probably out of date."
+            )
         binpkg_format = get_binpkg_format(binpkg_path)
 
         self.pkg_allocated_path = pkg.root_config.trees["bintree"].getname(

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Sam James]

commit:     0f075baba8e3accc124c9c599c90603e17cdf6a5
Author:     Benjamin Gordon <bmgordon <AT> chromium <DOT> org>
AuthorDate: Tue Feb  4 18:40:13 2020 +0000
Commit:     Sam James <sam <AT> gentoo <DOT> org>
CommitDate: Sat Feb 18 00:00:12 2023 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=0f075bab

emerge: Factor out two functions

Will be used in the next commit.

Reviewed-by: Mike Frysinger <vapier <AT> chromium.org>
Reviewed-by: Chris McDonald <cjmcdonald <AT> chromium.org>
Signed-off-by: Benjamin Gordon <bmgordon <AT> chromium.org>
Signed-off-by: Matt Turner <mattst88 <AT> gentoo.org>
Signed-off-by: Sam James <sam <AT> gentoo.org>

 lib/_emerge/PackageMerge.py | 42 +++++++++++++++++++++++++-----------------
 1 file changed, 25 insertions(+), 17 deletions(-)

diff --git a/lib/_emerge/PackageMerge.py b/lib/_emerge/PackageMerge.py
index 8272511a1..f9e6bf8fe 100644
--- a/lib/_emerge/PackageMerge.py
+++ b/lib/_emerge/PackageMerge.py
@@ -9,13 +9,33 @@ from portage.output import colorize
 class PackageMerge(CompositeTask):
     __slots__ = ("merge", "postinst_failure")
 
+    def _should_show_status(self):
+        return (
+            not self.merge.build_opts.fetchonly
+            and not self.merge.build_opts.pretend
+            and not self.merge.build_opts.buildpkgonly
+        )
+
+    def _make_msg(self, pkg, action_desc, preposition, counter_str):
+        pkg_color = "PKG_MERGE"
+        if pkg.type_name == "binary":
+            pkg_color = "PKG_BINARY_MERGE"
+
+        msg = "{} {}{}".format(
+            action_desc,
+            counter_str,
+            colorize(pkg_color, pkg.cpv + _repo_separator + pkg.repo),
+        )
+
+        if pkg.root_config.settings["ROOT"] != "/":
+            msg += f" {preposition} {pkg.root}"
+
+        return msg
+
     def _start(self):
         self.scheduler = self.merge.scheduler
         pkg = self.merge.pkg
         pkg_count = self.merge.pkg_count
-        pkg_color = "PKG_MERGE"
-        if pkg.type_name == "binary":
-            pkg_color = "PKG_BINARY_MERGE"
 
         if pkg.installed:
             action_desc = "Uninstalling"
@@ -29,20 +49,8 @@ class PackageMerge(CompositeTask):
                 colorize("MERGE_LIST_PROGRESS", str(pkg_count.maxval)),
             )
 
-        msg = "{} {}{}".format(
-            action_desc,
-            counter_str,
-            colorize(pkg_color, pkg.cpv + _repo_separator + pkg.repo),
-        )
-
-        if pkg.root_config.settings["ROOT"] != "/":
-            msg += f" {preposition} {pkg.root}"
-
-        if (
-            not self.merge.build_opts.fetchonly
-            and not self.merge.build_opts.pretend
-            and not self.merge.build_opts.buildpkgonly
-        ):
+        if self._should_show_status():
+            msg = self._make_msg(pkg, action_desc, preposition, counter_str)
             self.merge.statusMessage(msg)
 
         task = self.merge.create_install_task()

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Sam James]

commit:     11b975054dc001b69f0138f1335b2d384ece184b
Author:     Berin Aniesh <berinaniesh <AT> gmail <DOT> com>
AuthorDate: Mon May 22 10:50:02 2023 +0000
Commit:     Sam James <sam <AT> gentoo <DOT> org>
CommitDate: Tue May 23 02:59:13 2023 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=11b97505

actions: drop duplication of --help logic

--help is already handled in `emerge_main()` and the deleted codeblock is not needed /
unreachable.

Signed-off-by: Berin Aniesh <berinaniesh <AT> gmail.com>
Closes: https://github.com/gentoo/portage/pull/1044
Signed-off-by: Sam James <sam <AT> gentoo.org>

 lib/_emerge/actions.py | 3 ---
 1 file changed, 3 deletions(-)

diff --git a/lib/_emerge/actions.py b/lib/_emerge/actions.py
index 0aceb2c44..60199dae6 100644
--- a/lib/_emerge/actions.py
+++ b/lib/_emerge/actions.py
@@ -3522,9 +3522,6 @@ def run_action(emerge_config):
             noiselevel=-1,
         )
         return 0
-    if emerge_config.action == "help":
-        emerge_help()
-        return 0
 
     spinner = stdout_spinner()
     if "candy" in emerge_config.target_config.settings.features:

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Sam James]

commit:     a818381d0faeec7aeefad0894df8b8fefa376dc5
Author:     Sam James <sam <AT> gentoo <DOT> org>
AuthorDate: Wed Jun 14 00:55:44 2023 +0000
Commit:     Sam James <sam <AT> gentoo <DOT> org>
CommitDate: Wed Jun 14 00:55:44 2023 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=a818381d

emerge: depgraph: fix comment typo

Reported by Arfrever at https://github.com/gentoo/portage/commit/4c25c0d7af7ad71fccbfafe1e5019116c691968e#r45851837.

Signed-off-by: Sam James <sam <AT> gentoo.org>

 lib/_emerge/depgraph.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/_emerge/depgraph.py b/lib/_emerge/depgraph.py
index 2c9820da1..861579a6d 100644
--- a/lib/_emerge/depgraph.py
+++ b/lib/_emerge/depgraph.py
@@ -2226,7 +2226,7 @@ class depgraph:
                 if parent in built_slot_operator_parents:
                     if hasattr(atom, "_orig_atom"):
                         # If atom is the result of virtual expansion, then
-                        # derefrence it to _orig_atom so that it will be correctly
+                        # dereference it to _orig_atom so that it will be correctly
                         # handled as a built slot operator dependency when
                         # appropriate (see bug 764764).
                         atom = atom._orig_atom

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Sam James]

commit:     7e8a730b85b77f9495f4999ac173dc726777203f
Author:     Berin Aniesh <berinaniesh <AT> gmail <DOT> com>
AuthorDate: Mon Jun 12 03:56:25 2023 +0000
Commit:     Sam James <sam <AT> gentoo <DOT> org>
CommitDate: Wed Jun 14 05:03:27 2023 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=7e8a730b

depgraph: type annotations

Signed-off-by: Berin Aniesh <berinaniesh <AT> gmail.com>
Closes: https://github.com/gentoo/portage/pull/1054
Signed-off-by: Sam James <sam <AT> gentoo.org>

 lib/_emerge/depgraph.py | 43 +++++++++++++++++++++++++++++++++++++++----
 1 file changed, 39 insertions(+), 4 deletions(-)

diff --git a/lib/_emerge/depgraph.py b/lib/_emerge/depgraph.py
index 861579a6d..94a4fff3e 100644
--- a/lib/_emerge/depgraph.py
+++ b/lib/_emerge/depgraph.py
@@ -100,6 +100,10 @@ from _emerge.resolver.slot_collision import slot_conflict_handler
 from _emerge.resolver.circular_dependency import circular_dependency_handler
 from _emerge.resolver.output import Display, format_unmatched_atom
 
+# Type annotation imports
+from typing import Any, Optional, Dict, List, Tuple, Union
+from _emerge.stdout_spinner import stdout_spinner
+
 # Exposes a depgraph interface to dep_check.
 _dep_check_graph_interface = collections.namedtuple(
     "_dep_check_graph_interface",
@@ -11385,8 +11389,17 @@ def _spinner_stop(spinner):
     portage.writemsg_stdout(f"Dependency resolution took {darkgreen(time_fmt)} s.\n\n")
 
 
-def backtrack_depgraph(settings, trees, myopts, myparams, myaction, myfiles, spinner):
+def backtrack_depgraph(
+    settings: portage.package.ebuild.config.config,
+    trees: portage._trees_dict,
+    myopts: Dict[str, Any],
+    myparams: Dict[str, Union[int, str, bool]],
+    myaction: Optional[str],
+    myfiles: List[str],
+    spinner: stdout_spinner,
+) -> Tuple[Any, depgraph, List[str]]:
     """
+
     Raises PackageSetNotFound if myfiles contains a missing package set.
     """
     _spinner_start(spinner, myopts)
@@ -11398,7 +11411,15 @@ def backtrack_depgraph(settings, trees, myopts, myparams, myaction, myfiles, spi
         _spinner_stop(spinner)
 
 
-def _backtrack_depgraph(settings, trees, myopts, myparams, myaction, myfiles, spinner):
+def _backtrack_depgraph(
+    settings: portage.package.ebuild.config.config,
+    trees: portage._trees_dict,
+    myopts: Dict[str, Any],
+    myparams: Dict[str, Union[int, str, bool]],
+    myaction: Optional[str],
+    myfiles: List[str],
+    spinner: stdout_spinner,
+) -> Tuple[Any, depgraph, List[str]]:
     debug = "--debug" in myopts
     mydepgraph = None
     max_retries = myopts.get("--backtrack", 10)
@@ -11494,7 +11515,14 @@ def _backtrack_depgraph(settings, trees, myopts, myparams, myaction, myfiles, sp
     return (success, mydepgraph, favorites)
 
 
-def resume_depgraph(settings, trees, mtimedb, myopts, myparams, spinner):
+def resume_depgraph(
+    settings: portage.package.ebuild.config.config,
+    trees: portage._trees_dict,
+    mtimedb: Any,
+    myopts: Dict[str, str],
+    myparams: Dict[str, Any],
+    spinner: stdout_spinner,
+):
     """
     Raises PackageSetNotFound if myfiles contains a missing package set.
     """
@@ -11505,7 +11533,14 @@ def resume_depgraph(settings, trees, mtimedb, myopts, myparams, spinner):
         _spinner_stop(spinner)
 
 
-def _resume_depgraph(settings, trees, mtimedb, myopts, myparams, spinner):
+def _resume_depgraph(
+    settings: portage.package.ebuild.config.config,
+    trees: portage._trees_dict,
+    mtimedb: Any,
+    myopts: Dict[str, str],
+    myparams: Dict[str, Any],
+    spinner: stdout_spinner,
+):
     """
     Construct a depgraph for the given resume list. This will raise
     PackageNotFound or depgraph.UnsatisfiedResumeDep when necessary.

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Sam James]

commit:     a5b4eb5a69be4b0abe0c0ee682c40f1881be084f
Author:     Sam James <sam <AT> gentoo <DOT> org>
AuthorDate: Wed Jun 14 05:06:25 2023 +0000
Commit:     Sam James <sam <AT> gentoo <DOT> org>
CommitDate: Wed Jun 14 05:06:25 2023 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=a5b4eb5a

depgraph: drop unnecessary spinner import

Signed-off-by: Sam James <sam <AT> gentoo.org>

 lib/_emerge/depgraph.py | 9 ++++-----
 1 file changed, 4 insertions(+), 5 deletions(-)

diff --git a/lib/_emerge/depgraph.py b/lib/_emerge/depgraph.py
index 94a4fff3e..77133e99c 100644
--- a/lib/_emerge/depgraph.py
+++ b/lib/_emerge/depgraph.py
@@ -102,7 +102,6 @@ from _emerge.resolver.output import Display, format_unmatched_atom
 
 # Type annotation imports
 from typing import Any, Optional, Dict, List, Tuple, Union
-from _emerge.stdout_spinner import stdout_spinner
 
 # Exposes a depgraph interface to dep_check.
 _dep_check_graph_interface = collections.namedtuple(
@@ -11396,7 +11395,7 @@ def backtrack_depgraph(
     myparams: Dict[str, Union[int, str, bool]],
     myaction: Optional[str],
     myfiles: List[str],
-    spinner: stdout_spinner,
+    spinner: "_emerge.stdout_spinner.stdout_spinner",
 ) -> Tuple[Any, depgraph, List[str]]:
     """
 
@@ -11418,7 +11417,7 @@ def _backtrack_depgraph(
     myparams: Dict[str, Union[int, str, bool]],
     myaction: Optional[str],
     myfiles: List[str],
-    spinner: stdout_spinner,
+    spinner: "_emerge.stdout_spinner.stdout_spinner",
 ) -> Tuple[Any, depgraph, List[str]]:
     debug = "--debug" in myopts
     mydepgraph = None
@@ -11521,7 +11520,7 @@ def resume_depgraph(
     mtimedb: Any,
     myopts: Dict[str, str],
     myparams: Dict[str, Any],
-    spinner: stdout_spinner,
+    spinner: "_emerge.stdout_spinner.stdout_spinner",
 ):
     """
     Raises PackageSetNotFound if myfiles contains a missing package set.
@@ -11539,7 +11538,7 @@ def _resume_depgraph(
     mtimedb: Any,
     myopts: Dict[str, str],
     myparams: Dict[str, Any],
-    spinner: stdout_spinner,
+    spinner: "_emerge.stdout_spinner.stdout_spinner",
 ):
     """
     Construct a depgraph for the given resume list. This will raise

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Sam James]

commit:     043ce446b6451fc9e74539634fc2e8e266c8b39e
Author:     Berin Aniesh <berinaniesh <AT> gmail <DOT> com>
AuthorDate: Mon Jun 19 01:32:12 2023 +0000
Commit:     Sam James <sam <AT> gentoo <DOT> org>
CommitDate: Thu Jun 29 08:19:28 2023 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=043ce446

actions.py: Remove unused variable

Signed-off-by: Berin Aniesh <berinaniesh <AT> gmail.com>
Closes: https://github.com/gentoo/portage/pull/1058
Signed-off-by: Sam James <sam <AT> gentoo.org>

 lib/_emerge/actions.py | 3 ---
 1 file changed, 3 deletions(-)

diff --git a/lib/_emerge/actions.py b/lib/_emerge/actions.py
index 7dddd4760..0fee21459 100644
--- a/lib/_emerge/actions.py
+++ b/lib/_emerge/actions.py
@@ -1908,7 +1908,6 @@ def action_info(settings, trees, myopts, myfiles):
     append(
         getportageversion(
             settings["PORTDIR"],
-            None,
             settings.profile_path,
             chost,
             trees[settings["EROOT"]]["vartree"].dbapi,
@@ -2858,7 +2857,6 @@ def get_profile_version(
 
 def getportageversion(
     portdir: str,
-    _unused: None,
     profile: str,
     chost: str,
     vardb: portage.dbapi.vartree.vardbapi,
@@ -3527,7 +3525,6 @@ def run_action(emerge_config):
         writemsg_stdout(
             getportageversion(
                 emerge_config.target_config.settings["PORTDIR"],
-                None,
                 emerge_config.target_config.settings.profile_path,
                 emerge_config.target_config.settings.get("CHOST"),
                 emerge_config.target_config.trees["vartree"].dbapi,

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Sam James]

commit:     e29cb58272b26fb6a4eeae27f4b127f80b34a88c
Author:     Berin Aniesh <berinaniesh <AT> gmail <DOT> com>
AuthorDate: Mon Jun 19 01:30:25 2023 +0000
Commit:     Sam James <sam <AT> gentoo <DOT> org>
CommitDate: Thu Jun 29 08:19:28 2023 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=e29cb582

actions.py: More type annotations

Signed-off-by: Berin Aniesh <berinaniesh <AT> gmail.com>
Signed-off-by: Sam James <sam <AT> gentoo.org>

 lib/_emerge/actions.py | 17 ++++++++++++++---
 1 file changed, 14 insertions(+), 3 deletions(-)

diff --git a/lib/_emerge/actions.py b/lib/_emerge/actions.py
index 01bf82fcb..7dddd4760 100644
--- a/lib/_emerge/actions.py
+++ b/lib/_emerge/actions.py
@@ -105,6 +105,9 @@ from _emerge.UnmergeDepPriority import UnmergeDepPriority
 from _emerge.UseFlagDisplay import pkg_use_display
 from _emerge.UserQuery import UserQuery
 
+# Type annotation imports
+from typing import List
+
 
 def action_build(
     emerge_config,
@@ -2790,7 +2793,7 @@ def relative_profile_path(portdir, abs_profile):
     return profilever
 
 
-def get_libc_version(vardb):
+def get_libc_version(vardb: portage.dbapi.vartree.vardbapi) -> List[str]:
     libcver = []
     libclist = set()
     for atom in expand_new_virt(vardb, portage.const.LIBC_PACKAGE_ATOM):
@@ -2807,7 +2810,9 @@ def get_libc_version(vardb):
     return libcver
 
 
-def get_profile_version(portdir, profile, vardb):
+def get_profile_version(
+    portdir: str, profile: str, vardb: portage.dbapi.vartree.vardbapi
+) -> str:
     profilever = None
     repositories = vardb.settings.repositories
     if profile:
@@ -2851,7 +2856,13 @@ def get_profile_version(portdir, profile, vardb):
     return profilever
 
 
-def getportageversion(portdir, _unused, profile, chost, vardb):
+def getportageversion(
+    portdir: str,
+    _unused: None,
+    profile: str,
+    chost: str,
+    vardb: portage.dbapi.vartree.vardbapi,
+) -> str:
     pythonver = (
         "python"
         f" {sys.version_info[0]}"

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Sam James]

commit:     5de6c63d0779eee0942a294b3a53ee1cce9d6d07
Author:     Berin Aniesh <berinaniesh <AT> gmail <DOT> com>
AuthorDate: Sun Jun 18 03:41:12 2023 +0000
Commit:     Sam James <sam <AT> gentoo <DOT> org>
CommitDate: Thu Jun 29 08:19:27 2023 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=5de6c63d

actions.py: refactor getportageversion() into smaller functions

Signed-off-by: Berin Aniesh <berinaniesh <AT> gmail.com>
Signed-off-by: Sam James <sam <AT> gentoo.org>

 lib/_emerge/actions.py | 51 ++++++++++++++++++++++++++++----------------------
 1 file changed, 29 insertions(+), 22 deletions(-)

diff --git a/lib/_emerge/actions.py b/lib/_emerge/actions.py
index 60199dae6..dae6b6b7b 100644
--- a/lib/_emerge/actions.py
+++ b/lib/_emerge/actions.py
@@ -2789,16 +2789,23 @@ def relative_profile_path(portdir, abs_profile):
         profilever = None
     return profilever
 
+def get_libc_version(vardb):
+    libcver = []
+    libclist = set()
+    for atom in expand_new_virt(vardb, portage.const.LIBC_PACKAGE_ATOM):
+        if not atom.blocker:
+            libclist.update(vardb.match(atom))
+    if libclist:
+        for cpv in sorted(libclist):
+            libc_split = portage.catpkgsplit(cpv)[1:]
+            if libc_split[-1] == "r0":
+                libc_split = libc_split[:-1]
+            libcver.append("-".join(libc_split))
+    else:
+        libcver = ["unavailable"]
+    return libcver
 
-def getportageversion(portdir, _unused, profile, chost, vardb):
-    pythonver = (
-        "python"
-        f" {sys.version_info[0]}"
-        f".{sys.version_info[1]}"
-        f".{sys.version_info[2]}"
-        f"-{sys.version_info[3]}"
-        f"-{sys.version_info[4]}"
-    )
+def get_profile_version(portdir, profile, vardb):
     profilever = None
     repositories = vardb.settings.repositories
     if profile:
@@ -2838,21 +2845,21 @@ def getportageversion(portdir, _unused, profile, chost, vardb):
 
     if profilever is None:
         profilever = "unavailable"
+    
+    return profilever
 
-    libcver = []
-    libclist = set()
-    for atom in expand_new_virt(vardb, portage.const.LIBC_PACKAGE_ATOM):
-        if not atom.blocker:
-            libclist.update(vardb.match(atom))
-    if libclist:
-        for cpv in sorted(libclist):
-            libc_split = portage.catpkgsplit(cpv)[1:]
-            if libc_split[-1] == "r0":
-                libc_split = libc_split[:-1]
-            libcver.append("-".join(libc_split))
-    else:
-        libcver = ["unavailable"]
+def getportageversion(portdir, _unused, profile, chost, vardb):
+    pythonver = (
+        "python"
+        f" {sys.version_info[0]}"
+        f".{sys.version_info[1]}"
+        f".{sys.version_info[2]}"
+        f"-{sys.version_info[3]}"
+        f"-{sys.version_info[4]}"
+    )
 
+    profilever = get_profile_version(portdir, profile, vardb)
+    libcver = get_libc_version(vardb)
     gccver = getgccversion(chost)
     unameout = platform.release() + " " + platform.machine()
 

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Sam James]

commit:     f228252f4b4c3b33ff1e199f55bec9a6a104b80c
Author:     Berin Aniesh <berinaniesh <AT> gmail <DOT> com>
AuthorDate: Mon Jul 10 03:10:07 2023 +0000
Commit:     Sam James <sam <AT> gentoo <DOT> org>
CommitDate: Tue Jul 11 05:00:43 2023 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=f228252f

depgraph.py: fix wrong type annotations

Fixes: 7e8a730b85b77f9495f4999ac173dc726777203f
Bug: https://bugs.gentoo.org/910171
Signed-off-by: Berin Aniesh <berinaniesh <AT> gmail.com>
Closes: https://github.com/gentoo/portage/pull/1067
Signed-off-by: Sam James <sam <AT> gentoo.org>

 lib/_emerge/depgraph.py | 18 +++++++++++-------
 1 file changed, 11 insertions(+), 7 deletions(-)

diff --git a/lib/_emerge/depgraph.py b/lib/_emerge/depgraph.py
index 1aeae6257..72747d7bc 100644
--- a/lib/_emerge/depgraph.py
+++ b/lib/_emerge/depgraph.py
@@ -101,7 +101,11 @@ from _emerge.resolver.circular_dependency import circular_dependency_handler
 from _emerge.resolver.output import Display, format_unmatched_atom
 
 # Type annotation imports
-from typing import Any, Optional, Dict, List, Tuple, Union
+from typing import Any, Optional, Dict, List, Tuple, Union, TYPE_CHECKING
+
+if TYPE_CHECKING:
+    import _emerge.stdout_spinner.stdout_spinner
+
 
 # Exposes a depgraph interface to dep_check.
 _dep_check_graph_interface = collections.namedtuple(
@@ -11409,7 +11413,7 @@ def _spinner_stop(spinner):
 def backtrack_depgraph(
     settings: portage.package.ebuild.config.config,
     trees: portage._trees_dict,
-    myopts: Dict[str, Any],
+    myopts: Dict[str, Union[str, int, bool]],
     myparams: Dict[str, Union[int, str, bool]],
     myaction: Optional[str],
     myfiles: List[str],
@@ -11431,7 +11435,7 @@ def backtrack_depgraph(
 def _backtrack_depgraph(
     settings: portage.package.ebuild.config.config,
     trees: portage._trees_dict,
-    myopts: Dict[str, Any],
+    myopts: Dict[str, Union[str, int, bool]],
     myparams: Dict[str, Union[int, str, bool]],
     myaction: Optional[str],
     myfiles: List[str],
@@ -11536,8 +11540,8 @@ def resume_depgraph(
     settings: portage.package.ebuild.config.config,
     trees: portage._trees_dict,
     mtimedb: Any,
-    myopts: Dict[str, str],
-    myparams: Dict[str, Any],
+    myopts: Dict[str, Union[str, int, bool]],
+    myparams: Dict[str, Union[str, bool]],
     spinner: "_emerge.stdout_spinner.stdout_spinner",
 ):
     """
@@ -11554,8 +11558,8 @@ def _resume_depgraph(
     settings: portage.package.ebuild.config.config,
     trees: portage._trees_dict,
     mtimedb: Any,
-    myopts: Dict[str, str],
-    myparams: Dict[str, Any],
+    myopts: Dict[str, Union[str, int, bool]],
+    myparams: Dict[str, Union[str, bool]],
     spinner: "_emerge.stdout_spinner.stdout_spinner",
 ):
     """

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Sam James]

commit:     f5081b7def54bc9e49aac117ce2b1ebc89379c79
Author:     Sam James <sam <AT> gentoo <DOT> org>
AuthorDate: Fri Sep 15 04:00:32 2023 +0000
Commit:     Sam James <sam <AT> gentoo <DOT> org>
CommitDate: Fri Sep 15 10:36:47 2023 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=f5081b7d

_emerge: use binpkg coloring for failures too

Bug: https://bugs.gentoo.org/914159
Signed-off-by: Sam James <sam <AT> gentoo.org>

 lib/_emerge/Scheduler.py | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

diff --git a/lib/_emerge/Scheduler.py b/lib/_emerge/Scheduler.py
index db9615d406..2d1872feba 100644
--- a/lib/_emerge/Scheduler.py
+++ b/lib/_emerge/Scheduler.py
@@ -1260,12 +1260,14 @@ class Scheduler(PollScheduler):
             and not mod_echo_output
         ):
             for mysettings, key, logentries in self._failed_pkgs_die_msgs:
+                color = "PKG_BINARY_MERGE" if pkg.built else "INFORM"
+
                 root_msg = ""
                 if mysettings["ROOT"] != "/":
                     root_msg = f" merged to {mysettings['ROOT']}"
                 print()
                 printer.einfo(
-                    f"Error messages for package {colorize('INFORM', key)}{root_msg}:"
+                    f"Error messages for package {colorize(color, key)}{root_msg}:"
                 )
                 print()
                 for phase in portage.const.EBUILD_PHASES:
@@ -2001,7 +2003,10 @@ class Scheduler(PollScheduler):
 
     def _failed_pkg_msg(self, failed_pkg, action, preposition):
         pkg = failed_pkg.pkg
-        msg = f"{bad('Failed')} to {action} {colorize('INFORM', pkg.cpv)}"
+
+        color = "PKG_BINARY_MERGE" if failed_pkg.pkg.built else "INFORM"
+
+        msg = f"{bad('Failed')} to {action} {colorize(color, pkg.cpv)}"
         if pkg.root_config.settings["ROOT"] != "/":
             msg += f" {preposition} {pkg.root}"
 

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Sam James]

commit:     7280f1b3dc19ae1be1cc7a930346127b1b7e7686
Author:     Sam James <sam <AT> gentoo <DOT> org>
AuthorDate: Fri Sep 15 03:38:13 2023 +0000
Commit:     Sam James <sam <AT> gentoo <DOT> org>
CommitDate: Fri Sep 15 10:36:47 2023 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=7280f1b3

_emerge: use binpkg coloring for pkg_pretend

For the 'Running pre-merge checks' message, we write:
```
>>> Running pre-merge checks for X
```

X is currently always in green, while the emerge list above might have the atom
X listed in purple if it's a binpkg.

Change X to be colored based on if it's a binpkg or not.

Bug: https://bugs.gentoo.org/914159
Signed-off-by: Sam James <sam <AT> gentoo.org>

 lib/_emerge/Scheduler.py | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/lib/_emerge/Scheduler.py b/lib/_emerge/Scheduler.py
index ece3f27f7c..db9615d406 100644
--- a/lib/_emerge/Scheduler.py
+++ b/lib/_emerge/Scheduler.py
@@ -865,12 +865,13 @@ class Scheduler(PollScheduler):
             if self._terminated_tasks:
                 raise asyncio.CancelledError
 
-            out_str = "Running pre-merge checks for " + colorize("INFORM", x.cpv)
-            self._status_msg(out_str)
-
             root_config = x.root_config
             settings = self._allocate_config(root_config.root)
             settings.setcpv(x)
+
+            color = "PKG_BINARY_MERGE" if x.built else "INFORM"
+            self._status_msg(f"Running pre-merge checks for {colorize(color, x.cpv)}")
+
             if not x.built:
                 # Get required SRC_URI metadata (it's not cached in x.metadata
                 # because some packages have an extremely large SRC_URI value).

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Sam James]

commit:     c6ee4348df65210ee4c052b894cbc9f4f54f6ce8
Author:     Daniel Harding <dharding <AT> living180 <DOT> net>
AuthorDate: Sun Sep 10 06:51:13 2023 +0000
Commit:     Sam James <sam <AT> gentoo <DOT> org>
CommitDate: Thu Sep 21 15:46:00 2023 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=c6ee4348

depgraph: improve reverse dep caching

Avoid applying @functools.lru_cache() to the
depgraph._slot_operator_check_reverse_depencencies() method, as applying
the lru_cache decorator to a method can prevent the class instances from
being garbage collected due to self being stored in the cache as part of
the args key[1].  Instead, set up a per-instance cache in the __init__()
method.

[1] https://stackoverflow.com/q/33672412/3573385

Bug: https://bugs.gentoo.org/883071
Signed-off-by: Daniel Harding <dharding <AT> living180.net>
Signed-off-by: Sam James <sam <AT> gentoo.org>

 lib/_emerge/depgraph.py | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/lib/_emerge/depgraph.py b/lib/_emerge/depgraph.py
index 01a49bcb53..0629acab22 100644
--- a/lib/_emerge/depgraph.py
+++ b/lib/_emerge/depgraph.py
@@ -679,6 +679,12 @@ class depgraph:
 
         self.query = UserQuery(myopts).query
 
+        # Set up a per-instance memoization cache for the
+        # _slot_operator_check_reverse_dependencies() method:
+        self._slot_operator_check_reverse_dependencies = functools.lru_cache(
+            maxsize=100
+        )(self._slot_operator_check_reverse_dependencies)
+
     def _index_binpkgs(self):
         for root in self._frozen_config.trees:
             bindb = self._frozen_config.trees[root]["bintree"].dbapi
@@ -2214,7 +2220,8 @@ class depgraph:
 
         return None
 
-    @functools.lru_cache(maxsize=100)
+    # This method is memoized on a per-instance basis via a decorator applied
+    # in __init__().
     def _slot_operator_check_reverse_dependencies(
         self, existing_pkg, candidate_pkg, replacement_parent=None
     ):

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Sam James]

commit:     53236c55ba8c8106afc80fa113a876ae5784f1ed
Author:     Daniel Harding <dharding <AT> living180 <DOT> net>
AuthorDate: Thu Sep 21 07:42:50 2023 +0000
Commit:     Sam James <sam <AT> gentoo <DOT> org>
CommitDate: Thu Sep 21 15:46:04 2023 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=53236c55

depgraph: increase reverse dep cache size

To avoid blowing out the cache and destroying performance, increase the
maximum size of the memoization cache for the depgraph
_slot_opererator_check_reverse_dependencies() method from 100 to 1000.

Bug: https://bugs.gentoo.org/883071
Signed-off-by: Daniel Harding <dharding <AT> living180.net>
Closes: https://github.com/gentoo/portage/pull/1092
Signed-off-by: Sam James <sam <AT> gentoo.org>

 lib/_emerge/depgraph.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/_emerge/depgraph.py b/lib/_emerge/depgraph.py
index 0629acab22..ef7dd54052 100644
--- a/lib/_emerge/depgraph.py
+++ b/lib/_emerge/depgraph.py
@@ -682,7 +682,7 @@ class depgraph:
         # Set up a per-instance memoization cache for the
         # _slot_operator_check_reverse_dependencies() method:
         self._slot_operator_check_reverse_dependencies = functools.lru_cache(
-            maxsize=100
+            maxsize=1000
         )(self._slot_operator_check_reverse_dependencies)
 
     def _index_binpkgs(self):

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     8e88810e61b6b36bd36e8fb1e5ae76715a72dcd9
Author:     Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Thu Oct  5 04:17:10 2023 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Thu Oct  5 04:18:31 2023 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=8e88810e

EbuildFetcherProcess: Migrate to ForkProcess target parameter

Bug: https://bugs.gentoo.org/915099
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/EbuildFetcher.py | 35 +++++++++++++++++++++++------------
 1 file changed, 23 insertions(+), 12 deletions(-)

diff --git a/lib/_emerge/EbuildFetcher.py b/lib/_emerge/EbuildFetcher.py
index 7e28be4938..edabe54456 100644
--- a/lib/_emerge/EbuildFetcher.py
+++ b/lib/_emerge/EbuildFetcher.py
@@ -1,7 +1,8 @@
-# Copyright 1999-2020 Gentoo Authors
+# Copyright 1999-2023 Gentoo Authors
 # Distributed under the terms of the GNU General Public License v2
 
 import copy
+import functools
 import io
 import sys
 
@@ -246,6 +247,14 @@ class _EbuildFetcherProcess(ForkProcess):
 
         self._settings = settings
         self.log_filter_file = settings.get("PORTAGE_LOG_FILTER_FILE_CMD")
+        self.target = functools.partial(
+            self._target,
+            self._settings,
+            self._get_digests,
+            self._get_manifest,
+            self._uri_map,
+            self.fetchonly,
+        )
         ForkProcess._start(self)
 
         # Free settings now since it's no longer needed in
@@ -254,25 +263,27 @@ class _EbuildFetcherProcess(ForkProcess):
         settings = None
         self._settings = None
 
-    def _run(self):
+    @staticmethod
+    def _target(settings, get_digests, get_manifest, uri_map, fetchonly):
+        """
+        TODO: Make all arguments picklable for the multiprocessing spawn start method.
+        """
         # Force consistent color output, in case we are capturing fetch
         # output through a normal pipe due to unavailability of ptys.
-        portage.output.havecolor = self._settings.get("NOCOLOR") not in ("yes", "true")
+        portage.output.havecolor = settings.get("NOCOLOR") not in ("yes", "true")
 
         # For userfetch, drop privileges for the entire fetch call, in
         # order to handle DISTDIR on NFS with root_squash for bug 601252.
-        if _want_userfetch(self._settings):
-            _drop_privs_userfetch(self._settings)
+        if _want_userfetch(settings):
+            _drop_privs_userfetch(settings)
 
         rval = 1
-        allow_missing = (
-            self._get_manifest().allow_missing or "digest" in self._settings.features
-        )
+        allow_missing = get_manifest().allow_missing or "digest" in settings.features
         if fetch(
-            self._uri_map,
-            self._settings,
-            fetchonly=self.fetchonly,
-            digests=copy.deepcopy(self._get_digests()),
+            uri_map,
+            settings,
+            fetchonly=fetchonly,
+            digests=copy.deepcopy(get_digests()),
             allow_missing_digests=allow_missing,
         ):
             rval = os.EX_OK

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     5f5012803b2b647f6b7528f48579a8c27c8a3fd7
Author:     Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Sat Oct 14 19:54:39 2023 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Sat Oct 14 19:55:49 2023 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=5f501280

depgraph: Skip _eliminate_rebuilds for --emptytree

Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/depgraph.py | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/lib/_emerge/depgraph.py b/lib/_emerge/depgraph.py
index 4e4452dad1..deab51718b 100644
--- a/lib/_emerge/depgraph.py
+++ b/lib/_emerge/depgraph.py
@@ -3605,6 +3605,9 @@ class depgraph:
         careful to obey the user's wishes if they have explicitly requested
         for a package to be rebuilt or reinstalled for some reason.
         """
+        if "empty" in self._dynamic_config.myparams:
+            return False
+
         modified = False
         selective = "selective" in self._dynamic_config.myparams
         for root, atom in self._dynamic_config._slot_operator_replace_installed:

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     935d1432b2ea42fc0a49b9e3368a1766a673e1a9
Author:     Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Sat Oct 21 01:31:44 2023 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Sun Oct 22 04:36:43 2023 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=935d1432

_emerge: Fix NameError 'pkg' bug 915551

Use (pkg.cpv, pkg.root) to lookup package
instances corresponding to die messages.

Bug: https://bugs.gentoo.org/915551
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/Scheduler.py | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/lib/_emerge/Scheduler.py b/lib/_emerge/Scheduler.py
index 2d1872feba..c7e465731a 100644
--- a/lib/_emerge/Scheduler.py
+++ b/lib/_emerge/Scheduler.py
@@ -1259,8 +1259,13 @@ class Scheduler(PollScheduler):
             and self._failed_pkgs_die_msgs
             and not mod_echo_output
         ):
+            failed_pkg_map = {}
+            for pkg in self._failed_pkgs_all:
+                failed_pkg_map[(pkg.cpv, pkg.root)] = pkg
+
             for mysettings, key, logentries in self._failed_pkgs_die_msgs:
-                color = "PKG_BINARY_MERGE" if pkg.built else "INFORM"
+                pkg = failed_pkg_map.get((key, mysettings["EROOT"]))
+                color = "PKG_BINARY_MERGE" if pkg and pkg.built else "INFORM"
 
                 root_msg = ""
                 if mysettings["ROOT"] != "/":

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Sam James]

commit:     bb82666b48e18f448661a1a8bf6a39b773cc4b1c
Author:     Eli Schwartz <eschwartz93 <AT> gmail <DOT> com>
AuthorDate: Sun Oct 22 16:53:23 2023 +0000
Commit:     Sam James <sam <AT> gentoo <DOT> org>
CommitDate: Sun Oct 22 22:09:32 2023 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=bb82666b

emerge: avoid spamming too much info about unused binpkgs

Specifically:

- the binhost might have multiple binpkgs for the same cpv, same USE
  flags, but with different build ids, in which case the same line is
  repeated twice

- many packages can be participating in the recursive deptree, but not
  get selected for actual merging, and may not even be installed at all;
  these were still reported as mismatched-USE although the packages
  themselves were not shown in show_merge_list

Both cases are useless noise. Prune them before printing a status
report.

Before:
```
Local copy of remote index is up-to-date and will be used.

Local copy of remote index is up-to-date and will be used.

These are the packages that would be merged, in order:

Calculating dependencies... done!
Dependency resolution took 17.29 s.

[binary     U  ] sys-apps/hwdata-0.374-1 [0.373]
[binary  r  U  ] dev-libs/libsodium-1.0.19-r1-1 [1.0.18_p20220618]
[binary     U  ] net-dns/libidn2-2.3.4-r1-1 [2.3.4]
[ebuild   R    ] dev-python/fs-2.4.16  PYTHON_TARGETS="(-python3_12%*)"
[ebuild     U  ] sys-apps/iproute2-6.5.0 [6.4.0]
[binary     U  ] media-video/ffmpeg-6.0-r9-2 [6.0-r6]
[ebuild  rR    ] app-editors/vim-9.0.1627

The following packages are causing rebuilds:

  (dev-libs/libsodium-1.0.19-r1-1:0/26::gentoo, binary scheduled for merge) causes rebuilds for:
    (app-editors/vim-9.0.1627:0/0::gentoo, ebuild scheduled for merge)

!!! The following binary packages have been ignored due to non matching USE:

    =app-shells/bash-5.1_p16-r6 -plugins
    =sys-apps/util-linux-2.38.1-r2 -caps split-usr
    =sys-apps/util-linux-2.38.1-r2 -caps split-usr
    =sys-process/procps-3.3.17-r2 split-usr
    =sys-process/procps-3.3.17-r2 split-usr -systemd
    =app-editors/vim-9.0.1627 -python
    =app-editors/vim-9.0.1627 -python
    =dev-lang/python-3.11.5 -lto -pgo
    =dev-lang/python-3.11.5 -lto -pgo
    =dev-lang/python-3.11.5 -lto -pgo
    =media-fonts/noto-20230831 X
    =net-misc/networkmanager-1.42.6-r2 bluetooth gtk-doc ppp
    =net-misc/networkmanager-1.42.6-r2 bluetooth gtk-doc ppp
    =net-misc/networkmanager-1.42.6-r2 bluetooth gtk-doc ppp
    =net-misc/networkmanager-1.42.6-r2 bluetooth gtk-doc ppp
    =sys-devel/gdb-13.2-r1 -lzma nls -source-highlight -xxhash -zstd
    =x11-base/xorg-server-21.1.8-r2 -xvfb
    =media-libs/libsdl2-2.28.3 gles2 pulseaudio wayland
    =media-libs/libsdl2-2.28.3 gles2 pipewire pulseaudio wayland
    =sys-devel/clang-16.0.6 abi_x86_32
    =sys-devel/clang-16.0.6 -xml
    =app-arch/zstd-1.5.5 split-usr
    =app-arch/zstd-1.5.5 split-usr
    =sys-apps/sandbox-2.38 -abi_x86_32
    =dev-python/certifi-3021.3.16-r3 -python_targets_python3_12
    =dev-python/certifi-3021.3.16-r3 -python_targets_python3_12
    =dev-libs/libffi-3.4.4-r1 abi_x86_32
    =sys-libs/ncurses-6.4_p20230401 abi_x86_32 -gpm split-usr
    =sys-libs/readline-8.1_p2-r1 split-usr
    =dev-db/sqlite-3.43.2 -icu
    =dev-libs/icu-73.2 abi_x86_32
    =sys-libs/gpm-1.20.7-r5 abi_x86_32 split-usr
    =app-crypt/pinentry-1.2.1-r3 keyring qt5 wayland
    =dev-libs/glib-2.76.4 sysprof
    =sys-apps/systemd-253.11 split-usr
    =dev-libs/libxml2-2.11.5 abi_x86_32
    =dev-libs/libxml2-2.11.5 -icu
    =media-libs/mesa-23.1.8 video_cards_intel video_cards_nouveau
    =media-libs/mesa-23.1.8 video_cards_intel video_cards_nouveau
    =media-libs/mesa-23.1.8 video_cards_intel video_cards_nouveau
    =media-libs/mesa-23.1.8 video_cards_intel video_cards_nouveau
    =media-libs/mesa-23.1.8 video_cards_intel video_cards_nouveau
    =x11-libs/libxkbcommon-1.5.0 wayland
    =x11-libs/libxkbcommon-1.5.0 wayland
    =x11-libs/libxkbcommon-1.5.0 wayland
    =sys-devel/llvm-16.0.6 abi_x86_32
    =sys-devel/llvm-16.0.6 -xml
    =x11-misc/xdg-utils-1.1.3_p20210805-r1 gnome
    =dev-libs/libpcre2-10.42-r1 split-usr
    =sys-auth/polkit-122-r1 kde
    =net-libs/nghttp2-1.57.0 -systemd -xml
    =sys-libs/glibc-2.37-r7 -multilib -stack-realign -systemd
    =app-crypt/gpgme-1.21.0 -qt5
    =sys-devel/clang-runtime-16.0.6 -abi_x86_32
    =media-sound/mpg123-1.31.3-r1 pulseaudio
    =dev-libs/libpcre-8.45-r2 split-usr
    =net-wireless/wpa_supplicant-2.10-r3 qt5
    =media-libs/gst-plugins-base-1.20.6 wayland
    =media-libs/gst-plugins-base-1.20.6 theora wayland
    =media-libs/gst-plugins-base-1.20.6 wayland
    =net-libs/libproxy-0.5.3 kde
    =gnome-base/gvfs-1.50.6 gnome-online-accounts keyring
    =gnome-base/gvfs-1.50.6 gnome-online-accounts keyring
    =sys-libs/libblockdev-2.28-r1 introspection
    =dev-libs/libdbusmenu-16.04.0-r2 gtk
    =dev-libs/libdbusmenu-16.04.0-r2 gtk
    =dev-python/css-parser-1.0.9 -python_targets_python3_12
    =dev-qt/qtbase-6.5.2-r2 nls wayland
    =dev-qt/qtwebengine-6.5.2-r1 pulseaudio qml screencast
    =dev-qt/qtwebengine-6.5.2-r1 pulseaudio qml screencast
    =dev-qt/qtwebchannel-6.5.2-r2 qml
    =app-arch/unzip-6.0_p27-r1 -natspec
    =app-text/liblangtag-0.6.4 introspection
    =app-text/liblangtag-0.6.4 introspection
    =sys-apps/groff-1.22.4 -X
    =virtual/tmpfiles-0-r5 -systemd
    =sys-apps/systemd-utils-253.11-r1 split-usr
    =virtual/libcrypt-2-r1 -abi_x86_32
    =virtual/udev-217-r7 -systemd
    =net-misc/openssh-9.4_p1-r1 -X
    =dev-libs/libusb-1.0.26 split-usr
    =media-video/pipewire-0.3.80 bluetooth sound-server
    =media-video/pipewire-0.3.80 bluetooth sound-server
    =x11-misc/mate-notification-daemon-1.26.0-r1 wayland
    =x11-misc/mate-notification-daemon-1.26.0-r1 wayland
    =x11-misc/mate-notification-daemon-1.26.0-r1 wayland
    =kde-plasma/plasma-workspace-5.27.8-r1 screencast
    =kde-plasma/plasma-workspace-5.27.8-r1 screencast
    =kde-plasma/plasma-workspace-5.27.8-r1 screencast
    =sys-devel/llvm-15.0.7-r3 abi_x86_32

NOTE: The --binpkg-respect-use=n option will prevent emerge
      from ignoring these binary packages if possible.
      Using --binpkg-respect-use=y will silence this warning.

Would you like to merge these packages? [Yes/No]
```

After:
```
Local copy of remote index is up-to-date and will be used.

Local copy of remote index is up-to-date and will be used.

These are the packages that would be merged, in order:

Calculating dependencies... done!
Dependency resolution took 17.25 s.

[binary     U  ] sys-apps/hwdata-0.374-1 [0.373]
[binary  r  U  ] dev-libs/libsodium-1.0.19-r1-1 [1.0.18_p20220618]
[binary     U  ] net-dns/libidn2-2.3.4-r1-1 [2.3.4]
[ebuild   R    ] dev-python/fs-2.4.16  PYTHON_TARGETS="(-python3_12%*)"
[ebuild     U  ] sys-apps/iproute2-6.5.0 [6.4.0]
[binary     U  ] media-video/ffmpeg-6.0-r9-2 [6.0-r6]
[ebuild  rR    ] app-editors/vim-9.0.1627

The following packages are causing rebuilds:

  (dev-libs/libsodium-1.0.19-r1-1:0/26::gentoo, binary scheduled for merge) causes rebuilds for:
    (app-editors/vim-9.0.1627:0/0::gentoo, ebuild scheduled for merge)

!!! The following binary packages have been ignored due to non matching USE:

    =sys-apps/iproute2-6.5.0 split-usr
    =app-editors/vim-9.0.1627 -python

NOTE: The --binpkg-respect-use=n option will prevent emerge
      from ignoring these binary packages if possible.
      Using --binpkg-respect-use=y will silence this warning.

Would you like to merge these packages? [Yes/No]
```

Signed-off-by: Eli Schwartz <eschwartz93 <AT> gmail.com>
Closes: https://github.com/gentoo/portage/pull/1148
Signed-off-by: Sam James <sam <AT> gentoo.org>

 lib/_emerge/depgraph.py | 35 ++++++++++++++++++++++++-----------
 1 file changed, 24 insertions(+), 11 deletions(-)

diff --git a/lib/_emerge/depgraph.py b/lib/_emerge/depgraph.py
index deab51718b..85845dc1e2 100644
--- a/lib/_emerge/depgraph.py
+++ b/lib/_emerge/depgraph.py
@@ -1232,25 +1232,38 @@ class depgraph:
             self._show_ignored_binaries_changed_deps(ignored_binaries["changed_deps"])
 
     def _show_ignored_binaries_respect_use(self, respect_use):
-        writemsg(
-            "\n!!! The following binary packages have been ignored "
-            + "due to non matching USE:\n\n",
-            noiselevel=-1,
-        )
-
+        seen = {}
+        messages = []
+        merging = {pkg.cpv for pkg in self._dynamic_config._displayed_list}
         for pkg, flags in respect_use.items():
+            # Don't include recursive deps which aren't in the merge list anyway.
+            if pkg.cpv not in merging:
+                continue
             flag_display = []
             for flag in sorted(flags):
                 if flag not in pkg.use.enabled:
                     flag = "-" + flag
                 flag_display.append(flag)
             flag_display = " ".join(flag_display)
-            # The user can paste this line into package.use
-            writemsg(f"    ={pkg.cpv} {flag_display}", noiselevel=-1)
-            if pkg.root_config.settings["ROOT"] != "/":
-                writemsg(f" # for {pkg.root}", noiselevel=-1)
-            writemsg("\n", noiselevel=-1)
+            # We don't want to list the same USE flags for multiple build IDs
+            if pkg.cpv not in seen or flag_display not in seen[pkg.cpv]:
+                seen.setdefault(pkg.cpv, set()).add(flag_display)
+                # The user can paste this line into package.use
+                messages.append(f"    ={pkg.cpv} {flag_display}")
+                if pkg.root_config.settings["ROOT"] != "/":
+                    messages.append(f" # for {pkg.root}")
+                messages.append("\n")
+
+        if not messages:
+            return
 
+        writemsg(
+            "\n!!! The following binary packages have been ignored "
+            + "due to non matching USE:\n\n",
+            noiselevel=-1,
+        )
+        for line in messages:
+            writemsg(line, noiselevel=-1)
         msg = [
             "",
             "NOTE: The --binpkg-respect-use=n option will prevent emerge",

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     6963901bebf5403be5042fcbc54f4c950608204e
Author:     Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Sun Oct 22 22:50:23 2023 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Sun Oct 22 22:51:28 2023 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=6963901b

BinpkgVerifier: multiprocessing spawn compat

Solves this error with the multiprocessing spawn start method:

TypeError: cannot pickle 'generator' object

Bug: https://bugs.gentoo.org/916141
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/BinpkgVerifier.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/lib/_emerge/BinpkgVerifier.py b/lib/_emerge/BinpkgVerifier.py
index 58691e68b6..a7917453aa 100644
--- a/lib/_emerge/BinpkgVerifier.py
+++ b/lib/_emerge/BinpkgVerifier.py
@@ -1,4 +1,4 @@
-# Copyright 1999-2019 Gentoo Authors
+# Copyright 1999-2023 Gentoo Authors
 # Distributed under the terms of the GNU General Public License v2
 
 import errno
@@ -59,7 +59,7 @@ class BinpkgVerifier(CompositeTask):
         self._start_task(
             FileDigester(
                 file_path=self._pkg_path,
-                hash_names=(k for k in digests if k != "size"),
+                hash_names=[k for k in digests if k != "size"],
                 background=self.background,
                 logfile=self.logfile,
                 scheduler=self.scheduler,

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Sam James]

commit:     a6853d5493b7bed37e60c2e3b7536b209500ba3f
Author:     Sam James <sam <AT> gentoo <DOT> org>
AuthorDate: Thu Nov  2 13:59:05 2023 +0000
Commit:     Sam James <sam <AT> gentoo <DOT> org>
CommitDate: Mon Nov  6 15:57:56 2023 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=a6853d54

emerge: account for EROOT in _show_ignored_binaries_respect_use

Zac suggested this when reviewing the fix for bug #916336.

Bug: https://bugs.gentoo.org/916336
Signed-off-by: Sam James <sam <AT> gentoo.org>

 lib/_emerge/depgraph.py | 13 +++++++++----
 1 file changed, 9 insertions(+), 4 deletions(-)

diff --git a/lib/_emerge/depgraph.py b/lib/_emerge/depgraph.py
index 0717e0429f..d1fed0d652 100644
--- a/lib/_emerge/depgraph.py
+++ b/lib/_emerge/depgraph.py
@@ -1235,23 +1235,28 @@ class depgraph:
         seen = {}
         messages = []
         merging = {
-            pkg.cpv
+            (pkg.root, pkg.cpv)
             for pkg in self._dynamic_config._displayed_list
             if isinstance(pkg, Package)
         }
         for pkg, flags in respect_use.items():
             # Don't include recursive deps which aren't in the merge list anyway.
-            if pkg.cpv not in merging:
+            if (pkg.root, pkg.cpv) not in merging:
                 continue
+
             flag_display = []
             for flag in sorted(flags):
                 if flag not in pkg.use.enabled:
                     flag = "-" + flag
                 flag_display.append(flag)
             flag_display = " ".join(flag_display)
+
             # We don't want to list the same USE flags for multiple build IDs
-            if pkg.cpv not in seen or flag_display not in seen[pkg.cpv]:
-                seen.setdefault(pkg.cpv, set()).add(flag_display)
+            seen.setdefault(pkg.root, dict())
+            if (pkg.root, pkg.cpv) not in seen or flag_display not in seen[pkg.root][
+                pkg.cpv
+            ]:
+                seen[pkg.root].setdefault(pkg.cpv, set()).add(flag_display)
                 # The user can paste this line into package.use
                 messages.append(f"    ={pkg.cpv} {flag_display}")
                 if pkg.root_config.settings["ROOT"] != "/":

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     cfb856419711784fce2c5d54781338729cb94287
Author:     Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Fri Nov 10 05:34:53 2023 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Fri Nov 10 05:47:15 2023 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=cfb85641

emerge: Use finally instead of atexit for xtermTitleReset

The xtermTitleReset function will trigger a RuntimeError
when called via an atexit hook in python 3.12, since it
forks to run PROMPT_COMMAND. Avoid this problem by calling
it in a finally block instead of atexit.

Bug: https://bugs.gentoo.org/917033
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/actions.py | 10 ----------
 lib/_emerge/main.py    | 10 ++++++++++
 2 files changed, 10 insertions(+), 10 deletions(-)

diff --git a/lib/_emerge/actions.py b/lib/_emerge/actions.py
index 07d477a046..37264114e0 100644
--- a/lib/_emerge/actions.py
+++ b/lib/_emerge/actions.py
@@ -54,7 +54,6 @@ from portage.output import (
     darkgreen,
     red,
     xtermTitle,
-    xtermTitleReset,
 )
 
 good = create_color_func("GOOD")
@@ -3828,15 +3827,6 @@ def run_action(emerge_config):
 
     signal.signal(signal.SIGTERM, emergeexitsig)
 
-    def emergeexit():
-        """This gets out final log message in before we quit."""
-        if "--pretend" not in emerge_config.opts:
-            emergelog(xterm_titles, " *** terminating.")
-        if xterm_titles:
-            xtermTitleReset()
-
-    portage.atexit_register(emergeexit)
-
     if emerge_config.action in ("config", "metadata", "regen", "sync"):
         if "--pretend" in emerge_config.opts:
             sys.stderr.write(

diff --git a/lib/_emerge/main.py b/lib/_emerge/main.py
index d92f069901..6a52404c88 100644
--- a/lib/_emerge/main.py
+++ b/lib/_emerge/main.py
@@ -12,9 +12,11 @@ portage.proxy.lazyimport.lazyimport(
     globals(),
     "logging",
     "portage.dep:Atom",
+    "portage.output:xtermTitleReset",
     "portage.util:writemsg_level",
     "textwrap",
     "_emerge.actions:load_emerge_config,run_action," + "validate_ebuild_environment",
+    "_emerge.emergelog:emergelog",
     "_emerge.help:emerge_help",
     "_emerge.is_valid_package_atom:insert_category_into_atom",
 )
@@ -1310,3 +1312,11 @@ def emerge_main(args: Optional[list[str]] = None):
             if "porttree" in x.lazy_items:
                 continue
             x["porttree"].dbapi.close_caches()
+
+        # This gets out final log message in before we quit.
+        xterm_titles = "notitles" not in emerge_config.target_config.settings.features
+        if "--pretend" not in emerge_config.opts:
+            emergelog(xterm_titles, " *** terminating.")
+
+        if xterm_titles:
+            xtermTitleReset()

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Sam James]

commit:     37ee6ef700f4512999dc2e3bd8fcd8a8d9d36315
Author:     Sam James <sam <AT> gentoo <DOT> org>
AuthorDate: Sat Nov 11 03:03:29 2023 +0000
Commit:     Sam James <sam <AT> gentoo <DOT> org>
CommitDate: Sat Nov 11 07:23:08 2023 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=37ee6ef7

_emerge: fix _eval_visiblity typo

Signed-off-by: Sam James <sam <AT> gentoo.org>

 lib/_emerge/Package.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/lib/_emerge/Package.py b/lib/_emerge/Package.py
index 873edbf81d..79011380d5 100644
--- a/lib/_emerge/Package.py
+++ b/lib/_emerge/Package.py
@@ -222,7 +222,7 @@ class Package(Task):
     @property
     def visible(self):
         if self._visible is None:
-            self._visible = self._eval_visiblity(self.masks)
+            self._visible = self._eval_visibility(self.masks)
         return self._visible
 
     @property
@@ -473,7 +473,7 @@ class Package(Task):
 
         return masks
 
-    def _eval_visiblity(self, masks):
+    def _eval_visibility(self, masks):
         if masks is not False:
             if "EAPI.unsupported" in masks:
                 return False

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     bf61ab8c1ae0eb3311fea20f1bdc5cf7d69fd586
Author:     Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Mon Nov 13 23:17:26 2023 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Mon Nov 13 23:17:30 2023 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=bf61ab8c

_emerge: Fix _failed_pkg cpv AttributeError

Bug: https://bugs.gentoo.org/917315
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/Scheduler.py | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/lib/_emerge/Scheduler.py b/lib/_emerge/Scheduler.py
index c7e465731a..ceff2be0f7 100644
--- a/lib/_emerge/Scheduler.py
+++ b/lib/_emerge/Scheduler.py
@@ -1,4 +1,4 @@
-# Copyright 1999-2021 Gentoo Authors
+# Copyright 1999-2023 Gentoo Authors
 # Distributed under the terms of the GNU General Public License v2
 
 from collections import deque
@@ -1260,7 +1260,8 @@ class Scheduler(PollScheduler):
             and not mod_echo_output
         ):
             failed_pkg_map = {}
-            for pkg in self._failed_pkgs_all:
+            for failed_pkg in self._failed_pkgs_all:
+                pkg = failed_pkg.pkg
                 failed_pkg_map[(pkg.cpv, pkg.root)] = pkg
 
             for mysettings, key, logentries in self._failed_pkgs_die_msgs:

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     c8e599af298a500334e6566d1518b35a531ec99e
Author:     Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Sun Nov 19 19:52:01 2023 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Sat Nov 25 06:31:31 2023 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=c8e599af

_run_pkg_pretend: Refer to emerge-fetch.log for binpkg parallel-fetch

If there is a parallel-fetch task fetching a binpkg in the
background, refer to emerge-fetch.log for status updates,
just like the Binpkg class does prior to pkg_setup.

Bug: https://bugs.gentoo.org/760893
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/Scheduler.py | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/lib/_emerge/Scheduler.py b/lib/_emerge/Scheduler.py
index ceff2be0f7..e1179e0f13 100644
--- a/lib/_emerge/Scheduler.py
+++ b/lib/_emerge/Scheduler.py
@@ -948,6 +948,16 @@ class Scheduler(PollScheduler):
                             # handles fetch, verification, and the
                             # bintree.inject call which moves the file.
                             fetched = fetcher.pkg_path
+                        else:
+                            msg = (
+                                "Fetching in the background:",
+                                fetcher.pkg_path,
+                                "To view fetch progress, run in another terminal:",
+                                f"tail -f {self._fetch_log}",
+                            )
+                            out = portage.output.EOutput()
+                            for l in msg:
+                                out.einfo(l)
                         if await fetcher.async_wait() != os.EX_OK:
                             failures += 1
                             self._record_pkg_failure(x, settings, fetcher.returncode)

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     9b711229ef913e8772e9a3a1cce744c87ee468cf
Author:     Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Mon Nov 27 01:55:04 2023 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Tue Nov 28 22:47:53 2023 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=9b711229

Cancel prefetchers that have not started yet

Cancel prefetchers that	have not started yet. This will cause the
parallel-fetch SequentialTaskQueue to simply discard them, and
a regular fetcher will be instantiated immediately. This prevents
errors when accessing the pkg_path attribute of BinpkgPrefetcher
instances that have not started yet.

Bug: https://bugs.gentoo.org/918636
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/Binpkg.py        | 6 +++++-
 lib/_emerge/CompositeTask.py | 8 +++++++-
 lib/_emerge/EbuildBuild.py   | 4 ++++
 lib/_emerge/Scheduler.py     | 8 +++++++-
 4 files changed, 23 insertions(+), 3 deletions(-)

diff --git a/lib/_emerge/Binpkg.py b/lib/_emerge/Binpkg.py
index fac8722064..9b1036538a 100644
--- a/lib/_emerge/Binpkg.py
+++ b/lib/_emerge/Binpkg.py
@@ -1,4 +1,4 @@
-# Copyright 1999-2021 Gentoo Authors
+# Copyright 1999-2023 Gentoo Authors
 # Distributed under the terms of the GNU General Public License v2
 
 import functools
@@ -110,6 +110,10 @@ class Binpkg(CompositeTask):
         # use the scheduler and fetcher methods to
         # synchronize with the fetcher.
         prefetcher = self.prefetcher
+        if prefetcher is not None and not prefetcher.isAlive():
+            # Cancel it because it hasn't started yet.
+            prefetcher.cancel()
+            self.prefetcher = prefetcher = None
         if prefetcher is None:
             pass
         elif prefetcher.isAlive() and prefetcher.poll() is None:

diff --git a/lib/_emerge/CompositeTask.py b/lib/_emerge/CompositeTask.py
index a0203e71f3..50c1b8c5f3 100644
--- a/lib/_emerge/CompositeTask.py
+++ b/lib/_emerge/CompositeTask.py
@@ -1,4 +1,4 @@
-# Copyright 1999-2018 Gentoo Foundation
+# Copyright 1999-2023 Gentoo Authors
 # Distributed under the terms of the GNU General Public License v2
 
 from _emerge.AsynchronousTask import AsynchronousTask
@@ -10,6 +10,12 @@ class CompositeTask(AsynchronousTask):
 
     _TASK_QUEUED = -1
 
+    def isAlive(self):
+        """
+        Returns True if started and returncode has not been set.
+        """
+        return self.returncode is None and self._current_task is not None
+
     def _cancel(self):
         if self._current_task is not None:
             if self._current_task is self._TASK_QUEUED:

diff --git a/lib/_emerge/EbuildBuild.py b/lib/_emerge/EbuildBuild.py
index 079a67ffbe..81cbfdc085 100644
--- a/lib/_emerge/EbuildBuild.py
+++ b/lib/_emerge/EbuildBuild.py
@@ -96,6 +96,10 @@ class EbuildBuild(CompositeTask):
             return
 
         prefetcher = self.prefetcher
+        if prefetcher is not None and not prefetcher.isAlive():
+            # Cancel it because it hasn't started yet.
+            prefetcher.cancel()
+            self.prefetcher = prefetcher = None
         if prefetcher is None:
             pass
         elif prefetcher.isAlive() and prefetcher.poll() is None:

diff --git a/lib/_emerge/Scheduler.py b/lib/_emerge/Scheduler.py
index e1179e0f13..66eca4c650 100644
--- a/lib/_emerge/Scheduler.py
+++ b/lib/_emerge/Scheduler.py
@@ -940,6 +940,10 @@ class Scheduler(PollScheduler):
                     # is consuming time here.
                     if bintree.isremote(x.cpv):
                         fetcher = self._get_prefetcher(x)
+                        if fetcher is not None and not fetcher.isAlive():
+                            # Cancel it because it hasn't started yet.
+                            fetcher.cancel()
+                            fetcher = None
                         if fetcher is None:
                             fetcher = BinpkgFetcher(pkg=x, scheduler=loop)
                             fetcher.start()
@@ -1963,7 +1967,9 @@ class Scheduler(PollScheduler):
             # CPython 2.7, so it may be possible for CPython to raise KeyError
             # here as well.
             prefetcher = None
-        if prefetcher is not None and not prefetcher.isAlive():
+        if prefetcher is not None and (
+            prefetcher.cancelled or not prefetcher.isAlive()
+        ):
             try:
                 self._task_queues.fetch._task_queue.remove(prefetcher)
             except ValueError:

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     9823f70c6e4ef3cdd6abb4d9fc599ce02a138125
Author:     Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Tue Nov 28 23:58:23 2023 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Wed Nov 29 00:00:18 2023 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=9823f70c

Prefer installed leaves in runtime cycle topological sort

In order to avoid possibly merging a package too early, prefer
installed leaves in runtime cycle topological sort. This fixes
an AlternativesGzipTestCase failure that arose after 2e298ea7ba36
caused leaves to be selected in a slightly different order.

Bug: https://bugs.gentoo.org/917259
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/depgraph.py | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/lib/_emerge/depgraph.py b/lib/_emerge/depgraph.py
index da37f980ad..9f041f83a9 100644
--- a/lib/_emerge/depgraph.py
+++ b/lib/_emerge/depgraph.py
@@ -9498,6 +9498,12 @@ class depgraph:
                     if smallest_leaves is None:
                         smallest_leaves = [cycle_digraph.order[-1]]
 
+                    # Prefer installed leaves, in order to avoid
+                    # merging something too early.
+                    installed_leaves = [pkg for pkg in smallest_leaves if pkg.installed]
+                    if installed_leaves:
+                        smallest_leaves = installed_leaves
+
                     # Only harvest one node at a time, in order to
                     # minimize the number of ignored dependencies.
                     cycle_digraph.remove(smallest_leaves[0])

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     ec938c4599e3250d6fd38b9f381d57376bd66df6
Author:     Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Wed Nov 29 15:57:09 2023 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Wed Nov 29 19:56:19 2023 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=ec938c45

Account for runtime_slot_op in DepPriority sort for digraph.debug_print()

Since 2e298ea7ba36, the digraph.debug_print() output may not
display runtime_slot_op when appropriate, since it relies
on DepPriority sort order to determine which priority is
most relevant to display. Adjust DepPriority sort order to
account for this, which makes the debug output for
AlternativesGzipTestCase display runtime_slot_op where
appropriate:

runtime cycle digraph (9 nodes):

(app-alternatives/gzip-1:0/0::test_repo, ebuild scheduled for merge) depends on
  (app-arch/pigz-2.8:0/0::test_repo, ebuild scheduled for merge) (runtime)
(sys-libs/zlib-1.3-r1:0/1::test_repo, ebuild scheduled for merge) depends on
  (sys-devel/automake-1.16.5-r1:0/0::test_repo, installed) (buildtime)
(sys-devel/automake-1.16.5-r1:0/0::test_repo, installed) depends on
  (dev-lang/perl-5.36.1-r3:0/0::test_repo, installed) (runtime)
  (app-alternatives/gzip-1:0/0::test_repo, ebuild scheduled for merge) (optional)
(app-arch/pigz-2.8:0/0::test_repo, ebuild scheduled for merge) depends on
  (sys-libs/zlib-1.3-r1:0/1::test_repo, ebuild scheduled for merge) (buildtime)
  (app-alternatives/gzip-1:0/0::test_repo, ebuild scheduled for merge) (runtime_post)
(dev-lang/perl-5.36.1-r3:0/0::test_repo, installed) depends on
  (sys-libs/zlib-1.3-r1:0/1::test_repo, ebuild scheduled for merge) (runtime)
  (virtual/libcrypt-2-r1:0/2::test_repo, installed) (runtime_slot_op)
(sys-libs/glibc-2.37-r7:0/0::test_repo, installed) depends on
  (dev-lang/perl-5.36.1-r3:0/0::test_repo, installed) (runtime)
  (sys-apps/locale-gen-2.23-r1:0/0::test_repo, installed) (runtime)
(sys-apps/locale-gen-2.23-r1:0/0::test_repo, installed) depends on
  (app-alternatives/gzip-1:0/0::test_repo, ebuild scheduled for merge) (runtime)
(virtual/libcrypt-2-r1:0/2::test_repo, installed) depends on
  (sys-libs/libxcrypt-4.4.36:0/0::test_repo, installed) (runtime)
(sys-libs/libxcrypt-4.4.36:0/0::test_repo, installed) depends on
  (sys-libs/glibc-2.37-r7:0/0::test_repo, installed) (runtime)

Bug: https://bugs.gentoo.org/918683
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/DepPriority.py | 19 +++++++++++--------
 1 file changed, 11 insertions(+), 8 deletions(-)

diff --git a/lib/_emerge/DepPriority.py b/lib/_emerge/DepPriority.py
index ed4adc5ab7..99d38477e2 100644
--- a/lib/_emerge/DepPriority.py
+++ b/lib/_emerge/DepPriority.py
@@ -19,24 +19,27 @@ class DepPriority(AbstractDepPriority):
 
         buildtime_slot_op                       0
         buildtime                              -1
-        runtime                                -2
-        runtime_post                           -3
-        optional                               -4
-        (none of the above)                    -5
+        runtime_slot_op                        -2
+        runtime                                -3
+        runtime_post                           -4
+        optional                               -5
+        (none of the above)                    -6
 
         """
 
         if self.optional:
-            return -4
+            return -5
         if self.buildtime_slot_op:
             return 0
         if self.buildtime:
             return -1
-        if self.runtime:
+        if self.runtime_slot_op:
             return -2
-        if self.runtime_post:
+        if self.runtime:
             return -3
-        return -5
+        if self.runtime_post:
+            return -4
+        return -6
 
     def __str__(self):
         if self.ignored:

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     5977d8cba5bae12248188960b189012a318c8fc1
Author:     Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Wed Nov 29 20:03:02 2023 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Wed Nov 29 20:04:10 2023 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=5977d8cb

depgraph: Handle empty leaves during runtime cycle topological sort

Fixes: 31832c7faf5b ("Optimize runtime cycle ignore_priority leaf selection loop for topological sort")
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/depgraph.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/lib/_emerge/depgraph.py b/lib/_emerge/depgraph.py
index 15c3e3ca7b..d3b5756429 100644
--- a/lib/_emerge/depgraph.py
+++ b/lib/_emerge/depgraph.py
@@ -9485,10 +9485,10 @@ class depgraph:
                         )
                         if leaves:
                             # Select leaves with minimum ignore_priority,
-                            # in order to ingore as few deps as possible.
+                            # in order to ignore as few deps as possible.
                             break
 
-                    if leaves is None:
+                    if not leaves:
                         leaves = [cycle_digraph.order[-1]]
 
                     # Prefer installed leaves, in order to avoid

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Sam James]

commit:     09f642b894eff3a607026b1edf417f5bceee95f1
Author:     Eli Schwartz <eschwartz93 <AT> gmail <DOT> com>
AuthorDate: Sun Nov 26 02:25:06 2023 +0000
Commit:     Sam James <sam <AT> gentoo <DOT> org>
CommitDate: Mon Dec  4 06:45:45 2023 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=09f642b8

emerge: avoid spamming too much info about unused binpkgs, redux

Specifically:

- many packages can be participating in the recursive deptree, but not
  get selected for actual merging, and may not even be installed at all;
  these were still reported as changed-deps although the packages
  themselves were not shown in show_merge_list

Previously done for mismatched-USE in commit:
bb82666b48e18f448661a1a8bf6a39b773cc4b1c

Before:
```

Local copy of remote index is up-to-date and will be used.

Local copy of remote index is up-to-date and will be used.

These are the packages that would be merged, in order:

Calculating dependencies... done!
Dependency resolution took 17.62 s.

[binary     U  ] dev-python/ensurepip-pip-23.3.1-1 [23.2.1]
[binary     U  ] sys-devel/binutils-2.40-r9-1 [2.40-r5] USE="-debuginfod%"
[binary     U  ] dev-python/trove-classifiers-2023.11.14-1 [2023.10.18]
[ebuild     U  ] dev-python/traitlets-5.13.0 [5.11.2] PYTHON_TARGETS="(-pypy3)"
[ebuild     U  ] dev-python/pip-23.3.1 [23.2.1]
[ebuild     U  ] dev-python/pycurl-7.45.2-r1 [7.45.2]
[ebuild     U  ] dev-python/cachetools-5.3.2 [5.3.1]
[ebuild     U  ] dev-python/pytest-7.4.3 [7.4.2]
[ebuild     U  ] dev-python/executing-2.0.1 [1.2.0] PYTHON_TARGETS="(-pypy3) -python3_12%"
[ebuild     U  ] dev-python/virtualenv-20.24.7 [20.24.5]

!!! The following binary packages have been ignored due to changed dependencies:

     kde-plasma/kde-cli-tools-5.27.9::gentoo
     kde-plasma/kde-cli-tools-5.27.8::gentoo

NOTE: The --binpkg-changed-deps=n option will prevent emerge
      from ignoring these binary packages if possible.
      Using --binpkg-changed-deps=y will silence this warning.
```

After:
```

Local copy of remote index is up-to-date and will be used.

Local copy of remote index is up-to-date and will be used.

These are the packages that would be merged, in order:

Calculating dependencies... done!
Dependency resolution took 17.62 s.

[binary     U  ] dev-python/ensurepip-pip-23.3.1-1 [23.2.1]
[binary     U  ] sys-devel/binutils-2.40-r9-1 [2.40-r5] USE="-debuginfod%"
[binary     U  ] dev-python/trove-classifiers-2023.11.14-1 [2023.10.18]
[ebuild     U  ] dev-python/traitlets-5.13.0 [5.11.2] PYTHON_TARGETS="(-pypy3)"
[ebuild     U  ] dev-python/pip-23.3.1 [23.2.1]
[ebuild     U  ] dev-python/pycurl-7.45.2-r1 [7.45.2]
[ebuild     U  ] dev-python/cachetools-5.3.2 [5.3.1]
[ebuild     U  ] dev-python/pytest-7.4.3 [7.4.2]
[ebuild     U  ] dev-python/executing-2.0.1 [1.2.0] PYTHON_TARGETS="(-pypy3) -python3_12%"
[ebuild     U  ] dev-python/virtualenv-20.24.7 [20.24.5]
```

Signed-off-by: Eli Schwartz <eschwartz93 <AT> gmail.com>
Closes: https://github.com/gentoo/portage/pull/1194
Signed-off-by: Sam James <sam <AT> gentoo.org>

 lib/_emerge/depgraph.py | 28 ++++++++++++++++++++++------
 1 file changed, 22 insertions(+), 6 deletions(-)

diff --git a/lib/_emerge/depgraph.py b/lib/_emerge/depgraph.py
index d3b5756429..9b09701021 100644
--- a/lib/_emerge/depgraph.py
+++ b/lib/_emerge/depgraph.py
@@ -1288,17 +1288,33 @@ class depgraph:
             writemsg(line + "\n", noiselevel=-1)
 
     def _show_ignored_binaries_changed_deps(self, changed_deps):
-        writemsg(
-            "\n!!! The following binary packages have been "
-            "ignored due to changed dependencies:\n\n",
-            noiselevel=-1,
-        )
+        merging = {
+            (pkg.root, pkg.cpv)
+            for pkg in self._dynamic_config._displayed_list or ()
+            if isinstance(pkg, Package)
+        }
+        messages = []
 
         for pkg in changed_deps:
+            # Don't include recursive deps which aren't in the merge list anyway.
+            if (pkg.root, pkg.cpv) not in merging:
+                continue
+
             msg = f"     {pkg.cpv}{_repo_separator}{pkg.repo}"
             if pkg.root_config.settings["ROOT"] != "/":
                 msg += f" for {pkg.root}"
-            writemsg(f"{msg}\n", noiselevel=-1)
+            messages.append(f"{msg}\n")
+
+        if not messages:
+            return
+
+        writemsg(
+            "\n!!! The following binary packages have been "
+            "ignored due to changed dependencies:\n\n",
+            noiselevel=-1,
+        )
+        for line in messages:
+            writemsg(line, noiselevel=-1)
 
         msg = [
             "",

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Sam James]

commit:     69ce8a71cc806b4b333c2e707e7a9291c3d84664
Author:     Sam James <sam <AT> gentoo <DOT> org>
AuthorDate: Sun Dec 10 22:13:09 2023 +0000
Commit:     Sam James <sam <AT> gentoo <DOT> org>
CommitDate: Sun Dec 10 22:29:08 2023 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=69ce8a71

_emerge: BinpkgVerifier: give better error message on stale binpkg index

portage-3.0.52 defaults to FEATURES="pkgdir-index-trusted" (see NEWS) which
has a few benefits, but means that manually editing PKGDIR without regenerating
the index with 'emaint binhost -f' will confuse Portage.

Give a better error message mentioning that command if we fail to fetch a binpkg
but bintree.dbapi.cpv_exists says it should exist.

Bug: https://bugs.gentoo.org/915474
Bug: https://bugs.gentoo.org/918597
Signed-off-by: Sam James <sam <AT> gentoo.org>

 lib/_emerge/BinpkgVerifier.py | 22 +++++++++++++++++-----
 1 file changed, 17 insertions(+), 5 deletions(-)

diff --git a/lib/_emerge/BinpkgVerifier.py b/lib/_emerge/BinpkgVerifier.py
index a7917453aa..7e044c6c43 100644
--- a/lib/_emerge/BinpkgVerifier.py
+++ b/lib/_emerge/BinpkgVerifier.py
@@ -41,11 +41,23 @@ class BinpkgVerifier(CompositeTask):
         except OSError as e:
             if e.errno not in (errno.ENOENT, errno.ESTALE):
                 raise
-            self.scheduler.output(
-                f"!!! Fetching Binary failed for '{self.pkg.cpv}'\n",
-                log_path=self.logfile,
-                background=self.background,
-            )
+
+            # We might end up here with FEATURES="pkgdir-index-trusted" if
+            # binpkgs have been removed manually without refreshing the index.
+            if bintree.dbapi.cpv_exists(self.pkg.cpv):
+                self.scheduler.output(
+                    f"!!! Tried to use non-existent binary for '{self.pkg.cpv}'\n"
+                    + f"!!! Likely caused by an outdated index. Run 'emaint binhost -f'.\n",
+                    log_path=self.logfile,
+                    background=self.background,
+                )
+            else:
+                self.scheduler.output(
+                    f"!!! Fetching Binary failed for '{self.pkg.cpv}'\n",
+                    log_path=self.logfile,
+                    background=self.background,
+                )
+
             self.returncode = 1
             self._async_wait()
             return

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     1477124cc20c8ff3ca3be07efe98fafeb9999990
Author:     Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Mon Dec 18 18:35:14 2023 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Mon Dec 18 18:35:46 2023 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=1477124c

Scheduler: Reset current_task in _run_pkg_pretend

For completed tasks, reset current_task to None so that
it does not cause the clean phase to execute with an invalid
environment in the finally block.

Bug: https://bugs.gentoo.org/920258
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/Scheduler.py | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/lib/_emerge/Scheduler.py b/lib/_emerge/Scheduler.py
index 66eca4c650..4193ffdd03 100644
--- a/lib/_emerge/Scheduler.py
+++ b/lib/_emerge/Scheduler.py
@@ -930,6 +930,7 @@ class Scheduler(PollScheduler):
                     current_task = clean_phase
                     clean_phase.start()
                     await clean_phase.async_wait()
+                    current_task = None
 
                 if x.built:
                     tree = "bintree"
@@ -981,6 +982,7 @@ class Scheduler(PollScheduler):
                         self._record_pkg_failure(x, settings, verifier.returncode)
                         continue
 
+                    current_task = None
                     if fetched:
                         bintree.inject(
                             x.cpv,
@@ -1030,6 +1032,8 @@ class Scheduler(PollScheduler):
                 current_task = pretend_phase
                 pretend_phase.start()
                 ret = await pretend_phase.async_wait()
+                # Leave current_task assigned in order to trigger clean
+                # on success in the below finally block.
                 if ret != os.EX_OK:
                     failures += 1
                     self._record_pkg_failure(x, settings, ret)

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     fbd6909c24f51ad79fb313ddd7043b4a40de91a6
Author:     Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Tue Dec 26 21:37:24 2023 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Tue Dec 26 22:01:51 2023 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=fbd6909c

Scheduler: Handle unpack_metadata SignatureException in _run_pkg_pretend

This will handle the InvalidSignature exception that triggered
bug 920258, allowing emerge --keep-going to skip the package
if possible. All of the gpg stderr output is dumped before the
exception is raised, as shown in bug 920258 comment #0.

Bug: https://bugs.gentoo.org/920258
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/Scheduler.py | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)

diff --git a/lib/_emerge/Scheduler.py b/lib/_emerge/Scheduler.py
index 4193ffdd03..620d513511 100644
--- a/lib/_emerge/Scheduler.py
+++ b/lib/_emerge/Scheduler.py
@@ -992,7 +992,16 @@ class Scheduler(PollScheduler):
 
                     infloc = os.path.join(build_dir_path, "build-info")
                     ensure_dirs(infloc)
-                    await bintree.dbapi.unpack_metadata(settings, infloc, loop=loop)
+                    try:
+                        await bintree.dbapi.unpack_metadata(settings, infloc, loop=loop)
+                    except portage.exception.SignatureException as e:
+                        writemsg(
+                            f"!!! Invalid binary package: '{bintree.getname(x.cpv)}', {e}\n",
+                            noiselevel=-1,
+                        )
+                        failures += 1
+                        self._record_pkg_failure(x, settings, 1)
+                        continue
                     ebuild_path = os.path.join(infloc, x.pf + ".ebuild")
                     settings.configdict["pkg"]["EMERGE_FROM"] = "binary"
                     settings.configdict["pkg"]["MERGE_TYPE"] = "binary"

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Sam James]

commit:     8b167b0adffac5bdf1d7e43f2ca3da5f8d604b33
Author:     Eli Schwartz <eschwartz93 <AT> gmail <DOT> com>
AuthorDate: Wed Dec 27 02:12:03 2023 +0000
Commit:     Sam James <sam <AT> gentoo <DOT> org>
CommitDate: Wed Dec 27 13:30:19 2023 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=8b167b0a

emerge: enable "avoid spamming too much info about unused binpkgs" again

In commit a6853d5493b7bed37e60c2e3b7536b209500ba3f this code in
_show_ignored_binaries_respect_use was refactored to handle pkg.root.
But it was refactored incorrectly -- the storage/lookup key started off
as:

```
seen[pkg.cpv]
```

and was refactored so the storage key was:
```
seen[pkg.root][pkg.cpv]
```

and the lookup key was
```
seen[(pkg.root, pkg.cpv)]
```

As a result, we never detected a previously seen USE flags combo, and
the logic to avoid spamming too much info was a no-op; the info was
spammed, instead.

Note that even though we have more code than before this patch, we
actually have less code. The black code formatter decided that since the
line length was decreased, this entire code block should be reformatted,
murdering the diff view and dividing less code across *more* lines. This
is not my fault and I refuse to be held accountable for it -- if you git
blame this and do not know what happened, understand that it happens
despite my objections.

Signed-off-by: Eli Schwartz <eschwartz93 <AT> gmail.com>
Closes: https://github.com/gentoo/portage/pull/1219
Signed-off-by: Sam James <sam <AT> gentoo.org>

 lib/_emerge/depgraph.py | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/lib/_emerge/depgraph.py b/lib/_emerge/depgraph.py
index 6ee4471bbe..efe084a780 100644
--- a/lib/_emerge/depgraph.py
+++ b/lib/_emerge/depgraph.py
@@ -1256,9 +1256,10 @@ class depgraph:
 
             # We don't want to list the same USE flags for multiple build IDs
             seen.setdefault(pkg.root, dict())
-            if (pkg.root, pkg.cpv) not in seen or flag_display not in seen[pkg.root][
-                pkg.cpv
-            ]:
+            if (
+                pkg.cpv not in seen[pkg.root]
+                or flag_display not in seen[pkg.root][pkg.cpv]
+            ):
                 seen[pkg.root].setdefault(pkg.cpv, set()).add(flag_display)
                 # The user can paste this line into package.use
                 messages.append(f"    ={pkg.cpv} {flag_display}")

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     89df7574a355a245e19ba297c3685997eec6bbbe
Author:     Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Sat Dec 30 18:21:08 2023 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Sat Dec 30 18:27:21 2023 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=89df7574

EbuildBinpkg: Handle binarytree inject failure

Capture stdout and stderr for logging during
binarytree inject, and set returncode to 1 when
inject fails and returns None.

Bug: https://bugs.gentoo.org/921089
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/EbuildBinpkg.py | 37 ++++++++++++++++++++++++++++++-------
 1 file changed, 30 insertions(+), 7 deletions(-)

diff --git a/lib/_emerge/EbuildBinpkg.py b/lib/_emerge/EbuildBinpkg.py
index cb01f73d73..97b69ae018 100644
--- a/lib/_emerge/EbuildBinpkg.py
+++ b/lib/_emerge/EbuildBinpkg.py
@@ -1,6 +1,9 @@
-# Copyright 1999-2012 Gentoo Foundation
+# Copyright 1999-2023 Gentoo Authors
 # Distributed under the terms of the GNU General Public License v2
 
+import io
+import sys
+
 from _emerge.CompositeTask import CompositeTask
 from _emerge.EbuildPhase import EbuildPhase
 
@@ -55,14 +58,34 @@ class EbuildBinpkg(CompositeTask):
 
         pkg = self.pkg
         bintree = pkg.root_config.trees["bintree"]
-        self._binpkg_info = bintree.inject(
-            pkg.cpv,
-            current_pkg_path=self._binpkg_tmpfile,
-            allocated_pkg_path=self.pkg_allocated_path,
-        )
+
+        stdout_orig = sys.stdout
+        stderr_orig = sys.stderr
+        out = io.StringIO()
+        try:
+            sys.stdout = out
+            sys.stderr = out
+
+            self._binpkg_info = bintree.inject(
+                pkg.cpv,
+                current_pkg_path=self._binpkg_tmpfile,
+                allocated_pkg_path=self.pkg_allocated_path,
+            )
+
+        finally:
+            sys.stdout = stdout_orig
+            sys.stderr = stderr_orig
+
+            output_value = out.getvalue()
+            if output_value:
+                self.scheduler.output(
+                    output_value,
+                    log_path=self.settings.get("PORTAGE_LOG_FILE"),
+                    background=self.background,
+                )
 
         self._current_task = None
-        self.returncode = os.EX_OK
+        self.returncode = 1 if self._binpkg_info is None else os.EX_OK
         self.wait()
 
     def get_binpkg_info(self):

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Sam James]

commit:     d8285bad5815f0f2db64657dd245efc3fe948210
Author:     Sam James <sam <AT> gentoo <DOT> org>
AuthorDate: Wed Jan  3 06:08:53 2024 +0000
Commit:     Sam James <sam <AT> gentoo <DOT> org>
CommitDate: Wed Jan  3 19:59:51 2024 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=d8285bad

BinpkgFetcher: preserve mtime

The FileCopier from 1db44d18578a7aee58449cb97e1991cb06c915c3 doesn't preserve
its timestamp so there's a mismatch between the expected mtime in the index
and the binpkg.

Bug: https://bugs.gentoo.org/921208
Thanks-to: Zac Medico <zmedico <AT> gentoo.org>
Signed-off-by: Sam James <sam <AT> gentoo.org>

 lib/_emerge/BinpkgFetcher.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/lib/_emerge/BinpkgFetcher.py b/lib/_emerge/BinpkgFetcher.py
index 01b2bae637..a1524dc009 100644
--- a/lib/_emerge/BinpkgFetcher.py
+++ b/lib/_emerge/BinpkgFetcher.py
@@ -106,7 +106,8 @@ class BinpkgFetcher(CompositeTask):
                 finally:
                     if copier.isAlive():
                         copier.cancel()
-
+                if copier.returncode == os.EX_OK:
+                    fetcher.sync_timestamp()
             else:
                 fetcher.start()
                 try:

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     0ff49114cec79edce723da1190087a41699f6b2f
Author:     Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Thu Jan  4 05:12:24 2024 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Thu Jan  4 06:13:25 2024 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=0ff49114

binarytree: Handle inject failures

Capture stdout and stderr for logging purposes during binarytree
inject calls, and use the inject return value to report success
or failure.

For failures prior to pkg_pretend, use an eerror/elog message
to indicate that the binary package is not usable. Move
corresponding elog_process call to a finally block so that it
is called for all pkg_pretend failures.

Bug: https://bugs.gentoo.org/921327
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/Binpkg.py           | 34 ++++++++++++++++++++++++++++++----
 lib/_emerge/BinpkgPrefetcher.py | 36 +++++++++++++++++++++++++++++-------
 lib/_emerge/Scheduler.py        | 36 +++++++++++++++++++++++-------------
 3 files changed, 82 insertions(+), 24 deletions(-)

diff --git a/lib/_emerge/Binpkg.py b/lib/_emerge/Binpkg.py
index 9b1036538a..299ae7fbc9 100644
--- a/lib/_emerge/Binpkg.py
+++ b/lib/_emerge/Binpkg.py
@@ -1,6 +1,8 @@
-# Copyright 1999-2023 Gentoo Authors
+# Copyright 1999-2024 Gentoo Authors
 # Distributed under the terms of the GNU General Public License v2
 
+import io
+import sys
 import functools
 import _emerge.emergelog
 from _emerge.EbuildPhase import EbuildPhase
@@ -244,12 +246,36 @@ class Binpkg(CompositeTask):
         pkg_count = self.pkg_count
 
         if self._fetched_pkg:
-            pkg_path = self._bintree.getname(
-                self._bintree.inject(
+            stdout_orig = sys.stdout
+            stderr_orig = sys.stderr
+            out = io.StringIO()
+            try:
+                sys.stdout = out
+                sys.stderr = out
+
+                injected_pkg = self._bintree.inject(
                     pkg.cpv,
                     current_pkg_path=self._fetched_pkg,
                     allocated_pkg_path=self._pkg_allocated_path,
-                ),
+                )
+            finally:
+                sys.stdout = stdout_orig
+                sys.stderr = stderr_orig
+
+                output_value = out.getvalue()
+                if output_value:
+                    self.scheduler.output(
+                        output_value,
+                        log_path=self.settings.get("PORTAGE_LOG_FILE"),
+                        background=self.background,
+                    )
+
+            if injected_pkg is None:
+                self._async_unlock_builddir(returncode=1)
+                return
+
+            pkg_path = self._bintree.getname(
+                injected_pkg,
                 allocate_new=False,
             )
         else:

diff --git a/lib/_emerge/BinpkgPrefetcher.py b/lib/_emerge/BinpkgPrefetcher.py
index 37dbe0a40f..ed68d2852c 100644
--- a/lib/_emerge/BinpkgPrefetcher.py
+++ b/lib/_emerge/BinpkgPrefetcher.py
@@ -1,6 +1,9 @@
-# Copyright 1999-2009 Gentoo Foundation
+# Copyright 1999-2024 Gentoo Authors
 # Distributed under the terms of the GNU General Public License v2
 
+import io
+import sys
+
 from _emerge.BinpkgFetcher import BinpkgFetcher
 from _emerge.CompositeTask import CompositeTask
 from _emerge.BinpkgVerifier import BinpkgVerifier
@@ -45,12 +48,31 @@ class BinpkgPrefetcher(CompositeTask):
             self.wait()
             return
 
-        self._bintree.inject(
-            self.pkg.cpv,
-            current_pkg_path=self.pkg_path,
-            allocated_pkg_path=self.pkg_allocated_path,
-        )
+        stdout_orig = sys.stdout
+        stderr_orig = sys.stderr
+        out = io.StringIO()
+        try:
+            sys.stdout = out
+            sys.stderr = out
+
+            injected_pkg = self._bintree.inject(
+                self.pkg.cpv,
+                current_pkg_path=self.pkg_path,
+                allocated_pkg_path=self.pkg_allocated_path,
+            )
+
+        finally:
+            sys.stdout = stdout_orig
+            sys.stderr = stderr_orig
+
+            output_value = out.getvalue()
+            if output_value:
+                self.scheduler.output(
+                    output_value,
+                    log_path=self.scheduler.fetch.log_file,
+                    background=self.background,
+                )
 
         self._current_task = None
-        self.returncode = os.EX_OK
+        self.returncode = 1 if injected_pkg is None else os.EX_OK
         self.wait()

diff --git a/lib/_emerge/Scheduler.py b/lib/_emerge/Scheduler.py
index 620d513511..ae01214d32 100644
--- a/lib/_emerge/Scheduler.py
+++ b/lib/_emerge/Scheduler.py
@@ -1,4 +1,4 @@
-# Copyright 1999-2023 Gentoo Authors
+# Copyright 1999-2024 Gentoo Authors
 # Distributed under the terms of the GNU General Public License v2
 
 from collections import deque
@@ -984,11 +984,19 @@ class Scheduler(PollScheduler):
 
                     current_task = None
                     if fetched:
-                        bintree.inject(
+                        if not bintree.inject(
                             x.cpv,
                             current_pkg_path=fetched,
                             allocated_pkg_path=fetcher.pkg_allocated_path,
-                        )
+                        ):
+                            eerror(
+                                "Binary package is not usable",
+                                phase="pretend",
+                                key=x.cpv,
+                            )
+                            failures += 1
+                            self._record_pkg_failure(x, settings, 1)
+                            continue
 
                     infloc = os.path.join(build_dir_path, "build-info")
                     ensure_dirs(infloc)
@@ -1046,20 +1054,22 @@ class Scheduler(PollScheduler):
                 if ret != os.EX_OK:
                     failures += 1
                     self._record_pkg_failure(x, settings, ret)
-                portage.elog.elog_process(x.cpv, settings)
             finally:
                 if current_task is not None:
                     if current_task.isAlive():
                         current_task.cancel()
-                    if current_task.returncode == os.EX_OK:
-                        clean_phase = EbuildPhase(
-                            background=False,
-                            phase="clean",
-                            scheduler=sched_iface,
-                            settings=settings,
-                        )
-                        clean_phase.start()
-                        await clean_phase.async_wait()
+
+                portage.elog.elog_process(x.cpv, settings)
+
+                if current_task is not None and current_task.returncode == os.EX_OK:
+                    clean_phase = EbuildPhase(
+                        background=False,
+                        phase="clean",
+                        scheduler=sched_iface,
+                        settings=settings,
+                    )
+                    clean_phase.start()
+                    await clean_phase.async_wait()
 
                 await build_dir.async_unlock()
                 self._deallocate_config(settings)

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     da5b18a418233e289229d3d124f6352e531c9ffb
Author:     Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Wed Jan 17 12:49:32 2024 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Wed Jan 17 12:50:46 2024 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=da5b18a4

depgraph: Fix SyntaxWarning: 'str' object is not callable

Fixes: a5078e577493 ("emerge: backtrack consistently regardless of --fetchonly")
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/depgraph.py | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/lib/_emerge/depgraph.py b/lib/_emerge/depgraph.py
index 7491d970c3..1cbbbaf815 100644
--- a/lib/_emerge/depgraph.py
+++ b/lib/_emerge/depgraph.py
@@ -3794,7 +3794,8 @@ class depgraph:
                     except KeyError:
                         optional_msg = " ({} previously removed from graph)"
                         warnings.warn(
-                            f"_eliminate_rebuilds inconsistency: parent priorities missing for {parent} -> {pkg} edge"(
+                            f"_eliminate_rebuilds inconsistency: parent priorities missing for {parent} -> {pkg} edge"
+                            + (
                                 optional_msg.format("parent and child")
                                 if parent in removed and pkg in removed
                                 else optional_msg.format("parent")
@@ -3815,7 +3816,8 @@ class depgraph:
                     except KeyError:
                         optional_msg = " ({} previously removed from graph)"
                         warnings.warn(
-                            f"_eliminate_rebuilds inconsistency: parent priorities missing for {pkg} -> {child} edge"(
+                            f"_eliminate_rebuilds inconsistency: parent priorities missing for {pkg} -> {child} edge"
+                            + (
                                 optional_msg.format("parent and child")
                                 if pkg in removed and child in removed
                                 else optional_msg.format("parent")

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     f9ea958018c02b3ce07761c2d965260498add2af
Author:     Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Tue Feb 13 04:04:34 2024 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Wed Feb 21 15:27:30 2024 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=f9ea9580

MetadataRegen: Use EbuildMetadataPhase deallocate_config

For EbuildMetadataPhase consumers like MetadataRegen and
depgraph, store a pool of config instances in a config_pool
list, and return instaces to the list when the deallocate_config
future is done.

Bug: https://bugs.gentoo.org/924319
Fixes: c95fc64abf96 ("EbuildPhase: async_check_locale")
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/MetadataRegen.py | 16 ++++++++++++++--
 lib/_emerge/depgraph.py      | 11 +++++++++++
 2 files changed, 25 insertions(+), 2 deletions(-)

diff --git a/lib/_emerge/MetadataRegen.py b/lib/_emerge/MetadataRegen.py
index d29722b94c..538a94b450 100644
--- a/lib/_emerge/MetadataRegen.py
+++ b/lib/_emerge/MetadataRegen.py
@@ -1,4 +1,4 @@
-# Copyright 1999-2020 Gentoo Authors
+# Copyright 1999-2024 Gentoo Authors
 # Distributed under the terms of the GNU General Public License v2
 
 from _emerge.EbuildMetadataPhase import EbuildMetadataPhase
@@ -44,6 +44,7 @@ class MetadataRegen(AsyncScheduler):
         valid_pkgs = self._valid_pkgs
         cp_set = self._cp_set
         consumer = self._consumer
+        config_pool = []
 
         portage.writemsg_stdout("Regenerating cache entries...\n")
         for cp in self._cp_iter:
@@ -73,12 +74,23 @@ class MetadataRegen(AsyncScheduler):
                             consumer(cpv, repo_path, metadata, ebuild_hash, True)
                         continue
 
+                    if config_pool:
+                        settings = config_pool.pop()
+                    else:
+                        settings = portage.config(clone=portdb.settings)
+
+                    deallocate_config = self.scheduler.create_future()
+                    deallocate_config.add_done_callback(
+                        lambda future: config_pool.append(future.result())
+                    )
+
                     yield EbuildMetadataPhase(
                         cpv=cpv,
                         ebuild_hash=ebuild_hash,
                         portdb=portdb,
                         repo_path=repo_path,
-                        settings=portdb.doebuild_settings,
+                        settings=settings,
+                        deallocate_config=deallocate_config,
                         write_auxdb=self._write_auxdb,
                     )
 

diff --git a/lib/_emerge/depgraph.py b/lib/_emerge/depgraph.py
index 1674fa289e..70b83ee1f4 100644
--- a/lib/_emerge/depgraph.py
+++ b/lib/_emerge/depgraph.py
@@ -754,6 +754,7 @@ class depgraph:
 
     def _dynamic_deps_preload(self, fake_vartree):
         portdb = fake_vartree._portdb
+        config_pool = []
         for pkg in fake_vartree.dbapi:
             self._spinner_update()
             self._dynamic_config._package_tracker.add_installed_pkg(pkg)
@@ -768,12 +769,22 @@ class depgraph:
             if metadata is not None:
                 fake_vartree.dynamic_deps_preload(pkg, metadata)
             else:
+                if config_pool:
+                    settings = config_pool.pop()
+                else:
+                    settings = portage.config(clone=portdb.settings)
+
+                deallocate_config = portdb._event_loop.create_future()
+                deallocate_config.add_done_callback(
+                    lambda future: config_pool.append(future.result())
+                )
                 proc = EbuildMetadataPhase(
                     cpv=pkg.cpv,
                     ebuild_hash=ebuild_hash,
                     portdb=portdb,
                     repo_path=repo_path,
                     settings=portdb.doebuild_settings,
+                    deallocate_config=deallocate_config,
                 )
                 proc.addExitListener(self._dynamic_deps_proc_exit(pkg, fake_vartree))
                 yield proc

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     69fbd8f9f76df32d9bc72510e5d9348eb8f059bc
Author:     Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Tue Feb 13 04:04:53 2024 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Wed Feb 21 15:27:30 2024 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=69fbd8f9

EbuildMetadataPhase: Add deallocate_config future

Use a deallocate_config future to release self.settings when
it is no longer needed. It's necessary to manage concurrency
since commit c95fc64abf96 because mutation of self.settings
is no longer limited to the EbuildMetadataPhase _start method,
where exclusive access was guaranteed within the main thread.

Add support to the isAlive() method to detect when the
EbuildMetadataPhase has started but the pid is not yet
available (due to async_check_locale usage from commit
c95fc64abf96). This can be used to check if an
EbuildMetadataPhase instance has been successfully started
so that it can be relied upon to set the result of the
deallocate_config future.

Bug: https://bugs.gentoo.org/924319
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/EbuildMetadataPhase.py | 34 ++++++++++++++++++++++++++++++++++
 lib/_emerge/SubProcess.py          |  5 ++++-
 2 files changed, 38 insertions(+), 1 deletion(-)

diff --git a/lib/_emerge/EbuildMetadataPhase.py b/lib/_emerge/EbuildMetadataPhase.py
index f4f685e81c..249086f8af 100644
--- a/lib/_emerge/EbuildMetadataPhase.py
+++ b/lib/_emerge/EbuildMetadataPhase.py
@@ -14,6 +14,7 @@ from portage import os
 from portage import _encodings
 from portage import _unicode_decode
 from portage import _unicode_encode
+from portage.util.futures import asyncio
 
 import fcntl
 
@@ -33,6 +34,7 @@ class EbuildMetadataPhase(SubProcess):
         "portdb",
         "repo_path",
         "settings",
+        "deallocate_config",
         "write_auxdb",
     ) + (
         "_eapi",
@@ -127,6 +129,15 @@ class EbuildMetadataPhase(SubProcess):
             returnproc=True,
         )
         settings.pop("PORTAGE_PIPE_FD", None)
+        # At this point we can return settings to the caller
+        # since we never use it for anything more than an
+        # eapi_invalid call after this, and eapi_invalid is
+        # insensitive to concurrent modifications.
+        if (
+            self.deallocate_config is not None
+            and not self.deallocate_config.cancelled()
+        ):
+            self.deallocate_config.set_result(settings)
 
         os.close(slave_fd)
         null_input.close()
@@ -139,6 +150,29 @@ class EbuildMetadataPhase(SubProcess):
 
         self._proc = retval
 
+        asyncio.ensure_future(
+            self._async_start(), loop=self.scheduler
+        ).add_done_callback(self._async_start_done)
+
+    async def _async_start(self):
+        # Call async check_locale here for bug 923841, but code
+        # also needs to migrate from _start to here, including
+        # the self.deallocate_config set_result call.
+        pass
+
+    def _async_start_done(self, future):
+        future.cancelled() or future.result()
+        if not self._was_cancelled() and future.cancelled():
+            self.cancel()
+            self._was_cancelled()
+
+        if self.deallocate_config is not None and not self.deallocate_config.done():
+            self.deallocate_config.set_result(self.settings)
+
+        if self.returncode is not None:
+            self._unregister()
+            self.wait()
+
     def _output_handler(self):
         while True:
             buf = self._read_buf(self._files.ebuild)

diff --git a/lib/_emerge/SubProcess.py b/lib/_emerge/SubProcess.py
index 029bbc3f44..057e0adc24 100644
--- a/lib/_emerge/SubProcess.py
+++ b/lib/_emerge/SubProcess.py
@@ -18,9 +18,12 @@ class SubProcess(AbstractPollTask):
     # we've sent a kill signal to our subprocess.
     _cancel_timeout = 1  # seconds
 
+    def isAlive(self):
+        return (self._registered or self.pid is not None) and self.returncode is None
+
     @property
     def pid(self):
-        return self._proc.pid
+        return None if self._proc is None else self._proc.pid
 
     def _poll(self):
         # Simply rely on _async_waitpid_cb to set the returncode.

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     789493d521eb09f4aca2a8552b5262e2c0f5bc51
Author:     Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Tue Feb 13 08:19:59 2024 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Wed Feb 21 15:27:31 2024 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=789493d5

EbuildMetadataPhase: Migrate to _async_start

Bug: https://bugs.gentoo.org/923841
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/EbuildMetadataPhase.py | 17 ++++++-----------
 1 file changed, 6 insertions(+), 11 deletions(-)

diff --git a/lib/_emerge/EbuildMetadataPhase.py b/lib/_emerge/EbuildMetadataPhase.py
index 249086f8af..fc64d84c94 100644
--- a/lib/_emerge/EbuildMetadataPhase.py
+++ b/lib/_emerge/EbuildMetadataPhase.py
@@ -46,6 +46,12 @@ class EbuildMetadataPhase(SubProcess):
     _files_dict = slot_dict_class(_file_names, prefix="")
 
     def _start(self):
+        asyncio.ensure_future(
+            self._async_start(), loop=self.scheduler
+        ).add_done_callback(self._async_start_done)
+        self._registered = True
+
+    async def _async_start(self):
         ebuild_path = self.ebuild_hash.location
 
         with open(
@@ -116,7 +122,6 @@ class EbuildMetadataPhase(SubProcess):
         self._raw_metadata = []
         files.ebuild = master_fd
         self.scheduler.add_reader(files.ebuild, self._output_handler)
-        self._registered = True
 
         retval = portage.doebuild(
             ebuild_path,
@@ -150,16 +155,6 @@ class EbuildMetadataPhase(SubProcess):
 
         self._proc = retval
 
-        asyncio.ensure_future(
-            self._async_start(), loop=self.scheduler
-        ).add_done_callback(self._async_start_done)
-
-    async def _async_start(self):
-        # Call async check_locale here for bug 923841, but code
-        # also needs to migrate from _start to here, including
-        # the self.deallocate_config set_result call.
-        pass
-
     def _async_start_done(self, future):
         future.cancelled() or future.result()
         if not self._was_cancelled() and future.cancelled():

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     d06515e1424be60c8ed7af6ed2a80d56145b85c5
Author:     Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Sat Feb 24 13:28:22 2024 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Sat Feb 24 13:39:29 2024 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=d06515e1

_dynamic_deps_preload: Fix settings reference

Use the settings reference from the config pool as
intended to prevent a KeyError similar to bug 924319
but triggered by emerge --dynamic-deps.

Fixes: f9ea958018c0 ("MetadataRegen: Use EbuildMetadataPhase deallocate_config")
Bug: https://bugs.gentoo.org/925350
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/depgraph.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/_emerge/depgraph.py b/lib/_emerge/depgraph.py
index ea96bd58c4..6853ec4911 100644
--- a/lib/_emerge/depgraph.py
+++ b/lib/_emerge/depgraph.py
@@ -783,7 +783,7 @@ class depgraph:
                     ebuild_hash=ebuild_hash,
                     portdb=portdb,
                     repo_path=repo_path,
-                    settings=portdb.doebuild_settings,
+                    settings=settings,
                     deallocate_config=deallocate_config,
                 )
                 proc.addExitListener(self._dynamic_deps_proc_exit(pkg, fake_vartree))

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     c3ed026be40b4cf857b33854e70f59f7e016e3b5
Author:     Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Sat Mar  9 23:23:28 2024 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Sat Mar  9 23:24:44 2024 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=c3ed026b

_setup_repo_revisions: Skip async_aux_get for INHERITED

Since INHERITED is always available for the ebuild environment,
get it from settings.configdict["pkg"]["INHERITED"].

Fixes: d55d415a4f68 ("EbuildPhase/EbuildBinpkg: Ensure PKGDIR subdirectory permissions")
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/EbuildPhase.py | 13 +++----------
 1 file changed, 3 insertions(+), 10 deletions(-)

diff --git a/lib/_emerge/EbuildPhase.py b/lib/_emerge/EbuildPhase.py
index 73c284fe41..9e5268ff73 100644
--- a/lib/_emerge/EbuildPhase.py
+++ b/lib/_emerge/EbuildPhase.py
@@ -92,17 +92,10 @@ async def _setup_repo_revisions(settings):
         and repo_name
         and "PORTAGE_REPO_REVISIONS" not in settings.configdict["pkg"]
     ):
-        inherits = frozenset(
-            (
-                await db.async_aux_get(
-                    settings.mycpv,
-                    ["INHERITED"],
-                    myrepo=repo_name,
-                )
-            )[0].split()
-        )
         repo = db.repositories[repo_name]
-        ec_dict = repo.eclass_db.get_eclass_data(inherits)
+        ec_dict = repo.eclass_db.get_eclass_data(
+            settings.configdict["pkg"]["INHERITED"].split()
+        )
         referenced_repos = {repo.name: repo}
         for ec_info in ec_dict.values():
             ec_repo = db.repositories.get_repo_for_location(

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     08a2bc3800ea2e997716903244ec041339b45b06
Author:     Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Sun Mar 24 22:11:54 2024 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Thu Mar 28 14:47:49 2024 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=08a2bc38

BinpkgPrefetcher: Emit eerror message for binarytree inject failure

Since it is confusing when emerge exits due to an error that only shows
in the fetch log, emit an eerror message when binarytree inject fails:

>>> Running pre-merge checks for sys-libs/glibc-2.38-r10
 * Fetching in the background:
 * /var/cache/binpkgs/sys-libs/glibc/glibc-2.38-r10-5.gpkg.tar.partial
 * To view fetch progress, run in another terminal:
 * tail -f /var/log/emerge-fetch.log
>>> Failed to emerge sys-libs/glibc-2.38-r10

 * Messages for package sys-libs/glibc-2.38-r10:

 * Binary package is not usable:
 *      !!!
 *      gpg: keyblock resource '/etc/portage/gnupg/pubring.kbx': No such file or directory
 *     	[GNUPG:] ERROR add_keyblock_resource 33587281
 *     	[GNUPG:] PLAINTEXT 74 0
 *     	[GNUPG:] NEWSIG
 *     	gpg: Signature made Wed 20 Mar 2024 10:34:45 PM CET
 *     	gpg:                using RSA key 534E4209AB49EEE1C19D96162C44695DB9F6043D
 *     	[GNUPG:] ERROR keydb_search 33554445
 *     	[GNUPG:] ERROR keydb_search 33554445
 *     	[GNUPG:] ERRSIG 2C44695DB9F6043D 1 10 01 1710970485 9 534E4209AB49EEE1C19D96162C44695DB9F6043D
 *     	[GNUPG:] NO_PUBKEY 2C44695DB9F6043D
 *      !!! Invalid binary package: '/var/cache/binpkgs/sys-devel/binutils/binutils-2.41-r5-4.gpkg.tar.partial', GPG verify failed

Fixes: 0ff49114cec7 ("binarytree: Handle inject failures")
Bug: https://bugs.gentoo.org/927632
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/BinpkgPrefetcher.py | 35 ++++++++++++++++++++++++++++++-----
 1 file changed, 30 insertions(+), 5 deletions(-)

diff --git a/lib/_emerge/BinpkgPrefetcher.py b/lib/_emerge/BinpkgPrefetcher.py
index ed68d2852c..a8af30ca80 100644
--- a/lib/_emerge/BinpkgPrefetcher.py
+++ b/lib/_emerge/BinpkgPrefetcher.py
@@ -7,7 +7,10 @@ import sys
 from _emerge.BinpkgFetcher import BinpkgFetcher
 from _emerge.CompositeTask import CompositeTask
 from _emerge.BinpkgVerifier import BinpkgVerifier
+import portage
 from portage import os
+from portage.elog import messages as elog_messages
+from portage.util import no_color
 
 
 class BinpkgPrefetcher(CompositeTask):
@@ -48,6 +51,7 @@ class BinpkgPrefetcher(CompositeTask):
             self.wait()
             return
 
+        injected_pkg = None
         stdout_orig = sys.stdout
         stderr_orig = sys.stderr
         out = io.StringIO()
@@ -67,12 +71,33 @@ class BinpkgPrefetcher(CompositeTask):
 
             output_value = out.getvalue()
             if output_value:
-                self.scheduler.output(
-                    output_value,
-                    log_path=self.scheduler.fetch.log_file,
-                    background=self.background,
-                )
+                if injected_pkg is None:
+                    msg = ["Binary package is not usable:"]
+                    msg.extend("\t" + line for line in output_value.splitlines())
+                    self._elog("eerror", msg)
+                else:
+                    self.scheduler.output(
+                        output_value,
+                        log_path=self.scheduler.fetch.log_file,
+                        background=self.background,
+                    )
 
         self._current_task = None
         self.returncode = 1 if injected_pkg is None else os.EX_OK
         self.wait()
+
+    def _elog(self, elog_funcname, lines, phase="other"):
+        out = io.StringIO()
+        elog_func = getattr(elog_messages, elog_funcname)
+        global_havecolor = portage.output.havecolor
+        try:
+            portage.output.havecolor = not no_color(self._bintree.settings)
+            for line in lines:
+                elog_func(line, phase=phase, key=self.pkg.cpv, out=out)
+        finally:
+            portage.output.havecolor = global_havecolor
+        msg = out.getvalue()
+        if msg:
+            self.scheduler.output(
+                msg, background=self.background, log_path=self.scheduler.fetch.log_file
+            )

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Sam James]

commit:     7c9bb2caeaff59232cb44e84b585db3feb2dd5fe
Author:     Sam James <sam <AT> gentoo <DOT> org>
AuthorDate: Sat May  4 12:05:17 2024 +0000
Commit:     Sam James <sam <AT> gentoo <DOT> org>
CommitDate: Sat May  4 12:05:17 2024 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=7c9bb2ca

actions: handle pkgmoves for autoconf, automake, libtool

Signed-off-by: Sam James <sam <AT> gentoo.org>

 lib/_emerge/actions.py | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/lib/_emerge/actions.py b/lib/_emerge/actions.py
index d36a799244..b93c08aa07 100644
--- a/lib/_emerge/actions.py
+++ b/lib/_emerge/actions.py
@@ -2070,11 +2070,11 @@ def action_info(settings, trees, myopts, myfiles):
         append(ccache_str)
 
     myvars = [
-        "sys-devel/autoconf",
-        "sys-devel/automake",
+        "dev-build/autoconf",
+        "dev-build/automake",
         "virtual/os-headers",
         "sys-devel/binutils",
-        "sys-devel/libtool",
+        "dev-build/libtool",
         "dev-lang/python",
     ]
     myvars += portage.util.grabfile(settings["PORTDIR"] + "/profiles/info_pkgs")

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Sam James]

commit:     b16ab6c7088cfde7664f1728542545e6dafce99d
Author:     Bryan Gardiner <bog <AT> khumba <DOT> net>
AuthorDate: Sat May 11 03:18:56 2024 +0000
Commit:     Sam James <sam <AT> gentoo <DOT> org>
CommitDate: Mon May 13 05:43:48 2024 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=b16ab6c7

depgraph.py: Fix the URL for the changed-deps wiki page

Signed-off-by: Bryan Gardiner <bog <AT> khumba.net>
Signed-off-by: Sam James <sam <AT> gentoo.org>

 lib/_emerge/depgraph.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/_emerge/depgraph.py b/lib/_emerge/depgraph.py
index 6853ec4911..9673d85f87 100644
--- a/lib/_emerge/depgraph.py
+++ b/lib/_emerge/depgraph.py
@@ -1181,7 +1181,7 @@ class depgraph:
                     "NOTE: Refer to the following page for more information about dependency",
                     "      change(s) without revision bump:",
                     "",
-                    "          https://wiki.gentoo.org/wiki/Project:Portage/Changed_Deps",
+                    "          https://wiki.gentoo.org/wiki/Project:Portage/Changed_dependencies",
                     "",
                     "      In order to suppress reports about dependency changes, add",
                     "      --changed-deps-report=n to the EMERGE_DEFAULT_OPTS variable in",

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     cd755af8347c32ea5247b68e6193c8428c9ffd7b
Author:     Florian Schmaus <flow <AT> gentoo <DOT> org>
AuthorDate: Tue May 21 12:55:16 2024 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Tue May 21 16:07:45 2024 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=cd755af8

SchedulerInterface/PollScheduler: _loadavg_latency's unit is seconds

The unit of _loadavg_latency is seconds, not milliseconds, as it is
used as first argument to an eventloop's call_later() function.

Signed-off-by: Florian Schmaus <flow <AT> gentoo.org>
Closes: https://github.com/gentoo/portage/pull/1325
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/PollScheduler.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/_emerge/PollScheduler.py b/lib/_emerge/PollScheduler.py
index b5bfd20b76..fdf88dd04d 100644
--- a/lib/_emerge/PollScheduler.py
+++ b/lib/_emerge/PollScheduler.py
@@ -11,7 +11,7 @@ from _emerge.getloadavg import getloadavg
 
 
 class PollScheduler:
-    # max time between loadavg checks (milliseconds)
+    # max time between loadavg checks (seconds)
     _loadavg_latency = None
 
     def __init__(self, main=False, event_loop=None):

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     685908978c48f6bfeab99f47ba7d78ee00294a64
Author:     Tatsuyuki Ishi <ishitatsuyuki <AT> google <DOT> com>
AuthorDate: Wed May 15 07:46:36 2024 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Sun May 26 18:01:22 2024 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=68590897

Fix a depgraph crash with useoldpkg and subslot bumps

This resolves a "list index out of range" error that could happen with
when a package's subslot is bumped and --useoldpkg is used.

Due to the subslot bump, the code will filter matched_packages to
only contain the latest version. However, this didn't update
matched_oldpkg, which would now contain stale packages and cause logic
errors.

Fix this by filtering matched_oldpkg again before its use.

Signed-off-by: Tatsuyuki Ishi <ishitatsuyuki <AT> google.com>
Bug: https://bugs.gentoo.org/932804
Closes: https://github.com/gentoo/portage/pull/1322
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/depgraph.py | 1 +
 1 file changed, 1 insertion(+)

diff --git a/lib/_emerge/depgraph.py b/lib/_emerge/depgraph.py
index 13add990e6..6b91d5c42d 100644
--- a/lib/_emerge/depgraph.py
+++ b/lib/_emerge/depgraph.py
@@ -8177,6 +8177,7 @@ class depgraph:
                     ):
                         return pkg, existing_node
 
+            matched_oldpkg = [x for x in matched_oldpkg if x in matched_packages]
             visible_matches = []
             if matched_oldpkg:
                 visible_matches = [

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Sam James]

commit:     aa0a0c6f74ae28eb209ee933946e82917612b21e
Author:     Florian Schmaus <flow <AT> gentoo <DOT> org>
AuthorDate: Tue May 21 07:33:05 2024 +0000
Commit:     Sam James <sam <AT> gentoo <DOT> org>
CommitDate: Sun May 26 23:28:13 2024 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=aa0a0c6f

Scheduler: convert state_change to boolean in _schedule_tasks_imp()

Signed-off-by: Florian Schmaus <flow <AT> gentoo.org>
Closes: https://github.com/gentoo/portage/pull/1324
Signed-off-by: Sam James <sam <AT> gentoo.org>

 lib/_emerge/Scheduler.py | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/lib/_emerge/Scheduler.py b/lib/_emerge/Scheduler.py
index c547c309c2..d913cd2dc6 100644
--- a/lib/_emerge/Scheduler.py
+++ b/lib/_emerge/Scheduler.py
@@ -1953,11 +1953,11 @@ class Scheduler(PollScheduler):
         @return: True if state changed, False otherwise.
         """
 
-        state_change = 0
+        state_change = False
 
         while True:
             if not self._keep_scheduling():
-                return bool(state_change)
+                return state_change
 
             if (
                 self._choose_pkg_return_early
@@ -1966,13 +1966,13 @@ class Scheduler(PollScheduler):
                 or not self._can_add_job()
                 or self._job_delay()
             ):
-                return bool(state_change)
+                return state_change
 
             pkg = self._choose_pkg()
             if pkg is None:
-                return bool(state_change)
+                return state_change
 
-            state_change += 1
+            state_change = True
 
             if not pkg.installed:
                 self._pkg_count.curval += 1

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Sam James]

commit:     f1843024c34b8915786d3044f6f1b3a3b72284bf
Author:     Florian Schmaus <flow <AT> gentoo <DOT> org>
AuthorDate: Tue May 21 07:35:09 2024 +0000
Commit:     Sam James <sam <AT> gentoo <DOT> org>
CommitDate: Sun May 26 23:28:12 2024 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=f1843024

Scheduler: coalesce common code in _schedule_tasks_imp()

Signed-off-by: Florian Schmaus <flow <AT> gentoo.org>
Signed-off-by: Sam James <sam <AT> gentoo.org>

 lib/_emerge/Scheduler.py | 15 +++++----------
 1 file changed, 5 insertions(+), 10 deletions(-)

diff --git a/lib/_emerge/Scheduler.py b/lib/_emerge/Scheduler.py
index 5c318f89b9..77318b0b3e 100644
--- a/lib/_emerge/Scheduler.py
+++ b/lib/_emerge/Scheduler.py
@@ -1985,15 +1985,6 @@ class Scheduler(PollScheduler):
                 self._task_queues.merge.addFront(merge)
                 merge.addExitListener(self._merge_exit)
 
-            elif pkg.built:
-                self._jobs += 1
-                self._previous_job_start_time = time.time()
-                self._status_display.running = self._jobs
-                self._running_tasks[id(task)] = task
-                task.scheduler = self._sched_iface
-                self._task_queues.jobs.add(task)
-                task.addExitListener(self._extract_exit)
-
             else:
                 self._jobs += 1
                 self._previous_job_start_time = time.time()
@@ -2001,7 +1992,11 @@ class Scheduler(PollScheduler):
                 self._running_tasks[id(task)] = task
                 task.scheduler = self._sched_iface
                 self._task_queues.jobs.add(task)
-                task.addExitListener(self._build_exit)
+
+                if pkg.built:
+                    task.addExitListener(self._extract_exit)
+                else:
+                    task.addExitListener(self._build_exit)
 
         return bool(state_change)
 

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Sam James]

commit:     305d843be43c79d6b09584280a67e16060e85df0
Author:     Florian Schmaus <flow <AT> gentoo <DOT> org>
AuthorDate: Tue May 21 16:11:53 2024 +0000
Commit:     Sam James <sam <AT> gentoo <DOT> org>
CommitDate: Sun May 26 23:28:13 2024 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=305d843b

Scheduler: drop unreachable return statement in _schedule_tasks_imp()

Signed-off-by: Florian Schmaus <flow <AT> gentoo.org>
Signed-off-by: Sam James <sam <AT> gentoo.org>

 lib/_emerge/Scheduler.py | 2 --
 1 file changed, 2 deletions(-)

diff --git a/lib/_emerge/Scheduler.py b/lib/_emerge/Scheduler.py
index 77318b0b3e..c547c309c2 100644
--- a/lib/_emerge/Scheduler.py
+++ b/lib/_emerge/Scheduler.py
@@ -1998,8 +1998,6 @@ class Scheduler(PollScheduler):
                 else:
                     task.addExitListener(self._build_exit)
 
-        return bool(state_change)
-
     def _get_prefetcher(self, pkg):
         try:
             prefetcher = self._prefetchers.pop(pkg, None)

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[James Le Cuirot]

commit:     a28a0fd6600242a2e062a0fd2d7d5be95296b7ae
Author:     Florian Schmaus <flow <AT> gentoo <DOT> org>
AuthorDate: Tue Jun 18 09:40:27 2024 +0000
Commit:     James Le Cuirot <chewi <AT> gentoo <DOT> org>
CommitDate: Fri Aug  2 13:35:07 2024 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=a28a0fd6

JobStatusDisplay: increase the default width to 100

Conservatively modernize the default width value from 80 to
100. Usually, I do not care much about the width, but the job status
display holds more information these days, compared to the times where
this value was set.

Plus, self._isatty has the tendency to return false, even though
portage is actually run within a tty. The prime examples causing this
are systemd-run (and run0). Therefore, a conservatively increasing the
value is sensible.

Signed-off-by: Florian Schmaus <flow <AT> gentoo.org>
Closes: https://github.com/gentoo/portage/pull/1348
Signed-off-by: James Le Cuirot <chewi <AT> gentoo.org>

 lib/_emerge/JobStatusDisplay.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/_emerge/JobStatusDisplay.py b/lib/_emerge/JobStatusDisplay.py
index f9e034c6e5..78fd8f7618 100644
--- a/lib/_emerge/JobStatusDisplay.py
+++ b/lib/_emerge/JobStatusDisplay.py
@@ -65,7 +65,7 @@ class JobStatusDisplay:
         if self._isatty:
             width = portage.output.get_term_size()[1]
         else:
-            width = 80
+            width = 100
         self._set_width(width)
 
     def _set_width(self, width):

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     74e29110d86a55ac74fdde3e1b79506b3fb695c2
Author:     Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Wed Aug 14 05:49:30 2024 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Wed Aug 14 15:08:12 2024 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=74e29110

_EbuildFetcherProcess: Suppress CancelledError

Suppress CancelledError when attempting to cache the
result in the _async_uri_map method. The cancelled
result is returned for the caller to handle.

Bug: https://bugs.gentoo.org/937888
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/EbuildFetcher.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/_emerge/EbuildFetcher.py b/lib/_emerge/EbuildFetcher.py
index 994271236c..422d7c0a06 100644
--- a/lib/_emerge/EbuildFetcher.py
+++ b/lib/_emerge/EbuildFetcher.py
@@ -373,7 +373,7 @@ class _EbuildFetcherProcess(ForkProcess):
         def cache_result(result):
             try:
                 self._uri_map = result.result()
-            except Exception:
+            except (CancelledError, Exception):
                 # The caller handles this when it retrieves the result.
                 pass
 

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     e4f0fa4a316aba2cf2b1a47c9d2974710a1d8bd3
Author:     Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Wed Aug 14 16:03:52 2024 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Wed Aug 14 16:04:34 2024 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=e4f0fa4a

_EbuildFetcherProcess: Suppress asyncio.CancelledError

Fixes: 74e29110d86a ("_EbuildFetcherProcess: Suppress CancelledError")
Bug: https://bugs.gentoo.org/937888
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/EbuildFetcher.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/_emerge/EbuildFetcher.py b/lib/_emerge/EbuildFetcher.py
index 422d7c0a06..554cd95973 100644
--- a/lib/_emerge/EbuildFetcher.py
+++ b/lib/_emerge/EbuildFetcher.py
@@ -373,7 +373,7 @@ class _EbuildFetcherProcess(ForkProcess):
         def cache_result(result):
             try:
                 self._uri_map = result.result()
-            except (CancelledError, Exception):
+            except (asyncio.CancelledError, Exception):
                 # The caller handles this when it retrieves the result.
                 pass
 

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Sam James]

commit:     9f919c387fbc4c8526535b03fb781ab2a01ddfdd
Author:     Florian Schmaus <flow <AT> gentoo <DOT> org>
AuthorDate: Sat Aug 31 08:35:11 2024 +0000
Commit:     Sam James <sam <AT> gentoo <DOT> org>
CommitDate: Wed Sep 11 01:38:59 2024 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=9f919c38

JobStatusDisplay: fix setting the default width to 100

This fixes a28a0fd66002 ("JobStatusDisplay: increase the default width
to 100") which attempted to set the default width to 100. However, the
out of bounds check in _set_width() was not adjusted. Therefore, the
default width was effectively still 80.

Fixes: a28a0fd6600242a2e062a0fd2d7d5be95296b7ae
Signed-off-by: Florian Schmaus <flow <AT> gentoo.org>
Signed-off-by: Sam James <sam <AT> gentoo.org>

 lib/_emerge/JobStatusDisplay.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/lib/_emerge/JobStatusDisplay.py b/lib/_emerge/JobStatusDisplay.py
index 78fd8f7618..6506aed7dc 100644
--- a/lib/_emerge/JobStatusDisplay.py
+++ b/lib/_emerge/JobStatusDisplay.py
@@ -71,8 +71,8 @@ class JobStatusDisplay:
     def _set_width(self, width):
         if width == getattr(self, "width", None):
             return
-        if width <= 0 or width > 80:
-            width = 80
+        if width <= 0 or width > 100:
+            width = 100
         object.__setattr__(self, "width", width)
         object.__setattr__(self, "_jobs_column_width", width - 32)
 

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Sam James]

commit:     225e574108549ac85dd743bfbc0c3d5e2e740c60
Author:     Florian Schmaus <flow <AT> gentoo <DOT> org>
AuthorDate: Sat Aug 31 08:38:51 2024 +0000
Commit:     Sam James <sam <AT> gentoo <DOT> org>
CommitDate: Wed Sep 11 01:38:59 2024 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=225e5741

JobStatusDisplay: introduce max_display_width variable

Values that belong together and may only be changed at the same time
should be combined in a common constant. So let's do this.

Signed-off-by: Florian Schmaus <flow <AT> gentoo.org>
Closes: https://github.com/gentoo/portage/pull/1377
Signed-off-by: Sam James <sam <AT> gentoo.org>

 lib/_emerge/JobStatusDisplay.py | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)

diff --git a/lib/_emerge/JobStatusDisplay.py b/lib/_emerge/JobStatusDisplay.py
index 6506aed7dc..57495c5ae5 100644
--- a/lib/_emerge/JobStatusDisplay.py
+++ b/lib/_emerge/JobStatusDisplay.py
@@ -16,6 +16,9 @@ from _emerge.getloadavg import getloadavg
 
 
 class JobStatusDisplay:
+    # Used as maximum display width and default fallback value.
+    max_display_width = 100
+
     _bound_properties = ("curval", "failed", "running")
 
     # Don't update the display unless at least this much
@@ -65,14 +68,14 @@ class JobStatusDisplay:
         if self._isatty:
             width = portage.output.get_term_size()[1]
         else:
-            width = 100
+            width = self.max_display_width
         self._set_width(width)
 
     def _set_width(self, width):
         if width == getattr(self, "width", None):
             return
-        if width <= 0 or width > 100:
-            width = 100
+        if width <= 0 or width > self.max_display_width:
+            width = self.max_display_width
         object.__setattr__(self, "width", width)
         object.__setattr__(self, "_jobs_column_width", width - 32)
 

[gentoo-commits] proj/portage:master commit in: lib/_emerge/

[Zac Medico]

commit:     376ab2c56e3dc7e7904ec862128bdabc07f6a00a
Author:     Zac Medico <zmedico <AT> gentoo <DOT> org>
AuthorDate: Sun Sep 29 20:18:14 2024 +0000
Commit:     Zac Medico <zmedico <AT> gentoo <DOT> org>
CommitDate: Sun Sep 29 22:47:07 2024 +0000
URL:        https://gitweb.gentoo.org/proj/portage.git/commit/?id=376ab2c5

SpawnProcess: Remove unused _CGROUP_CLEANUP_RETRY_MAX

Left over from FEATURES="cgroup".

Fixes: f8e3b11496bd6d602a690535c4a3bb32bb8e9744
Bug: https://bugs.gentoo.org/894398
Signed-off-by: Zac Medico <zmedico <AT> gentoo.org>

 lib/_emerge/SpawnProcess.py | 4 ----
 1 file changed, 4 deletions(-)

diff --git a/lib/_emerge/SpawnProcess.py b/lib/_emerge/SpawnProcess.py
index 513a7b2fe4..5d8b3929e1 100644
--- a/lib/_emerge/SpawnProcess.py
+++ b/lib/_emerge/SpawnProcess.py
@@ -51,10 +51,6 @@ class SpawnProcess(SubProcess):
         )
     )
 
-    # Max number of attempts to kill the processes listed in cgroup.procs,
-    # given that processes may fork before they can be killed.
-    _CGROUP_CLEANUP_RETRY_MAX = 8
-
     def _start(self):
         if self.fd_pipes is None:
             self.fd_pipes = {}