[gentoo-portage-dev] [PATCH] Add @unsatisfied-deps package set (bug 248026)

[gentoo-portage-dev] [PATCH] Add @unsatisfied-deps package set (bug 248026)

[Zac Medico]

Bug: https://bugs.gentoo.org/248026
Signed-off-by: Zac Medico <zmedico@gentoo.org>
---
 cnf/sets/portage.conf      |  5 +++
 doc/config/sets.docbook    |  7 ++++
 lib/portage/_sets/dbapi.py | 73 +++++++++++++++++++++++++++++++++++++-
 3 files changed, 84 insertions(+), 1 deletion(-)

diff --git a/cnf/sets/portage.conf b/cnf/sets/portage.conf
index c4ad2efca..2bf38e414 100644
--- a/cnf/sets/portage.conf
+++ b/cnf/sets/portage.conf
@@ -115,3 +115,8 @@ class = portage.sets.dbapi.ChangedDepsSet
 class = portage.sets.dbapi.VariableSet
 variable = INHERITED
 includes = golang-base golang-build golang-vcs golang-vcs-snapshot go-module
+
+# Package set which contains all installed packages having one or more
+# unsatisfied runtime dependencies.
+[unsatisfied-deps]
+class = portage.sets.dbapi.UnsatisfiedDepsSet
diff --git a/doc/config/sets.docbook b/doc/config/sets.docbook
index eba98f468..1aa7e8aa1 100644
--- a/doc/config/sets.docbook
+++ b/doc/config/sets.docbook
@@ -610,6 +610,13 @@
 			</itemizedlist>
 			</sect3>
 		</sect2>
+		<sect2 id='config-set-classes-UnsatisfiedDepsSet'>
+		<title>portage.sets.dbapi.UnsatisfiedDepsSet</title>
+		<para>
+		Package set which contains all installed packages packages
+		having one or more unsatisfied runtime dependencies.
+		</para>
+		</sect2>
 	</sect1>
 	
 	<sect1 id='config-set-defaults'>
diff --git a/lib/portage/_sets/dbapi.py b/lib/portage/_sets/dbapi.py
index 8e1f19979..8e436980c 100644
--- a/lib/portage/_sets/dbapi.py
+++ b/lib/portage/_sets/dbapi.py
@@ -14,8 +14,16 @@ from portage._sets.base import PackageSet
 from portage._sets import SetConfigError, get_boolean
 import portage
 
+from _emerge.Package import Package
+from _emerge.RootConfig import RootConfig
+
 __all__ = ["CategorySet", "ChangedDepsSet", "DowngradeSet",
-	"EverythingSet", "OwnerSet", "SubslotChangedSet", "VariableSet"]
+	"EverythingSet",
+	"OwnerSet",
+	"SubslotChangedSet",
+	"UnsatisfiedDepsSet",
+	"VariableSet",
+]
 
 class EverythingSet(PackageSet):
 	_operations = ["merge"]
@@ -303,6 +311,69 @@ class UnavailableBinaries(EverythingSet):
 
 	singleBuilder = classmethod(singleBuilder)
 
+class UnsatisfiedDepsSet(PackageSet):
+
+	_operations = ["merge", "unmerge"]
+
+	description = (
+		"Package set which contains all installed packages packages "
+		"having one or more unsatisfied runtime dependencies."
+	)
+
+	def __init__(self, vardb=None):
+		super().__init__()
+		self._vardb = vardb
+
+	def load(self):
+		vardb = self._vardb
+		trees = {
+			vardb.settings["EROOT"]: {
+				"porttree": vardb.vartree,
+				"vartree": vardb.vartree,
+			}
+		}
+		root_config = RootConfig(vardb.settings, trees[vardb.settings["EROOT"]], None)
+		atoms = []
+		for pkg_str in vardb.cpv_all():
+			try:
+				metadata = dict(
+					zip(
+						vardb._aux_cache_keys,
+						vardb.aux_get(pkg_str, vardb._aux_cache_keys),
+					)
+				)
+			except KeyError:
+				continue
+			pkg = Package(
+				built=True,
+				cpv=pkg_str,
+				installed=True,
+				metadata=metadata,
+				root_config=root_config,
+				type_name="installed",
+			)
+
+			runtime_deps = " ".join(pkg._metadata[k] for k in Package._runtime_keys)
+
+			success, unsatisfied_deps = portage.dep_check(
+				runtime_deps,
+				None,
+				vardb.settings,
+				myuse=pkg.use.enabled,
+				myroot=vardb.settings["EROOT"],
+				trees=trees,
+			)
+
+			if not (success and not unsatisfied_deps):
+				atoms.append(pkg.slot_atom)
+
+		self._setAtoms(atoms)
+
+	def singleBuilder(cls, options, settings, trees):
+		return cls(vardb=trees["vartree"].dbapi)
+
+	singleBuilder = classmethod(singleBuilder)
+
 class CategorySet(PackageSet):
 	_operations = ["merge", "unmerge"]
 
-- 
2.26.2

[gentoo-portage-dev] [PATCH v2] Add @unsatisfied-deps package set (bug 248026)

[Zac Medico]

If emerge --depclean fails to resolve any dependencies, then it will
now suggest emerge @unsatisfied-deps as the simplest possible
solution, and will also suggest to unmerge @unavailable where
appropriate at the end:

$ emerge --depclean

Calculating dependencies... done!
 * Dependencies could not be completely resolved due to
 * the following required packages not being installed:
 *
 *   virtual/cdrtools pulled in by:
 *     app-cdr/cdrdao-1.2.4
 *
 * Have you forgotten to resolve unsatisfied dependencies prior to
 * depclean? The simplest possible command for this purpose is as
 * follows:
 *
 *   emerge @unsatisfied-deps
 *
 * The most comprehensive possible update command is this:
 *
 *   emerge --update --newuse --deep --with-bdeps=y @world
 *
 * Note that the --with-bdeps=y option is not required in many
 * situations. Refer to the emerge manual page (run `man emerge`)
 * for more information about --with-bdeps.
 *
 * Also, note that it may be necessary to manually uninstall
 * packages that no longer exist in the repository, since it may not
 * be possible to satisfy their dependencies. The simplest possible
 * command for this purpose is as follows, but be careful to examine
 * the resulting package list carefully:
 *
 *   emerge --ask --unmerge @unavailable
 *

Bug: https://bugs.gentoo.org/248026
Signed-off-by: Zac Medico <zmedico@gentoo.org>
---
[PATCH v2] Update --depclean message to suggest @unsatisfied-deps
           and unmerge @unavailable where appropriate.

 cnf/sets/portage.conf         |  5 +++
 doc/config/sets.docbook       |  7 ++++
 lib/_emerge/actions.py        | 19 +++++++--
 lib/portage/_sets/__init__.py |  4 ++
 lib/portage/_sets/dbapi.py    | 73 ++++++++++++++++++++++++++++++++++-
 5 files changed, 104 insertions(+), 4 deletions(-)

diff --git a/cnf/sets/portage.conf b/cnf/sets/portage.conf
index c4ad2efca..2bf38e414 100644
--- a/cnf/sets/portage.conf
+++ b/cnf/sets/portage.conf
@@ -115,3 +115,8 @@ class = portage.sets.dbapi.ChangedDepsSet
 class = portage.sets.dbapi.VariableSet
 variable = INHERITED
 includes = golang-base golang-build golang-vcs golang-vcs-snapshot go-module
+
+# Package set which contains all installed packages having one or more
+# unsatisfied runtime dependencies.
+[unsatisfied-deps]
+class = portage.sets.dbapi.UnsatisfiedDepsSet
diff --git a/doc/config/sets.docbook b/doc/config/sets.docbook
index eba98f468..015ec0c05 100644
--- a/doc/config/sets.docbook
+++ b/doc/config/sets.docbook
@@ -610,6 +610,13 @@
 			</itemizedlist>
 			</sect3>
 		</sect2>
+		<sect2 id='config-set-classes-UnsatisfiedDepsSet'>
+		<title>portage.sets.dbapi.UnsatisfiedDepsSet</title>
+		<para>
+		Package set which contains all installed packages
+		having one or more unsatisfied runtime dependencies.
+		</para>
+		</sect2>
 	</sect1>
 	
 	<sect1 id='config-set-defaults'>
diff --git a/lib/_emerge/actions.py b/lib/_emerge/actions.py
index 1946f49df..ba2592bba 100644
--- a/lib/_emerge/actions.py
+++ b/lib/_emerge/actions.py
@@ -1000,11 +1000,18 @@ def _calc_depclean(settings, trees, ldpath_mtimes,
 				msg.append("    %s" % (parent,))
 				msg.append("")
 			msg.extend(textwrap.wrap(
-				"Have you forgotten to do a complete update prior " + \
-				"to depclean? The most comprehensive command for this " + \
+				"Have you forgotten to resolve unsatisfied dependencies prior "
+				"to depclean? The simplest possible command for this "
 				"purpose is as follows:", 65
 			))
 			msg.append("")
+			msg.append("  " + \
+				good("emerge @unsatisfied-deps"))
+			msg.append("")
+			msg.extend(textwrap.wrap(
+				"The most comprehensive possible update command is this:", 65
+			))
+			msg.append("")
 			msg.append("  " + \
 				good("emerge --update --newuse --deep --with-bdeps=y @world"))
 			msg.append("")
@@ -1018,8 +1025,14 @@ def _calc_depclean(settings, trees, ldpath_mtimes,
 			msg.extend(textwrap.wrap(
 				"Also, note that it may be necessary to manually uninstall " + \
 				"packages that no longer exist in the repository, since " + \
-				"it may not be possible to satisfy their dependencies.", 65
+				"it may not be possible to satisfy their dependencies."
+				" The simplest possible command for this purpose is as follows,"
+				" but be careful to examine the resulting package list carefully:", 65
 			))
+			msg.append("")
+			msg.append("  " + \
+				good("emerge --ask --unmerge @unavailable"))
+			msg.append("")
 			if action == "prune":
 				msg.append("")
 				msg.append("If you would like to ignore " + \
diff --git a/lib/portage/_sets/__init__.py b/lib/portage/_sets/__init__.py
index efa6314ba..ea48f6eb0 100644
--- a/lib/portage/_sets/__init__.py
+++ b/lib/portage/_sets/__init__.py
@@ -136,6 +136,10 @@ class SetConfig:
 		parser.add_section("preserved-rebuild")
 		parser.set("preserved-rebuild", "class", "portage.sets.libs.PreservedLibraryConsumerSet")
 
+		parser.remove_section("unsatisfied-deps")
+		parser.add_section("unsatisfied-deps")
+		parser.set("unsatisfied-deps", "class", "portage.sets.dbapi.UnsatisfiedDepsSet")
+
 		parser.remove_section("x11-module-rebuild")
 		parser.add_section("x11-module-rebuild")
 		parser.set("x11-module-rebuild", "class", "portage.sets.dbapi.OwnerSet")
diff --git a/lib/portage/_sets/dbapi.py b/lib/portage/_sets/dbapi.py
index 8e1f19979..c82c4a4cd 100644
--- a/lib/portage/_sets/dbapi.py
+++ b/lib/portage/_sets/dbapi.py
@@ -14,8 +14,16 @@ from portage._sets.base import PackageSet
 from portage._sets import SetConfigError, get_boolean
 import portage
 
+from _emerge.Package import Package
+from _emerge.RootConfig import RootConfig
+
 __all__ = ["CategorySet", "ChangedDepsSet", "DowngradeSet",
-	"EverythingSet", "OwnerSet", "SubslotChangedSet", "VariableSet"]
+	"EverythingSet",
+	"OwnerSet",
+	"SubslotChangedSet",
+	"UnsatisfiedDepsSet",
+	"VariableSet",
+]
 
 class EverythingSet(PackageSet):
 	_operations = ["merge"]
@@ -303,6 +311,69 @@ class UnavailableBinaries(EverythingSet):
 
 	singleBuilder = classmethod(singleBuilder)
 
+class UnsatisfiedDepsSet(PackageSet):
+
+	_operations = ["merge", "unmerge"]
+
+	description = (
+		"Package set which contains all installed packages "
+		"having one or more unsatisfied runtime dependencies."
+	)
+
+	def __init__(self, vardb=None):
+		super().__init__()
+		self._vardb = vardb
+
+	def load(self):
+		vardb = self._vardb
+		trees = {
+			vardb.settings["EROOT"]: {
+				"porttree": vardb.vartree,
+				"vartree": vardb.vartree,
+			}
+		}
+		root_config = RootConfig(vardb.settings, trees[vardb.settings["EROOT"]], None)
+		atoms = []
+		for pkg_str in vardb.cpv_all():
+			try:
+				metadata = dict(
+					zip(
+						vardb._aux_cache_keys,
+						vardb.aux_get(pkg_str, vardb._aux_cache_keys),
+					)
+				)
+			except KeyError:
+				continue
+			pkg = Package(
+				built=True,
+				cpv=pkg_str,
+				installed=True,
+				metadata=metadata,
+				root_config=root_config,
+				type_name="installed",
+			)
+
+			runtime_deps = " ".join(pkg._metadata[k] for k in Package._runtime_keys)
+
+			success, unsatisfied_deps = portage.dep_check(
+				runtime_deps,
+				None,
+				vardb.settings,
+				myuse=pkg.use.enabled,
+				myroot=vardb.settings["EROOT"],
+				trees=trees,
+			)
+
+			if not (success and not unsatisfied_deps):
+				atoms.append(pkg.slot_atom)
+
+		self._setAtoms(atoms)
+
+	def singleBuilder(cls, options, settings, trees):
+		return cls(vardb=trees["vartree"].dbapi)
+
+	singleBuilder = classmethod(singleBuilder)
+
 class CategorySet(PackageSet):
 	_operations = ["merge", "unmerge"]
 
-- 
2.26.2

[gentoo-portage-dev] Re: [PATCH v2] Add @unsatisfied-deps package set (bug 248026)

[Zac Medico]

[-- Attachment #1.1: Type: text/plain, Size: 927 bytes --]

On 6/18/21 6:01 PM, Zac Medico wrote:
> If emerge --depclean fails to resolve any dependencies, then it will
> now suggest emerge @unsatisfied-deps as the simplest possible
> solution, and will also suggest to unmerge @unavailable where
> appropriate at the end:
> 
> $ emerge --depclean
> 
> Calculating dependencies... done!
>  * Dependencies could not be completely resolved due to
>  * the following required packages not being installed:
>  *
>  *   virtual/cdrtools pulled in by:
>  *     app-cdr/cdrdao-1.2.4
>  *
>  * Have you forgotten to resolve unsatisfied dependencies prior to
>  * depclean? The simplest possible command for this purpose is as
>  * follows:
>  *
>  *   emerge @unsatisfied-deps

It turns out that @unsatisfied-deps is often unsuitable here because it
pulls in a bunch of @installed packages, when you really want to use
@world as the source of truth.
-- 
Thanks,
Zac


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 981 bytes --]

[gentoo-portage-dev] Re: [PATCH v2] Add @unsatisfied-deps package set (bug 248026)

[Zac Medico]

[-- Attachment #1.1: Type: text/plain, Size: 1201 bytes --]

On 6/18/21 8:13 PM, Zac Medico wrote:
> On 6/18/21 6:01 PM, Zac Medico wrote:
>> If emerge --depclean fails to resolve any dependencies, then it will
>> now suggest emerge @unsatisfied-deps as the simplest possible
>> solution, and will also suggest to unmerge @unavailable where
>> appropriate at the end:
>>
>> $ emerge --depclean
>>
>> Calculating dependencies... done!
>>  * Dependencies could not be completely resolved due to
>>  * the following required packages not being installed:
>>  *
>>  *   virtual/cdrtools pulled in by:
>>  *     app-cdr/cdrdao-1.2.4
>>  *
>>  * Have you forgotten to resolve unsatisfied dependencies prior to
>>  * depclean? The simplest possible command for this purpose is as
>>  * follows:
>>  *
>>  *   emerge @unsatisfied-deps
> 
> It turns out that @unsatisfied-deps is often unsuitable here because it
> pulls in a bunch of @installed packages, when you really want to use
> @world as the source of truth.

The underlying reason is the same as the reason that we've never used
@installed updates as a substitute for @world updates. It just doesn't
work, because @installed is polluted in comparison to @world.
-- 
Thanks,
Zac


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 981 bytes --]

[gentoo-portage-dev] Re: [PATCH v2] Add @unsatisfied-deps package set (bug 248026)

[Zac Medico]

[-- Attachment #1.1: Type: text/plain, Size: 1413 bytes --]

On 6/18/21 8:29 PM, Zac Medico wrote:
> On 6/18/21 8:13 PM, Zac Medico wrote:
>> On 6/18/21 6:01 PM, Zac Medico wrote:
>>> If emerge --depclean fails to resolve any dependencies, then it will
>>> now suggest emerge @unsatisfied-deps as the simplest possible
>>> solution, and will also suggest to unmerge @unavailable where
>>> appropriate at the end:
>>>
>>> $ emerge --depclean
>>>
>>> Calculating dependencies... done!
>>>  * Dependencies could not be completely resolved due to
>>>  * the following required packages not being installed:
>>>  *
>>>  *   virtual/cdrtools pulled in by:
>>>  *     app-cdr/cdrdao-1.2.4
>>>  *
>>>  * Have you forgotten to resolve unsatisfied dependencies prior to
>>>  * depclean? The simplest possible command for this purpose is as
>>>  * follows:
>>>  *
>>>  *   emerge @unsatisfied-deps
>>
>> It turns out that @unsatisfied-deps is often unsuitable here because it
>> pulls in a bunch of @installed packages, when you really want to use
>> @world as the source of truth.
> 
> The underlying reason is the same as the reason that we've never used
> @installed updates as a substitute for @world updates. It just doesn't
> work, because @installed is polluted in comparison to @world.
> 

My plan is it introduce an @unsatisfied-world set which is equivalent to
@unsatisfied-deps but filters out any non @world packages.
-- 
Thanks,
Zac


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 981 bytes --]