[gentoo-portage-dev] [PATCHES] @changed-deps + missing slot check reposted for bernalex

[gentoo-portage-dev] [PATCHES] @changed-deps + missing slot check reposted for bernalex

[Michał Górny]

Hello, Friend!

This is a repost of my previous patches that I'd like to see applied
soon. I've also fixed the commit messages.

Short description:

1. fixes Portage not to replace non-installed 'dev-foo/bar:=' deps with
plain 'dev-foo/bar'. This affects e.g. || () dependencies where
the remaining branches may have referred to non-installed packages. By
keeping the := operator in those dependencies, @changed-deps is able to
correctly match old and new depstring.

2. adds a @changed-deps set to rebuild packages for which dependencies
have changed (ebuild compared to vdb).

3. adds a repoman check for dependencies that refer to packages that
have more than one slot yet do not specify a slot explicitly or use
a proper slot operator. This is meant to help fixing mistakes like:

  dev-libs/openssl -> dev-libs/openssl:0
    [:0.9.8 comes with no headers]

--
Best regards,
Michał Górny

[gentoo-portage-dev] [PATCH 1/3] vardb: Do not remove := from vdb on unmatched dependencies

[Michał Górny]

Previously, the slot operator processing code used to remove := operator
from vdb dependency entries when the underlying package was not
installed. With regards to a dependency alike || ( A:= B:= ), this
suggested that the particular slot/subslot of installed package could
be replaced by any slot/subslot of the other package. In particular, it
confused dependency comparison in @changed-deps.

Instead, store the original := (or :slot=) atom in vdb. It does not
trigger any special behavior in the current dependency resolver, yet
keeps the extra information. It also makes proper @changed-deps
comparisons possible since the comparison code can know if the original
ebuild had the slot operator.
---
 pym/portage/dep/_slot_operator.py | 27 ++++++++++++++++++---------
 1 file changed, 18 insertions(+), 9 deletions(-)

diff --git a/pym/portage/dep/_slot_operator.py b/pym/portage/dep/_slot_operator.py
index 7b64444..8b67fc5 100644
--- a/pym/portage/dep/_slot_operator.py
+++ b/pym/portage/dep/_slot_operator.py
@@ -1,4 +1,4 @@
-# Copyright 2012-2013 Gentoo Foundation
+# Copyright 2012-2014 Gentoo Foundation
 # Distributed under the terms of the GNU General Public License v2
 
 from __future__ import unicode_literals
@@ -67,6 +67,11 @@ def evaluate_slot_operator_equal_deps(settings, use, trees):
 	return result
 
 def _eval_deps(dep_struct, vardbs):
+	# TODO: we'd use a better || () handling, i.e. || ( A:= B:= ) with both A
+	# and B installed should record subslot on A only since the package is
+	# supposed to link against that anyway, and we have no guarantee that B
+	# has matching ABI.
+
 	for i, x in enumerate(dep_struct):
 		if isinstance(x, list):
 			_eval_deps(x, vardbs)
@@ -87,11 +92,15 @@ def _eval_deps(dep_struct, vardbs):
 						dep_struct[i] = x
 						break
 			else:
-				# this dep could not be resolved, so remove the operator
-				# (user may be using package.provided and managing rebuilds
-				# manually)
-				if x.slot:
-					x = x.with_slot(x.slot)
-				else:
-					x = x.without_slot
-				dep_struct[i] = x
+				# this dep could not be resolved, possibilities include:
+				# 1. unsatisfied branch of || () dep,
+				# 2. package.provided,
+				# 3. --nodeps.
+				#
+				# just leave it as-is for now. this does not cause any special
+				# behavior while keeping the information in vdb -- necessary
+				# e.g. for @changed-deps to work properly.
+				#
+				# TODO: make it actually cause subslot rebuilds when switching
+				# || () branches.
+				pass
-- 
2.0.4

[gentoo-portage-dev] [PATCH 2/3] sets: Introduce @changed-deps to update packages which need dep changes

[Michał Górny]

The @changed-deps set tries to compare RDEPEND and PDEPEND entries of
installed packages with ebuild counterparts, and pulls the ebuild
whenever the two are not in sync. This could be used, for example, to
clean up the system after disabling --dynamic-deps.
---
 cnf/sets/portage.conf      |  5 +++
 pym/portage/_sets/dbapi.py | 81 ++++++++++++++++++++++++++++++++++++++++++++--
 2 files changed, 84 insertions(+), 2 deletions(-)

diff --git a/cnf/sets/portage.conf b/cnf/sets/portage.conf
index b73afb1..fd2c387 100644
--- a/cnf/sets/portage.conf
+++ b/cnf/sets/portage.conf
@@ -84,3 +84,8 @@ class = portage.sets.dbapi.UnavailableSet
 # are not available.
 [unavailable-binaries]
 class = portage.sets.dbapi.UnavailableBinaries
+
+# Installed packages for which vdb *DEPEND entries are outdated compared
+# to the matching portdb entry.
+[changed-deps]
+class = portage.sets.dbapi.ChangedDepsSet
diff --git a/pym/portage/_sets/dbapi.py b/pym/portage/_sets/dbapi.py
index 817bcd7..299cb81 100644
--- a/pym/portage/_sets/dbapi.py
+++ b/pym/portage/_sets/dbapi.py
@@ -3,17 +3,19 @@
 
 from __future__ import division
 
+import re
 import time
 
 from portage import os
 from portage.versions import best, catsplit, vercmp
-from portage.dep import Atom
+from portage.dep import Atom, use_reduce
+from portage.exception import InvalidAtom
 from portage.localization import _
 from portage._sets.base import PackageSet
 from portage._sets import SetConfigError, get_boolean
 import portage
 
-__all__ = ["CategorySet", "DowngradeSet",
+__all__ = ["CategorySet", "ChangedDepsSet", "DowngradeSet",
 	"EverythingSet", "OwnerSet", "VariableSet"]
 
 class EverythingSet(PackageSet):
@@ -458,3 +460,78 @@ class RebuiltBinaries(EverythingSet):
 			bindb=trees["bintree"].dbapi)
 
 	singleBuilder = classmethod(singleBuilder)
+
+class ChangedDepsSet(PackageSet):
+
+	_operations = ["merge", "unmerge"]
+
+	description = "Package set which contains all installed " + \
+		"packages for which the vdb *DEPEND entries are outdated " + \
+		"compared to corresponding portdb entries."
+
+	def __init__(self, portdb=None, vardb=None):
+		super(ChangedDepsSet, self).__init__()
+		self._portdb = portdb
+		self._vardb = vardb
+
+	def load(self):
+		depvars = ('RDEPEND', 'PDEPEND')
+
+		# regexp used to match atoms using subslot operator :=
+		subslot_repl_re = re.compile(r':[^[]*=')
+
+		atoms = []
+		for cpv in self._vardb.cpv_all():
+			# no ebuild, no update :).
+			if not self._portdb.cpv_exists(cpv):
+				continue
+
+			# USE flags used to build the ebuild and EAPI
+			# (needed for Atom & use_reduce())
+			use, eapi = self._vardb.aux_get(cpv, ('USE', 'EAPI'))
+			usel = use.split()
+
+			# function used to recursively process atoms in nested lists.
+			def clean_subslots(depatom, usel=None):
+				if isinstance(depatom, list):
+					# process the nested list.
+					return [clean_subslots(x, usel) for x in depatom]
+				else:
+					try:
+						# this can be either an atom or some special operator.
+						# in the latter case, we get InvalidAtom and pass it as-is.
+						a = Atom(depatom)
+					except InvalidAtom:
+						return depatom
+					else:
+						# if we're processing portdb, we need to evaluate USE flag
+						# dependency conditionals to make them match vdb. this
+						# requires passing the list of USE flags, so we reuse it
+						# as conditional for the operation as well.
+						if usel is not None:
+							a = a.evaluate_conditionals(usel)
+
+						# replace slot operator := dependencies with plain :=
+						# since we can't properly compare expanded slots
+						# in vardb to abstract slots in portdb.
+						return subslot_repl_re.sub(':=', a)
+
+			# get all *DEPEND variables from vdb & portdb and compare them.
+			# we need to do some cleaning up & expansion to make matching
+			# meaningful since vdb dependencies are conditional-free.
+			vdbvars = [clean_subslots(use_reduce(x, uselist=usel, eapi=eapi))
+					for x in self._vardb.aux_get(cpv, depvars)]
+			pdbvars = [clean_subslots(use_reduce(x, uselist=usel, eapi=eapi), usel)
+					for x in self._portdb.aux_get(cpv, depvars)]
+
+			# if dependencies don't match, trigger the rebuild.
+			if vdbvars != pdbvars:
+				atoms.append('=%s' % cpv)
+
+		self._setAtoms(atoms)
+
+	def singleBuilder(cls, options, settings, trees):
+		return cls(portdb=trees["porttree"].dbapi,
+			vardb=trees["vartree"].dbapi)
+
+	singleBuilder = classmethod(singleBuilder)
-- 
2.0.4

[gentoo-portage-dev] [PATCH 3/3] repoman: Add check for missing slot values/slot operators

[Michał Górny]

The idea if that a particular dependency atom matches more than one slot
of a package, you are supposed to either use := or :* operator
(or a specific :slot dependency), whichever is appropriate.

This will help catching mistakes (when packages become slotted) and make
cross-slot behavior clear (it is undefined with no slot operator). I
will estimate the amount of new warnings later.
---
 bin/repoman                      |  6 ++++++
 pym/repoman/check_missingslot.py | 25 +++++++++++++++++++++++++
 2 files changed, 31 insertions(+)
 create mode 100644 pym/repoman/check_missingslot.py

diff --git a/bin/repoman b/bin/repoman
index 5a6ee5b..6c54917 100755
--- a/bin/repoman
+++ b/bin/repoman
@@ -58,6 +58,7 @@ from portage import _encodings
 from portage import _unicode_encode
 import repoman.checks
 from repoman.checks import run_checks
+from repoman.check_missingslot import check_missingslot
 from repoman import utilities
 from repoman.herdbase import make_herd_base
 from _emerge.Package import Package
@@ -299,6 +300,7 @@ qahelp = {
 	"dependency.badindev": "User-visible ebuilds with unsatisfied dependencies (matched against *visible* ebuilds) in developing arch",
 	"dependency.badmaskedindev": "Masked ebuilds with unsatisfied dependencies (matched against *all* ebuilds) in developing arch",
 	"dependency.badtilde": "Uses the ~ dep operator with a non-zero revision part, which is useless (the revision is ignored)",
+	"dependency.missingslot": "RDEPEND matches more than one SLOT but does not specify a slot and/or use the := or :* slot operator",
 	"dependency.perlcore": "This ebuild directly depends on a package in perl-core; it should use the corresponding virtual instead.",
 	"dependency.syntax": "Syntax error in dependency string (usually an extra/missing space/parenthesis)",
 	"dependency.unknown": "Ebuild has a dependency that refers to an unknown package (which may be valid if it is a blocker for a renamed/removed package, or is an alternative choice provided by an overlay)",
@@ -386,6 +388,7 @@ qawarnings = set((
 "dependency.badindev",
 "dependency.badmaskedindev",
 "dependency.badtilde",
+"dependency.missingslot",
 "dependency.perlcore",
 "DESCRIPTION.toolong",
 "EAPI.deprecated",
@@ -2090,6 +2093,9 @@ for x in effective_scanlist:
 							 " with a non-zero revision:" + \
 							 " '%s'") % (mytype, atom))
 
+					check_missingslot(atom, mytype, eapi, portdb, stats, fails,
+						relative_path)
+
 			type_list.extend([mytype] * (len(badsyntax) - len(type_list)))
 
 		for m, b in zip(type_list, badsyntax):
diff --git a/pym/repoman/check_missingslot.py b/pym/repoman/check_missingslot.py
new file mode 100644
index 0000000..1da2ac1
--- /dev/null
+++ b/pym/repoman/check_missingslot.py
@@ -0,0 +1,25 @@
+# repoman: missing slot check
+# Copyright 2014 Gentoo Foundation
+# Distributed under the terms of the GNU General Public License v2
+
+"""This module contains the check used to find missing slot values
+in dependencies."""
+
+from portage.eapi import eapi_has_slot_operator
+
+def check_missingslot(atom, mytype, eapi, portdb, stats, fails, relative_path):
+	# If no slot or slot operator is specified in RDEP...
+	if (not atom.blocker and not atom.slot and not atom.slot_operator
+			and mytype == 'RDEPEND' and eapi_has_slot_operator(eapi)):
+		# Check whether it doesn't match more than one.
+		atom_matches = portdb.xmatch("match-all", atom)
+		dep_slots = frozenset(
+				portdb.aux_get(cpv, ['SLOT'])[0].split('/')[0]
+					for cpv in atom_matches)
+
+		if len(dep_slots) > 1:
+			stats["dependency.missingslot"] += 1
+			fails["dependency.missingslot"].append(
+				relative_path +
+				": %s: '%s' matches more than one slot, please specify an explicit slot and/or use the := or :* slot operator" %
+				(mytype, atom))
-- 
2.0.4

Re: [gentoo-portage-dev] [PATCHES] @changed-deps + missing slot check reposted for bernalex

[Alexander Berntsen]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

These look good. ACK on 1 and 2 from me, but I would like Sebastian,
Arfrever, Brian, or someone else at least to look over them. These
patches are too "interesting" to be pushed after a single ACK.

For 3, I think it looks nice but you should check with Brian & Tom
about whether to put it in now or in the refactor branch. Also, please
ask QA about it so we don't get another bikeshed marathon to deal with.


Lastly, I think the commit message of the first commit is very poorly
worded, but we've discussed that on IRC. Ping me there for more
details if you want.


Thanks for doing this!
- -- 
Alexander
bernalex@gentoo.org
https://secure.plaimi.net/~alexander
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iF4EAREIAAYFAlPr1B4ACgkQRtClrXBQc7VG7gEAllXiY0FUKn+5dUajk/tiZoj/
3H3KW9MPfmfeMh0Dnv8BAJX+y8hsBnppm9W+ap8VfzkrxGeme6J6VSAdYCQd+Dea
=VX87
-----END PGP SIGNATURE-----

Re: [gentoo-portage-dev] [PATCHES] @changed-deps + missing slot check reposted for bernalex

[Brian Dolbec]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

On Wed, 13 Aug 2014 23:09:50 +0200
Alexander Berntsen <bernalex@gentoo.org> wrote:

>  
> For 3, I think it looks nice but you should check with Brian & Tom
> about whether to put it in now or in the refactor branch. Also, please
> ask QA about it so we don't get another bikeshed marathon to deal
> with.
> 
> Thanks for doing this!
> - -- 
> Alexander
> bernalex@gentoo.org

I've already reviewed this patch in the original thread.  I also had
him re-do it into a separate function so it would be easy to insert
into the current master without disrupting the rewrite.  The rewrite
should be easily rebased including this patch plus the few others
recently applied.

But yes, having QA's approval is needed...

I know it is already scheduled with them. :)

- -- 
Brian Dolbec <dolsen>

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (GNU/Linux)

iQF8BAEBCgBmBQJT7F4TXxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w
ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXQ4Njg4RkQxQ0M3MUMxQzA0RUFFQTQyMzcy
MjE0RDkwQTAxNEYxN0NCAAoJECIU2QoBTxfL82oIANWK+BzWLXVg6A7+sXWAlVWH
J+WHtubENpq/Cl25H/MSF7altoZ4ZpZVJ15E1ZvGIV47g3A8mZJQEmwSvQ6ZYdll
YotyIGQTsETgyAUe4THngsOx8+mP1MP8mHYl8nHAz6Tu5+OJQxgrdHai3JAsKH/X
KhGOHhhIBsThP9ooj2WoLjijwZ0epgmyN3T6SDy3GxOs18ugdXQuR3EGi5Wa2JCo
57xHdfZIvPFL2tBiLQuDwiN62Ygb5RfyNfiqYTy9zypK1tnNihUWIF8QYZ3NgFVc
pYWYXLVTbvd8/SDAUTcOy+Z5a9hVBZzudXJbi4YVRpSkRHV4YZx1AkojwbKyye0=
=H614
-----END PGP SIGNATURE-----

Re: [gentoo-portage-dev] [PATCHES] @changed-deps + missing slot check reposted for bernalex

[Brian Dolbec]

On Wed, 13 Aug 2014 19:20:32 +0200
Michał Górny <mgorny@gentoo.org> wrote:

> Hello, Friend!
> 
> This is a repost of my previous patches that I'd like to see applied
> soon. I've also fixed the commit messages.
> 
> Short description:
> 
> 1. fixes Portage not to replace non-installed 'dev-foo/bar:=' deps
> with plain 'dev-foo/bar'. This affects e.g. || () dependencies where
> the remaining branches may have referred to non-installed packages. By
> keeping the := operator in those dependencies, @changed-deps is able
> to correctly match old and new depstring.
> 
> 2. adds a @changed-deps set to rebuild packages for which dependencies
> have changed (ebuild compared to vdb).
> 


Your clear to merge 1 & 2



> 3. adds a repoman check for dependencies that refer to packages that
> have more than one slot yet do not specify a slot explicitly or use
> a proper slot operator. This is meant to help fixing mistakes like:
> 
>   dev-libs/openssl -> dev-libs/openssl:0
>     [:0.9.8 comes with no headers]
> 
> --
> Best regards,
> Michał Górny
> 
> 

You need to get an ack from Chris and/or Tom for the repoman check

-- 
Brian Dolbec <dolsen>

Re: [gentoo-portage-dev] [PATCHES] @changed-deps + missing slot check reposted for bernalex

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 593 bytes --]

Dnia 2014-08-13, o godz. 19:20:32
Michał Górny <mgorny@gentoo.org> napisał(a):

> 1. fixes Portage not to replace non-installed 'dev-foo/bar:=' deps with
> plain 'dev-foo/bar'. This affects e.g. || () dependencies where
> the remaining branches may have referred to non-installed packages. By
> keeping the := operator in those dependencies, @changed-deps is able to
> correctly match old and new depstring.
> 
> 2. adds a @changed-deps set to rebuild packages for which dependencies
> have changed (ebuild compared to vdb).

Pushed these two.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 949 bytes --]