[gentoo-portage-dev] [PATCH] repoman: filter out duplicate dependencies in error messages

[gentoo-portage-dev] [PATCH] repoman: filter out duplicate dependencies in error messages

[Mike Frysinger]

Some packages list the same atom multiple times (e.g. behind diff USE
flags).  If one of them throws an error, we end up listing it more than
once, and the output can get verbose/useless.
---
 pym/repoman/scanner.py | 13 +++++++++++--
 1 file changed, 11 insertions(+), 2 deletions(-)

diff --git a/pym/repoman/scanner.py b/pym/repoman/scanner.py
index d1c10d7..94ada90 100644
--- a/pym/repoman/scanner.py
+++ b/pym/repoman/scanner.py
@@ -704,13 +704,22 @@ class Scanner(object):
 
 								# we have some unsolvable deps
 								# remove ! deps, which always show up as unsatisfiable
-								atoms = [
+								all_atoms = [
 									str(atom.unevaluated_atom)
 									for atom in atoms if not atom.blocker]
 
 								# if we emptied out our list, continue:
-								if not atoms:
+								if not all_atoms:
 									continue
+
+								# Filter out duplicates.  We do this by hand (rather
+								# than use a set) so the order is stable and better
+								# matches the order that's in the ebuild itself.
+								atoms = []
+								for atom in all_atoms:
+									if atom not in atoms:
+										atoms.append(atom)
+
 								if self.options.output_style in ['column']:
 									self.qatracker.add_error(mykey,
 										"%s: %s: %s(%s) %s"
-- 
2.6.2

Re: [gentoo-portage-dev] [PATCH] repoman: filter out duplicate dependencies in error messages

[Brian Dolbec]

On Mon,  4 Jan 2016 16:30:30 -0500
Mike Frysinger <vapier@gentoo.org> wrote:

> Some packages list the same atom multiple times (e.g. behind diff USE
> flags).  If one of them throws an error, we end up listing it more
> than once, and the output can get verbose/useless.
> ---
>  pym/repoman/scanner.py | 13 +++++++++++--
>  1 file changed, 11 insertions(+), 2 deletions(-)
> 
> diff --git a/pym/repoman/scanner.py b/pym/repoman/scanner.py
> index d1c10d7..94ada90 100644
> --- a/pym/repoman/scanner.py
> +++ b/pym/repoman/scanner.py
> @@ -704,13 +704,22 @@ class Scanner(object):
>  
>  								# we
> have some unsolvable deps # remove ! deps, which always show up as
> unsatisfiable
> -
> atoms = [
> +
> all_atoms = [ str(atom.unevaluated_atom)
>  									for
> atom in atoms if not atom.blocker] 
>  								# if
> we emptied out our list, continue:
> -								if
> not atoms:
> +								if
> not all_atoms: continue
> +
> +								#
> Filter out duplicates.  We do this by hand (rather
> +								#
> than use a set) so the order is stable and better
> +								#
> matches the order that's in the ebuild itself.
> +
> atoms = []
> +								for
> atom in all_atoms:
> +
> if atom not in atoms:
> +
> atoms.append(atom) +
>  								if
> self.options.output_style in ['column']:
> self.qatracker.add_error(mykey, "%s: %s: %s(%s) %s"


I immediately want to say REJECT!, REJECT!, REJECT!,...

I just spent a marathon week working on stage2 of the repoman rewrite.

I have all the checks and vcs related code in 2 plugin systems.  I have
to move the vcs plugins to their final destination path still (minor
move)

I am going to start cleaning up the commits, do some rebasing and
unifying of them all now that I have it split up and working.  Still
some more testing/debugging to do.

Hopefully be the end of this week it'll be ready for review and merge
soon.

If this is applied to current repoman, it may cause some rebase hell.

 https://github.com/dol-sen/portage/tree/repoman

That is the repoman branch with the individualized checks that run in 3
small loops in scanner.py now.  There is no code in scanner that does
any checks.  Those are all in pym/repoman/modules/scan/*/*.py.  Some
modules contain several different files and class definitions.  There
are a bunch of new ones that I created from the code that still
remained in scanner.py's _scan_ebuilds().  I'll push the changes to teh
main gentoo portage repo's repoman branch once I have it cleaned up.

I would much prefer you re-base your patch on the rewrite code.

I will make a wiki page for the module definition requirements, with a
section on the vcs system as well.  But the modules are quite simple,
only small changes from the initial code split we did already.  So new
modules are easy to create and add in to the sequence of checks to
perform.  You just have to be careful where you insert checks.  As the
dynamic_data used and updated by the modules varies as it progresses
through the sequence.  I have yet to document the data changed/required
by each of the modules.  But they are quite clear looking at the code.



-- 
Brian Dolbec <dolsen>

Re: [gentoo-portage-dev] [PATCH] repoman: filter out duplicate dependencies in error messages

[Brian Dolbec]

On Mon, 4 Jan 2016 19:17:41 -0800
Brian Dolbec <dolsen@gentoo.org> wrote:

> On Mon,  4 Jan 2016 16:30:30 -0500
> Mike Frysinger <vapier@gentoo.org> wrote:
> 
> > Some packages list the same atom multiple times (e.g. behind diff
> > USE flags).  If one of them throws an error, we end up listing it
> > more than once, and the output can get verbose/useless.
> > ---
> >  pym/repoman/scanner.py | 13 +++++++++++--
> >  1 file changed, 11 insertions(+), 2 deletions(-)
> > 
> > diff --git a/pym/repoman/scanner.py b/pym/repoman/scanner.py
> > index d1c10d7..94ada90 100644
> > --- a/pym/repoman/scanner.py
> > +++ b/pym/repoman/scanner.py
> > @@ -704,13 +704,22 @@ class Scanner(object):
> >  
> >  								#
> > we have some unsolvable deps # remove ! deps, which always show up
> > as unsatisfiable
> > -
> > atoms = [
> > +
> > all_atoms = [ str(atom.unevaluated_atom)
> >  									for
> > atom in atoms if not atom.blocker] 
> >  								#
> > if we emptied out our list, continue:
> > -								if
> > not atoms:
> > +								if
> > not all_atoms: continue
> > +
> > +								#
> > Filter out duplicates.  We do this by hand (rather
> > +								#
> > than use a set) so the order is stable and better
> > +								#
> > matches the order that's in the ebuild itself.
> > +
> > atoms = []
> > +								for
> > atom in all_atoms:
> > +
> > if atom not in atoms:
> > +
> > atoms.append(atom) +
> >  								if
> > self.options.output_style in ['column']:
> > self.qatracker.add_error(mykey, "%s: %s: %s(%s) %s"  
> 
> 
> I immediately want to say REJECT!, REJECT!, REJECT!,...
> 
> I just spent a marathon week working on stage2 of the repoman rewrite.
> 

looks like this is now the pym/repoman/modules/scan/depend/profile.py
check

-- 
Brian Dolbec <dolsen>

Re: [gentoo-portage-dev] [PATCH] repoman: filter out duplicate dependencies in error messages

[Mike Frysinger]

[-- Attachment #1: Type: text/plain, Size: 233 bytes --]

On 04 Jan 2016 19:17, Brian Dolbec wrote:
> I immediately want to say REJECT!, REJECT!, REJECT!,...
> ...
> I would much prefer you re-base your patch on the rewrite code.

reject != delay ... i don't mind waiting and rebasing
-mike

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [gentoo-portage-dev] [PATCH] repoman: filter out duplicate dependencies in error messages

[Zac Medico]

On 01/04/2016 01:30 PM, Mike Frysinger wrote:
> +								# Filter out duplicates.  We do this by hand (rather
> +								# than use a set) so the order is stable and better
> +								# matches the order that's in the ebuild itself.
> +								atoms = []
> +								for atom in all_atoms:
> +									if atom not in atoms:
> +										atoms.append(atom)
> +

Alternative implementation using OrderedDict which has O(1) average
complexity for containment checks, rather than O(N):

    atoms = OrderedDict()
    for atom in all_atoms:
        atoms.setdefault(atom, atom)
    atoms = list(atoms)

-- 
Thanks,
Zac