[gentoo-dev] Things one could be upset about

[gentoo-dev] Things one could be upset about

[Patrick Lauer]

Here's a random unsorted list of things that it would make sense to be upset 
about. Some issues that people have successfully ignored for a few years ...

In no way exhaustive list, feel free to remember a dozen things I forgot ;)
(If you suggest other things please try to offer constructive criticism,
i.e. possible strategies to fix issues ... whining by itself is not very 
useful) 

* stable genkernel still doesn't enable all useful kernel features
   e.g. accounting statistics are absent, so iotop doesn't work ootb
   See for example #442936 (from 2012 ?!)
   Fix: Stable newer versions

* stage3 still enables bindist in make.conf
   See https://bugs.gentoo.org/show_bug.cgi?id=473332
   Causes random breakage of tools like wget/curl, etc.
   Fix: poke releng until they stop being silly 

* AutoRepoman catches on average maybe 2 user-visible breakages.
    Mostly removing stable on HPPA ;)
    Fix: Make repoman faster (tree-wide scans take ~2 CPU-hours)
    Fix: Remind people that using repoman is not optional

* Portage is too slow
    On 'small' hardware emerge -upNDv @world can take enough time 
    to make updates prohibitive - on an 800Mhz machine it took me 
    about 3 days to figure out a solution to some silly blockers due to the
    very slow change test cycle
    Fix: Do some profiling and un-refactoring to remove useless code
    Fix: Set up a reference system (CI) to catch future regressions

* Stage3 archives are too fat
    See https://bugs.gentoo.org/show_bug.cgi?id=531632
    We're now shipping three python versions and glib for extra fun!
    Fix: Motivate releng to stop being silly

* AutoRepoman catches issues, but no one but me seems to care
    Fix: Remind people of http://packages.gentooexperimental.org/repoman-current-issues.txt
    Fix: Make it yell louder? It currently reports on IRC to #gentoo-{bugs,qa}

* AutoRepoman still runs on my hardware
    Fix: Remind infra of https://bugs.gentoo.org/show_bug.cgi?id=484064

* mail archives have been broken since 2012
    Fix: get infra to either fix it, or provide enough information that it can 
    be fixed. See https://bugs.gentoo.org/show_bug.cgi?id=424647#c27

* git.overlays.gentoo.org only partially functional (web interface / cgit 
down)

* euscan doesn't run on infra hardware
    Fix: https://bugs.gentoo.org/show_bug.cgi?id=453886

* libreoffice-bin isn't built on infra hardware
    Fix: Remind infra to set up a build environment for that

* Some stable bugs are left alone for months
   See e.g. https://bugs.gentoo.org/show_bug.cgi?id=485632
   Fix: Have more people work on stable bugs
   Fix: Motivate people to file more stable bugs (continuous updates)

* Updates from very old installs are still difficult
   Fix: Collect stage3 archives and tree snapshots maybe every 3 months apart
     then update from one snapshot to the next. Possibly fix upgrade bugs 
     retroactively in the snapshots and automate the process


It'd be nice to have such things fixed, but alas, many rely on someone else to 
respond. And for some there's no 'easy' solution.

But they can all be fixed.

Let's not tolerate mediocrity.

Take care,

Patrick

Re: [gentoo-dev] Things one could be upset about

[Dirkjan Ochtman]

On Sat, Jan 17, 2015 at 12:35 PM, Patrick Lauer <patrick@gentoo.org> wrote:
> In no way exhaustive list, feel free to remember a dozen things I forgot ;)
> (If you suggest other things please try to offer constructive criticism,
> i.e. possible strategies to fix issues ... whining by itself is not very
> useful)

I think this is a good list, thanks for coming up with it. I very much
agree that whining by itself is not very useful, and we should find
ways to move things forward (even if sometimes it's not the most
straightforward way?).

> * AutoRepoman catches on average maybe 2 user-visible breakages.
>     Mostly removing stable on HPPA ;)
>     Fix: Make repoman faster (tree-wide scans take ~2 CPU-hours)
>     Fix: Remind people that using repoman is not optional
>
> * AutoRepoman catches issues, but no one but me seems to care
>     Fix: Remind people of http://packages.gentooexperimental.org/repoman-current-issues.txt
>     Fix: Make it yell louder? It currently reports on IRC to #gentoo-{bugs,qa}

I'll happily fix my repoman issues when I notice them. I try to always
run repoman full on a package, but like you say, a tree-wide scan
isn't really viable on a per-commit basis. Can we have AutoRepoman
report open issues to gentoo-dev on a weekly basis? That seems like a
fine idea to me. Per-maintainer and per-team/project/herd RSS/Atom
feeds would also be pretty nice to have.

Also, I hate something like
"['dev-python/restkit[python_targets_python2_7(-)?,-python_single_target_python2_7(-)]']".
What the hell kind of warning is that? I guess maybe these are the
results of USE_EXPAND trickery and what not, but it would sure be nice
to have something more readable.

> * Portage is too slow
>     On 'small' hardware emerge -upNDv @world can take enough time
>     to make updates prohibitive - on an 800Mhz machine it took me
>     about 3 days to figure out a solution to some silly blockers due to the
>     very slow change test cycle
>     Fix: Do some profiling and un-refactoring to remove useless code
>     Fix: Set up a reference system (CI) to catch future regressions

Why not use paludis, or another package manager?

> * Stage3 archives are too fat
>     See https://bugs.gentoo.org/show_bug.cgi?id=531632
>     We're now shipping three python versions and glib for extra fun!
>     Fix: Motivate releng to stop being silly

Why the heck do we ship both 3.3 and 3.4? I forget the exact situation
with 2.x and 3.x, but I don't think setting PYTHON_TARGETS to 2.7-only
is a great option if that remains the default after installation
(although it would be fine for just the initial stages).

> * AutoRepoman still runs on my hardware
>     Fix: Remind infra of https://bugs.gentoo.org/show_bug.cgi?id=484064
>
> * euscan doesn't run on infra hardware
>     Fix: https://bugs.gentoo.org/show_bug.cgi?id=453886
>
> * mail archives have been broken since 2012
>     Fix: get infra to either fix it, or provide enough information that it can
>     be fixed. See https://bugs.gentoo.org/show_bug.cgi?id=424647#c27
>
> * libreoffice-bin isn't built on infra hardware
>     Fix: Remind infra to set up a build environment for that

These ones, the euscan one in particular, also have been stuff I cared
about, but after trying a bunch of times to start helping infra out,
apparently they don't have any ways to absorb new contributors. This
does seem like a big problem given the amount of technical debt on the
infra-side you're laying out here.

I understand that it's a big job and that there are security aspects
to it, but if there are no ways to empower new people to help them
out, that is worrying to me.

> * Some stable bugs are left alone for months
>    See e.g. https://bugs.gentoo.org/show_bug.cgi?id=485632
>    Fix: Have more people work on stable bugs
>    Fix: Motivate people to file more stable bugs (continuous updates)

This is a thorny problem as well. I worry that we lose momentum here
due to size and perfectionism (e.g. we can only stable new gcc once we
fix all the blockers, and we don't have enough active maintainers on
some of those blockers). I think we should maybe stabilize more
optimistically at least on common architectures, e.g. by having a
lighter-weight upfront testing process and relying more on maintainers
keeping up to speed on subsequent bugs.

I also wonder if we could sort of crowd-source archtesting, maybe by
having people contribute their package.keywords through gentoo-stats
or some such to see how well an unstable package is being tested on
stable systems already.

Or if we should have a different process for e.g. Python/Ruby/Perl/PHP
packages compared to C/C++ packages, since the former are much less
sensitive to architecture differences.

Also, one thing you didn't mention is the git migration; I think our
usage of CVS definitely counts as a big gob of technical debt. What's
that currently blocked on, anyway?

Cheers,

Dirkjan

Re: [gentoo-dev] Things one could be upset about

[Patrick Lauer]

On Saturday 17 January 2015 13:44:21 Dirkjan Ochtman wrote:
> On Sat, Jan 17, 2015 at 12:35 PM, Patrick Lauer <patrick@gentoo.org> wrote:

> > * AutoRepoman catches issues, but no one but me seems to care
> > 
> >     Fix: Remind people of
> >     http://packages.gentooexperimental.org/repoman-current-issues.txt
> >     Fix: Make it yell louder? It currently reports on IRC to
> >     #gentoo-{bugs,qa}
> I'll happily fix my repoman issues when I notice them. I try to always
> run repoman full on a package, but like you say, a tree-wide scan
> isn't really viable on a per-commit basis. Can we have AutoRepoman
> report open issues to gentoo-dev on a weekly basis? That seems like a
> fine idea to me. Per-maintainer and per-team/project/herd RSS/Atom
> feeds would also be pretty nice to have.

Most issues are transient, often fixed with either a keywording bug, or careful 
masking of useflags / pruning of old versions. Per-maintainer doesn't really 
make sense as most issues are indirect - things like "removing package.mask 
entry for new version causes dependency breakage on ia64 to become visible", 
or "dependency of dependency changed". In both cases it's often hard enough to 
figure out what caused the issue.
 
> Also, I hate something like
> "['dev-python/restkit[python_targets_python2_7(-)?,-python_single_target_pyt
> hon2_7(-)]']". What the hell kind of warning is that? I guess maybe these
> are the results of USE_EXPAND trickery and what not, but it would sure be
> nice to have something more readable.

Indeed, portage output is quite complex and often hard to read. I'm not sure 
what's the best way to improve it - I've filed a few small bugs for output 
prettyfication, but I don't even know what I'd want to be displayed for these 
useflag / blocker issues.

> 
> > * Portage is too slow
> > 
> >     On 'small' hardware emerge -upNDv @world can take enough time
> >     to make updates prohibitive - on an 800Mhz machine it took me
> >     about 3 days to figure out a solution to some silly blockers due to
> >     the
> >     very slow change test cycle
> >     Fix: Do some profiling and un-refactoring to remove useless code
> >     Fix: Set up a reference system (CI) to catch future regressions
> 
> Why not use paludis, or another package manager?

Last time I tested paludis it was slower (and building it OOMed on that 
machine with default settings, so that's quite amusing)

Also it suffers from a hostile upstream, makes bugreports very tricky to handle 
etc.etc.

Pkgcore is in hibernation, it needs a bit more work to become really useful 
again. Possibly some ideas from pkgcore can be migrated to portage again, but 
that'd need someone with lots of free time.


Thanks for your feedback,

Patrick

Re: [gentoo-dev] Things one could be upset about

[Dirkjan Ochtman]

On Sat, Jan 17, 2015 at 2:03 PM, Patrick Lauer <patrick@gentoo.org> wrote:
> Most issues are transient, often fixed with either a keywording bug, or careful
> masking of useflags / pruning of old versions. Per-maintainer doesn't really
> make sense as most issues are indirect - things like "removing package.mask
> entry for new version causes dependency breakage on ia64 to become visible",
> or "dependency of dependency changed". In both cases it's often hard enough to
> figure out what caused the issue.

I think louder in the sense of a weekly -dev email could still makes
sense. #-bugs and #-qa are way too noisy for me.

Cheers,

Dirkjan

Re: [gentoo-dev] Things one could be upset about

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 197 bytes --]

On Sat, 17 Jan 2015 21:03:30 +0800
Patrick Lauer <patrick@gentoo.org> wrote:
> Last time I tested paludis it was slower

You've yet to do a like-for-like comparison...

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: [gentoo-dev] Things one could be upset about

[Patrick Lauer]

On Saturday 17 January 2015 14:32:03 Ciaran McCreesh wrote:
> On Sat, 17 Jan 2015 21:03:30 +0800
> 
> Patrick Lauer <patrick@gentoo.org> wrote:
> > Last time I tested paludis it was slower
> 
> You've yet to do a like-for-like comparison...

Hello hostile upstream.

It was as "like for like" as possible, even when there had to be workaround to 
make paludis build (and wait multiple hours as it is kinda horribly C++) as it 
tried to OOM very nicely.

You can continue whining about it as much as you want, my results were 
duplicated by multiple people and you never offered a tweaked comparison that 
is to your liking. So stop whining (which is amusingly the trigger for my 
initial email - people whining about irrelevant things instead of doing 
something useful).

Re: [gentoo-dev] Things one could be upset about

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 463 bytes --]

On Sat, 17 Jan 2015 22:58:33 +0800
Patrick Lauer <patrick@gentoo.org> wrote:
> On Saturday 17 January 2015 14:32:03 Ciaran McCreesh wrote:
> > On Sat, 17 Jan 2015 21:03:30 +0800
> > Patrick Lauer <patrick@gentoo.org> wrote:
> > > Last time I tested paludis it was slower
> > 
> > You've yet to do a like-for-like comparison...
> 
> Hello hostile upstream.
> 
> It was as "like for like" as possible

Well that's the point...

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: [gentoo-dev] Things one could be upset about

[Patrick Lauer]

On Saturday 17 January 2015 15:03:05 Ciaran McCreesh wrote:
> On Sat, 17 Jan 2015 22:58:33 +0800
> 
> Patrick Lauer <patrick@gentoo.org> wrote:
> > On Saturday 17 January 2015 14:32:03 Ciaran McCreesh wrote:
> > > On Sat, 17 Jan 2015 21:03:30 +0800
> > > 
> > > Patrick Lauer <patrick@gentoo.org> wrote:
> > > > Last time I tested paludis it was slower
> > > 
> > > You've yet to do a like-for-like comparison...
> > 
> > Hello hostile upstream.
> > 
> > It was as "like for like" as possible
> 
> Well that's the point...

A rare case of violent agreement?

Excellent. And/or you're terminally bored - either way you're not adding 
anything relevant to this conversation.

Re: [gentoo-dev] Things one could be upset about

[Pacho Ramos]

El sáb, 17-01-2015 a las 13:44 +0100, Dirkjan Ochtman escribió:
[...]
> Also, I hate something like
> "['dev-python/restkit[python_targets_python2_7(-)?,-python_single_target_python2_7(-)]']".
> What the hell kind of warning is that? I guess maybe these are the
> results of USE_EXPAND trickery and what not, but it would sure be nice
> to have something more readable.

Yeah, sometimes the output are really fat, not sure if some heuristic
could be done to, for example, collate the exact same errors that are
coming from every single subprofile.

Re: [gentoo-dev] Things one could be upset about

[Joshua Kinard]

On 01/17/2015 08:21, Pacho Ramos wrote:
> El sáb, 17-01-2015 a las 13:44 +0100, Dirkjan Ochtman escribió:
> [...]
>> Also, I hate something like
>> "['dev-python/restkit[python_targets_python2_7(-)?,-python_single_target_python2_7(-)]']".
>> What the hell kind of warning is that? I guess maybe these are the
>> results of USE_EXPAND trickery and what not, but it would sure be nice
>> to have something more readable.
> 
> Yeah, sometimes the output are really fat, not sure if some heuristic
> could be done to, for example, collate the exact same errors that are
> coming from every single subprofile.

I've been spending the better half of the last two days trying to kickstart
catalyst runs on two of my SGI systems, one doing o32 and the other n32.  Using
seed stage3 stages I built 6 months ago (but never released due to getting
sidetracked), I run into errors like this:

!!! Multiple package instances within a single package slot have been pulled
!!! into the dependency graph, resulting in a slot conflict:

dev-lang/perl:0

  (dev-lang/perl-5.20.1-r4:0/5.20::gentoo, ebuild scheduled for merge) pulled in by
    =dev-lang/perl-5.20* required by
(virtual/perl-ExtUtils-ParseXS-3.240.0:0/0::gentoo, ebuild scheduled for merge)
    ^              ^^^^^
    (and 16 more with the same problem)

  (dev-lang/perl-5.18.2-r2:0/5.18::gentoo, ebuild scheduled for merge) pulled in by
    dev-lang/perl:0/5.18=[-build(-)] required by
(dev-perl/libintl-perl-1.230.0:0/0::gentoo, installed)
                 ^^^^^^^^
    =dev-lang/perl-5.18* required by
(virtual/perl-ExtUtils-Manifest-1.630.0-r1:0/0::gentoo, installed)
    ^              ^^^^^
    (and 2 more with the same problems)

It's hard to read mess like that and trace down the offending package, fix it,
and make catalyst happy.  Got bit by the splitting of libltdl and libtool as
well.  Several packages included a block on <=libtool-2.4.2, which was in my
stage3 builds from last summer.  Not an easy way to work around those in catalyst.

Eventually just unpacked the seed stage3 on both systems, updated
libtool/libltdl, repacked them, and used those as the as seed stages.  Kinda
defeats the purpose of catalyst in the first place.  Looks like I have to
repeat for perl now, which seems to do this every major update.

-- 
Joshua Kinard
Gentoo/MIPS
kumba@gentoo.org
4096R/D25D95E3 2011-03-28

"The past tempts us, the present confuses us, the future frightens us.  And our
lives slip away, moment by moment, lost in that vast, terrible in-between."

--Emperor Turhan, Centauri Republic

Re: [gentoo-dev] Things one could be upset about

[Peter Stuge]

Joshua Kinard wrote:
> Using seed stage3 stages I built 6 months ago (but never released due
> to getting sidetracked), I run into errors like this:
> 
> !!! Multiple package instances within a single package slot have been pulled
> !!! into the dependency graph, resulting in a slot conflict:
> 
> dev-lang/perl:0
> 
>   (dev-lang/perl-5.20.1-r4:0/5.20::gentoo, ebuild scheduled for merge) pulled in by
>     =dev-lang/perl-5.20* required by
> (virtual/perl-ExtUtils-ParseXS-3.240.0:0/0::gentoo, ebuild scheduled for merge)
>     ^              ^^^^^
>     (and 16 more with the same problem)
> 
>   (dev-lang/perl-5.18.2-r2:0/5.18::gentoo, ebuild scheduled for merge) pulled in by
>     dev-lang/perl:0/5.18=[-build(-)] required by
> (dev-perl/libintl-perl-1.230.0:0/0::gentoo, installed)
>                  ^^^^^^^^
>     =dev-lang/perl-5.18* required by
> (virtual/perl-ExtUtils-Manifest-1.630.0-r1:0/0::gentoo, installed)
>     ^              ^^^^^
>     (and 2 more with the same problems)
> 
> It's hard to read mess like that and trace down the offending package,
> fix it, and make catalyst happy.

Lots of dev-perl packages have specific minor version dependencies on
dev-lang/perl, maybe because sometimes the package is included in perl
and sometimes not. It's a f*ing mess. You have to look up all your
installed dev-perl packages manually and find which ones are either
too old to know about perl-5.20 or not compatible with it, and then
you have to unmerge those manually.


> Kinda defeats the purpose of catalyst in the first place.

The proper way is to build stage1+2+3 yourself, then this mess
doesn't happen. But like you I too cheat a little, and have to deal
with the mess.


//Peter

Re: [gentoo-dev] Things one could be upset about

[Joshua Kinard]

On 01/22/2015 10:19, Peter Stuge wrote:
> Joshua Kinard wrote:
>> Using seed stage3 stages I built 6 months ago (but never released due
>> to getting sidetracked), I run into errors like this:
>>
>> !!! Multiple package instances within a single package slot have been pulled
>> !!! into the dependency graph, resulting in a slot conflict:
>>
>> dev-lang/perl:0
>>
>>   (dev-lang/perl-5.20.1-r4:0/5.20::gentoo, ebuild scheduled for merge) pulled in by
>>     =dev-lang/perl-5.20* required by
>> (virtual/perl-ExtUtils-ParseXS-3.240.0:0/0::gentoo, ebuild scheduled for merge)
>>     ^              ^^^^^
>>     (and 16 more with the same problem)
>>
>>   (dev-lang/perl-5.18.2-r2:0/5.18::gentoo, ebuild scheduled for merge) pulled in by
>>     dev-lang/perl:0/5.18=[-build(-)] required by
>> (dev-perl/libintl-perl-1.230.0:0/0::gentoo, installed)
>>                  ^^^^^^^^
>>     =dev-lang/perl-5.18* required by
>> (virtual/perl-ExtUtils-Manifest-1.630.0-r1:0/0::gentoo, installed)
>>     ^              ^^^^^
>>     (and 2 more with the same problems)
>>
>> It's hard to read mess like that and trace down the offending package,
>> fix it, and make catalyst happy.
> 
> Lots of dev-perl packages have specific minor version dependencies on
> dev-lang/perl, maybe because sometimes the package is included in perl
> and sometimes not. It's a f*ing mess. You have to look up all your
> installed dev-perl packages manually and find which ones are either
> too old to know about perl-5.20 or not compatible with it, and then
> you have to unmerge those manually.

In the past, it's been possible to have Portage deal with the updates to Perl,
but only as long as you hit all of the packages in the same update run to
satisfy the dependency chain.  Newer portage seems to not do that anymore.  But
that output is horrible.  Even with the color coding, it's not directly
apparent which package is the problem package.

I once had a Perl update issue bad enough that I removed all perl packages
entirely, then remerged them from scratch.  Took a while, but it fixed things.


>> Kinda defeats the purpose of catalyst in the first place.
> 
> The proper way is to build stage1+2+3 yourself, then this mess
> doesn't happen. But like you I too cheat a little, and have to deal
> with the mess.

Well, I was trying to do it the right way by going stage1 -> stage2 -> stage3.
 I was using a stage3 that I built over the summer as the seed stage for the
new stage1 when I started running into problems with Perl.  I finally fixed
that, got stage1 built, then got bit by Bug #447126 while trying to build the
stage2.  So now, I have to start a stage2 run, then after the unpack (but
before catalyst drops into the chroot), edit the chroot's make.conf and remove
sandbox from FEATURES, which is apparently part of the problem.

Just irritating.  And I know I'm earning no sympathy when I point out that my
build machines (an Octane and an Onyx2) aren't the fastest things on the
planet, nor the most power efficient (1 kW between the both of them).  But I'd
at least like to waste that power on actual compile jobs, not watching emerge's
little spinner all the time as I try to fix various dependency bugs or other
oddities that seemingly came out of nowhere (because the summertime stage runs
were flawless in execution).

-- 
Joshua Kinard
Gentoo/MIPS
kumba@gentoo.org
4096R/D25D95E3 2011-03-28

"The past tempts us, the present confuses us, the future frightens us.  And our
lives slip away, moment by moment, lost in that vast, terrible in-between."

--Emperor Turhan, Centauri Republic

Re: [gentoo-dev] Things one could be upset about

[William Hubbs]

[-- Attachment #1: Type: text/plain, Size: 1181 bytes --]

On Sat, Jan 17, 2015 at 01:44:21PM +0100, Dirkjan Ochtman wrote:
> On Sat, Jan 17, 2015 at 12:35 PM, Patrick Lauer <patrick@gentoo.org> wrote:
> > * Stage3 archives are too fat
> >     See https://bugs.gentoo.org/show_bug.cgi?id=531632
> >     We're now shipping three python versions and glib for extra fun!
> >     Fix: Motivate releng to stop being silly
> 
> Why the heck do we ship both 3.3 and 3.4? I forget the exact situation
> with 2.x and 3.x, but I don't think setting PYTHON_TARGETS to 2.7-only
> is a great option if that remains the default after installation
> (although it would be fine for just the initial stages).

I'm going to be very blunt. I am sick of the finger being pointed
only at releng for this.

The issue is package dependencies.

If even one package in the tree has a dumb dependency on python, e.g.
dev-lang/python, it will pull in all stable versionf of python.

If all the dependencies are correct, on the other hand, it is a matter
of setting python_targets to include the versions of python we want in
the stages and making sure that everything we include in the stages
works with those versions of python.

William


[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: [gentoo-dev] Things one could be upset about

[Matt Turner]

On Sat, Jan 17, 2015 at 12:00 PM, William Hubbs <williamh@gentoo.org> wrote:
> On Sat, Jan 17, 2015 at 01:44:21PM +0100, Dirkjan Ochtman wrote:
>> On Sat, Jan 17, 2015 at 12:35 PM, Patrick Lauer <patrick@gentoo.org> wrote:
>> > * Stage3 archives are too fat
>> >     See https://bugs.gentoo.org/show_bug.cgi?id=531632
>> >     We're now shipping three python versions and glib for extra fun!
>> >     Fix: Motivate releng to stop being silly
>>
>> Why the heck do we ship both 3.3 and 3.4? I forget the exact situation
>> with 2.x and 3.x, but I don't think setting PYTHON_TARGETS to 2.7-only
>> is a great option if that remains the default after installation
>> (although it would be fine for just the initial stages).
>
> I'm going to be very blunt. I am sick of the finger being pointed
> only at releng for this.

Well, there are trivial things that could be done on the stage
building end... and complete intransigence toward totally reasonable
ideas like not building with default options for a particularly
specialized target (stages, livecds, etc.).

Blame whomever you like, but ultimately the stages are a product of
the release engineering team who has the ability to fix the issue
themselves but choose not to.

Re: [gentoo-dev] Things one could be upset about

[Patrick Lauer]

On Saturday 17 January 2015 14:00:34 William Hubbs wrote:
> On Sat, Jan 17, 2015 at 01:44:21PM +0100, Dirkjan Ochtman wrote:
> > On Sat, Jan 17, 2015 at 12:35 PM, Patrick Lauer <patrick@gentoo.org> 
wrote:
> > > * Stage3 archives are too fat
> > > 
> > >     See https://bugs.gentoo.org/show_bug.cgi?id=531632
> > >     We're now shipping three python versions and glib for extra fun!
> > >     Fix: Motivate releng to stop being silly
> > 
> > Why the heck do we ship both 3.3 and 3.4? I forget the exact situation
> > with 2.x and 3.x, but I don't think setting PYTHON_TARGETS to 2.7-only
> > is a great option if that remains the default after installation
> > (although it would be fine for just the initial stages).
> 
> I'm going to be very blunt. I am sick of the finger being pointed
> only at releng for this.
> 
> The issue is package dependencies.
> 
> If even one package in the tree has a dumb dependency on python, e.g.
> dev-lang/python, it will pull in all stable versionf of python.

Only if you absolutely insist that releng can never deviate from tree.

Which is a silly idea, see bindist useflag, which is locally enabled for stage 
building and then removed. Oh wait, it's not removed because we can't deviate 
while deviating. So users regularly find ssl 'broken' ...

It took me about 2h of fiddling around to find a few spots where stage3 has 
useless bloat:
- pkgconfig pulling in 30MB of glib
- python installing tests (that's 3x 25MB now ...)
- having more than one python installed (which is not really absolutely 
necessary, and could easily be reduced to one)

Out of 700MB on-disk I could prune about 150MB - about 20%, without affecting 
functionality

It's not about pointing a finger, it's about fixing issues when they are easy to 
fix and not hiding behind a fake complexity argument.
(I would fix things, if I had access and/or certainty that patches provided 
would be integrated ...)

Have fun,

Patrick

Re: [gentoo-dev] Things one could be upset about

[Jorge Manuel B. S. Vicetto]

On Sun, 18 Jan 2015, Patrick Lauer wrote:

> On Saturday 17 January 2015 14:00:34 William Hubbs wrote:
>> On Sat, Jan 17, 2015 at 01:44:21PM +0100, Dirkjan Ochtman wrote:
>>> On Sat, Jan 17, 2015 at 12:35 PM, Patrick Lauer <patrick@gentoo.org>
> wrote:
>>>> * Stage3 archives are too fat
>>>>
>>>>     See https://bugs.gentoo.org/show_bug.cgi?id=531632
>>>>     We're now shipping three python versions and glib for extra fun!
>>>>     Fix: Motivate releng to stop being silly
>>>
>>> Why the heck do we ship both 3.3 and 3.4? I forget the exact situation
>>> with 2.x and 3.x, but I don't think setting PYTHON_TARGETS to 2.7-only
>>> is a great option if that remains the default after installation
>>> (although it would be fine for just the initial stages).
>>
>> I'm going to be very blunt. I am sick of the finger being pointed
>> only at releng for this.
>>
>> The issue is package dependencies.
>>
>> If even one package in the tree has a dumb dependency on python, e.g.
>> dev-lang/python, it will pull in all stable versionf of python.
>
> Only if you absolutely insist that releng can never deviate from tree.

What RelEng has insisted in is building what we have in the tree - which, 
curiously enough, is a good way to test the tree.
Anyway, I've started working on a portdir config for the default stages, 
not because of this, but because of the caps issue (bug 531788).

> Which is a silly idea, see bindist useflag, which is locally enabled for stage
> building and then removed. Oh wait, it's not removed because we can't deviate
> while deviating. So users regularly find ssl 'broken' ...

Building stages with USE="bindist" isn't a question of being silly but of 
making sure we don't give anyone a reason to "sue" Gentoo. We don't want 
to force USE="bindist" in the released stages, but that means we add one 
more workaround to catalyst code or we provide a way to specify which use 
flags we want to use for building the stage and which we want to be kept 
in make.conf.

> It took me about 2h of fiddling around to find a few spots where stage3 has
> useless bloat:
> - pkgconfig pulling in 30MB of glib
> - python installing tests (that's 3x 25MB now ...)

internal-glib is not something that should be used in the stages.

> - having more than one python installed (which is not really absolutely
> necessary, and could easily be reduced to one)

If your system has 3 python versions installed because the tree deps make 
portage install 3 versions, you should expect that to happen in your 
stages.
Since I'll have to work with portdir to address the caps issue above, I'll 
also take care of this, but if you want to blame someone about this, 
releng doesn't "maintain the tree".
Also, by adding this to the releng repo, that means we're going to have 
more work as we'll have to track new python versions.

> Out of 700MB on-disk I could prune about 150MB - about 20%, without affecting
> functionality
>
> It's not about pointing a finger, it's about fixing issues when they are easy to
> fix and not hiding behind a fake complexity argument.
> (I would fix things, if I had access and/or certainty that patches provided
> would be integrated ...)

I don't recall ever having seen a patch from you to catalyst, so before 
you "suggest" we don't want to accept your patches, you may want to send 
one ;-)

> Have fun,
>
> Patrick

Have fun,
Jorge

Re: [gentoo-dev] Things one could be upset about

[Dirkjan Ochtman]

On Sat, Jan 17, 2015 at 9:00 PM, William Hubbs <williamh@gentoo.org> wrote:
>> Why the heck do we ship both 3.3 and 3.4? I forget the exact situation
>> with 2.x and 3.x, but I don't think setting PYTHON_TARGETS to 2.7-only
>> is a great option if that remains the default after installation
>> (although it would be fine for just the initial stages).
>
> I'm going to be very blunt. I am sick of the finger being pointed
> only at releng for this.

To be quite clear, I didn't intend at all to point the finger at
anyone. I mostly reckon it's due to an unfortunate way things hang
together, definitely not any one person or group to blame. I just
really don't understand how it happens.

Cheers,

Dirkjan

Re: [gentoo-dev] Things one could be upset about

[William Hubbs]

[-- Attachment #1: Type: text/plain, Size: 976 bytes --]

On Sun, Jan 18, 2015 at 01:21:56PM +0100, Dirkjan Ochtman wrote:
> On Sat, Jan 17, 2015 at 9:00 PM, William Hubbs <williamh@gentoo.org> wrote:
> >> Why the heck do we ship both 3.3 and 3.4? I forget the exact situation
> >> with 2.x and 3.x, but I don't think setting PYTHON_TARGETS to 2.7-only
> >> is a great option if that remains the default after installation
> >> (although it would be fine for just the initial stages).
> >
> > I'm going to be very blunt. I am sick of the finger being pointed
> > only at releng for this.
> 
> To be quite clear, I didn't intend at all to point the finger at
> anyone. I mostly reckon it's due to an unfortunate way things hang
> together, definitely not any one person or group to blame. I just
> really don't understand how it happens.
> 
> Cheers,
> 
> Dirkjan
 
 Hey Dirkjan,

 I didn't mean to imply that *you* personally were pointing the finger
 at anyone, sorry about the misunderstanding.

 William


[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: [gentoo-dev] Things one could be upset about

[Róbert Čerňanský]

On Sat, 17 Jan 2015 13:44:21 +0100
Dirkjan Ochtman <djc@gentoo.org> wrote:

> On Sat, Jan 17, 2015 at 12:35 PM, Patrick Lauer <patrick@gentoo.org>
> wrote:
> 
> > * Some stable bugs are left alone for months
> >    See e.g. https://bugs.gentoo.org/show_bug.cgi?id=485632
> >    Fix: Have more people work on stable bugs
> >    Fix: Motivate people to file more stable bugs (continuous
> > updates)
> 
> This is a thorny problem as well. I worry that we lose momentum here
> due to size and perfectionism (e.g. we can only stable new gcc once we
> fix all the blockers, and we don't have enough active maintainers on
> some of those blockers). I think we should maybe stabilize more
> optimistically
[...]
> I also wonder if we could sort of crowd-source archtesting, maybe by
> having people contribute their package.keywords through gentoo-stats
> or some such to see how well an unstable package is being tested on
> stable systems already.

This could help in a sense that developers would have more confidence
when stabilizing packages.  But even without such statistics there is
for example the number of open bugs that gives a clue about the
quality of a package.  It should be used to drive stabilizations in
the spirit of good old rule "1 month without bugs => stabilize". At
least for less critical packages, mainly end-user applications.  I
recall that some time ago there were some activities regarding this
rule but I am not sure if it is in place.  So I would add one more fix
for this issue:
Fix: Apply "1 month without bugs => stabilize" rule more often.

Also I think that end-user applications could be stabilized little
more aggresivelly while libraries could keep current conservative
approach.

For example, having installed ~1700 packages of which ~500 are in
world file, recent update world after two months gave me: 292 packages
(183 upgrades, 60 new, 4 in new slots, 45 reinstalls).  However
counting number of end-user applications that were updated I end up
with 9 of them of which only 6 was a somewhat major update that could
bring new features.  (I do not consider system utilites -- like for
example lsof -- as end-user applications even that number of them are
in my world file.)

So if you look to it from this perspective such update does not look
very efficient since out of ~300 builded packages only 6 brings
potential to increase productivity/bring new features.  Such
experiences brings me to conclusion that end-user applications may be
stabilized more often.

Regards,
Robert


-- 
Róbert Čerňanský
E-mail: openhs@tightmail.com
Jabber: hs@jabber.sk

Re: [gentoo-dev] Things one could be upset about

[Jeroen Roovers]

On Sat, 17 Jan 2015 13:44:21 +0100
Dirkjan Ochtman <djc@gentoo.org> wrote:

> Also, I hate something like
> "['dev-python/restkit[python_targets_python2_7(-)?,-python_single_target_python2_7(-)]']".
> What the hell kind of warning is that? I guess maybe these are the
> results of USE_EXPAND trickery and what not, but it would sure be nice
> to have something more readable.

https://bugs.gentoo.org/show_bug.cgi?id=534022


     jer

Re: [gentoo-dev] Things one could be upset about

[Pacho Ramos]

El sáb, 17-01-2015 a las 19:35 +0800, Patrick Lauer escribió:
[...]
> * stable genkernel still doesn't enable all useful kernel features
>    e.g. accounting statistics are absent, so iotop doesn't work ootb
>    See for example #442936 (from 2012 ?!)
>    Fix: Stable newer versions
> 

I see there isn't even any stable bug report for that, is anyone major
blocking newer versions to be stabilized? 

> * AutoRepoman catches issues, but no one but me seems to care
>     Fix: Remind people of http://packages.gentooexperimental.org/repoman-current-issues.txt
>     Fix: Make it yell louder? It currently reports on IRC to #gentoo-{bugs,qa}
> 

Maybe some warnings could be raised to fatal to prevent to commit them
(that shouldn't be a problem as that are usually easy to fix problems).
For example, the dodoc "LICENSE" stuff and WANT_AUTOCONF=latest

> * mail archives have been broken since 2012
>     Fix: get infra to either fix it, or provide enough information that it can 
>     be fixed. See https://bugs.gentoo.org/show_bug.cgi?id=424647#c27
> 

Is there any advantage of setting them up again instead of relying on
other existing external resources? :/

Re: [gentoo-dev] Things one could be upset about

[Andrew Savchenko]

[-- Attachment #1: Type: text/plain, Size: 886 bytes --]

On Sat, 17 Jan 2015 19:35:09 +0800 Patrick Lauer wrote:
> * Portage is too slow
>     On 'small' hardware emerge -upNDv @world can take enough time 
>     to make updates prohibitive - on an 800Mhz machine it took me 
>     about 3 days to figure out a solution to some silly blockers due to the
>     very slow change test cycle
>     Fix: Do some profiling and un-refactoring to remove useless code
>     Fix: Set up a reference system (CI) to catch future regressions

There is some progress here. In portage-2.2.15 profile based
optimizations are included (see bugs 529660, 530010). On my
hardware (Athlon-XP, 2200 MHz and Intel Atom N270, 1600 MHz) they
speed up dependency resolution by ~ factor 2.

Of course it will be great to see further optimizations, though as
far as I remember this will require more complicated changes.

Best regards,
Andrew Savchenko

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: [gentoo-dev] Things one could be upset about

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 541 bytes --]

On Sat, 17 Jan 2015 17:39:29 +0300
Andrew Savchenko <bircoph@gentoo.org> wrote:
> There is some progress here. In portage-2.2.15 profile based
> optimizations are included (see bugs 529660, 530010). On my
> hardware (Athlon-XP, 2200 MHz and Intel Atom N270, 1600 MHz) they
> speed up dependency resolution by ~ factor 2.

The problem isn't the constants, though. The problem is the resolution
algorithm. There's not much point tweaking performance until the
resolver is fixed to produce a correct answer...

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: [gentoo-dev] Things one could be upset about

[Patrick Lauer]

On Saturday 17 January 2015 14:45:51 Ciaran McCreesh wrote:
> On Sat, 17 Jan 2015 17:39:29 +0300
> 
> Andrew Savchenko <bircoph@gentoo.org> wrote:
> > There is some progress here. In portage-2.2.15 profile based
> > optimizations are included (see bugs 529660, 530010). On my
> > hardware (Athlon-XP, 2200 MHz and Intel Atom N270, 1600 MHz) they
> > speed up dependency resolution by ~ factor 2.
> 
> The problem isn't the constants, though. The problem is the resolution
> algorithm. There's not much point tweaking performance until the
> resolver is fixed to produce a correct answer...

Patches welcome :)

Re: [gentoo-dev] Things one could be upset about

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 447 bytes --]

On Sat, 17 Jan 2015 22:59:08 +0800
Patrick Lauer <patrick@gentoo.org> wrote:
> > The problem isn't the constants, though. The problem is the
> > resolution algorithm. There's not much point tweaking performance
> > until the resolver is fixed to produce a correct answer...
> 
> Patches welcome :)

If I send you a patch that updates all the documentation to use Paludis
rather than Portage, will you welcome it?

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: [gentoo-dev] Things one could be upset about

[Luca Barbato]

On 17/01/15 16:03, Ciaran McCreesh wrote:
> On Sat, 17 Jan 2015 22:59:08 +0800
> Patrick Lauer <patrick@gentoo.org> wrote:
>>> The problem isn't the constants, though. The problem is the
>>> resolution algorithm. There's not much point tweaking performance
>>> until the resolver is fixed to produce a correct answer...
>>
>> Patches welcome :)
>
> If I send you a patch that updates all the documentation to use Paludis
> rather than Portage, will you welcome it?
>

If you rewrite paludis in C99 or equally language with non-brittle 
runtime (libc++ isn't really viable yet =/) I would seriously consider it.

lu

Re: [gentoo-dev] Things one could be upset about

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 922 bytes --]

On Tue, 20 Jan 2015 12:01:13 +0100
Luca Barbato <lu_zero@gentoo.org> wrote:
> On 17/01/15 16:03, Ciaran McCreesh wrote:
> > On Sat, 17 Jan 2015 22:59:08 +0800
> > Patrick Lauer <patrick@gentoo.org> wrote:
> >>> The problem isn't the constants, though. The problem is the
> >>> resolution algorithm. There's not much point tweaking performance
> >>> until the resolver is fixed to produce a correct answer...
> >>
> >> Patches welcome :)
> >
> > If I send you a patch that updates all the documentation to use
> > Paludis rather than Portage, will you welcome it?
> >
> 
> If you rewrite paludis in C99 or equally language with non-brittle 
> runtime (libc++ isn't really viable yet =/) I would seriously
> consider it.

There's always -static-libstdc++. Although Gentoo is still the only
distribution which has trouble with having more than one libstdc++
visible at a time...

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: [gentoo-dev] Things one could be upset about

[Andrew Savchenko]

[-- Attachment #1: Type: text/plain, Size: 2175 bytes --]

On Sat, 17 Jan 2015 14:45:51 +0000 Ciaran McCreesh wrote:
> On Sat, 17 Jan 2015 17:39:29 +0300
> Andrew Savchenko <bircoph@gentoo.org> wrote:
> > There is some progress here. In portage-2.2.15 profile based
> > optimizations are included (see bugs 529660, 530010). On my
> > hardware (Athlon-XP, 2200 MHz and Intel Atom N270, 1600 MHz) they
> > speed up dependency resolution by ~ factor 2.
> 
> The problem isn't the constants, though. The problem is the resolution
> algorithm. There's not much point tweaking performance until the
> resolver is fixed to produce a correct answer...

Oh, this was discussed so many times already... There is NO single
correct solution to such problems. And some mathematically correct
solutions are impractical (e.g. half of the tree rebuild), so other
ones which are good enough are preferred. As long as imperfect
solution works fine, I'm ok with it.

As for paludis, I tried it. Everything mentioned below is my
_personal_ experience and not supposed to pretend to be objective.
So, from my experience paludis turned out to be:

- slower then portage (I don't care here if it is more correct or
not, I just want to do my daily job);

- not fully compatible with portage: it triggers a lot of problems
portage doesn't. While this may be good for QA and tree
improvement, I don't want to hang my workflow due to these issues;

- importare instead of local overlay is a complete nightmare:
usually I don't want to install package exactly as make install
does: often it lacks some required files (e.g. init scripts) or
installs something unneeded on Gentoo system.
Besides I use local overlay to test packages before committing to
public overlays or Gentoo main tree. Lack of local overlay support
is sufficient to send paludis straight to waste bin on my systems.

- completely insane command line options, arguments required to do
what I want to do are quite cumbersome, portage is much saner here.

Everything above is just my personal experience, so your mileage
may vary. But after all that issues I don't even want to try
paludis in the foreseeable future.

Best regards,
Andrew Savchenko

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: [gentoo-dev] Things one could be upset about

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 2232 bytes --]

On Sat, 17 Jan 2015 18:33:45 +0300
Andrew Savchenko <bircoph@gentoo.org> wrote:
> Oh, this was discussed so many times already... There is NO single
> correct solution to such problems. And some mathematically correct
> solutions are impractical (e.g. half of the tree rebuild), so other
> ones which are good enough are preferred. As long as imperfect
> solution works fine, I'm ok with it.

And this is how errors accumulate until your system is a broken mess...

> - slower then portage

Not to do the same thing it isn't.

> (I don't care here if it is more correct or not, I just want to do my
> daily job);

Well, either you spend a tiny bit more time every now and again
avoiding problems, or you spend a huge amount of time when something
breaks at the least convenient possible moment.

> - not fully compatible with portage: it triggers a lot of problems
> portage doesn't. While this may be good for QA and tree
> improvement, I don't want to hang my workflow due to these issues;

Most of the problems you'll encounter are a one-off thing when
switching, particularly if you've allowed your system to be full of
broken dependencies etc by long-term use of Portage (see above). Once
you've cleaned up your system and fixed all the breakages you've
introduced by general sloppiness over the years, the problems you'll see
are genuine ones that need to be dealt with properly.

> - importare instead of local overlay is a complete nightmare:
> usually I don't want to install package exactly as make install
> does: often it lacks some required files (e.g. init scripts) or
> installs something unneeded on Gentoo system.
> Besides I use local overlay to test packages before committing to
> public overlays or Gentoo main tree. Lack of local overlay support
> is sufficient to send paludis straight to waste bin on my systems.

Uh, Paludis supports overlays, and always has.

> - completely insane command line options, arguments required to do
> what I want to do are quite cumbersome, portage is much saner here.

"emerge -uUDpvN --with-bdeps=y" vs "cave resolve --complete"?

Here's a quiz for you: exactly what do -u and -D even do with Portage?

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: [gentoo-dev] Things one could be upset about

[Сергей]

Any random user can tell you: -u  means UPDATE, -D means DEEP (follow
dependencies).

Re: [gentoo-dev] Things one could be upset about

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 232 bytes --]

On Sat, 17 Jan 2015 19:49:24 +0400
Сергей <protserovsd@gmail.com> wrote:
> Any random user can tell you: -u  means UPDATE, -D means DEEP (follow
> dependencies).

And what do those actually mean?

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: [gentoo-dev] Things one could be upset about

[Sergey Popov]

[-- Attachment #1: Type: text/plain, Size: 817 bytes --]

17.01.2015 18:51, Ciaran McCreesh пишет:
> On Sat, 17 Jan 2015 19:49:24 +0400
> Сергей <protserovsd@gmail.com> wrote:
>> Any random user can tell you: -u  means UPDATE, -D means DEEP (follow
>> dependencies).
> 
> And what do those actually mean?
> 

Do you need citation from 'man portage'? :-)

-D usually adds to deptree more deps, than usual 'world'(if we are still
talking about 'emerge -uDN world') like deps of deps, etc, until full
tree will be built.

Maybe i am not totally correct, cause i am not portage developer, but
that's my point of view.

And 'man portage' is telling me pretty much the same things

-- 
Best regards, Sergey Popov
Gentoo developer
Gentoo Desktop Effects project lead
Gentoo Quality Assurance project lead
Gentoo Proxy maintainers project lead


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [gentoo-dev] Things one could be upset about

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 1011 bytes --]

On Mon, 19 Jan 2015 12:42:45 +0300
Sergey Popov <pinkbyte@gentoo.org> wrote:
> 17.01.2015 18:51, Ciaran McCreesh пишет:
> > On Sat, 17 Jan 2015 19:49:24 +0400
> > Сергей <protserovsd@gmail.com> wrote:
> >> Any random user can tell you: -u  means UPDATE, -D means DEEP
> >> (follow dependencies).
> > 
> > And what do those actually mean?
> 
> Do you need citation from 'man portage'? :-)

Well no, because that doesn't answer the question...

> -D usually adds to deptree more deps, than usual 'world'(if we are
> still talking about 'emerge -uDN world') like deps of deps, etc,
> until full tree will be built.
> 
> Maybe i am not totally correct

You're not totally correct.

> And 'man portage' is telling me pretty much the same things

Right, which brings us back to the original point: very few people
actually understand what those options *really* do, so claiming that
Portage has an intuitive or obvious commandline is rather questionable.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: [gentoo-dev] Things one could be upset about

[Róbert Čerňanský]

On Sat, 17 Jan 2015 18:33:45 +0300
Andrew Savchenko <bircoph@gentoo.org> wrote:

> On Sat, 17 Jan 2015 14:45:51 +0000 Ciaran McCreesh wrote:
> > The problem isn't the constants, though. The problem is the
> > resolution algorithm. There's not much point tweaking performance
> > until the resolver is fixed to produce a correct answer...
> 
> Oh, this was discussed so many times already... There is NO single
> correct solution to such problems. And some mathematically correct
> solutions are impractical (e.g. half of the tree rebuild), so other
> ones which are good enough are preferred. As long as imperfect
> solution works fine, I'm ok with it.

From my point of view it would do much help if portage resolves USE
dependencies automatically instead of telling the user to change USE
flags manually (I am talking about bug #258371).

Robert


-- 
Róbert Čerňanský
E-mail: openhs@tightmail.com
Jabber: hs@jabber.sk

Re: [gentoo-dev] Things one could be upset about

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 465 bytes --]

On Mon, 19 Jan 2015 21:44:25 +0100
Róbert Čerňanský <openhs@tightmail.com> wrote:
> From my point of view it would do much help if portage resolves USE
> dependencies automatically instead of telling the user to change USE
> flags manually (I am talking about bug #258371).

This is only possible in carefully selected circumstances, and to get
it to work more generally would require a lot of hinting from package
maintainers.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: [gentoo-dev] Things one could be upset about

[Róbert Čerňanský]

On Mon, 19 Jan 2015 20:51:31 +0000
Ciaran McCreesh <ciaran.mccreesh@googlemail.com> wrote:

> On Mon, 19 Jan 2015 21:44:25 +0100
> Róbert Čerňanský <openhs@tightmail.com> wrote:
> > From my point of view it would do much help if portage resolves USE
> > dependencies automatically instead of telling the user to change USE
> > flags manually (I am talking about bug #258371).
> 
> This is only possible in carefully selected circumstances, and to get
> it to work more generally would require a lot of hinting from package
> maintainers.

But portage already knows that.  It tells the user which USE flags
needs to be changed in order to emerge a package.  It should just go
one step further - to make the proposed change happen by itself.

Robert


-- 
Róbert Čerňanský
E-mail: openhs@tightmail.com
Jabber: hs@jabber.sk

Re: [gentoo-dev] Things one could be upset about

[Andrew Savchenko]

[-- Attachment #1: Type: text/plain, Size: 1460 bytes --]

On Tue, 20 Jan 2015 06:51:01 +0100 Róbert Čerňanský wrote:
> On Mon, 19 Jan 2015 20:51:31 +0000
> Ciaran McCreesh <ciaran.mccreesh@googlemail.com> wrote:
> 
> > On Mon, 19 Jan 2015 21:44:25 +0100
> > Róbert Čerňanský <openhs@tightmail.com> wrote:
> > > From my point of view it would do much help if portage resolves USE
> > > dependencies automatically instead of telling the user to change USE
> > > flags manually (I am talking about bug #258371).
> > 
> > This is only possible in carefully selected circumstances, and to get
> > it to work more generally would require a lot of hinting from package
> > maintainers.
> 
> But portage already knows that.  It tells the user which USE flags
> needs to be changed in order to emerge a package.  It should just go
> one step further - to make the proposed change happen by itself.

And ofter it proposes multiple alternative ways to fix this. How do
you suppose to select between multiple possible alternative
solutions then? Another issues is that sometimes it will be
preferred by the user to disable offending functionality at all.

Good example here is sci-libs/hdf5. Set USE="cxx threads mpi" in
make.conf. Have fun. Especially if in one application (depending on
hdf5) you really need cxx support and in another one (also
depending on hdf5) mpi support is really needed. In some cases it is
preferred to disable hdf5 support at all.

Best regards,
Andrew Savchenko

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

[gentoo-dev] Re: Things one could be upset about

[Duncan]

Róbert Čerňanský posted on Tue, 20 Jan 2015 06:51:01 +0100 as excerpted:

> On Mon, 19 Jan 2015 20:51:31 +0000 Ciaran McCreesh
> <ciaran.mccreesh@googlemail.com> wrote:
> 
>> On Mon, 19 Jan 2015 21:44:25 +0100 Róbert Čerňanský
>> <openhs@tightmail.com> wrote:
>> > From my point of view it would do much help if portage resolves USE
>> > dependencies automatically instead of telling the user to change USE
>> > flags manually (I am talking about bug #258371).
>> 
>> This is only possible in carefully selected circumstances, and to get
>> it to work more generally would require a lot of hinting from package
>> maintainers.
> 
> But portage already knows that.  It tells the user which USE flags needs
> to be changed in order to emerge a package.  It should just go one step
> further - to make the proposed change happen by itself.

Actually, current portage (2.2.15 is what I have installed here ATM) does 
exactly that, making changes to the appropriate package.* files as 
necessary, mediated only by the usual CONFIG_PROTECT variables.

Since /etc/portage is CONFIG_PROTECTed by default, these changes normally 
first appear in that feature's .* files, to be merged by the admin using 
etc_update or whatever, but if you CONFIG_PROTECT_MASK /etc/portage, 
portage will (I believe, I've never been stupid enough to try it) happily 
write those changes without admin mediation.

And a number of users, including me, objected and still don't like that 
feature, tho with the CONFIG_PROTECT mediation it's at least tolerable.  
As others in-thread have stated, we don't believe that's something 
portage should be messing with.

In particular, I have a specific scheme for my package.* directories and 
the files under them, and portage always wants to write the changes to 
the wrong one... based on my experience so far, apparently the last 
existing file in the appropriate package.* directory, when it should go 
in some other file.

Picking an example out of the air, I have USE=pdf set globally, with
USE=-pdf set for specific packages in a file named after the flag (not 
the package it's set for) and to reflect the fact that I'm disabling it:

/etc/portage/package.use/0-pdf

But if portage were to need to set USE=-pdf on another package, instead 
of writing the changes to apply to that file, it would write the changes 
to apply to...

/etc/portage/package.use/zzpkg-ncmpc

... which is there because ncmpc has several very specific USE flags that 
will never apply to anything else and are best grouped together, and 
which shouldn't contain a USE=-pdf setting for ANY package, INCLUDING 
ncmpc, because that's a more generic USE flag, that belongs in the 
previously mentioned /etc/portage/package.use/0-pdf file.

Of course in practice, no automated ruleset is ever going to make sense 
of the way I have things organized, and I don't expect it to.  Which is 
my point.  Portage shouldn't be editing those files AT ALL.  It should 
spit out the suggested change, and let me edit what I, as the *ADMIN*, 
consider the appropriate file to make that change, assuming of course 
that I actually agree with the changed flag in the first place, and don't 
want to make some other adjustment that I consider more appropriate to 
the specific situation.

At least with CONFIG_PROTECT, tho, portage doesn't just edit the file if 
I accidentally hit an enter when I run my usual post-sync
emerge --ask --update --newuse ... @world.  It simply creates a dot-file 
which I have to delete later, after I've decided the appropriate action 
and taken it.  Which is why I characterize it as at least tolerable.  But 
were that feature to have never been merged, I'd be /much/ happier.


Meanwhile, if you're not seeing that feature yet, then you're not running 
current portage, maybe because you're on stale^h^hble portage or 
something.  But it's certainly there, to my unhappiness if toleration 
because at /least/ it obeys CONFIG_PROTECT.

So this subthread could go away... perhaps to be replaced with one 
griping about portage trying to mess with its own config and screwing up 
badly, with only CONFIG_PROTECT stopping it from /really/ screwing up an 
admin's painstaking portage config.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-dev] Re: Things one could be upset about

[Róbert Čerňanský]

On Wed, 21 Jan 2015 01:57:27 +0000 (UTC)
Duncan <1i5t5.duncan@cox.net> wrote:

> Róbert Čerňanský posted on Tue, 20 Jan 2015 06:51:01 +0100 as
> excerpted:
> 
> > On Mon, 19 Jan 2015 20:51:31 +0000 Ciaran McCreesh
> > <ciaran.mccreesh@googlemail.com> wrote:
> > 
> >> On Mon, 19 Jan 2015 21:44:25 +0100 Róbert Čerňanský
> >> <openhs@tightmail.com> wrote:
> >> > From my point of view it would do much help if portage resolves
> >> > USE dependencies automatically instead of telling the user to
> >> > change USE flags manually (I am talking about bug #258371).
> >> 
> >> This is only possible in carefully selected circumstances, and to
> >> get it to work more generally would require a lot of hinting from
> >> package maintainers.
> > 
> > But portage already knows that.  It tells the user which USE flags
> > needs to be changed in order to emerge a package.  It should just
> > go one step further - to make the proposed change happen by itself.
> 
> Actually, current portage (2.2.15 is what I have installed here ATM)
> does exactly that, making changes to the appropriate package.* files
> as necessary, mediated only by the usual CONFIG_PROTECT variables.

No, no, no that is not the right solution.  Portage should _not_ touch
my precious config files crafted for many years.  It should store the
USE related dependencies info in its _internal_ structures (somewhere
in /var/db/pkg I presume).  Sorry I was not clear previously.  Moreover
it should be able to "depclean" them - revert the USE changes once the
dependency is no longer needed (for example with new emerge option
--use-depclean).  Just like with standard package dependencies.

> Since /etc/portage is CONFIG_PROTECTed by default, these changes
> normally first appear in that feature's .* files, to be merged by the
[...]
> tolerable. As others in-thread have stated, we don't believe that's
> something portage should be messing with.

Totally agree, it should mess only with /var/db/pkg or so, not /etc.

[... sniped great explanation of current testing portage behaviour in
this regard; thanks for that, even though it is not what I crave for]

Robert


-- 
Róbert Čerňanský
E-mail: openhs@tightmail.com
Jabber: hs@jabber.sk

Re: [gentoo-dev] Things one could be upset about

[Daniel Campbell]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 01/19/2015 12:44 PM, Róbert Čerňanský wrote:
> On Sat, 17 Jan 2015 18:33:45 +0300 Andrew Savchenko
> <bircoph@gentoo.org> wrote:
> 
>> On Sat, 17 Jan 2015 14:45:51 +0000 Ciaran McCreesh wrote:
>>> The problem isn't the constants, though. The problem is the 
>>> resolution algorithm. There's not much point tweaking
>>> performance until the resolver is fixed to produce a correct
>>> answer...
>> 
>> Oh, this was discussed so many times already... There is NO
>> single correct solution to such problems. And some mathematically
>> correct solutions are impractical (e.g. half of the tree
>> rebuild), so other ones which are good enough are preferred. As
>> long as imperfect solution works fine, I'm ok with it.
> 
> From my point of view it would do much help if portage resolves
> USE dependencies automatically instead of telling the user to
> change USE flags manually (I am talking about bug #258371).
> 
> Robert
> 
> 

As a user, the last thing I want happening is Portage making USE
decisions for me. I'm completely in support of an emerge *flag*, but
not doing it unconditionally.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQEcBAEBAgAGBQJUvXBBAAoJEJUrb08JgYgHqsQH/1m/Dgu547RTcooHhZ+B4gt1
FvjPGy1qEKB4W2ErxDj6J6TLlP09ASIiJ/7hrndKonDNd1aP4gAi7tKI5XzetWVt
cWYG3UWLhxJRvMc2y7kbOyDSIy68Sz/r1Bruwymqdn+N6ooqnHVK252OJgaMGQHP
aDa+ibNAywE7t/CTWS6rQU/ilEHsXIps+c4gmvEGv5iWiCKxlQF5fNKfWjOGEr9c
NN23RaSEJj7BCEfFaFgmjd7P0akz/yzg/sr8xuaaEwUv5/KFJp7SI/Q/6GzG48rg
H6TiNIYm3Gs0ucEWISCZx+qon5EmkkSREaQ5xeqBBRklNN63evH1pttjFg9rX6o=
=RjLq
-----END PGP SIGNATURE-----

Re: [gentoo-dev] Things one could be upset about

[Andrew Savchenko]

[-- Attachment #1: Type: text/plain, Size: 1352 bytes --]

On Mon, 19 Jan 2015 21:44:25 +0100 Róbert Čerňanský wrote:
> On Sat, 17 Jan 2015 18:33:45 +0300
> Andrew Savchenko <bircoph@gentoo.org> wrote:
> 
> > On Sat, 17 Jan 2015 14:45:51 +0000 Ciaran McCreesh wrote:
> > > The problem isn't the constants, though. The problem is the
> > > resolution algorithm. There's not much point tweaking performance
> > > until the resolver is fixed to produce a correct answer...
> > 
> > Oh, this was discussed so many times already... There is NO single
> > correct solution to such problems. And some mathematically correct
> > solutions are impractical (e.g. half of the tree rebuild), so other
> > ones which are good enough are preferred. As long as imperfect
> > solution works fine, I'm ok with it.
> 
> From my point of view it would do much help if portage resolves USE
> dependencies automatically instead of telling the user to change USE
> flags manually (I am talking about bug #258371).

As the user the last thing I want is to have some USE flags changed
without my permission ending up stuff I need to be omitted or stuff
I don't want to see on my system to be installed. Of course if
someone prefers USE flags to be randomly changed I don't mind if
such option will exist (as proposed in bug #258371) as long as it
is disabled by default.

Best regards,
Andrew Savchenko

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: [gentoo-dev] Things one could be upset about

[Róbert Čerňanský]

On Tue, 20 Jan 2015 00:14:29 +0300
Andrew Savchenko <bircoph@gentoo.org> wrote:

> On Mon, 19 Jan 2015 21:44:25 +0100 Róbert Čerňanský wrote:
> > From my point of view it would do much help if portage resolves USE
> > dependencies automatically instead of telling the user to change USE
> > flags manually (I am talking about bug #258371).
> 
> As the user the last thing I want is to have some USE flags changed
> without my permission ending up stuff I need to be omitted or stuff
> I don't want to see on my system to be installed. Of course if
> someone prefers USE flags to be randomly changed I don't mind if
> such option will exist (as proposed in bug #258371) as long as it
> is disabled by default.

Is is of course perfectly fine to have that option disabled by
default.  But I am afraid that I do not quite understand yours and
Daniel's concerns.  If I paraphrase your statement with regards to
current dependency handling, it is as if you were saying: "I do not
want portage to install any package automatically by its own, I want
to install all the dependencies explicitly."

Why we allow portage to install dependencies and still have things
under control?  Well we have --pretend and --ask options and we can
see what would/will be done.  This would be the same with USE flags.
You would see in the --pretend output which flags would be changed.

To match the package dependency handling, there should be also an
option equivalent to --depclean.  It would revert the USE changes
which were done because of dependencies requirements if no longer
needed.

For example, lets have following packages:

- libbar
- libfoo with IUSE=bar, which depends on: bar? ( libbar )
- foo, which depends on: libfoo[bar]

USE flag bar is not enabled.  You want to install application foo.

Current behaviour:

# emerge -av foo
...
Required USE changes:
libfoo +bar
... (sorry for not exact emerge output but you get the idea)

Now, you either fire up your favorite editor and add "libfoo +bar" to
/etc/portage/package.use, re-run emerge and have libbar installed even
you do not want it.  The other option is not to install application
foo at all.

If you choose to emerge it and after year or so you decide to unmerge
it because you do not need it anymore you still will have a leftover
in package.use file - the line "libfoo +foo" even if you run emerge
--depclean.

New behaviour with automatic USE dependencies resolution:

emerge -av foo
[ebuild  N     ] libbar
[ebuild  N     ] libfoo +bar
[ebuild  N     ] foo

Now, you can hit Y if you agree what portage wants to do or N and not
to install foo at all.

After unmerging it you run emerge --depclean which removes libfoo with
its changed USE flag and libbar, no leftovers. (In this case new
--use-depclean command is not required since libfoo was removed.)

If libfoo would not be removed because some other package needs it,
then emerge --use-depclean would revert the +bar USE flag to its
default state and re-emerge libfoo.

Robert


-- 
Róbert Čerňanský
E-mail: openhs@tightmail.com
Jabber: hs@jabber.sk

Re: [gentoo-dev] Things one could be upset about

[Andrew Savchenko]

[-- Attachment #1: Type: text/plain, Size: 4562 bytes --]

On Tue, 20 Jan 2015 07:46:32 +0100 Róbert Čerňanský wrote:
> On Tue, 20 Jan 2015 00:14:29 +0300
> Andrew Savchenko <bircoph@gentoo.org> wrote:
> 
> > On Mon, 19 Jan 2015 21:44:25 +0100 Róbert Čerňanský wrote:
> > > From my point of view it would do much help if portage resolves USE
> > > dependencies automatically instead of telling the user to change USE
> > > flags manually (I am talking about bug #258371).
> > 
> > As the user the last thing I want is to have some USE flags changed
> > without my permission ending up stuff I need to be omitted or stuff
> > I don't want to see on my system to be installed. Of course if
> > someone prefers USE flags to be randomly changed I don't mind if
> > such option will exist (as proposed in bug #258371) as long as it
> > is disabled by default.
> 
> Is is of course perfectly fine to have that option disabled by
> default.  But I am afraid that I do not quite understand yours and
> Daniel's concerns.  If I paraphrase your statement with regards to
> current dependency handling, it is as if you were saying: "I do not
> want portage to install any package automatically by its own, I want
> to install all the dependencies explicitly."

The paraphrasing above is wrong. What I want to say is "I don't want
portage to automagically change _functionality_ of my packages on
its own, even in order to solve dependencies."
 
> Why we allow portage to install dependencies and still have things
> under control?  Well we have --pretend and --ask options and we can
> see what would/will be done.  This would be the same with USE flags.
> You would see in the --pretend output which flags would be changed.

No this wouldn't be the same. USE flags are _not_ equal to package
dependencies, sometimes they will not trigger dependency change at
all. USE flags are about _functionality_, dependencies are about
the _means_ to implement that functionality (as well as base
functionality of a package).

> To match the package dependency handling, there should be also an
> option equivalent to --depclean.  It would revert the USE changes
> which were done because of dependencies requirements if no longer
> needed.

This will dramatically increase complexity of dependencies metadata
as well as of algorithms to handle it (and they are already slow),
with no clear benefits therefore. No, thanks.

> For example, lets have following packages:
> 
> - libbar
> - libfoo with IUSE=bar, which depends on: bar? ( libbar )
> - foo, which depends on: libfoo[bar]
> 
> USE flag bar is not enabled.  You want to install application foo.
> 
> Current behaviour:
> 
> # emerge -av foo
> ...
> Required USE changes:
> libfoo +bar
> ... (sorry for not exact emerge output but you get the idea)
> 
> Now, you either fire up your favorite editor and add "libfoo +bar" to
> /etc/portage/package.use, re-run emerge and have libbar installed even
> you do not want it.  The other option is not to install application
> foo at all.
> 
> If you choose to emerge it and after year or so you decide to unmerge
> it because you do not need it anymore you still will have a leftover
> in package.use file - the line "libfoo +foo" even if you run emerge
> --depclean.
> 
> New behaviour with automatic USE dependencies resolution:
> 
> emerge -av foo
> [ebuild  N     ] libbar
> [ebuild  N     ] libfoo +bar
> [ebuild  N     ] foo
> 
> Now, you can hit Y if you agree what portage wants to do or N and not
> to install foo at all.

And if I don't agree? What if for some reason I don't want to
have libfoo[bar] on my system. What If preferred solution will
be not to use libbar at all and to use libclue instread? 

World update on my systems usually involves about 2000 of packages.
It's already hard to read emerge -DNuav output in order to check
for new packages, dependencies, downgraded and removed packages.
Automagick dependency change will make this even harder, much
harder because it will involve multiple additional emerge runs in
order to deal with issues properly. And each run takes about half
an hour.

Yet again, Gentoo is all about choise. If someone wants that
automagick to be implemented, go ahead and send patches for
developers to review. But please keep it disabled by default, in
most cases it will provide bizarre results anyway. (Think about
conflicting alternatives, no sane way to prefer one above another
automatically and a chain of such issues during world update.) 

Best regards,
Andrew Savchenko

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: [gentoo-dev] Things one could be upset about

[Róbert Čerňanský]

On Tue, 20 Jan 2015 11:08:19 +0300
Andrew Savchenko <bircoph@gentoo.org> wrote:

> On Tue, 20 Jan 2015 07:46:32 +0100 Róbert Čerňanský wrote:
> > On Tue, 20 Jan 2015 00:14:29 +0300
> > Andrew Savchenko <bircoph@gentoo.org> wrote:
> > > On Mon, 19 Jan 2015 21:44:25 +0100 Róbert Čerňanský wrote:
> > > > From my point of view it would do much help if portage resolves
> > > > USE dependencies automatically
[...]
> > > As the user the last thing I want is to have some USE flags
> > > changed without my permission
[...]
> > Is is of course perfectly fine to have that option disabled by
> > default.  But I am afraid that I do not quite understand yours and
> > Daniel's concerns.  If I paraphrase your statement with regards to
> > current dependency handling, it is as if you were saying: "I do not
> > want portage to install any package automatically by its own, I want
> > to install all the dependencies explicitly."
> 
> The paraphrasing above is wrong. What I want to say is "I don't want
> portage to automagically change _functionality_ of my packages on
> its own, even in order to solve dependencies."
>  
> > Why we allow portage to install dependencies and still have things
> > under control?  Well we have --pretend and --ask options and we can
> > see what would/will be done.  This would be the same with USE flags.
> > You would see in the --pretend output which flags would be changed.
> 
> No this wouldn't be the same. USE flags are _not_ equal to package
> dependencies, sometimes they will not trigger dependency change at
> all. USE flags are about _functionality_, dependencies are about
> the _means_ to implement that functionality (as well as base
> functionality of a package).

I see your point about functionality.  There are USE flags that
_changes_ functionality (like threads) and there are others which _adds_
functionality (like for example speex adds possibility to play files in
that particular format in mplayer).  The former I consider similar to
package dependencies because they too can add functionality to the
system (for example if ruby is installed as dependency it adds
possibility to execute ruby scripts) same as those USE flags adds
functionality to an application.

However, even if portage wants me to change USE flag that will change
functionality, in 99% of time I end up adding to my package.use what it
wants, since my goal is to install some application or update my
applications.  So just reviewing the changes that portage wants to do
and hit Y.  And in those rare cases when I really do not want some flag
enabled or some package installed I hit N and tweak things.

> > To match the package dependency handling, there should be also an
> > option equivalent to --depclean.  It would revert the USE changes
> > which were done because of dependencies requirements if no longer
> > needed.
> 
> This will dramatically increase complexity of dependencies metadata
> as well as of algorithms to handle it (and they are already slow),
> with no clear benefits therefore. No, thanks.

Are you talking about proposed new USE specific depclean option or
emerge in general?  If it is the new depclean command that would run
slow then I am quite sure it will still be faster than me or anyone else
going manually through package.use and for each and every USE flag there
trying to remove it, run emerge, see if it passes, if not, add it back.

If it is emerge in general that would be slower that it is still better
than run it, see it fail and telling you what USE to change, you making
the change, and run it again.

> > For example, lets have following packages:
> > 
> > - libbar
> > - libfoo with IUSE=bar, which depends on: bar? ( libbar )
> > - foo, which depends on: libfoo[bar]
[...]
> > New behaviour with automatic USE dependencies resolution:
> > 
> > emerge -av foo
> > [ebuild  N     ] libbar
> > [ebuild  N     ] libfoo +bar
> > [ebuild  N     ] foo
> > 
> > Now, you can hit Y if you agree what portage wants to do or N and
> > not to install foo at all.
> 
> And if I don't agree? What if for some reason I don't want to
> have libfoo[bar] on my system. What If preferred solution will
> be not to use libbar at all and to use libclue instread? 

In this example, if you do not agree, you have no other option how to
install foo (with or without automatic USE deps).  Because foo depends
on libfoo[bar] unconditionally.

If however foo would depend either on libfoo[bar] or libclue then the
portage would do the same thing as today (I guess it would take the
first dependency as mandatory or the one which is already installed if
any.) In this case, yes, your preference might be libclue instead what
portage chooses.  But I see no difference in the way how user choose it
comparing to todays '|| ( libfoo libclue )' -like dependencies.

> World update on my systems usually involves about 2000 of packages.
> It's already hard to read emerge -DNuav output in order to check
> for new packages, dependencies, downgraded and removed packages.
> Automagick dependency change will make this even harder, much
> harder because it will involve multiple additional emerge runs in
> order to deal with issues properly. And each run takes about half
> an hour.

Hmm, mine is about 1700 and on my 10 years old AMD64 it also takes long
time.  But my impression is that when portage resolves USE deps
automatically it will actually decrease the number of emerge runs.
Only in those rare cases when I really do not want some dependency (USE
or package) installed on my system and I am prepared to sacrify some of
the current functionality to get rid of it I need to tweak and re-run
emerge.

Of course there are blockers, like you pointed out in your other mail
with sci-libs/hdf5 and USE flags "cxx threads mpi".  But blockers are
here even today.  And moreover I am not sure what would portage do
today in such case.  Tell you to enable cxx and mpi, you do it and it
will fail again?

> Yet again, Gentoo is all about choise. If someone wants that

I agree, but I must say I am quite stunned that there are strong voices
against it.  I somehow thought that edit the overgrowing package.use
file upon each emerge world annoys anyone the same as me.

How do you handling it?  How do you clean it?  eix-test-obsolete does
not catch unnecessary USE flags there.

Regards,
Robert


-- 
Róbert Čerňanský
E-mail: openhs@tightmail.com
Jabber: hs@jabber.sk

Re: [gentoo-dev] Things one could be upset about

[Alexey Mishustin]

2015-01-20 14:42 GMT+04:00 Róbert Čerňanský <openhs@tightmail.com>:
> On Tue, 20 Jan 2015 11:08:19 +0300
> Andrew Savchenko <bircoph@gentoo.org> wrote:
>
>> On Tue, 20 Jan 2015 07:46:32 +0100 Róbert Čerňanský wrote:
>> > On Tue, 20 Jan 2015 00:14:29 +0300
>> > Andrew Savchenko <bircoph@gentoo.org> wrote:
>> > > On Mon, 19 Jan 2015 21:44:25 +0100 Róbert Čerňanský wrote:
>> > For example, lets have following packages:
>> >
>> > - libbar
>> > - libfoo with IUSE=bar, which depends on: bar? ( libbar )
>> > - foo, which depends on: libfoo[bar]
> [...]
>> > New behaviour with automatic USE dependencies resolution:
>> >
>> > emerge -av foo
>> > [ebuild  N     ] libbar
>> > [ebuild  N     ] libfoo +bar
>> > [ebuild  N     ] foo
>> >
>> > Now, you can hit Y if you agree what portage wants to do or N and
>> > not to install foo at all.
>>
>> And if I don't agree? What if for some reason I don't want to
>> have libfoo[bar] on my system. What If preferred solution will
>> be not to use libbar at all and to use libclue instread?
>
> In this example, if you do not agree, you have no other option how to
> install foo (with or without automatic USE deps).  Because foo depends
> on libfoo[bar] unconditionally.

Perfect! May be I will prefer to refuse to install that package, after
seeing its dependencies.

>> Yet again, Gentoo is all about choise. If someone wants that
>
> I agree, but I must say I am quite stunned that there are strong voices
> against it.  I somehow thought that edit the overgrowing package.use
> file upon each emerge world annoys anyone the same as me.

But for me this is one of the most useful and convenient options in
Gentoo. Yes, I do edit package.use almost every emerge world. And I
like to do it. And I don't want to delegate this right to any program
- portage, or any other.

-- 
Regards,
Alex

Re: [gentoo-dev] Things one could be upset about

[Róbert Čerňanský]

On Sat, 24 Jan 2015 21:54:06 +0400
Alexey Mishustin <shumkar@shumkar.ru> wrote:

> 2015-01-20 14:42 GMT+04:00 Róbert Čerňanský <openhs@tightmail.com>:
> > On Tue, 20 Jan 2015 11:08:19 +0300
> > Andrew Savchenko <bircoph@gentoo.org> wrote:
> >
> >> On Tue, 20 Jan 2015 07:46:32 +0100 Róbert Čerňanský wrote:
> >> > On Tue, 20 Jan 2015 00:14:29 +0300
> >> > Andrew Savchenko <bircoph@gentoo.org> wrote:
> >> > > On Mon, 19 Jan 2015 21:44:25 +0100 Róbert Čerňanský wrote:
> >> > For example, lets have following packages:
> >> >
> >> > - libbar
> >> > - libfoo with IUSE=bar, which depends on: bar? ( libbar )
> >> > - foo, which depends on: libfoo[bar]
> > [...]
> >> > New behaviour with automatic USE dependencies resolution:
> >> >
> >> > emerge -av foo
> >> > [ebuild  N     ] libbar
> >> > [ebuild  N     ] libfoo +bar
> >> > [ebuild  N     ] foo
> >> >
> >> > Now, you can hit Y if you agree what portage wants to do or N and
> >> > not to install foo at all.
> >>
> >> And if I don't agree? What if for some reason I don't want to
> >> have libfoo[bar] on my system. What If preferred solution will
> >> be not to use libbar at all and to use libclue instread?
> >
> > In this example, if you do not agree, you have no other option how
> > to install foo (with or without automatic USE deps).  Because foo
> > depends on libfoo[bar] unconditionally.
> 
> Perfect! May be I will prefer to refuse to install that package, after
> seeing its dependencies.
> 
> >> Yet again, Gentoo is all about choise. If someone wants that
> >
> > I agree, but I must say I am quite stunned that there are strong
> > voices against it.  I somehow thought that edit the overgrowing
> > package.use file upon each emerge world annoys anyone the same as
> > me.
> 
> But for me this is one of the most useful and convenient options in
> Gentoo. Yes, I do edit package.use almost every emerge world. And I
> like to do it. And I don't want to delegate this right to any program
> - portage, or any other.

You would not loose that.  First of all, portage would do its changes
elsewhere (in /var).  Secondly, your settings in package.use would
still be respected.  But instead of portage telling you to edit
package.use it would show you required changes in emerge -av output.
If you do not like them then hit N and edit package.use, masks or
whatever.

In case of conflict, i.e. if portage would want to set a USE flag for a
package which you have disabled in package.use then it would fail
(similarly as it fails when some package has testing keyword or is
masked).  So if I have 'net-print/cups -usb' in package.use the
automatic USE depencencies would _never_ enable usb for cups.  But if I
have -python in _make.conf_ and want to emerge app-mobilephone/wammu
which depends on app-mobilephone/gammu[python] then portage would be
able to enable python for gammu.

The intention behind this automatic USE dependencies is to edit
package.use only when user wants, not when portage needs it.

Robert


-- 
Róbert Čerňanský
E-mail: openhs@tightmail.com
Jabber: hs@jabber.sk

[gentoo-dev] Re: Things one could be upset about

[Duncan]

Alexey Mishustin posted on Sat, 24 Jan 2015 21:54:06 +0400 as excerpted:

> 2015-01-20 14:42 GMT+04:00 Róbert Čerňanský <openhs@tightmail.com>:
>>
>> I somehow thought that edit the overgrowing package.use
>> file upon each emerge world annoys anyone the same as me.
> 
> But for me this is one of the most useful and convenient options in
> Gentoo. Yes, I do edit package.use almost every emerge world. And I like
> to do it. And I don't want to delegate this right to any program -
> portage, or any other.

Agreed that I don't want to (and won't) delegate that decision, but 
almost every emerge world?  Not here.  So ???

I do edit package.use (or my global USE flags) occasionally, but not as 
often as the above implies.  What might I be doing different?  Well, 
here's what I do:

1) I try to sync and update deep newuse @world once a week, tho sometimes 
it's every two weeks (but sometimes it's daily).  I suppose if people 
only get to it every couple months, they'll have more updates to do, and 
within that time yes, I guess it's pretty likely they'll have a few USE 
flag changes to work thru.

So maybe it's simply that I update frequently enough, tho I /do/ run 
~arch as well, which changes much faster than stable, and I even run some 
packages (mostly kde) from the overlays, before they even enter the main 
tree, so I'd /think/ every couple months updating stable would be like 
every week or two updating ~arch.

2) My global USE= starts with -*.  I got tired of worrying about what the 
profiles set, and what individual package IUSE-defaults were doing, so I 
simply set -* so everything would default to off like it USED to do, 
years ago, and I can turn on what I want/need when I decide I want/need 
it.  So if it's on, I turned it on.

3) I don't normally distinguish between local and global USE flags.  I 
normally treat them all as global and set them globally in my make.conf 
use file[1].  When I encounter a new USE flag, because of the -* in USE, 
it's off by default.  If I decide I want it off, no problem, it's off.  
If I decide I want it on, I run an equery hasuse <flag> to see if any 
other package uses it.  If nothing else uses it, I simply set it globally 
in my use file, turning it on for that package and anything else that 
might use it in the future.  If I want it on for the new package but 
something else uses it I obviously had it off for it before or it'd 
already be turned on in the use file, so I decide if I want to change the 
default to on or not.  If so, it goes in the use file.  If not, I have to 
decide if I REALLY want it on for the new package or not, and if so, I 
set a non-default entry in a package.use file.

Similarly, if I encounter a new USE flag that's on already, obviously 
some other package I use is already using it and the entry is in my use 
file or it wouldn't be on already, due to the -* in that use file.  
That's a strong hint what I'm likely to want the default to be.  If I 
decide I want it off anyway, I run an equery hasuse <flag> and check how 
many packages I have with it, and decide what I want the default to be 
based on that.  Only if I want a setting that's different than my 
default, will I put a new entry in my package.use files.

So for all flags, if I want the default off, due to the -* in my use 
file, it's off.  If I want the default on, it's in my use file, turning 
it on.

4) The result is that my package.use files contain ONLY non-default 
entries.

When I set such an entry, I prefix a comment line with the date and an 
explanation of WHY the entry needs to be there, different from my normal 
default settings.  Sometimes, it's due to a bug, and a couple versions 
later I can go thru and test with that entry commented, to see if the bug 
is fixed, yet.  Other times it's due to a use-dep from some other package 
I have installed.  Both qt and kde tend to have explicit use-deps for 
some of their dependencies, and I'll explain that in the comment.

Regardless of why it's there, however, because only non-default entries 
are in package.use, they're the obvious exception.  And as exceptions, 
they don't tend to change that often. =:^)  Generally, they'll only 
change with a major version bump -- say for the upcoming kde5[2], which 
I've already tried a couple times so have some of the flag changes I'll 
need for it made already.  Or sometimes they'll change if I unmerge a 
package that had a use-dep, and with it removed, I can remove my 
exception for that and get back to my normal defaults.

5) Because all my USE flag defaults are set in my global use file, and 
package.use contains only exceptions, organizing my package.use files by 
use flag makes more sense than the usual by-package organization.  So 
(with one exception) all my package.use files are named by flag, not by 
package, with all entries in a file being the same flag, but for 
different packages.

Further more, I name the file based on whether I'm turning the flag on or 
off.  In ordered to have all the per-package-turn-offs (where the default 
is on) sort first, I use a 0-* naming pattern for those.  The others, per-
package-turnons where the default is off, are simply named after the flag 
without the 0-*.

The one exception I mentioned is for the ncmpc package, which has several 
very unique flags that will only apply to it, so I do keep those flags 
together and out of my global use file.  It's named zzpkg-ncmpc to keep 
it last in the listing.

A partial listing, then...

$ cd /etc/portage/package.use
$ ls
0-bindist
0-curl
0-custom-cflags
...
binary
crypt
doc
...
zzpkg-ncmpc

I have bindist turned on by default, with an entry in my global use file, 
but turn it off in specific cases, so we have 0-bindist.  curl is 
similarly on by default, but it's off for systemd, which uses it for 
journal-upload, which I don't want/need, thus that file.

I have binary and crypt and doc off by default (not in my use file, 
ensured off no matter the profile or package use-default setting by the
-* in my global use file), but have them turned on for specific packages, 
thus those files.

And I explained zzpkg-ncmpc above...


So for a new installation until my package and USE flag lists settle 
down, I'd expect frequent changes, and I'd expect a bunch of changes when 
I install a major version bump of something big like kde5, or for other 
big changes like when I switched to systemd and changed a bunch of 
packages and USE flags as a result.

But other than that, they don't change that much; certainly not almost 
every update.

Yes, almost every update there's a handful of packages with new use 
flags.  And I always check them out and decide what I want to do with 
them.  But most of the time, I already have a default entry for the flag 
in my global use file, and most of the time I want to keep that default, 
so no changes necessary on my end.

So what's different between that and whatever others are doing such that 
they have package.use changes nearly every update?  I don't know, but 
whatever it is, I'm glad I'm doing the above, and it's not so often, here.

---
Footnotes:

[1]  My make.conf file itself is simply a list of include <file> lines, 
one for each type of setting.  So I have a use file that contains my USE= 
setting, with a line sourcing it in make.conf, not a USE= line in 
make.conf itself.

[2] KDE5.  I've actually already tried it a couple times and have many of 
the USE flag changes for it already made as a result.  But kwin was still 
crashing on my graphics hardware/drivers when I tried, so I didn't get 
far, and because it's not possible to have both it and kde4 installed at 
the same time, I had to unmerge it to get back to a working kde4.  I 
should try again one of these days as it has been a few months, and qt5 
is actually in the tree now altho I don't believe it's unmasked yet, but 
at least /that/ part should be considerably more stable than it was, when 
I was having to install it from the qt overlay.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-dev] Re: Things one could be upset about

[Róbert Čerňanský]

On Sun, 25 Jan 2015 04:29:43 +0000 (UTC)
Duncan <1i5t5.duncan@cox.net> wrote:

> Alexey Mishustin posted on Sat, 24 Jan 2015 21:54:06 +0400 as
> excerpted:
> 
> > 2015-01-20 14:42 GMT+04:00 Róbert Čerňanský <openhs@tightmail.com>:
> >>
> >> I somehow thought that edit the overgrowing package.use
> >> file upon each emerge world annoys anyone the same as me.
> > 
> > But for me this is one of the most useful and convenient options in
> > Gentoo. Yes, I do edit package.use almost every emerge world. And I
> > like to do it. And I don't want to delegate this right to any
> > program - portage, or any other.
> 
> Agreed that I don't want to (and won't) delegate that decision, but 
> almost every emerge world?  Not here.  So ???
> 
> I do edit package.use (or my global USE flags) occasionally, but not
> as often as the above implies.  What might I be doing different?
> Well, here's what I do:
> 
> 1) I try to sync and update deep newuse @world once a week, tho
> sometimes it's every two weeks (but sometimes it's daily).  I suppose
> if people only get to it every couple months, they'll have more
[...]
> So maybe it's simply that I update frequently enough, tho I /do/ run 
> ~arch as well, which changes much faster than stable, and I even run

More frequent updates is most likely the reason that you do not have
to edit use flags every time.  Running testing perhaps does not
increase the editing frequency because dependency changes are the same
regardles of how many bumps a package has.  For example in testing
you'll get following updates of package foo: foo-1.1, ~foo-1.2,
~foo-1.3, foo-1.4.  In stable, I would get: foo-1.1, foo-1.4.  If
dependency changes in 1.3, both of us could have to edit USE flags
once.

I update every 2-4 months (only GLSAs in between) therefore my
experience is that I have to edit it every time (not for GLSAs of
course).

> 2) My global USE= starts with -*.  I got tired of worrying about what
[...]
> 3) I don't normally distinguish between local and global USE flags.
> I normally treat them all as global and set them globally in my
> make.conf use file[1].  When I encounter a new USE flag, because of
> the -* in USE, it's off by default.  If I decide I want it off, no
> problem, it's off. If I decide I want it on, I run an equery hasuse
> <flag> to see if any other package uses it.  If nothing else uses it,
[...]
> Similarly, if I encounter a new USE flag that's on already, obviously 
> some other package I use is already using it and the entry is in my
> use file or it wouldn't be on already, due to the -* in that use
> file. That's a strong hint what I'm likely to want the default to
> be.  If I decide I want it off anyway, I run an equery hasuse <flag>
[...]
> So for all flags, if I want the default off, due to the -* in my use 
> file, it's off.  If I want the default on, it's in my use file,
> turning it on.
> 
> 4) The result is that my package.use files contain ONLY non-default 
> entries.

More or less same here, except -* as the default.  I trust developers
that they are choosing wisely in profile and ebuilds. ;-)  If not, I
see it in emerge -av output anyway and can disable/enable particular
flag.  But generally I have vast majority of flags in make.conf and
exceptions in package.use.

> When I set such an entry, I prefix a comment line with the date and
> an explanation of WHY the entry needs to be there, different from my
> normal default settings.  Sometimes, it's due to a bug, and a couple
> versions later I can go thru and test with that entry commented, to
> see if the bug is fixed, yet.  Other times it's due to a use-dep from
> some other package I have installed.  Both qt and kde tend to have

This where we get to the point.  If you set USE flag because of a bug
or because of dependency it is not because you *want to* but because
you *have to* (or portage *needs to*).  Therefore you need to add a
comment WHY you did it.

For example I have -doc in make.conf because I do *not want* docs
globally.  But I have 'dev-lang/python doc' in package.use because I
develop in Python and *want* the documentation for it.  Such entry
does not need a comment, because I simply know what I want.

Another example.  I have -python globally and have been using
app-mobilephone/gammu.  Now I want to emerge app-mobilephone/wammu.
But it needs +python for gammu, so I *have to* set
'app-mobilephone/gammu python' in package.use.  In this case I also
add a comment which explains the reason because it is not what *I
want* it is what *portage needs*.  Once this dependency is gone
(either because wammu stops requiring it or I unmerge it) it is on me
to detect it and remove the entry.  So effectively I manage portage's
stuff in my configuration file.

> Regardless of why it's there, however, because only non-default
> entries are in package.use, they're the obvious exception.

You somehow do not distinguish between those two types of exceptions
explained in examples above.

> And as exceptions, they don't tend to change that often. =:^)
> Generally,

They might change as often as package dependencies and for those we
have --depclean.  For use dependencies we have to go by hand through
package.use and check each entry if it is still needed or let have
bunch of unnecessary crap installed on our systems.  This is not very
good user (or admin) experience.

> 5) Because all my USE flag defaults are set in my global use file,
> and package.use contains only exceptions, organizing my package.use
> files by use flag makes more sense than the usual by-package
[...]
> 0-bindist
> 0-curl
> 0-custom-cflags
> ...
> binary
> crypt

Indeed an interesting organization of package.use.

Robert


-- 
Róbert Čerňanský
E-mail: openhs@tightmail.com
Jabber: hs@jabber.sk

[gentoo-dev] Re: Things one could be upset about

[Duncan]

Róbert Čerňanský posted on Sun, 25 Jan 2015 14:27:50 +0100 as excerpted:

> On Sun, 25 Jan 2015 04:29:43 +0000 (UTC)
> Duncan <1i5t5.duncan@cox.net> wrote:
>> [...]
> 
> More frequent updates is most likely the reason that you do not have to
> edit use flags every time.

With the information below, seems like it.

> I update every 2-4 months (only GLSAs in between) therefore my
> experience is that I have to edit it every time (not for GLSAs of
> course).

I'd worry about that.

Frankly, I think the GLSAs are often too slow to come out, but I'm 
honestly not sure what could practically be done to fix it.  The problem 
(as I understand/see it) is that by policy the GLSAs don't get released 
until all security-handled archs have stable-keyworded the fixed package-
version, and some of those security-supported archs are underpowered and/
or undermanned and simply take longer than I'd consider ideal to do that 
keywording, even at security-priority.  But the only real and practical 
fix would be either GLSA's per-arch as they stabilize, and accepting that 
such a policy means the slow archs /will/ be sitting there exposed 
without a stabilized fix or GLSA, OR a GLSA-on-first-stable policy, and 
accept that archs behind the first to stable would be getting GLSAs 
suggesting fixed packages that might not actually be tested and 
stabilized for a month on some affected archs.

The result of the current policy is that if you're waiting for the GLSA, 
unless it's _extreme_ priority (heartbleed level), on at least amd64, 
you're very often sitting there exposed for well over a week, and too 
often a month, after the fix is out there, actually installed on /my/ 
systems.  And to me that's a game of Russian Roulette odds that I'm 
simply not willing to play.

If you're going to do security-updates only, at least follow a good 
notification service (LWN's daily security updates notices are a start, 
if you compare against what you have installed on gentoo), and grab the 
corresponding gentoo updates before the GLSAs.  At least then, if it's at 
least high enough priority to hit the news, you'll have it manually 
installed and won't be sitting there with your *** hanging out for weeks 
at a time!

> 
>> 2) My global USE= starts with -*.  I got tired of worrying about what
> [...]
> More or less same here, except -* as the default.  I trust developers
> that they are choosing wisely in profile and ebuilds. ;-)

It's not that I don't trust 'em.  I'm running their ebuilds as root, 
after all. =:^)

It's more the certainty of knowing however it's set, I set it that way, 
and other than a flag name change or add/remove, it's going to stay that 
way unless I change it.  Also, figuring out where a tree default comes 
from isn't as easy or intuitive as it might be.  Both the profiles and 
the ebuilds (with eclasses) cascade, and while I agree with doing it that 
way (I was around before cascading profiles and when eclasses were much 
simpler than they are today, and believe me, this way's easier/better!), 
it doesn't make looking up things easier.

FWIW I negate my entire @system set and thus have an empty @system, with 
some otherwise @system packages in @world, and a few simply not installed 
as such, for similar reasons.  I want to control it myself.

Plus, after setting -* in my use file, all the -flag entries in the file 
could go away.  MUCH cleaner USE file now! =:^)  And it turned out there 
actually weren't that many changes after that, because I had negated 
profile or IUSE defaults enough already, with package.use where it 
deviated, that -* ended up being a rather large simplification of the use 
file itself, with few actual flag changes. =:^)

The one thing I /do/ need to be aware of, now, is that I /don't/ 
automatically see IUSE defaults in the various packages.  Thus, on new 
packages or major upgrades that significantly change USE flags, when I'm 
in doubt on a flag, I do occasionally actually check the ebuild, to see 
whether the maintainer considered it important enough to use-default it 
on.

>> When I set such an entry, I prefix a comment line with the date and an
>> explanation of WHY the entry needs to be there, different from my
>> normal default settings.  Sometimes, it's due to a bug, and a couple
>> versions later I can go thru and test with that entry commented, to see
>> if the bug is fixed, yet.  Other times it's due to a use-dep from some
>> other package I have installed.  Both qt and kde tend to have
> 
> This where we get to the point.  If you set USE flag because of a bug or
> because of dependency it is not because you *want to* but because you
> *have to* (or portage *needs to*).  Therefore you need to add a comment
> WHY you did it.
> 
> For example I have -doc in make.conf because I do *not want* docs
> globally.  But I have 'dev-lang/python doc' in package.use because I
> develop in Python and *want* the documentation for it.  Such entry does
> not need a comment, because I simply know what I want.

My comment for the latter is generally along the lines of "I want docs 
for this", or "docs here are useful".

Here, it's justifying the non-default with documentation.  It doesn't 
matter so much whether it's a non-default I want specifically, or one 
that was forced on me, there's a reason I set my USE flags as I do, and 
deviating from that needs justification, so I provide it... in a way that 
makes sense to me, at least.  I'd not expect those "docs are useful here" 
comments to make much sense to others, as it's begging the question (in 
the legal and original Latin circular reasoning sense).  But it's 
justification for the non-default, to me.

> Another example.  I have -python globally and have been using
> app-mobilephone/gammu.  Now I want to emerge app-mobilephone/wammu. But
> it needs +python for gammu, so I *have to* set 'app-mobilephone/gammu
> python' in package.use.  In this case I also add a comment which
> explains the reason because it is not what *I want* it is what *portage
> needs*.
> 
>> Regardless of why it's there, however, because only non-default entries
>> are in package.use, they're the obvious exception.
> 
> You somehow do not distinguish between those two types of exceptions
> explained in examples above.

Because to me, it's the exception that matters.  I document it, and when 
the reason goes away, so can the exception, but whether it's an exception 
forced on me by upstream or one I want, like docs for some particular 
package, it's an exception to a rule that's there for a reason, and that 
exception needs documentation.


But bottom line back on the original debate on every-update USE flag 
changes, yeah, if you're updating only every 2-4 months, then yes, I 
could see that being flag changes every update.  Update frequency is the 
(formerly) mystery reason for the difference, at least the main one!

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-dev] Re: Things one could be upset about

[Rich Freeman]

On Mon, Jan 26, 2015 at 8:21 AM, Duncan <1i5t5.duncan@cox.net> wrote:
> The result of the current policy is that if you're waiting for the GLSA,
> unless it's _extreme_ priority (heartbleed level), on at least amd64,
> you're very often sitting there exposed for well over a week, and too
> often a month, after the fix is out there, actually installed on /my/
> systems.  And to me that's a game of Russian Roulette odds that I'm
> simply not willing to play.

Agree.  Honestly, I think we should really reconsider the current GLSA
policy.  I half-consider unsubscribing to them since they often come
out weeks after a vulnerability is fixed on amd64, let alone
discovered.  If you're relying on glsa-check as the indicator as to
whether you should update, then you're probably going to be vulnerable
for weeks.

I wonder if it would make sense to just send them out on first-fix, or
even on stablereq.  The main reason that I'd hold off on sending them
out at first sign of vulnerability is that information on what
versions are/aren't vulnerable is going to be hazy, and it won't have
clear instructions on what to do.  You might end up picking the wrong
version to update to and then find yourself having to update again or
downgrading or running ~arch because the package maintainer decided to
do something different.  By the time you have a stablereq things have
settled down - maybe if a bug is found on another arch you might end
up with a revbump, but that is going to be minor impact and anybody
doing daily updates is going to get hit by that anyway.

From a PR standpoint we'll be communicating to some users that they
are vulnerable, and we haven't completely fixed the issue yet.  I
think we just need to reset expectations here.  The fact is that today
they're just as vulnerable, but we don't broadcast that.  Sending out
notice sooner will help out users who want to update based on GLSAs,
and if there isn't a stable version yet the user can decide whether to
just wait for testing or move ahead on their own.

It just seems to me that the current approach of sending out GLSAs a
month after the fix is available for 98% of our users makes them
fairly unuseful.

--
Rich

Re: [gentoo-dev] Re: Things one could be upset about

[Róbert Čerňanský]

On Mon, 26 Jan 2015 09:20:30 -0500
Rich Freeman <rich0@gentoo.org> wrote:

> On Mon, Jan 26, 2015 at 8:21 AM, Duncan <1i5t5.duncan@cox.net> wrote:
> > The result of the current policy is that if you're waiting for the
> > GLSA, unless it's _extreme_ priority (heartbleed level), on at
> > least amd64, you're very often sitting there exposed for well over
> > a week, and too often a month, after the fix is out there, actually
> > installed on /my/ systems.  And to me that's a game of Russian
> > Roulette odds that I'm simply not willing to play.
> 
> From a PR standpoint we'll be communicating to some users that they
> are vulnerable, and we haven't completely fixed the issue yet.  I
> think we just need to reset expectations here.  The fact is that today
> they're just as vulnerable, but we don't broadcast that.  Sending out
> notice sooner will help out users who want to update based on GLSAs,
> and if there isn't a stable version yet the user can decide whether to
> just wait for testing or move ahead on their own.

I do check also other sources of security related info and take
measures if it affects me (update affected package, change
configuration, ...).  I should say earlier "security updates" instead
of "GLSAs" which would be actually closer to reality.

I agree that (unfixed) security issues should be communicated so we do
not put false hopes to GLSA.

Robert


-- 
Róbert Čerňanský
E-mail: openhs@tightmail.com
Jabber: hs@jabber.sk

[gentoo-dev] Re: Things one could be upset about

[Duncan]

Rich Freeman posted on Mon, 26 Jan 2015 09:20:30 -0500 as excerpted:

> Honestly, I think we should really reconsider the current GLSA policy.

What's the next step, not to step on anyone's toes?  It's going to take 
coordination, so get a sense of the council first and then see what the 
security team and archs think?  Security team first, then archs and 
council?

Or start a new thread here with a security proposal title and see where 
that goes first?

Because it's nice to think we should reconsider current policy, and I 
obviously think so too, but way down in this thread as it is, this could 
easily get lost and nothing will change, at present.  And that would be a 
shame.  It might be that the changes we're talking have a fatal flaw, but 
if so, better to get a proposal out there and have it properly shot down, 
ideally with another proposal substituted, than to have it sink and never 
know what that fatal flaw was.

Because, well, I shudder to think of the actual security of a publicly 
accessible installation if they're waiting on GLSAs, the way thinks are 
now.  That being the case, it'd arguably be better not to even have the 
charade.  Yes, have a security team and have them do the updates they are 
doing now.  Just don't have GLSAs at the end.  So people can't have a 
false sense of security relying on them.

But if they can be fixed, say by sending the GLSA at first-to-stable or 
even stable-request, I think the GLSAs would actually be useful, and it 
seems you, at least, are of the same opinion.

So where /does/ it go from here?

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-dev] Things one could be upset about

[Andreas K. Huettel]

> * AutoRepoman catches on average maybe 2 user-visible breakages.
>     Fix: Make repoman faster (tree-wide scans take ~2 CPU-hours)
>     Fix: Remind people that using repoman is not optional

Fix: Make more repoman warnings fatal. Please.
Fix: Make the QA team deliver a friendly but stern warning to people who don't 
use repoman and thus commit crap. If repeated or blatantly wrong, revert [1].

Dream: If the git migration ever happens, make later git refuse pushes that 
break the tree. (if only by pushing to a staging area first, which is then 
automatically forwarded to the main tree if ok.)

> * AutoRepoman still runs on my hardware
>     Fix: Remind infra of https://bugs.gentoo.org/show_bug.cgi?id=484064
> * mail archives have been broken since 2012
>     Fix: get infra to either fix it, or provide enough information that it
>     can be fixed. See https://bugs.gentoo.org/show_bug.cgi?id=424647#c27
> * git.overlays.gentoo.org only partially functional (web interface / cgit
>     down)
> * euscan doesn't run on infra hardware
>     Fix: https://bugs.gentoo.org/show_bug.cgi?id=453886
> * libreoffice-bin isn't built on infra hardware
>     Fix: Remind infra to set up a build environment for that

Fix: Maybe accept more volunteers into infra? 
It would be interesting to hear if/how recruiting of new infra members 
progresses, in relation to workload.

[I don't buy the argument that the mail archives are unneeded, ever since I've 
tried assembling a council agenda and half the relevant e-mails were missing 
at gmane. AutoRepoman and irker are extremely useful and a direct help for all 
developers.]

* Our main website is still stuck in the 90ies.
Fix: migrate all remaining projects to the wiki, then set up something new as 
main page. 
Willing to help out with the first step (generating the wiki pages) if people 
are willing to accept help.

* We need a bit more Public Relations. Feeling a bit annoyed right now that we 
still don't have a FOSDEM booth (maybe we should just occupy the ChromeOS 
one). 
Fix: Next year, start getting annoyed in time and annoy some fellow Europeans. 
:]

[1] http://sources.gentoo.org/cgi-bin/viewvc.cgi/gentoo-x86/dev-libs/libusbhp/ChangeLog?hideattic=0&view=log

-- 
Andreas K. Huettel
Gentoo Linux developer (council, kde)
dilfridge@gentoo.org
http://www.akhuettel.de/

Re: [gentoo-dev] Things one could be upset about

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 680 bytes --]

Dnia 2015-01-17, o godz. 19:35:09
Patrick Lauer <patrick@gentoo.org> napisał(a):

> Here's a random unsorted list of things that it would make sense to be upset 
> about. Some issues that people have successfully ignored for a few years ...
> 
> In no way exhaustive list, feel free to remember a dozen things I forgot ;)
> (If you suggest other things please try to offer constructive criticism,
> i.e. possible strategies to fix issues ... whining by itself is not very 
> useful) 

 * people writing to the mailing lists and starting pointless
   discussions which never bring anything good and only waste people's
   time.

-- 
Best regards,
Michał Górny

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 949 bytes --]

Re: [gentoo-dev] Things one could be upset about

[Peter Stuge]

Patrick Lauer wrote:
> they can all be fixed.
> 
> Let's not tolerate mediocrity.

All you can do is to try to set an example, but you'll likely find
that most of the time, nobody is willing to live with the tradeoffs
for excellence - the obvious one being perceived slower development.

Countless others are perfectly happy with mediocrity, and wouldn't
care even if they were to understand just how mediocre.


//Peter

Re: [gentoo-dev] Things one could be upset about

[Sergei Trofimovich]

[-- Attachment #1: Type: text/plain, Size: 322 bytes --]

On Sat, 17 Jan 2015 19:35:09 +0800
Patrick Lauer <patrick@gentoo.org> wrote:

> * AutoRepoman catches issues, but no one but me seems to care
>     Fix: Remind people of http://packages.gentooexperimental.org/repoman-current-issues.txt

I've tweaked two random things in this list :)
Thank you!

-- 

  Sergei

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: [gentoo-dev] Things one could be upset about

[Zac Medico]

On 01/17/2015 03:35 AM, Patrick Lauer wrote:
> * Portage is too slow
>     On 'small' hardware emerge -upNDv @world can take enough time 
>     to make updates prohibitive - on an 800Mhz machine it took me 
>     about 3 days to figure out a solution to some silly blockers due to the
>     very slow change test cycle
>     Fix: Do some profiling and un-refactoring to remove useless code
>     Fix: Set up a reference system (CI) to catch future regressions

I'm certainly in favor of improving portage performance. However, for
"small" hardware (which includes many ARM and MIPS systems), we really
need to focus on binary package support. Note that dependency resolution
for installing binary packages tends to be much simpler than for
building ebuilds from source, and this translates into much shorter
dependency resolution time.

As I have expressed in other emails [1], I am currently working on
making Gentoo's binary package support more competitive with binary
distros. I have created a sys-apps/portage-9999 ebuild [2] for people
who would like to test features not released in mainline portage yet.

[1] http://thread.gmane.org/gmane.linux.gentoo.project/4205/focus=4217
[2] https://github.com/zmedico/portage-binpkg-support-overlay
-- 
Thanks,
Zac

Re: [gentoo-dev] Things one could be upset about

[Patrick Lauer]

On Saturday 17 January 2015 13:12:56 Zac Medico wrote:
> On 01/17/2015 03:35 AM, Patrick Lauer wrote:
> > * Portage is too slow
> > 
> >     On 'small' hardware emerge -upNDv @world can take enough time
> >     to make updates prohibitive - on an 800Mhz machine it took me
> >     about 3 days to figure out a solution to some silly blockers due to
> >     the
> >     very slow change test cycle
> >     Fix: Do some profiling and un-refactoring to remove useless code
> >     Fix: Set up a reference system (CI) to catch future regressions
> 
> I'm certainly in favor of improving portage performance. However, for
> "small" hardware (which includes many ARM and MIPS systems), we really
> need to focus on binary package support. Note that dependency resolution
> for installing binary packages tends to be much simpler than for
> building ebuilds from source, and this translates into much shorter
> dependency resolution time.
> 
That's an orthogonal problem. And that's coming from someone who extensively 
uses binpkgs already ...

The problem with (dependency resolution) performance is that in some 
scenarios, even on fast machines, it takes "too long" - especially if there's 
some silly useflag mismatch and two packages that have ~arch keywords, and now 
you need 12 attempts to figure out a solution that works for you ...
At 30 seconds for each resolution cycle that's 6 minutes waiting for portage.

If it were faster it'd almost be interactive ...

Also - just for comparison:
On my currently fastest box "emerge -ep @world" takes about 3 seconds since 
there's almost no packages installed.
On my currently slowest box same takes about 15 minutes, because (1) lots of 
packages installed and (2) slow CPU and (3) brutally slow IO

Binpkgs only help so much - if any dependencies change I need to sync the 
configuration between client and server, and to see if it resolves I need to do 
a dryrun on the client (or be very certain that the configuration really 
matches) - or risk that it's going to fail because of mismatches.


I haven't quite figured out why portage needs such humongous amounts of memory 
(>300MB ?!) and why it needs to spend a minute to figure out how to install a 
simple almost-no-dependencies tool like htop or iotop. There's some obvious 
badness, and just saying "well let's use binpkgs" won't fix it (and, well, on 
the binpkg server if you have 3k packages installed you get the same 
performance issues, so ignoring the issue is kinda not a good idea)

There's been some good progress, I remember you reducing memory usage already 
(>500MB -> 350MB or so for merging kernel sources) and there's some speedups 
from the last round of performance work. Still I think we can do a lot better 
:)

Thanks for your efforts,

Patrick

Re: [gentoo-dev] Things one could be upset about

[Zac Medico]

On 01/17/2015 04:46 PM, Patrick Lauer wrote:
> On Saturday 17 January 2015 13:12:56 Zac Medico wrote:
>> On 01/17/2015 03:35 AM, Patrick Lauer wrote:
>>> * Portage is too slow
>>>
>>>     On 'small' hardware emerge -upNDv @world can take enough time
>>>     to make updates prohibitive - on an 800Mhz machine it took me
>>>     about 3 days to figure out a solution to some silly blockers due to
>>>     the
>>>     very slow change test cycle
>>>     Fix: Do some profiling and un-refactoring to remove useless code
>>>     Fix: Set up a reference system (CI) to catch future regressions
>>
>> I'm certainly in favor of improving portage performance. However, for
>> "small" hardware (which includes many ARM and MIPS systems), we really
>> need to focus on binary package support. Note that dependency resolution
>> for installing binary packages tends to be much simpler than for
>> building ebuilds from source, and this translates into much shorter
>> dependency resolution time.
>>
> That's an orthogonal problem. And that's coming from someone who extensively 
> uses binpkgs already ...

Sure, but building from source on "small" hardware is not necessarily
the best practice. If our binary package support was better, then people
might be more likely to use "big" hardware to build packages for
distribution to "small" hardware.

> The problem with (dependency resolution) performance is that in some 
> scenarios, even on fast machines, it takes "too long" - especially if there's 
> some silly useflag mismatch and two packages that have ~arch keywords, and now 
> you need 12 attempts to figure out a solution that works for you ...
> At 30 seconds for each resolution cycle that's 6 minutes waiting for portage.
> 
> If it were faster it'd almost be interactive ...

Yeah, that would be great. On a related note, I think that the default
emerge --backtrack=10 setting is too high, causing emerge to waste lots
of cpu time before it ultimately fails to find a valid solution. I've
filed a bug to make it default to --backtrack=3 [1].

> Also - just for comparison:
> On my currently fastest box "emerge -ep @world" takes about 3 seconds since 
> there's almost no packages installed.
> On my currently slowest box same takes about 15 minutes, because (1) lots of 
> packages installed and (2) slow CPU and (3) brutally slow IO
> 
> Binpkgs only help so much - if any dependencies change I need to sync the 
> configuration between client and server, and to see if it resolves I need to do 
> a dryrun on the client (or be very certain that the configuration really 
> matches) - or risk that it's going to fail because of mismatches.

For some, maybe a nice way to sync configuration would be to create a
customized profile. Then, emerge --sync would pull in your configuration
updates from the server.

With "profile-formats = portage-2" in metadata/layout.conf of your
profiles repository, you can put things like
"gentoo:default/linux/amd64/13.0/desktop" in the parent file of your
profile, so it inherits from a profile in the "gentoo" repository.

> I haven't quite figured out why portage needs such humongous amounts of memory 
> (>300MB ?!)

Yes, I would like to work on reducing the memory consumption.

> and why it needs to spend a minute to figure out how to install a 
> simple almost-no-dependencies tool like htop or iotop.

Note that the emerge --complete-graph-if-new-ver and
--complete-graph-if-new-use options are enabled by default. These
options will force emerge to create a complete dependency graph, in
order to ensure that all reverse-dependencies are respected.

> There's some obvious 
> badness, and just saying "well let's use binpkgs" won't fix it (and, well, on 
> the binpkg server if you have 3k packages installed you get the same 
> performance issues, so ignoring the issue is kinda not a good idea)

Of course not.

> There's been some good progress, I remember you reducing memory usage already 
> (>500MB -> 350MB or so for merging kernel sources) and there's some speedups 
> from the last round of performance work. Still I think we can do a lot better 
> :)

Sure we can.

> Thanks for your efforts,
> 
> Patrick

[1] https://bugs.gentoo.org/show_bug.cgi?id=536926
-- 
Thanks,
Zac

Re: [gentoo-dev] Things one could be upset about

[Andrew Savchenko]

[-- Attachment #1: Type: text/plain, Size: 2725 bytes --]

On Sat, 17 Jan 2015 17:43:17 -0800 Zac Medico wrote:
> On 01/17/2015 04:46 PM, Patrick Lauer wrote:
> > On Saturday 17 January 2015 13:12:56 Zac Medico wrote:
> >> On 01/17/2015 03:35 AM, Patrick Lauer wrote:
> >>> * Portage is too slow
> >>>
> >>>     On 'small' hardware emerge -upNDv @world can take enough time
> >>>     to make updates prohibitive - on an 800Mhz machine it took me
> >>>     about 3 days to figure out a solution to some silly blockers due to
> >>>     the
> >>>     very slow change test cycle
> >>>     Fix: Do some profiling and un-refactoring to remove useless code
> >>>     Fix: Set up a reference system (CI) to catch future regressions
> >>
> >> I'm certainly in favor of improving portage performance. However, for
> >> "small" hardware (which includes many ARM and MIPS systems), we really
> >> need to focus on binary package support. Note that dependency resolution
> >> for installing binary packages tends to be much simpler than for
> >> building ebuilds from source, and this translates into much shorter
> >> dependency resolution time.
> >>
> > That's an orthogonal problem. And that's coming from someone who extensively 
> > uses binpkgs already ...
> 
> Sure, but building from source on "small" hardware is not necessarily
> the best practice. If our binary package support was better, then people
> might be more likely to use "big" hardware to build packages for
> distribution to "small" hardware.

Maybe I'm going offtopic, but emerge performance is not limiting
factor here, at least from my experience; of course performance
improvements in emerge are always welcomed.

The largest problem with "small" hardware is cross-compilation from
"large" hardware. Less powerful systems often have completely
different architecture, e.g. I'd like to install Gentoo on
Paspberry Pi B (why? just for fun and to compare its performance
to Raspbian), but I don't have any other ARM hosts, so I have to
build packages on ~amd64. Setting distcc is not enough here,
because it can't handle configure, autotools, linking, non-C/C+
+/ObjC code and so no. And cross-compilation was always a black
magick.

I tried to setup ~amd64 host to build arbitrary arm packages, but I
failed. The main problem is that too many packages bootstrap during
compilation phase, e.g. compile some binary/library which is used
later during compilation. Of course, code compiled for arm will not
run on amd64, so package fails to build. Probably I should try to
use QEMU as described in embedded handbook and offload compilation
to distcc, even distcc on the same host in order to move most
compilation process out of QEMU VM.
 
Best regards,
Andrew Savchenko

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: [gentoo-dev] Things one could be upset about

[Jeroen Roovers]

On Sat, 17 Jan 2015 19:35:09 +0800
Patrick Lauer <patrick@gentoo.org> wrote:

> * AutoRepoman catches on average maybe 2 user-visible breakages.
>     Mostly removing stable on HPPA ;)
>     Fix: Make repoman faster (tree-wide scans take ~2 CPU-hours)
>     Fix: Remind people that using repoman is not optional

repoman doesn't check reverse dependencies for the package you're
working on.

eshowkw(1) displays keywording in a pretty nice graph. Avoiding removing
the (last) stable ebuild probably involves having that automatically
invoked on entering (or working in, at some point) a package directory
and actually reading the output before you decide to remove any ebuild.


     jer

Re: [gentoo-dev] Things one could be upset about

[Patrick Lauer]

On 01/19/15 17:47, Jeroen Roovers wrote:
> On Sat, 17 Jan 2015 19:35:09 +0800
> Patrick Lauer <patrick@gentoo.org> wrote:
> 
>> * AutoRepoman catches on average maybe 2 user-visible breakages.
>>     Mostly removing stable on HPPA ;)
>>     Fix: Make repoman faster (tree-wide scans take ~2 CPU-hours)
>>     Fix: Remind people that using repoman is not optional
> 
> repoman doesn't check reverse dependencies for the package you're
> working on.

If it were fast enough we could do that.

pcheck from pkgcore was at one point down to under 3 minutes for a full
tree scan, that's pretty much "fast enough" so that people could run it
on almost every ebuild removal. repoman takes around 30 minutes when
parallelized, on the fastest hardware I currently have access to. Or
about 2 CPU-hours with a single thread ... that's prohibitively slow.

Re: [gentoo-dev] Things one could be upset about

[Tim Harder]

[-- Attachment #1: Type: text/plain, Size: 1083 bytes --]

On 2015-01-19 07:28, Patrick Lauer wrote:
> On 01/19/15 17:47, Jeroen Roovers wrote:
> > On Sat, 17 Jan 2015 19:35:09 +0800
> > Patrick Lauer <patrick@gentoo.org> wrote:
> > 
> >> * AutoRepoman catches on average maybe 2 user-visible breakages.
> >>     Mostly removing stable on HPPA ;)
> >>     Fix: Make repoman faster (tree-wide scans take ~2 CPU-hours)
> >>     Fix: Remind people that using repoman is not optional
> > 
> > repoman doesn't check reverse dependencies for the package you're
> > working on.
> 
> If it were fast enough we could do that.
> 
> pcheck from pkgcore was at one point down to under 3 minutes for a full
> tree scan, that's pretty much "fast enough" so that people could run it
> on almost every ebuild removal. repoman takes around 30 minutes when
> parallelized, on the fastest hardware I currently have access to. Or
> about 2 CPU-hours with a single thread ... that's prohibitively slow.

I'd be interested to hear if pcheck has regressed significantly at all
for you when running it now on a similar set up.

Thanks,
Tim

[-- Attachment #2: Type: application/pgp-signature, Size: 473 bytes --]

Re: [gentoo-dev] Things one could be upset about

[Rich Freeman]

On Mon, Jan 19, 2015 at 4:47 AM, Jeroen Roovers <jer@gentoo.org> wrote:
>
> repoman doesn't check reverse dependencies for the package you're
> working on.
>

Indeed, it doesn't even check forward dependencies which are blockers.
kmod-19 was just stabilized accidentally despite having a blocker on
all stable versions of systemd.  That resulted in a big scramble to
backport a fix to systemd (and incidentally discover that openrc
suffered from the same problem, thus resulting in yet another rushed
fix).  It would have been helpful if repoman could have noticed this.

-- 
Rich

Re: [gentoo-dev] Things one could be upset about

[hasufell]

Patrick Lauer:
> Here's a random unsorted list of things that it would make sense to be upset 
> about. Some issues that people have successfully ignored for a few years ...
> 
> In no way exhaustive list, feel free to remember a dozen things I forgot ;)
> (If you suggest other things please try to offer constructive criticism,
> i.e. possible strategies to fix issues ... whining by itself is not very 
> useful) 
> 

Thanks, that's an excellent thread.

I think you forgot an important point:

* lack of practical QA: no review workflow and no appropriate tools for
reviewing

I could start a long text block about why reviewing is mandatory for QA,
but let's just think about it this way:
What do you think would happen if the linux kernel switched to CVS and
gave the most active 250 collaborators direct push access to the main
Linus repository?

I hope greg k-h does not read this. He'd probably get a heart attack.

Also: people seem to think we don't have enough manpower for a review
workflow. No, it's really the other way around. If you make
collaboration difficult, then you need a lot more manpower.

Re: [gentoo-dev] Things one could be upset about

[Luca Barbato]

On 19/01/15 16:47, hasufell wrote:
> I think you forgot an important point:
>
> * lack of practical QA: no review workflow and no appropriate tools for
> reviewing
>
> I could start a long text block about why reviewing is mandatory for QA,
> but let's just think about it this way:
> What do you think would happen if the linux kernel switched to CVS and
> gave the most active 250 collaborators direct push access to the main
> Linus repository?
>
> I hope greg k-h does not read this. He'd probably get a heart attack.
>
> Also: people seem to think we don't have enough manpower for a review
> workflow. No, it's really the other way around. If you make
> collaboration difficult, then you need a lot more manpower.
>

I already pointed out that there are _not_ good review tools. There are 
not for a by-email workflow we have in Libav, there aren't really for a 
tool-mediated workflow we could have in Gentoo.

I have no problems in devoting some time on preparing a tool suited for 
our purpose (once we switch to git), but I'd need more volunteers to 
help me with it.

lu