[gentoo-dev] Is || ( Atom... ) broken?

[gentoo-dev] Is || ( Atom... ) broken?

[Greg Turner]

[-- Attachment #1: Type: text/plain, Size: 201 bytes --]

WTF is up with it?  Why does it love the first Atom so much more than the
others?

It could be such a useful feature, but, in practice, it just never seems to
do what I want it to.  Is it a bug?

-gmt

[-- Attachment #2: Type: text/html, Size: 286 bytes --]

[gentoo-dev] Re: Is || ( Atom... ) broken?

[Duncan]

Greg Turner posted on Mon, 07 Jul 2014 03:14:14 -0700 as excerpted:

> WTF is up with it?  Why does it love the first Atom so much more than
> the others?
> 
> It could be such a useful feature, but, in practice, it just never seems
> to do what I want it to.  Is it a bug?

[FWIW, the HTML isn't particularly appreciated.]

AFAIK, the first atom is simply the one chosen to fill the dependency if 
none of the || choices are currently installed.  As such the first one is 
the default, but if one of the others is installed that should fill the 
dependency just as well.

As well, I /believe/ (but don't know for sure) that the resolver prefers 
other slots of installed packages to those not installed at all, so a new 
slot of something already installed but not in the first/default position 
should be preferred over the first/default dependency.  Obviously this 
would be particularly important for subslot dependencies.

Is that not the behavior you're seeing?

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-dev] Is || ( Atom... ) broken?

[Rich Freeman]

On Mon, Jul 7, 2014 at 6:14 AM, Greg Turner <gmt@malth.us> wrote:
> WTF is up with it?  Why does it love the first Atom so much more than the
> others?
>
> It could be such a useful feature, but, in practice, it just never seems to
> do what I want it to.  Is it a bug?

Well, more like unspecified behavior.  PMS just says that the PM has
to accept any package in the list.  It is silent on the matter of
which one is to be preferred, or to what degree.

As we saw with upower portage will jump through quite a few hoops to
install the first dependency - it doesn't always figure out that
installing one of the others is easier.  It is a bit hard to
algorithmically define "easier" - should portage favor fewer package
installs, fewer removals, fewer config file changes, avoiding changing
the init system (and what constitutes an init system), etc?  Plus,
there are a lot of potential permutations to deal with.

You'd probably need to be more specific as to what is going on to get further.

I think most would agree that there is room for improvement here.

Rich

Re: [gentoo-dev] Is || ( Atom... ) broken?

[James Potts]

On Mon, Jul 7, 2014 at 6:14 AM, Rich Freeman <rich0@gentoo.org> wrote:
>
> On Mon, Jul 7, 2014 at 6:14 AM, Greg Turner <gmt@malth.us> wrote:
> > WTF is up with it?  Why does it love the first Atom so much more than the
> > others?
> >
> > It could be such a useful feature, but, in practice, it just never seems to
> > do what I want it to.  Is it a bug?
>
> Well, more like unspecified behavior.  PMS just says that the PM has
> to accept any package in the list.  It is silent on the matter of
> which one is to be preferred, or to what degree.
>
> As we saw with upower portage will jump through quite a few hoops to
> install the first dependency - it doesn't always figure out that
> installing one of the others is easier.  It is a bit hard to
> algorithmically define "easier" - should portage favor fewer package
> installs, fewer removals, fewer config file changes, avoiding changing
> the init system (and what constitutes an init system), etc?  Plus,
> there are a lot of potential permutations to deal with.
>
> You'd probably need to be more specific as to what is going on to get further.
>
> I think most would agree that there is room for improvement here.
>
> Rich
>

In this case, it would be nice if Portage would see if one package of
the set could be resolved without blocks or required config changes
(i.e. if one package can be installed *now* choose it over
earlier-listed not-installable packages).  The problem with this is
that it would take longer to resolve || () deps if the first one isn't
installable.  Not only that, but the workaround is easy:  Either
install the package you want first (upower-pm-utils, for example), or
at the same time as your "target" package, so I also don't see this as
high-priority.  I also don't see this as something needing changed in
PMS, as other PMs have different ways of handling the issue.

--James

Re: [gentoo-dev] Re: Is || ( Atom... ) broken?

[Greg Turner]

On Mon, Jul 7, 2014 at 4:02 AM, Duncan <1i5t5.duncan@cox.net> wrote:
> [FWIW, the HTML isn't particularly appreciated.]

Ugh, embarrassing.  Been having mail client trouble for the past
several years :)  Starting to come to the conclusion that aside from
the many respectable curses-based readers, kmail:3.5 remains, after
all these years, the only fat-client GUI mail-reader/writer that
doesn't suck even less than the ajax tools and I should just go back
to it.

Cooking up a reply to some of the substantive stuff in this thread,
but won't have time to copy-edit myself properly until later today.

-gmt

Re: [gentoo-dev] Is || ( Atom... ) broken?

[Kent Fredric]

[-- Attachment #1: Type: text/plain, Size: 1623 bytes --]

On 8 July 2014 00:45, James Potts <arek75@gmail.com> wrote:

>
> In this case, it would be nice if Portage would see if one package of
> the set could be resolved without blocks or required config changes
> (i.e. if one package can be installed *now* choose it over
> earlier-listed not-installable packages).  The problem with this is
> that it would take longer to resolve || () deps if the first one isn't
> installable.  Not only that, but the workaround is easy:  Either
> install the package you want first (upower-pm-utils, for example), or
> at the same time as your "target" package, so I also don't see this as
> high-priority.  I also don't see this as something needing changed in
> PMS, as other PMs have different ways of handling the issue.
>
> --James
>
>
I sometimes wonder if it would be easier in some way if we just employed
more useflags to make the alternation less magical.

And then  perhaps fall back to the existing system of automagicking it only
if no relevant useflags were used.

Just there, I don't recall there being an easy way to say "do X only if !Y
!Z !A"

REQUIRED_USE="x? ( !y !z  ) y? ( !z )"
DEPEND="
x? (  a )
y? (  b )
z? (  c )
!x? ( !y? ( !z? ( || ( a b c ) )))
"

As it stands its useful, ... until portage takes the wrong path, and then
you have to fix the problem manually by convincing it which path to take by
lining up the dependencies yourself, instead of just declaring "look, the
path you should be doing is this one, forget that other stuff" and let
portage handle the rest.

Though this is Probably crazy talk.

-- 
Kent

*KENTNL* - https://metacpan.org/author/KENTNL

[-- Attachment #2: Type: text/html, Size: 2854 bytes --]

Re: [gentoo-dev] Is || ( Atom... ) broken?

[Samuli Suominen]

On 07/07/14 13:14, Greg Turner wrote:
> WTF is up with it?  Why does it love the first Atom so much more than
> the others?
>
> It could be such a useful feature, but, in practice, it just never
> seems to do what I want it to.  Is it a bug?
>
> -gmt

short answer, yes, specially in latest ~arch sys-apps/portage, see:

http://bugs.gentoo.org/515230

Re: [gentoo-dev] Is || ( Atom... ) broken?

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 731 bytes --]

On Mon, 7 Jul 2014 03:14:14 -0700
Greg Turner <gmt@malth.us> wrote:
> WTF is up with it?  Why does it love the first Atom so much more than
> the others?

Historically, because it was defined that way. Now there are so many
special cases, heuristics and exceptions that there's no way of telling
what it will do...

> It could be such a useful feature, but, in practice, it just never
> seems to do what I want it to.  Is it a bug?

The problem is that developers started wanting || to do "what they want
it to do" rather than "what it is defined to do", and that Portage
tried to give them that. Unfortunately, getting software to do "what you
expect" sometimes looks OK in the short term.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: [gentoo-dev] Is || ( Atom... ) broken?

[Greg Turner]

On Mon, Jul 7, 2014 at 4:14 AM, Rich Freeman <rich0@gentoo.org> wrote:
> Well, more like unspecified behavior.  PMS just says that the PM has
> to accept any package in the list.  It is silent on the matter of
> which one is to be preferred, or to what degree.

In general the PMS dependency language seems quite liberal here.  As
best I can tell, the PMS only stipulates something to the effect of:
"the PM has to meet the conditions matching the dependency specifiers
in the ebuild before running the phase functions" but doesn't provide
any guidance as to /how/ the PM should make it so, if it isn't at the
outset.  Assuming I haven't missed something that contradicts or
narrows that reading, I have mixed feelings about it.  While it feels
inscrutable, it gives us a lot of freedom to fix the problem.

> As we saw with upower portage will jump through quite a few hoops to
> install the first dependency - it doesn't always figure out that
> installing one of the others is easier.  It is a bit hard to
> algorithmically define "easier" - should portage favor fewer package
> installs, fewer removals, fewer config file changes, avoiding changing
> the init system (and what constitutes an init system), etc?  Plus,
> there are a lot of potential permutations to deal with.

Yes, agreed.  The.  However, the case where the PM has more than one
possible way to get there, although tricky/interesting, is not what
inspired me to post this.

I was hacking on my private overlay's pipelight-plugin ebuild, which
had deps like:

DEPEND="|| (
    >=app-emulation/wine-1.7.21[pipelight]
    >=app-emulation/wine-compholio-1.7.14
)"

when I tried to emerge, it insisted on wine[pipelight] even though I
already had wine-compholio installed.

Now that I think about it, the probable explanation was pretty
obvious: portage saw I had wine installed, and therefore said "that's
the one," and headed obstinately down that path, finally deciding,
"OK, but you need to enable the pipelight use-flag and rebuild wine".
I wonder what would have happened if I'd had --autounmask=n in the
command-line.

> I think most would agree that there is room for improvement here.

Some in this thread have argued that since there's an easy workaround
this is not a particularly important issue.  Fair enough, but while,
for most of the people reading this thread, that may be so, I have a
couple of friends who are casual users of portage, and they finds
these situations to be more-or-less utterly intractable.  I don't mind
them asking me for help, but what does suck is I usually wind up
having to tell them to use -O, a feature I'd rather they didn't need.

It seems pretty clear to me that, recently, people who "didn't do
anything crazy" are experiencing depsolving failures more and more
(perhaps due to the explosion of multilib-build ebuilds in the tree --
no pun intended :)).

I can only imagine how difficult the experience of someone trying
Gentoo for the first time must be today: Casual experimenters
presumably give up after encountering a failure or three justified by
more screenfulls of colorful gibberish than their consoles can hold in
scroll-back before even getting grub installed.  One can see how they
might conclude Gentoo is either broken, or only for for people more
technically inclined than they.

Indeed, truth be told, a reasonable person could defend either proposition.

One other note. If  we think "X would make portage dep-solving even
slower, so, screw that, we have bigger fish to fry," that means slow
dep-solving has effectively become a blocker for enhancement/bugfix X.
Not saying it shouldn't be, just that it's food for thought*.

As for the idea of reverting portage -- with some effort, we could
revert the dep-solving changes while leaving any important bugfixes
intact (effectively, I guess, backporting the latest portage to the
old portage.dep).

But, I suspect, the result would be, we'd find things were not so rosy
a year ago or whenever.  Gentoo-x86 had a /lot/ less hairy EAPI4+
use-conditional dependency cascades a year ago, so of course things
were faster.  If someone were to prove me wrong with some benchmarks,
though, I'd happily admit I was mistaken.

TBH I keep flirting with the Gordian-knot theory that only a complete
gut-rehab of portage.dep can rescue us...

--
gmt

*Indeed, I'd estimate I often have to wait a half-hour on my 32GB
workstation to get a plan from emerge -DuavN @world, so for me that's
a bigger problem than anything else about portage right now.

[gentoo-dev] Re: Is || ( Atom... ) broken?

[Duncan]

Greg Turner posted on Mon, 07 Jul 2014 16:42:08 -0700 as excerpted:

> *Indeed, I'd estimate I often have to wait a half-hour on my 32GB
> workstation to get a plan from emerge -DuavN @world, so for me that's a
> bigger problem than anything else about portage right now.

Wow.  I didn't realize and to some extent had forgotten how bad things 
had gotten, as I (1) run an SSD-based tree now, and (2) set backtrack=0 
so portage does its best only one time thru and spits out the result, 
allowing me to at least see what it's doing in reasonable time, such that 
I can resolve the thing manually if I need to.

I haven't had to resort to --nodeps yet, but I do often emerge subsets of 
the whole, reducing the number of updates until I have a much smaller 
update set to work with, then sometimes run --tree and/or even go look at 
individual ebuild deps, to see why something's getting pulled in.  And I 
guess running no-multilib means I bypass at least some of the issues, 
too.  That and a USE=-* base probably help a lot, but I've no idea 
whether the fully negated @system so there's no system set to deal with 
helps or hurts.

But from the sound of things, default backtrack=10, multilib, full 
@system set and default profile use, on spinning rust, is getting all but 
intolerable now. =:^(

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-dev] Re: Is || ( Atom... ) broken?

[Kent Fredric]

[-- Attachment #1: Type: text/plain, Size: 1139 bytes --]

On 8 July 2014 16:39, Duncan <1i5t5.duncan@cox.net> wrote:

> But from the sound of things, default backtrack=10, multilib, full
> @system set and default profile use, on spinning rust, is getting all but
> intolerable now. =:^(
>

I'm not sure I follow what scenario is happening to make it so slow.

4 minutes isn't great, but its far shy of 30 minutes.

I have only spinning rust, only 8G of ram, and I had firefox eating 100%
cpu(optimus--) while I did this, and I don't exactly have the fastest CPU (
frequent overheating problems trigger thermal throttling )

To suggest its a nightmare for all users seems a stretch given that. It
could be better, but everything can always be better.


time emerge --version
Portage 2.2.10 (default/linux/amd64/13.0, gcc-4.8.3, glibc-2.19-r1,
3.15.2-aufs x86_64)

real    0m4.072s
user    0m3.347s
sys    0m0.545s

find /var/db/pkg/ -name "DEPEND" | wc -l
1941

wc -l /var/lib/portage/world
339 /var/lib/portage/world

time  schedtool -D -n 20 -e ionice -c 3 emerge -uvtDN --pretend @world
real    4m20.247s
user    4m1.973s
sys    0m14.507s
-- 
Kent

*KENTNL* - https://metacpan.org/author/KENTNL

[-- Attachment #2: Type: text/html, Size: 2141 bytes --]

Re: [gentoo-dev] Re: Is || ( Atom... ) broken?

[Greg Turner]

On Mon, Jul 7, 2014 at 9:39 PM, Duncan <1i5t5.duncan@cox.net> wrote:
> But from the sound of things, default backtrack=10, multilib, full
> @system set and default profile use, on spinning rust, is getting all but
> intolerable now. =:^(

Actually a mix of RAID0 and RAID1 over fast, new-ish SSDs -- anyhow I
doubt it would matter on spinning rust.  When I sync, I run egencache
on everything and then emerge --metadata, so by the time I run emerge
-u*, it's against a warm cache; the disk light basically stays dark
the whole time.  But, I have overlays with forced eclass overrides,
and over 2000 packages installed.  Furthermore, after a sync, I'm
often waiting for emerge to fail, not to succeed.

For example, say I've multilibutized something upstream hasn't, and
upstream has version-bumped their non-multilibutized ebuild (but not
multilibutized it); and say I have packages installed requiring
multilib in the same slot -- that's a scenario I frequently find
myself in.  Worse, I will usually provide --complete-graph to portage
(because otherwise, portage can fail to show me the clues I need to
detect precisely this problem).  Basically, you could say my use case
is the perfect storm of ways to make emerge slow.  I probably wouldn't
be kept waiting for those 30 minutes unless I'd waited a week or two,
and let gx86 provide newer versions of several of my ebuilds.  I'm
pretty sure this triggers massive amounts of backtracking.

By the way, I've profiled depsolving on my system.  Portage spends a
/majority/ of it's depsolving time solving regexes on behalf of
portage.dep.Atom.  I have an untested theory that, without fixing any
algorithmic stuff, we could reap a meaningful O(1) gain implementing,
say, a CustomAtom (inherited by Atom) in cpython that does most of its
string parsing in hand-coded C.

-gmt

[gentoo-dev] Re: Is || ( Atom... ) broken?

[Duncan]

Greg Turner posted on Wed, 09 Jul 2014 02:52:31 -0700 as excerpted:

> On Mon, Jul 7, 2014 at 9:39 PM, Duncan <1i5t5.duncan@cox.net> wrote:
>> But from the sound of things, default backtrack=10, multilib, full
>> @system set and default profile use, on spinning rust, is getting all
>> but intolerable now. =:^(
> 
> Actually a mix of RAID0 and RAID1 over fast, new-ish SSDs -- anyhow I
> doubt it would matter on spinning rust.  When I sync, I run egencache on
> everything and then emerge --metadata, so by the time I run emerge -u*,
> it's against a warm cache; the disk light basically stays dark the whole
> time.  But, I have overlays with forced eclass overrides, and over 2000
> packages installed.  Furthermore, after a sync, I'm often waiting for
> emerge to fail, not to succeed.
> 
> Worse, I will usually provide --complete-graph to portage (because
> otherwise, portage can fail to show me the clues I need[.] Basically,
> you could say my use case is the perfect storm of ways to make emerge
> slow.

OK; makes more sense, now.


> I probably wouldn't be kept
> waiting for those 30 minutes unless I'd waited a week or two, and let
> gx86 provide newer versions of several of my ebuilds.  I'm pretty sure
> this triggers massive amounts of backtracking.

Have you /tried/ backtrack=0?  Someone suggested it and I was a bit 
skeptical, but one day after waiting for the 10 backtracks to complete, 
only to see portage tell me it needed my help to solve the problem 
anyway, I decided to try it.  It's likely I'm doing a bit more work 
myself now, but not /that/ much more, and at least it gets done faster, 
giving me enough information to actually do something about it sooner.

If as you say you're often waiting for emerge to fail, then backtrack=0 
may well shave significant time off that failure and let you deal with it 
sooner, as it seems to have done for me.

> By the way, I've profiled depsolving on my system.  Portage spends a
> /majority/ of it's depsolving time solving regexes on behalf of
> portage.dep.Atom.  I have an untested theory that, without fixing any
> algorithmic stuff, we could reap a meaningful O(1) gain implementing,
> say, a CustomAtom (inherited by Atom) in cpython that does most of its
> string parsing in hand-coded C.

Sounds reasonable, but that's well out of my league to code up, so...

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-dev] Re: Is || ( Atom... ) broken?

[Greg Turner]

On Wed, Jul 9, 2014 at 5:34 AM, Duncan <1i5t5.duncan@cox.net> wrote:
> Have you /tried/ backtrack=0?

Yeah.  I had backtrack=1 as my default for a good while before my
overlay started getting cray-cray. It's fast -- well, less slow -- but
diagnostically unhelpful when anything goes "wrong".

I guess, specifically, the things it deals with poorly are:

* precisely the issue I started this thread about (first option in
dependency disjunction not a match)

* ebuild A matches dependency of selected ebuild B, but is rejected by
selected ebuild C, which requires A', A' < A

* trying to switch between packages solving a virtual

* probably more stuff....

-gmt