[gentoo-user] A portage nuisance

[gentoo-user] A portage nuisance

[Helmut Jarausch]

Hi,
I have a problem with emerge for a long time.
Sometimes I need to (re-)emerge many packages like in an
emerge --emptytree @world

Because I use several overlays, there are problems with a lot of
packages.
Unfortunately, emerge shows me just the first problem (like a missing USE-flags)
and then terminates.
Is there any means to let emerge go and report several (all) problems which
are independent of each other?

Many thanks for some hints,
Helmut

Re: [gentoo-user] A portage nuisance

[Peter Humphrey]

On Fri, 27 Oct 2017 12:52:54 -0000
Helmut Jarausch <jarausch@skynet.be> wrote:

> I have a problem with emerge for a long time.
> Sometimes I need to (re-)emerge many packages like in an
> emerge --emptytree @world
> 
> Because I use several overlays, there are problems with a lot of
> packages.
> Unfortunately, emerge shows me just the first problem (like a missing
> USE-flags) and then terminates.
> Is there any means to let emerge go and report several (all) problems
> which are independent of each other?

EMERGE_DEFAULT_OPTS="--keep-going" ?

-- 
Regards,
Peter.

Re: [gentoo-user] A portage nuisance

[Andrew Savchenko]

[-- Attachment #1: Type: text/plain, Size: 1132 bytes --]

On Fri, 27 Oct 2017 14:58:13 +0100 Peter Humphrey wrote:
> On Fri, 27 Oct 2017 12:52:54 -0000
> Helmut Jarausch <jarausch@skynet.be> wrote:
> 
> > I have a problem with emerge for a long time.
> > Sometimes I need to (re-)emerge many packages like in an
> > emerge --emptytree @world
> > 
> > Because I use several overlays, there are problems with a lot of
> > packages.
> > Unfortunately, emerge shows me just the first problem (like a missing
> > USE-flags) and then terminates.
> > Is there any means to let emerge go and report several (all) problems
> > which are independent of each other?
> 
> EMERGE_DEFAULT_OPTS="--keep-going" ?

No, --keep-going allows to continue as long as possible after a
build failure. Helmut asks about dependecies resolution failures,
e.g. in some package REQUIRED_USE is not met, or circular
dependency occurs and so on.

AFAIK there is no way to use keep-going like option for deps
resolution, because first error may trigger a lot of others and
there will be inevitably false errors, because the dependency tree
was not fully built.

Best regards,
Andrew Savchenko

[-- Attachment #2: Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-user] A portage nuisance

[Wols Lists]

On 28/10/17 15:52, Andrew Savchenko wrote:
> On Fri, 27 Oct 2017 14:58:13 +0100 Peter Humphrey wrote:
>> > On Fri, 27 Oct 2017 12:52:54 -0000
>> > Helmut Jarausch <jarausch@skynet.be> wrote:
>> > 
>>> > > I have a problem with emerge for a long time.
>>> > > Sometimes I need to (re-)emerge many packages like in an
>>> > > emerge --emptytree @world
>>> > > 
>>> > > Because I use several overlays, there are problems with a lot of
>>> > > packages.
>>> > > Unfortunately, emerge shows me just the first problem (like a missing
>>> > > USE-flags) and then terminates.
>>> > > Is there any means to let emerge go and report several (all) problems
>>> > > which are independent of each other?
>> > 
>> > EMERGE_DEFAULT_OPTS="--keep-going" ?
> No, --keep-going allows to continue as long as possible after a
> build failure. Helmut asks about dependecies resolution failures,
> e.g. in some package REQUIRED_USE is not met, or circular
> dependency occurs and so on.

What I would like - a bit like --keep-going - is some option that tries
again.

When I do an "emerge -u" it sometimes blows up with this massive load of
dependency failures. So what I end up doing is emerge a few packages
that look like they're going to work, and then try my full update again.
After several cycles through this, suddenly everything works.

So my spec for what I would like is basically, as each package
successfully resolves its dependencies, add it to a "try again" list. If
the current list blows up in dependency hell, restart the emerge with
just the packages in the "try again" list.

When you haven't updated for a while and you've got a lot of packages,
this "emerge what you can" approach certainly seems to work for me, it
would just be nice if it was automated because it can take a lot of
attempts (and time) before the system finally succeeds in updating itself.

Cheers,
Wol

Re: [gentoo-user] A portage nuisance

[Alan McKinnon]

On 28/10/2017 19:58, Wols Lists wrote:
> On 28/10/17 15:52, Andrew Savchenko wrote:
>> On Fri, 27 Oct 2017 14:58:13 +0100 Peter Humphrey wrote:
>>>> On Fri, 27 Oct 2017 12:52:54 -0000
>>>> Helmut Jarausch <jarausch@skynet.be> wrote:
>>>>
>>>>>> I have a problem with emerge for a long time.
>>>>>> Sometimes I need to (re-)emerge many packages like in an
>>>>>> emerge --emptytree @world
>>>>>>
>>>>>> Because I use several overlays, there are problems with a lot of
>>>>>> packages.
>>>>>> Unfortunately, emerge shows me just the first problem (like a missing
>>>>>> USE-flags) and then terminates.
>>>>>> Is there any means to let emerge go and report several (all) problems
>>>>>> which are independent of each other?
>>>>
>>>> EMERGE_DEFAULT_OPTS="--keep-going" ?
>> No, --keep-going allows to continue as long as possible after a
>> build failure. Helmut asks about dependecies resolution failures,
>> e.g. in some package REQUIRED_USE is not met, or circular
>> dependency occurs and so on.
> 
> What I would like - a bit like --keep-going - is some option that tries
> again.
> 
> When I do an "emerge -u" it sometimes blows up with this massive load of
> dependency failures. So what I end up doing is emerge a few packages
> that look like they're going to work, and then try my full update again.
> After several cycles through this, suddenly everything works.
> 
> So my spec for what I would like is basically, as each package
> successfully resolves its dependencies, add it to a "try again" list. If
> the current list blows up in dependency hell, restart the emerge with
> just the packages in the "try again" list.


That is completely contrary to how portage is supposed to work and
produces an unpredictable result.

*You* know what you want and can do it, *you* are a human with a brain
who understands meaning.

Portage cannot do that, it is backed by silicon and has no concept of
meaning. So it has only one real choice - it can do it all or it does
not try.

I'm not surprised Zac never tried implementing partial graph resolution
for the very simple reason that if you try do it, you have no idea what
is going to be built. That is the opposite of what portage must deliver.
-- 
Alan McKinnon
alan.mckinnon@gmail.com

Re: [gentoo-user] A portage nuisance

[Anthony Youngman]

On 28/10/17 20:54, Alan McKinnon wrote:
> Portage cannot do that, it is backed by silicon and has no concept of
> meaning. So it has only one real choice - it can do it all or it does
> not try.
> 
> I'm not surprised Zac never tried implementing partial graph resolution
> for the very simple reason that if you try do it, you have no idea what
> is going to be built. That is the opposite of what portage must deliver.

Why is it the opposite of what portage *must* deliver? All I'm asking is 
that portage build *what it can*. In other words, I know EXACTLY what it 
is going to deliver - its best effort!

And why does portage *have* to choose between all or nothing? All I'm 
asking is that if it can't resolve everything, I want it to resolve 
everything it can. Silicon is perfectly capable of making that decision.

If I say "emerge -u world" I have no idea what it's going to build, if I 
say "emerge -u best-efforts", I have no idea what it's going to build, 
where's the difference?

What I do know, is if I repeat "emerge -u best-efforts" several times, I 
will end up (in all likelihood) with the same result as "emerge -u world".

Cheers,
Wol

Re: [gentoo-user] A portage nuisance

[Alan McKinnon]

On 28/10/2017 22:39, Anthony Youngman wrote:
> On 28/10/17 20:54, Alan McKinnon wrote:
>> Portage cannot do that, it is backed by silicon and has no concept of
>> meaning. So it has only one real choice - it can do it all or it does
>> not try.
>>
>> I'm not surprised Zac never tried implementing partial graph resolution
>> for the very simple reason that if you try do it, you have no idea what
>> is going to be built. That is the opposite of what portage must deliver.
> 
> Why is it the opposite of what portage *must* deliver? All I'm asking is
> that portage build *what it can*. In other words, I know EXACTLY what it
> is going to deliver - its best effort!

Dude, calm down. Showing me that you are upset isn't going to change
jack shit.

> 
> And why does portage *have* to choose between all or nothing? All I'm
> asking is that if it can't resolve everything, I want it to resolve
> everything it can. Silicon is perfectly capable of making that decision.

because "everything it can" is not necessarily the same thing as
"everything it should" or "everything it was asked to do"

Portage has no idea at all why the depgraph failed to resolve fully at
the start. It knows about blockers that are in it's own ebuilds and can
deal with those; the way ebuilds are spec'ed a package manager can
cleave off an entire branch of the tree and build the remainder fully
confident that each part of what it then does build is identical to the
same parts if the blockers never hit the rest. It's deterministic and so
portage can continue.

For everything else, the input is not completely known therefore the
output cannot be completely known either and portage does the correct
thing and not try guess. This is a good thing.

> 
> If I say "emerge -u world" I have no idea what it's going to build, if I
> say "emerge -u best-efforts", I have no idea what it's going to build,
> where's the difference?
> 
> What I do know, is if I repeat "emerge -u best-efforts" several times, I
> will end up (in all likelihood) with the same result as "emerge -u world".

No, you will not. In this case, what portage will give is not the sum of
it's individual parts as there is a real risk that some attributes of
the various parts are WRONG. Therefore the output can be WRONG. It
really is that simple.

You even said it yourself when you used the phrase "in all likelihood".
What about when it isn't? This is a package manager and package managers
have one singular objective that the code must always deliver: the
output must be entirely deterministic. Anything other than that and you
do not have a package manager anymore, now you have software that puts
stuff on a disk.

Portage is doing the right thing when it detects invalid input that it
cannot reliably  - it stops.

But feel free to disagree. If you can do this, the proof is in code and
patches. Got any?


-- 
Alan McKinnon
alan.mckinnon@gmail.com

Re: [gentoo-user] A portage nuisance

[Anthony Youngman]

On 28/10/17 21:50, Alan McKinnon wrote:
> On 28/10/2017 22:39, Anthony Youngman wrote:
>> On 28/10/17 20:54, Alan McKinnon wrote:
>>> Portage cannot do that, it is backed by silicon and has no concept of
>>> meaning. So it has only one real choice - it can do it all or it does
>>> not try.
>>>
>>> I'm not surprised Zac never tried implementing partial graph resolution
>>> for the very simple reason that if you try do it, you have no idea what
>>> is going to be built. That is the opposite of what portage must deliver.
>>
>> Why is it the opposite of what portage *must* deliver? All I'm asking is
>> that portage build *what it can*. In other words, I know EXACTLY what it
>> is going to deliver - its best effort!
> 
> Dude, calm down. Showing me that you are upset isn't going to change
> jack shit.
> 
Sorry - I'm not upset. I just think you are missing what I'm asking for.
>>
>> And why does portage *have* to choose between all or nothing? All I'm
>> asking is that if it can't resolve everything, I want it to resolve
>> everything it can. Silicon is perfectly capable of making that decision.
> 
> because "everything it can" is not necessarily the same thing as
> "everything it should" or "everything it was asked to do"
> 
> Portage has no idea at all why the depgraph failed to resolve fully at
> the start. It knows about blockers that are in it's own ebuilds and can
> deal with those; the way ebuilds are spec'ed a package manager can
> cleave off an entire branch of the tree and build the remainder fully
> confident that each part of what it then does build is identical to the
> same parts if the blockers never hit the rest. It's deterministic and so
> portage can continue.
> 
And if we are understanding each other correctly, THIS is the point at 
which I would expect portage to give up and terminate. Except you seem 
to be saying it doesn't terminate here ...

> For everything else, the input is not completely known therefore the
> output cannot be completely known either and portage does the correct
> thing and not try guess. This is a good thing.
> 
You think it terminates here ...
>>
>> If I say "emerge -u world" I have no idea what it's going to build, if I
>> say "emerge -u best-efforts", I have no idea what it's going to build,
>> where's the difference?
>>
>> What I do know, is if I repeat "emerge -u best-efforts" several times, I
>> will end up (in all likelihood) with the same result as "emerge -u world".
> 
> No, you will not. In this case, what portage will give is not the sum of
> it's individual parts as there is a real risk that some attributes of
> the various parts are WRONG. Therefore the output can be WRONG. It
> really is that simple.

When I run portage, it displays a list of what it is going to emerge. As 
I understand it, as each package is displayed of what it is going to 
emerge, that means it has successfully resolved the dependency graph. It 
also displays packages that are blocked. And then sometimes it just goes 
into a tizzy and says "too many blocks - giving up".

All I'm asking is that as it progresses, it makes a list of those 
packages it can resolve the dependencies for. If it then gives up with 
the current list it's processing, eg "world", it then goes back to the 
list it thinks it can process, and has another go with them.

Because that's exactly what I do, take the first few packages off the 
list that look fine, and emerge them. I then re-run the original emerge, 
rinse and repeat, but it takes absolutely ages, and worse I have to 
babysit the emerge because I'm *expecting* it to hit a problem.
> 
> You even said it yourself when you used the phrase "in all likelihood".
> What about when it isn't? This is a package manager and package managers
> have one singular objective that the code must always deliver: the
> output must be entirely deterministic. Anything other than that and you
> do not have a package manager anymore, now you have software that puts
> stuff on a disk.
> 
> Portage is doing the right thing when it detects invalid input that it
> cannot reliably  - it stops.

Why can't it just discard the invalid input, and try again with input 
that appears valid?

Note I didn't say *continue* - I said "try again". From the beginning.

I'm sorry, but the current behaviour is reminiscent of the way 
programmers (used to) code huge green-screen input forms - when you get 
to the bottom and hit "submit" - one error and it throws the whole lot 
away and makes you start again from scratch!
> 
> But feel free to disagree. If you can do this, the proof is in code and
> patches. Got any?
> 
What language? Python? Sorry, at the moment I don't speak Python - maybe 
I should.

And I hate to say it, but demanding code and patches is NOT good form. 
Yes, far too many people do it, but not everybody is a programmer (I am, 
but it's C and DataBasic). And do you REALLY want jack-of-all-trades 
giving you dodgy code? I contribute elsewhere. Fixing emerge is not 
really my thing, I'm just chucking out an idea I would find very useful 
(and I expect other people will too). I'm one of those odd people who 
actually *enjoy* maintenance work, rather than new stuff, and I'm hoping 
someone like me in the gentoo project will pick up this idea and run 
with it. I'm busy doing maintenance work elsewhere with what time I have ...

To give you a very clear example of what I'm thinking ...

emerge -u world
A will be emerged with options ...
B will be emerged with options ...
C will be emerged with options ...
D is blocked by E
F will be emerged with options ...
G is blocked by H
Giving up, too many circular dependencies
emerge A B C F
...
...
...

That is what I do manually, it would be nice if emerge could do it for me.

Cheers,
Wol

Re: [gentoo-user] A portage nuisance

[Andrew Savchenko]

[-- Attachment #1: Type: text/plain, Size: 2556 bytes --]

On Sat, 28 Oct 2017 22:59:26 +0100 Anthony Youngman wrote:
[...]
> All I'm asking is that as it progresses, it makes a list of those 
> packages it can resolve the dependencies for. If it then gives up with 
> the current list it's processing, eg "world", it then goes back to the 
> list it thinks it can process, and has another go with them.
> 
> Because that's exactly what I do, take the first few packages off the 
> list that look fine, and emerge them. I then re-run the original emerge, 
> rinse and repeat, but it takes absolutely ages, and worse I have to 
> babysit the emerge because I'm *expecting* it to hit a problem.
[...]
> To give you a very clear example of what I'm thinking ...
> 
> emerge -u world
> A will be emerged with options ...
> B will be emerged with options ...
> C will be emerged with options ...
> D is blocked by E
> F will be emerged with options ...
> G is blocked by H
> Giving up, too many circular dependencies
> emerge A B C F

Ah, man, this is where your mistake is. You are assuming that it
 is possible to get a correct dependency subgraph without building
full correct dependency graph first. This is not possible and this
is math. While the approach you described abode may work in some
practical cases, it will be busted in general case.

The key moment here is that graph's root node may be changed during
dependency recalculation based on _how_ conflict is solved, the
same as all other nodes may be reordered. And dependencies which
appear to be valid before conflict is resolved may became invalid
after, consider the following dep tree:

  A
 / \
B   C
    |
 !{D,E}

- B and C depends on A;
- D conflicts with E and both depend on C;

You assume that !{D,E} conflict can be skipped and A, B, C canbe
emerged. But let's assume that you selected D later, but D depends
on F and F conflicts with A[some_flag]. So you'll have to choose
some alternative to A or change its USE flags, this may require to
rebuild the whole dependency tree (and build order may change as
well). In order to prevent dozens (sometimes hundreds or even
thousands) of useless rebuilds and to avoid leaving intermediate
tree in the utterly broken state emerge fails if it can't build the
dependency graph.

Maybe my example above is synthetic and not the best one, you
should understand that dependencies are very complex, may be
intricately interconnected and there is no way to tell which parts
are correct until all picture is seen.

Best regards,
Andrew Savchenko

[-- Attachment #2: Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-user] A portage nuisance

[Wols Lists]

On 28/10/17 23:45, Andrew Savchenko wrote:
>> emerge -u world
>> > A will be emerged with options ...
>> > B will be emerged with options ...
>> > C will be emerged with options ...
>> > D is blocked by E
>> > F will be emerged with options ...
>> > G is blocked by H
>> > Giving up, too many circular dependencies
>> > emerge A B C F
> Ah, man, this is where your mistake is. You are assuming that it
>  is possible to get a correct dependency subgraph without building
> full correct dependency graph first. This is not possible and this
> is math. While the approach you described abode may work in some
> practical cases, it will be busted in general case.
> 
> The key moment here is that graph's root node may be changed during
> dependency recalculation based on _how_ conflict is solved, the
> same as all other nodes may be reordered. And dependencies which
> appear to be valid before conflict is resolved may became invalid
> after, consider the following dep tree:
> 
>   A
>  / \
> B   C
>     |
>  !{D,E}
> 
> - B and C depends on A;
> - D conflicts with E and both depend on C;
> 
> You assume that !{D,E} conflict can be skipped and A, B, C canbe
> emerged. But let's assume that you selected D later, but D depends
> on F and F conflicts with A[some_flag]. So you'll have to choose
> some alternative to A or change its USE flags, this may require to
> rebuild the whole dependency tree (and build order may change as
> well). In order to prevent dozens (sometimes hundreds or even
> thousands) of useless rebuilds and to avoid leaving intermediate
> tree in the utterly broken state emerge fails if it can't build the
> dependency graph.

EXCEPT.

My "emerge A B C F" will presumably then fail with those conflicts, and
it will loop again and just do an "emerge A".

I'm not saying "prune the graph first time, then just emerge
everything". I'm saying "prune the graph, and try again with the reduced
set".

Thing is, I have always found that deleting blockers "emerge -C1", and
emerging everything that I can get to emerge, is almost guaranteed to
end up with a system where everything eventually emerges.

The problem I have is that, if I delay updating (or something like kde
with its gazillions of packages is upgraded), I end up with maybe 300
packages to be emerged. Trying to sort that mess out is a nightmare
especially when emerge starts blowing up with circular dependencies and
stuff, and old packages blocking new ones - "A version 2 is blocked by A
version 1" etc etc. And all too often deleting the current version A
(and the other packages - slated for replacement - that depend on A)
fixes the problem.

I think we're having a misunderstanding here. If emerge does what I'm
asking for, and leaves the system in a broken state (as you seem to
think it will), then that's a serious bug in emerge. You seem to think
I'm asking emerge to prune the dependency tree - I'm not. I'm asking for
it to prune the package list - if it can't sort out dependencies, drop
that package from the package list and then, when it's got a list of
packages that it thinks will work, go back to the start and try to
emerge just those (calculating a new dependency tree).

In other words, I'm asking emerge to automate what I do - look at the
output, find a subset of packages that I think will work, and then ask
it to try again with just those packages. I would be very surprised if
it iterated repeatedly down to the null set.

As for doing useless rebuilds, isn't that MY call, not yours? What is
most valuable - my time, or my computer's time? As it stands, I have to
babysit an emerge - calculating the full graph bombs, so I have to try
to emerge a small subset, which might bomb, and when that works I try
again with the full set that bombs, so I have to iterate again, ad
nauseam ... I like to work on the principle "if the computer can
automate what I do, then it should ..."

What would you recommend as the best way (for somebody who's knowledge
of gentoo is patchy) of dealing with the situation when an emerge blows
up with loads of circular dependencies finishing with the message "too
many errors, giving up"? As opposed to my way of "emerge what I can,
then see what more emerges", which I have always found eventually fixes
everything.

Cheers,
Wol

Re: [gentoo-user] A portage nuisance

[Michael Orlitzky]

On 10/28/2017 08:10 PM, Wols Lists wrote:
> 
> In other words, I'm asking emerge to automate what I do - look at the
> output, find a subset of packages that I think will work, and then ask
> it to try again with just those packages. I would be very surprised if
> it iterated repeatedly down to the null set.

This should be possible. I've lovingly referred to this option in the
past as something like "emerge --just-fucking-do-anything". I can have
300 packages in my update list, with a bunch of installed stuff
depending on a masked version of ruby, and a package that needs to be
manually uninstalled, and new REQUIRED_USE conflicts, and subslot
rebuilds of things that are now masked, and...

And yet portage still knows that timezone-data is going to get updated.
If I can look through the list and "emerge -1 timezone-data" and have
that work, then portage should be able to figure that out, too. No
matter how long it takes, it can't be slower than me doing it by hand.

Re: [gentoo-user] A portage nuisance

[Alan McKinnon]

On 27/10/2017 14:52, Helmut Jarausch wrote:
> Hi,
> I have a problem with emerge for a long time.
> Sometimes I need to (re-)emerge many packages like in an
> emerge --emptytree @world
> 
> Because I use several overlays, there are problems with a lot of
> packages.
> Unfortunately, emerge shows me just the first problem (like a missing USE-flags)
> and then terminates.
> Is there any means to let emerge go and report several (all) problems which
> are independent of each other?

Nothing to my knowledge, no.

It's that word "independant" that would appear to be the problem. You
know that a doc USE flag for a webserver and for a python are completely
unrelated, but portage doesn't, so stops after the first failure.
Continuing would be pointless as so often flags are not independant,
they are very much interdependant. I would also wager that any output
from portage if it did continue would be vastly more confusing to users
than what it does now.

The reason you know so much more than portage about these dependencies
is that you have a brain and comprehend meaning. Portage has silicon and
no clue about meaning of anyything


-- 
Alan McKinnon
alan.mckinnon@gmail.com