[gentoo-user] Emerge order not deterministic !?

[gentoo-user] Emerge order not deterministic !?

[Walter Dnes]

[-- Attachment #1: Type: text/plain, Size: 330 bytes --]

  Ongoing installation.  I looked at 2 instances of
"emerge -pv x11-base/xorg-server" and the order was somewhat different.
Here are a couple of outputs, just a few seconds apart.  Is this a bug
or a feature?  See attachments.

-- 
Walter Dnes <waltdnes@waltdnes.org>
I don't run "desktop environments"; I run useful applications

[-- Attachment #2: z1.txt.gz --]
[-- Type: application/octet-stream, Size: 1972 bytes --]

[-- Attachment #3: z2.txt.gz --]
[-- Type: application/octet-stream, Size: 1970 bytes --]

Re: [gentoo-user] Emerge order not deterministic !?

[Alan McKinnon]

On 11/11/2015 21:35, Walter Dnes wrote:
>   Ongoing installation.  I looked at 2 instances of
> "emerge -pv x11-base/xorg-server" and the order was somewhat different.
> Here are a couple of outputs, just a few seconds apart.  Is this a bug
> or a feature?  See attachments.
> 


Emerge order is not deterministic, especially with parallel builds. The
reason is that it does not need to be according to the dep graph - if
two packages are at the same level and do not depend on each other, then
the order they are built in does not affect the final result.
Practically all parallel processing works this way.

What is deterministic, is that if you build the same set of packages
twice and even if portage does them in different order, the binaries
produced are functionally identical

-- 
Alan McKinnon
alan.mckinnon@gmail.com

[gentoo-user] Re: Emerge order not deterministic !?

[Jörg Schaible]

Alan McKinnon wrote:

> On 11/11/2015 21:35, Walter Dnes wrote:
>>   Ongoing installation.  I looked at 2 instances of
>> "emerge -pv x11-base/xorg-server" and the order was somewhat different.
>> Here are a couple of outputs, just a few seconds apart.  Is this a bug
>> or a feature?  See attachments.
>> 
> 
> 
> Emerge order is not deterministic, especially with parallel builds. The
> reason is that it does not need to be according to the dep graph - if
> two packages are at the same level and do not depend on each other, then
> the order they are built in does not affect the final result.
> Practically all parallel processing works this way.
> 
> What is deterministic, is that if you build the same set of packages
> twice and even if portage does them in different order, the binaries
> produced are functionally identical

Hmmm. And how can you then ever use

  emerge --resume --skip-fist

if not even the first build is deterministic? I skip the first package 
anyway only if the problematic package is the first one to build after 
resume, but if I cannot even rely on that?

Cheers,
Jörg

Re: [gentoo-user] Re: Emerge order not deterministic !?

[Alan McKinnon]

On 12/11/2015 10:29, Jörg Schaible wrote:
> Alan McKinnon wrote:
> 
>> On 11/11/2015 21:35, Walter Dnes wrote:
>>>   Ongoing installation.  I looked at 2 instances of
>>> "emerge -pv x11-base/xorg-server" and the order was somewhat different.
>>> Here are a couple of outputs, just a few seconds apart.  Is this a bug
>>> or a feature?  See attachments.
>>>
>>
>>
>> Emerge order is not deterministic, especially with parallel builds. The
>> reason is that it does not need to be according to the dep graph - if
>> two packages are at the same level and do not depend on each other, then
>> the order they are built in does not affect the final result.
>> Practically all parallel processing works this way.
>>
>> What is deterministic, is that if you build the same set of packages
>> twice and even if portage does them in different order, the binaries
>> produced are functionally identical
> 
> Hmmm. And how can you then ever use
> 
>   emerge --resume --skip-fist
> 
> if not even the first build is deterministic? I skip the first package 
> anyway only if the problematic package is the first one to build after 
> resume, but if I cannot even rely on that?


Because it re-uses the previous build order, not re-generate a new one.


-- 
Alan McKinnon
alan.mckinnon@gmail.com

[gentoo-user] Re: Re: Emerge order not deterministic !?

[Jörg Schaible]

Alan McKinnon wrote:

> On 12/11/2015 10:29, Jörg Schaible wrote:
>> Alan McKinnon wrote:
>> 
>>> On 11/11/2015 21:35, Walter Dnes wrote:
>>>>   Ongoing installation.  I looked at 2 instances of
>>>> "emerge -pv x11-base/xorg-server" and the order was somewhat different.
>>>> Here are a couple of outputs, just a few seconds apart.  Is this a bug
>>>> or a feature?  See attachments.
>>>>
>>>
>>>
>>> Emerge order is not deterministic, especially with parallel builds. The
>>> reason is that it does not need to be according to the dep graph - if
>>> two packages are at the same level and do not depend on each other, then
>>> the order they are built in does not affect the final result.
>>> Practically all parallel processing works this way.
>>>
>>> What is deterministic, is that if you build the same set of packages
>>> twice and even if portage does them in different order, the binaries
>>> produced are functionally identical
>> 
>> Hmmm. And how can you then ever use
>> 
>>   emerge --resume --skip-fist
>> 
>> if not even the first build is deterministic? I skip the first package
>> anyway only if the problematic package is the first one to build after
>> resume, but if I cannot even rely on that?
> 
> 
> Because it re-uses the previous build order, not re-generate a new one.

That's simply not true. Emerge resume calculates the order again and for me 
it starts often with a different package.

Cheers,
Jörg

Re: [gentoo-user] Re: Re: Emerge order not deterministic !?

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 749 bytes --]

On Thu, 12 Nov 2015 09:48:48 +0100, Jörg Schaible wrote:

>> > Hmmm. And how can you then ever use
> >> 
> >>   emerge --resume --skip-fist
> >> 
> >> if not even the first build is deterministic? I skip the first
> >> package anyway only if the problematic package is the first one to
> >> build after resume, but if I cannot even rely on that?  
> > 
> > 
> > Because it re-uses the previous build order, not re-generate a new
> > one.  
> 
> That's simply not true. Emerge resume calculates the order again and
> for me it starts often with a different package.

Then use emerge --keep-going and portage will take care of skipping
failing merges for you.


-- 
Neil Bothwick

Every morning is the dawn of a new error...

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

[gentoo-user] Re: Re: Re: Emerge order not deterministic !?

[Jörg Schaible]

Neil Bothwick wrote:

> On Thu, 12 Nov 2015 09:48:48 +0100, Jörg Schaible wrote:
> 
>>> > Hmmm. And how can you then ever use
>> >> 
>> >>   emerge --resume --skip-fist
>> >> 
>> >> if not even the first build is deterministic? I skip the first
>> >> package anyway only if the problematic package is the first one to
>> >> build after resume, but if I cannot even rely on that?
>> > 
>> > 
>> > Because it re-uses the previous build order, not re-generate a new
>> > one.
>> 
>> That's simply not true. Emerge resume calculates the order again and
>> for me it starts often with a different package.
> 
> Then use emerge --keep-going and portage will take care of skipping
> failing merges for you.

Ah, no, that's not an option. It breaks for a reason. Sometimes I can ignore 
that and look for it later and in this case I skip it, but normally I fix 
the problem first. However, you have to take care, which package you're 
actually skipping. Especially if the build order is different with resume.

Cheers,
Jörg

Re: [gentoo-user] Re: Re: Re: Emerge order not deterministic !?

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 986 bytes --]

On Thu, 12 Nov 2015 10:35:14 +0100, Jörg Schaible wrote:

> > Then use emerge --keep-going and portage will take care of skipping
> > failing merges for you.  
> 
> Ah, no, that's not an option. It breaks for a reason. Sometimes I can
> ignore that and look for it later and in this case I skip it, but
> normally I fix the problem first. However, you have to take care, which
> package you're actually skipping. Especially if the build order is
> different with resume.

--keep-going will emerge all unaffected packages, meaning you are then
working with a much smaller list when you try to fix the problem. At
least, that's the approach that normally works for me.

--keep-going is intelligent enough to skip any packages that depend on
the failed package. That means you often end up with a package list that
is a single branch dependency tree, so the order is unlikely to change.


-- 
Neil Bothwick

Rainbows are just to look at, not to really understand.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

[gentoo-user] Re: Emerge order not deterministic !?

[Hans]

On 12/11/15 19:51, Neil Bothwick wrote:
> On Thu, 12 Nov 2015 10:35:14 +0100, Jörg Schaible wrote:
>
>>> Then use emerge --keep-going and portage will take care of skipping
>>> failing merges for you.
>>
>> Ah, no, that's not an option. It breaks for a reason. Sometimes I can
>> ignore that and look for it later and in this case I skip it, but
>> normally I fix the problem first. However, you have to take care, which
>> package you're actually skipping. Especially if the build order is
>> different with resume.
>
> --keep-going will emerge all unaffected packages, meaning you are then
> working with a much smaller list when you try to fix the problem. At
> least, that's the approach that normally works for me.
>
> --keep-going is intelligent enough to skip any packages that depend on
> the failed package. That means you often end up with a package list that
> is a single branch dependency tree, so the order is unlikely to change.
>
>
I use the following commands to upgrade my Gentoo boxes:
emerge --sync
emerge --update --deep --with-bdeps=y --newuse --backtrack=300 
--changed-deps=y --keep-going=y @world -va

When necessary adding, deleting or changing use flags, keywords, masks.

Followed by:
emerge --depclean
revdep-rebuild

No more problems since using this sequence unless there is a bug in a 
ebuild, like the one last one in busybox ebuild.

Re: [gentoo-user] Re: Re: Emerge order not deterministic !?

[Alan McKinnon]

On 12/11/2015 10:48, Jörg Schaible wrote:
> Alan McKinnon wrote:
> 
>> On 12/11/2015 10:29, Jörg Schaible wrote:
>>> Alan McKinnon wrote:
>>>
>>>> On 11/11/2015 21:35, Walter Dnes wrote:
>>>>>   Ongoing installation.  I looked at 2 instances of
>>>>> "emerge -pv x11-base/xorg-server" and the order was somewhat different.
>>>>> Here are a couple of outputs, just a few seconds apart.  Is this a bug
>>>>> or a feature?  See attachments.
>>>>>
>>>>
>>>>
>>>> Emerge order is not deterministic, especially with parallel builds. The
>>>> reason is that it does not need to be according to the dep graph - if
>>>> two packages are at the same level and do not depend on each other, then
>>>> the order they are built in does not affect the final result.
>>>> Practically all parallel processing works this way.
>>>>
>>>> What is deterministic, is that if you build the same set of packages
>>>> twice and even if portage does them in different order, the binaries
>>>> produced are functionally identical
>>>
>>> Hmmm. And how can you then ever use
>>>
>>>   emerge --resume --skip-fist
>>>
>>> if not even the first build is deterministic? I skip the first package
>>> anyway only if the problematic package is the first one to build after
>>> resume, but if I cannot even rely on that?
>>
>>
>> Because it re-uses the previous build order, not re-generate a new one.
> 
> That's simply not true. Emerge resume calculates the order again and for me 
> it starts often with a different package.

I've never noticed that. For me --skip-first has always skipped the
correct first package (the one that previously failed).

As long as a known build failure is not in the --resume list, I don't
care what the build order is because it is irrelevant. The only time it
becomes relevant is when an ebuild has a bug such as a missing dep. But
that's a bug in the ebuild and is fixed there.


-- 
Alan McKinnon
alan.mckinnon@gmail.com

[gentoo-user] Re: Re: Re: Emerge order not deterministic !?

[Jörg Schaible]

Alan McKinnon wrote:

> On 12/11/2015 10:48, Jörg Schaible wrote:
>> Alan McKinnon wrote:
>> 
>>> On 12/11/2015 10:29, Jörg Schaible wrote:
>>>> Alan McKinnon wrote:

[snip]

>>>> Hmmm. And how can you then ever use
>>>>
>>>>   emerge --resume --skip-fist
>>>>
>>>> if not even the first build is deterministic? I skip the first package
>>>> anyway only if the problematic package is the first one to build after
>>>> resume, but if I cannot even rely on that?
>>>
>>>
>>> Because it re-uses the previous build order, not re-generate a new one.
>> 
>> That's simply not true. Emerge resume calculates the order again and for
>> me it starts often with a different package.
> 
> I've never noticed that. For me --skip-first has always skipped the
> correct first package (the one that previously failed).

That's what I always did originally also, until my build suddenly broke at 
the same package again and I had to notice that it skipped a completely 
different.

> As long as a known build failure is not in the --resume list, I don't
> care what the build order is because it is irrelevant. The only time it
> becomes relevant is when an ebuild has a bug such as a missing dep. But
> that's a bug in the ebuild and is fixed there.

Well, normally I don't care about the sequence either, except when skipping 
the first ;-)

Cheers,
Jörg