Re: [gentoo-user] sys-apps/portage and binary packages

Re: [gentoo-user] sys-apps/portage and binary packages

[J. Aho]

On 13/07/2024 14.50, Peter Humphrey wrote:
> Where I live, updates to portage itself usually take longer to appear as a
> binary package than as source, so I can't 'getbinpkg'. Therefore I've set:
The main issue is that you aren't syncing portage towards the bin 
server, which makes things out of sync and those you will be building a 
lot of the packages instead of fetching the binary files when they are 
built. One way to come around this issue is that when you sync your 
portage, wait a day before you check what files to update, this way the 
binary packages should have been built, but sure it's a pita.

The proper solution had been that you could sync portage against the 
binhost and that only if the binhost do have two portage that is always 
in par with the packages that has been built. I have my own experience 
of this issue from the time when I had multiple machines at home running 
Gentoo and I had a build environment that built packages for me. I had 
to setup two copies of portage on the build machine, one was the one it 
synced and built against, and then there was the portage version that 
was provided to the clients in the LAN (sadly this has a complication 
for the clients using git instead of rsync), this was only updated after 
a successful build. All the client synced portage against the binhost. 
If they didn't, then I had the issue as you describe and I had 
discussion about this on the gentoo irc channels, but as back then few 
people used binhosts so they didn't understand the issue and those 
portage don't support to do what you want to do, to filter out new 
ebuilds that don't have a binary package at the binhost, just wait 
another 20 years and then maybe.


-- 

//Aho

Re: [gentoo-user] sys-apps/portage and binary packages

[Peter Humphrey]

On Monday, 15 July 2024 10:54:37 BST J. Aho wrote:

> The main issue is that you aren't syncing portage towards the bin
> server, which makes things out of sync and those you will be building a
> lot of the packages instead of fetching the binary files when they are
> built. One way to come around this issue is that when you sync your
> portage, wait a day before you check what files to update, this way the
> binary packages should have been built, but sure it's a pita.

You're right. I had set the sync-uri to mirror.bytemark.co.uk, which I got 
from the handbook. but I still had GENTOO_MIRRORS="http://
www.mirrorservice.org/sites/distfiles.gentoo.org/" from an older installation.

I've now set the mirror to bytemark; let's see if that helps.

Thanks.

-- 
Regards,
Peter.

Re: [gentoo-user] sys-apps/portage and binary packages

[Michael]

[-- Attachment #1: Type: text/plain, Size: 1237 bytes --]

On Monday, 15 July 2024 13:36:19 BST Peter Humphrey wrote:
> On Monday, 15 July 2024 10:54:37 BST J. Aho wrote:
> > The main issue is that you aren't syncing portage towards the bin
> > server, which makes things out of sync and those you will be building a
> > lot of the packages instead of fetching the binary files when they are
> > built. One way to come around this issue is that when you sync your
> > portage, wait a day before you check what files to update, this way the
> > binary packages should have been built, but sure it's a pita.
> 
> You're right. I had set the sync-uri to mirror.bytemark.co.uk, which I got
> from the handbook. but I still had GENTOO_MIRRORS="http://
> www.mirrorservice.org/sites/distfiles.gentoo.org/" from an older
> installation.
> 
> I've now set the mirror to bytemark; let's see if that helps.
> 
> Thanks.

I recall having some trouble with bytemark in the past.  A package build time 
dependency portage required to complete an update was not available on 
bytemark and the update failed.  I tried a day later and the same problem came 
up.  A cursory look revealed this mirror to be out of date on a number of 
packages.  Eventually I move to different mirrors and the problem disappeared.

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-user] sys-apps/portage and binary packages

[Peter Humphrey]

On Tuesday, 16 July 2024 09:50:00 BST Michael wrote:

> I recall having some trouble with bytemark in the past.

Now you mention it, so do I, but I've been using it for quite a while now with 
no problems.

I suppose servers come and go...

-- 
Regards,
Peter.

Re: [gentoo-user] sys-apps/portage and binary packages

[Eli Schwartz]

[-- Attachment #1.1: Type: text/plain, Size: 2304 bytes --]

On 7/15/24 5:54 AM, J. Aho wrote:
> The main issue is that you aren't syncing portage towards the bin
> server, which makes things out of sync and those you will be building a
> lot of the packages instead of fetching the binary files when they are
> built.


As noted earlier, it's more complicated than that. Portage can handle
the case you describe just fine, in the sense that you will be building
a lot of the packages instead of fetching the binary files.

In which case, you don't need to set FEATURES="-getbinpkg" on a per
package basis, since they will simply be built from source anyway.

The problem comes when portage says there *is* a binary package
available, because the index of packages says there is one, then it
tries to download the binary file itself and receives a 404 error. That
causes portage to crash since it doesn't expect the 404.


> If they didn't, then I had the issue as you describe and I had
> discussion about this on the gentoo irc channels, but as back then few
> people used binhosts so they didn't understand the issue and those
> portage don't support to do what you want to do, to filter out new
> ebuilds that don't have a binary package at the binhost, just wait
> another 20 years and then maybe.


I understand where you are coming from :) because I refused to become a
Gentoo user until there were official binhosts.

There is as of 2024 a tracking issue for the bug you described, and with
that public record of what to improve, some progress has been made to
get it to work. I expect it to take significantly less than 20 years to
deploy: the current version of portage stabilized for amd64 causes a
binhost to expose the git commit for gentoo.git that it was built for,
and the plan is that you should be able to have `emerge --sync` just
sync to that revision as announced by the binhost.

I will admit that it took us 5 months from time of reporting until
portage made the progress it has so far, but I'm inclined to blame that
on FOSS software being FOSS and people having other things to do with
their time. Hopefully we'll only need to wait another couple of months
for it to be full solved. :)

Please do consider watching https://bugs.gentoo.org/924772 for further
progress.


-- 
Eli Schwartz


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 236 bytes --]

[gentoo-user] sys-apps/portage and binary packages

[Peter Humphrey]

Hello list,

Where I live, updates to portage itself usually take longer to appear as a 
binary package than as source, so I can't 'getbinpkg'. Therefore I've set:

# cat /etc/portage/env/nobinpkg.conf
FEATURES="${FEATURES} -getbinpkg"

# cat /etc/portage/package.env
sys-apps/portage nobinpkg.conf

But still portage wants to fetch the binary.

What am I doing wrong?

-- 
Regards,
Peter.

Re: [gentoo-user] sys-apps/portage and binary packages

[Arve Barsnes]

On Sat, 13 Jul 2024 at 14:42, Peter Humphrey <peter@prh.myzen.co.uk> wrote:
>
> Hello list,
>
> Where I live, updates to portage itself usually take longer to appear as a
> binary package than as source, so I can't 'getbinpkg'. Therefore I've set:
>
> # cat /etc/portage/env/nobinpkg.conf
> FEATURES="${FEATURES} -getbinpkg"
>
> # cat /etc/portage/package.env
> sys-apps/portage nobinpkg.conf
>
> But still portage wants to fetch the binary.
>
> What am I doing wrong?

I don't know what you're doing wrong, but FEATURES is an additive
variable, so adding the ${FEATURES} in there is not necessary.

An alternative might be adding it to emerge default opts in make.conf:

EMERGE_DEFAULT_OPTS="${EMERGE_DEFAULT_OPTS} --usepkg-exclude 'sys-apps/portage'"

Regards,
Arve

Re: [gentoo-user] sys-apps/portage and binary packages

[Peter Humphrey]

On Saturday, 13 July 2024 14:18:09 BST Arve Barsnes wrote:
> On Sat, 13 Jul 2024 at 14:42, Peter Humphrey <peter@prh.myzen.co.uk> 
wrote:
> > Hello list,
> > 
> > Where I live, updates to portage itself usually take longer to appear as a
> > binary package than as source, so I can't 'getbinpkg'. Therefore I've set:
> > 
> > # cat /etc/portage/env/nobinpkg.conf
> > FEATURES="${FEATURES} -getbinpkg"
> > 
> > # cat /etc/portage/package.env
> > sys-apps/portage nobinpkg.conf
> > 
> > But still portage wants to fetch the binary.
> > 
> > What am I doing wrong?
> 
> I don't know what you're doing wrong, but FEATURES is an additive
> variable, so adding the ${FEATURES} in there is not necessary.

I've tried it without that parenthesis but with no difference.

> An alternative might be adding it to emerge default opts in make.conf:
> 
> EMERGE_DEFAULT_OPTS="${EMERGE_DEFAULT_OPTS} --usepkg-exclude
> 'sys-apps/portage'"

Interesting. I hadn't seen that construction before; I'll try it.

Thanks Arve.

-- 
Regards,
Peter.

Re: [gentoo-user] sys-apps/portage and binary packages

[Peter Humphrey]

On Saturday, 13 July 2024 15:49:00 BST Peter Humphrey wrote:
> On Saturday, 13 July 2024 14:18:09 BST Arve Barsnes wrote:
--->8
> > I don't know what you're doing wrong, but FEATURES is an additive
> > variable, so adding the ${FEATURES} in there is not necessary.
> 
> I've tried it without that parenthesis but with no difference.
> 
> > An alternative might be adding it to emerge default opts in make.conf:
> > 
> > EMERGE_DEFAULT_OPTS="${EMERGE_DEFAULT_OPTS} --usepkg-exclude
> > 'sys-apps/portage'"
> 
> Interesting. I hadn't seen that construction before; I'll try it.

It works, but what's wrong with the way I tried it?

-- 
Regards,
Peter.

Re: [gentoo-user] sys-apps/portage and binary packages

[netfab]

Le 14/07/24 à 02:18, Peter Humphrey a tapoté :
> It works, but what's wrong with the way I tried it?

Here is the explanation :

	https://bugs.gentoo.org/463964

> Currently, FEATURES=getbinpkg only works as a global setting.

And from :

	https://forums.gentoo.org/viewtopic-t-1166963.html

> FEATURES are not fully supported in package.env as they can cause
> circular effects (e.g. getbinpkg may cause a different version to get
> selected, which can result in the package.env entry no longer being
> applicable).

Re: [gentoo-user] sys-apps/portage and binary packages

[Eli Schwartz]

[-- Attachment #1.1: Type: text/plain, Size: 1100 bytes --]

On 7/13/24 8:42 AM, Peter Humphrey wrote:
> Hello list,
> 
> Where I live, updates to portage itself usually take longer to appear as a 
> binary package than as source, so I can't 'getbinpkg'. Therefore I've set:
> 
> # cat /etc/portage/env/nobinpkg.conf
> FEATURES="${FEATURES} -getbinpkg"
> 
> # cat /etc/portage/package.env
> sys-apps/portage nobinpkg.conf
> 
> But still portage wants to fetch the binary.
> 
> What am I doing wrong?


As a matter of curiosity, why do you need to do any such thing at all?

If there is no binary package available for portage yet, then portage
will automatically build it from source instead, which is exactly what
setting -getbinpkg for it would do. So why bother?

The only thing that setting -getbinpkg could do is prevent you from
using a binary on the off chance that it happens to appear faster for you.


Note that independent of whether it's useful to exclude this one
package, the functionality doesn't work. You cannot set per-package
getbinpkg, this is tracked as https://bugs.gentoo.org/463964


-- 
Eli Schwartz


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 236 bytes --]

Re: [gentoo-user] sys-apps/portage and binary packages

[Dale]

[-- Attachment #1: Type: text/plain, Size: 1859 bytes --]

Eli Schwartz wrote:
> On 7/13/24 8:42 AM, Peter Humphrey wrote:
>> Hello list,
>>
>> Where I live, updates to portage itself usually take longer to appear as a 
>> binary package than as source, so I can't 'getbinpkg'. Therefore I've set:
>>
>> # cat /etc/portage/env/nobinpkg.conf
>> FEATURES="${FEATURES} -getbinpkg"
>>
>> # cat /etc/portage/package.env
>> sys-apps/portage nobinpkg.conf
>>
>> But still portage wants to fetch the binary.
>>
>> What am I doing wrong?
> As a matter of curiosity, why do you need to do any such thing at all?
>
> If there is no binary package available for portage yet, then portage
> will automatically build it from source instead, which is exactly what
> setting -getbinpkg for it would do. So why bother?
>
> The only thing that setting -getbinpkg could do is prevent you from
> using a binary on the off chance that it happens to appear faster for you.
>
>
> Note that independent of whether it's useful to exclude this one
> package, the functionality doesn't work. You cannot set per-package
> getbinpkg, this is tracked as https://bugs.gentoo.org/463964
>
>
> -- Eli Schwartz

That's actually why he wants to do that.  It seems for some reason,
portage shows up as a source long before the binary packages do.  So, he
wants to make a exception for portage so he can just build it himself
locally.  Given it likely doesn't take long to build from source anyway,
why not.  I'd suspect tho that most packages show up as source before
binary packages do tho.  After all, one has to have the source to build
the binary ones.  I guess maybe portage lags behind further than others
or something, at least for the OP anyway. 

I never did the binary thing tho.  When compiles start taking to long,
time to build a faster rig.  :-D  Some devices tho, like Raspberry Pis,
don't have that option. 

Dale

:-)  :-) 

[-- Attachment #2: Type: text/html, Size: 2749 bytes --]

Re: [gentoo-user] sys-apps/portage and binary packages

[Peter Humphrey]

On Sunday, 14 July 2024 07:05:04 BST Eli Schwartz wrote:

> As a matter of curiosity, why do you need to do any such thing at all?
> 
> If there is no binary package available for portage yet, then portage
> will automatically build it from source instead, which is exactly what
> setting -getbinpkg for it would do. So why bother?

It doesn't do that here. It tries to fetch the binary and bombs out when it 
can't be found. Then I have to edit make.conf to update Gentoo, then put it 
back as it was for the rest of the system.

> The only thing that setting -getbinpkg could do is prevent you from
> using a binary on the off chance that it happens to appear faster for you.

That might be so if portage behaved as you said above.

> Note that independent of whether it's useful to exclude this one
> package, the functionality doesn't work. You cannot set per-package
> getbinpkg, this is tracked as https://bugs.gentoo.org/463964

Yes, I saw that from netfab's post.

-- 
Regards,
Peter.

Re: [gentoo-user] sys-apps/portage and binary packages

[Wols Lists]

On 14/07/2024 13:04, Peter Humphrey wrote:
> It doesn't do that here. It tries to fetch the binary and bombs out when it
> can't be found. Then I have to edit make.conf to update Gentoo, then put it
> back as it was for the rest of the system.

Do you have PORTAGE_DEFAULT_OPTIONS or whatever it's called set? That's 
caused me similar grief - it messed up my attempts to update my profile, 
it regularly messed up my virtualbox updates, etc etc.

And if you have to edit various environment variables in make.conf, you 
know you can override them on the command line?

PORTAGE_DEFAULT_OPTIONS="" emerge virtualbox-modules

as I had to do ...

Cheers,
Wol

Re: [gentoo-user] sys-apps/portage and binary packages

[Peter Humphrey]

On Sunday, 14 July 2024 13:22:14 BST Wols Lists wrote:
> On 14/07/2024 13:04, Peter Humphrey wrote:
> > It doesn't do that here. It tries to fetch the binary and bombs out when
> > it can't be found. Then I have to edit make.conf to update Gentoo, then
> > put it back as it was for the rest of the system.
> 
> Do you have PORTAGE_DEFAULT_OPTIONS or whatever it's called set? That's
> caused me similar grief - it messed up my attempts to update my profile,
> it regularly messed up my virtualbox updates, etc etc.

Yes, I have EMERGE_DEFAULT_OPTS="--jobs=8 --load-average=8 --autounmask=n --
keep-going  --nospinner"

Nothing contentious there, I'd have thought.

Then I added this after Arve's advice, and after my problem arose:

EMERGE_DEFAULT_OPTS="${EMERGE_DEFAULT_OPTS} --usepkg-exclude 'sys-apps/
portage'"

> And if you have to edit various environment variables in make.conf, you
> know you can override them on the command line?
> 
> PORTAGE_DEFAULT_OPTIONS="" emerge virtualbox-modules
> 
> as I had to do ...

Useful reminder; thanks.

-- 
Regards,
Peter.

Re: [gentoo-user] sys-apps/portage and binary packages

[Wol]

On 14/07/2024 14:15, Peter Humphrey wrote:
> Yes, I have EMERGE_DEFAULT_OPTS="--jobs=8 --load-average=8 --autounmask=n --
> keep-going  --nospinner"
> 
> Nothing contentious there, I'd have thought.

I didn't think I had anything contentious - --once-only and that was 
about it. I think that was actually Dale's suggestion - anything I 
emerge will get cleaned away by my next depclean unless I actively force 
it into the world file.

But a load of stuff just wouldn't --update or emerge as a result ...

Cheers,
Wol

Re: [gentoo-user] sys-apps/portage and binary packages

[Dale]

Wol wrote:
> On 14/07/2024 14:15, Peter Humphrey wrote:
>> Yes, I have EMERGE_DEFAULT_OPTS="--jobs=8 --load-average=8
>> --autounmask=n --
>> keep-going  --nospinner"
>>
>> Nothing contentious there, I'd have thought.
>
> I didn't think I had anything contentious - --once-only and that was
> about it. I think that was actually Dale's suggestion - anything I
> emerge will get cleaned away by my next depclean unless I actively
> force it into the world file.
>
> But a load of stuff just wouldn't --update or emerge as a result ...
>
> Cheers,
> Wol
>
>


Is this about the -1 or --oneshot option? 

Dale

:-)  :-) 

Re: [gentoo-user] sys-apps/portage and binary packages

[Wols Lists]

On 15/07/2024 07:22, Dale wrote:
> Is this about the -1 or --oneshot option?

Yes.

Once your system is stable it's a damn good idea :-)

Cheers,
Wol

Re: [gentoo-user] sys-apps/portage and binary packages

[Dale]

Wols Lists wrote:
> On 15/07/2024 07:22, Dale wrote:
>> Is this about the -1 or --oneshot option?
>
> Yes.
>
> Once your system is stable it's a damn good idea :-)
>
> Cheers,
> Wol
>
>


On my recent new rig build, once I installed everything from the world
file from my old system, I added that option to make.conf.  Why that tip
isn't in the install guide surprises me.  A lot of people while trying
to get emerge past a block or something ends up with entries in the
world file that shouldn't be there, especially libs or packages with
versions.  It shouldn't be the default by any means but toward the end
of the install guide, I think it deserves a mention and when to add it. 

Most of my upgrades go fairly smoothly.  One reason for that, nothing is
in the world file unless it really needs to be there.  I can't add
something by mistake. 

Yep.  Good idea.  Sad a lot of people don't know to add that at the
right time. 

Dale

:-)  :-) 

Re: [gentoo-user] sys-apps/portage and binary packages

[Eli Schwartz]

[-- Attachment #1.1: Type: text/plain, Size: 1955 bytes --]

On 7/14/24 8:04 AM, Peter Humphrey wrote:
> On Sunday, 14 July 2024 07:05:04 BST Eli Schwartz wrote:
> 
>> As a matter of curiosity, why do you need to do any such thing at all?
>>
>> If there is no binary package available for portage yet, then portage
>> will automatically build it from source instead, which is exactly what
>> setting -getbinpkg for it would do. So why bother?
> 
> It doesn't do that here. It tries to fetch the binary and bombs out when it 
> can't be found. Then I have to edit make.conf to update Gentoo, then put it 
> back as it was for the rest of the system.


This indicates that the Packages index on the binhost server doesn't
correspond to the actual packages which are available; it is
broadcasting availability of a new sys-apps/portage binpackage but that
binpackage is not actually present.

It has nothing to do with sys-apps/portage, since if there is a lack of
synchronization here then the specific details of packages does not
matter at all. It just happens, by coincidence, to trigger for you with
the sys-apps/portage package.

This is nominally speaking impossible to happen as it implies the
binhost is simply broken altogether, which of course never happens in
real life because we are all perfect, right?

But actual broken binhosts aside, I'm guessing your real issue is using
a DNS round robin load balancer or similar as your binhost url. You're
getting delivered the Packages index from one server, but actual
*.gpkg.tar requests are being serviced by a different load balancing
machine, and the two aren't precisely in sync.

There are some similar bugs being tracked:

https://bugs.gentoo.org/464906
https://bugs.gentoo.org/831657
https://bugs.gentoo.org/890491
https://bugs.gentoo.org/865845


The default gentoobinhost shipped in stage3 tarballs is
distfiles.gentoo.org, try changing it to a specific mirror and see if
that helps?

-- 
Eli Schwartz


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 236 bytes --]