[gentoo-user] Prebuilt packages

[gentoo-user] Prebuilt packages

[Peter Humphrey]

Greetings,

I was delighted to see Gentoo's venture into prebuilt packages, having two 
small machines here. However, experience has rubbed some of the shine off.

Any time I update the system, if there's no package with the right USE flags, 
or even no package at all, I've built from source in the traditional way. 
Later, on running 'eclean-pkg --deep --package-names', I get a load of 
invalid-package warnings. I just assumed that was harmless.

More serious, I'm sure, is the mess I'm getting into with 'emerge -k' and/or 
'emerge -K'. It became so bad that I tried reinstalling from scratch today. I 
couldn't make it work at all - at one stage I was told to emerge @preserved-
rebuild, whereupon it insisted on installing the nouveau drivers. 
Preposterous.

Can portage not operate a mixed prebuilt and home-grown system?

-- 
Regards,
Peter.

Re: [gentoo-user] Prebuilt packages

[Eli Schwartz]

[-- Attachment #1.1: Type: text/plain, Size: 2351 bytes --]

On 2/11/25 10:22 AM, Peter Humphrey wrote:
> Greetings,
> 
> I was delighted to see Gentoo's venture into prebuilt packages, having two 
> small machines here. However, experience has rubbed some of the shine off.
> 
> Any time I update the system, if there's no package with the right USE flags, 
> or even no package at all, I've built from source in the traditional way. 
> Later, on running 'eclean-pkg --deep --package-names', I get a load of 
> invalid-package warnings. I just assumed that was harmless.
> 
> More serious, I'm sure, is the mess I'm getting into with 'emerge -k' and/or 
> 'emerge -K'. It became so bad that I tried reinstalling from scratch today. I 
> couldn't make it work at all - at one stage I was told to emerge @preserved-
> rebuild, whereupon it insisted on installing the nouveau drivers. 
> Preposterous.
> 
> Can portage not operate a mixed prebuilt and home-grown system?



I don't understand what you're trying to say at all.

emerge --getbinpkg is *already* a mixed prebuilt and home-grown system.
And emerge --getbinpkgonly is a modification to getbinpkg that says you
don't want a mixed system, you want exclusively prebuilt. Using the
*only variant is a personal choice you aren't obligated to make.

You haven't actually provided details about why you think there's a
problem though. You have said:

- if there are no packages available for your configuration, you have
  "built from source in the traditional way", which emerge already does
  *automatically* unless you explicitly tell it not to. Should I be
  concluding from here that you first tried the *only variant, then when
  that failed you disabled fetching binary packages altogether in a fit
  of frustration?

- you installed Gentoo from scratch because "it" became so bad. What is
  "it"? I am finding it hard to imagine a situation where there could be
  an "it", let alone deduce the details of what "it" constitutes.

- who told you to emerge @preserved-rebuild, what was it in relation to,
  what was the dependency graph that claimed it was installing nouveau?

- "I get a load of invalid-package warnings. I just assumed that was
  harmless". Failure to delete a useless unneeded file is most likely
  harmless, yes. More details would require you saying more here.


-- 
Eli Schwartz

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 236 bytes --]

Re: [gentoo-user] Prebuilt packages

[Peter Humphrey]

On Tuesday 11 February 2025 16:02:04 Greenwich Mean Time Eli Schwartz wrote:

> emerge --getbinpkg is *already* a mixed prebuilt and home-grown system.
> And emerge --getbinpkgonly is a modification to getbinpkg that says you
> don't want a mixed system, you want exclusively prebuilt. Using the
> *only variant is a personal choice you aren't obligated to make.

...and I don't do that. It wouldn't work anyway. Firefox, for instance, 
doesn't have a prebuilt package that would install here:

!!! The following binary packages have been ignored due to non matching USE:
[...]
    =www-client/firefox-128.7.0 clang gnome-shell -hwaccel -l10n_en-GB
    =www-client/firefox-128.7.0 clang gnome-shell -hwaccel -l10n_en-GB pgo -
pulseaudio -wayland
    =www-client/firefox-128.7.0 clang gnome-shell -hwaccel -l10n_en-GB pgo -
wayland
    =www-client/firefox-128.7.0 clang gnome-shell -hwaccel -l10n_en-GB pgo -
pulseaudio
    =www-client/firefox-128.7.0 clang gnome-shell -hwaccel -l10n_en-GB pgo
    =www-client/firefox-128.7.0 clang -hwaccel -l10n_en-GB

If I'm reading that right, anyone who has either hardware acceleration or en-
GB localisation set has to compile it himself. I can switch off the hwaccel but 
I'm not willing to use the US variant of English.

> You haven't actually provided details about why you think there's a
> problem though. You have said:
> 
> - if there are no packages available for your configuration, you have
>   "built from source in the traditional way", which emerge already does
>   *automatically* unless you explicitly tell it not to. Should I be
>   concluding from here that you first tried the *only variant, then when
>   that failed you disabled fetching binary packages altogether in a fit
>   of frustration?

The problem seems to be that portage knows when a requested prebuilt package 
exists at gentoo.org, but over here the local mirrors take time to catch up. 
Perhaps I'm just causing trouble by trying to work around that latency myself. 
I've just tried using distfiles.gentoo.org instead, and it looks promising.

--->8

-- 
Regards,
Peter.

Re: [gentoo-user] Prebuilt packages

[Eli Schwartz]

[-- Attachment #1.1: Type: text/plain, Size: 2036 bytes --]

On 2/11/25 12:06 PM, Peter Humphrey wrote:
> If I'm reading that right, anyone who has either hardware acceleration or en-
> GB localisation set has to compile it himself. I can switch off the hwaccel but 
> I'm not willing to use the US variant of English.


Ok yeah that is a nasty problem unfortunately. Maybe this should use the
same split design that libreoffice does, where libreoffice depends on
libreoffice-l10n and the latter can be freely rebuilt and only installs
some small static files.

Did you try submitting a bug report?


>> You haven't actually provided details about why you think there's a
>> problem though. You have said:
>>
>> - if there are no packages available for your configuration, you have
>>   "built from source in the traditional way", which emerge already does
>>   *automatically* unless you explicitly tell it not to. Should I be
>>   concluding from here that you first tried the *only variant, then when
>>   that failed you disabled fetching binary packages altogether in a fit
>>   of frustration?
> 
> The problem seems to be that portage knows when a requested prebuilt package 
> exists at gentoo.org, but over here the local mirrors take time to catch up. 
> Perhaps I'm just causing trouble by trying to work around that latency myself. 
> I've just tried using distfiles.gentoo.org instead, and it looks promising.


The index of packages comes from the same website as the package files
themselves, so in theory they should always be consistent with each
other. https://bugs.gentoo.org/936288 may be relevant.

If you aren't dealing with a broken mirror (bytemark was outright
dropping some directories via rsync!) then it's just a matter of time.
The binhost starts up at around 5am each day, runs for a bit, and
publishes all packages for the day. I usually sync around noon. More
details at

https://wiki.gentoo.org/wiki/Gentoo_Binary_Host_Quickstart#I_used_to_have_a_binary_package_but_not_anymore.2C_what_happened.3F



-- 
Eli Schwartz

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 236 bytes --]

Re: [gentoo-user] Prebuilt packages

[Mickaël Bucas]

Le mar. 11 févr. 2025 à 18:07, Peter Humphrey <peter@prh.myzen.co.uk> a écrit :
>
> On Tuesday 11 February 2025 16:02:04 Greenwich Mean Time Eli Schwartz wrote:
>
> > emerge --getbinpkg is *already* a mixed prebuilt and home-grown system.
> > And emerge --getbinpkgonly is a modification to getbinpkg that says you
> > don't want a mixed system, you want exclusively prebuilt. Using the
> > *only variant is a personal choice you aren't obligated to make.
>
> ...and I don't do that. It wouldn't work anyway. Firefox, for instance,
> doesn't have a prebuilt package that would install here:
>
> !!! The following binary packages have been ignored due to non matching USE:
> [...]
>     =www-client/firefox-128.7.0 clang gnome-shell -hwaccel -l10n_en-GB
>     =www-client/firefox-128.7.0 clang gnome-shell -hwaccel -l10n_en-GB pgo -
> pulseaudio -wayland
>     =www-client/firefox-128.7.0 clang gnome-shell -hwaccel -l10n_en-GB pgo -
> wayland
>     =www-client/firefox-128.7.0 clang gnome-shell -hwaccel -l10n_en-GB pgo -
> pulseaudio
>     =www-client/firefox-128.7.0 clang gnome-shell -hwaccel -l10n_en-GB pgo
>     =www-client/firefox-128.7.0 clang -hwaccel -l10n_en-GB
>
> If I'm reading that right, anyone who has either hardware acceleration or en-
> GB localisation set has to compile it himself. I can switch off the hwaccel but
> I'm not willing to use the US variant of English.
>

I think Firefox is currently a special case, because there's also
www-client/firefox-bin which obviously is precompiled.
So prebuilt packages probably aren't built for www-client/firefox

There're a few like this, compiled by Gentoo :
app-office/libreoffice-bin
dev-lang/rust-bin
mail-client/thunderbird-bin

I found about 200 packages ending with "-bin", I didn't check further
if non-bin packages are available and then if prebuilt packages for
them exist, but it seems logical to me to not compile twice.
However if you want the binary version, you have to switch manually,
because emerge sees them as different packages.

I recently converted a fully locally *slowly* built machine to a mixed
prebuilt/local build
Of the 1857 packages on this machine, 1056 are now using prebuilt
packages and the others are local builds
USE flags and ~amd64 prevent use of prebuilt packages
But I adjusted USE flags until dev-qt/qtwebengine:6 found a prebuilt package !
And I switched manually to www-client/firefox-bin and dev-lang/rust-bin

> --
> Regards,
> Peter.

Best regards
Mickaël

Re: [gentoo-user] Prebuilt packages

[Mickaël Bucas]

Le mer. 12 févr. 2025 à 00:04, Mickaël Bucas <mbucas@gmail.com> a écrit :
>
> Le mar. 11 févr. 2025 à 18:07, Peter Humphrey <peter@prh.myzen.co.uk> a écrit :
> >
> > On Tuesday 11 February 2025 16:02:04 Greenwich Mean Time Eli Schwartz wrote:
> >
> > > emerge --getbinpkg is *already* a mixed prebuilt and home-grown system.
> > > And emerge --getbinpkgonly is a modification to getbinpkg that says you
> > > don't want a mixed system, you want exclusively prebuilt. Using the
> > > *only variant is a personal choice you aren't obligated to make.
> >
> > ...and I don't do that. It wouldn't work anyway. Firefox, for instance,
> > doesn't have a prebuilt package that would install here:
> >
> > !!! The following binary packages have been ignored due to non matching USE:
> > [...]
> >     =www-client/firefox-128.7.0 clang gnome-shell -hwaccel -l10n_en-GB
> >     =www-client/firefox-128.7.0 clang gnome-shell -hwaccel -l10n_en-GB pgo -
> > pulseaudio -wayland
> >     =www-client/firefox-128.7.0 clang gnome-shell -hwaccel -l10n_en-GB pgo -
> > wayland
> >     =www-client/firefox-128.7.0 clang gnome-shell -hwaccel -l10n_en-GB pgo -
> > pulseaudio
> >     =www-client/firefox-128.7.0 clang gnome-shell -hwaccel -l10n_en-GB pgo
> >     =www-client/firefox-128.7.0 clang -hwaccel -l10n_en-GB
> >
> > If I'm reading that right, anyone who has either hardware acceleration or en-
> > GB localisation set has to compile it himself. I can switch off the hwaccel but
> > I'm not willing to use the US variant of English.
> >
>
> I think Firefox is currently a special case, because there's also
> www-client/firefox-bin which obviously is precompiled.
> So prebuilt packages probably aren't built for www-client/firefox

I didn't read the output you provided : it clearly shows binary
packages for www-client/firefox
The main difference with the -bin package is on keywords :
www-client/firefox-128.7.0 amd64
www-client/firefox-135.0 ~amd64 <= unstable
www-client/firefox-bin-128.7.0 amd64
www-client/firefox-bin-135.0 amd64 <= stable

> > --
> > Regards,
> > Peter.
>
> Best regards
> Mickaël

Re: [gentoo-user] Prebuilt packages

[Peter Humphrey]

On Tuesday 11 February 2025 17:26:29 Greenwich Mean Time Eli Schwartz wrote:
> On 2/11/25 12:06 PM, Peter Humphrey wrote:
> > If I'm reading that right, anyone who has either hardware acceleration or
> > en- GB localisation set has to compile it himself. I can switch off the
> > hwaccel but I'm not willing to use the US variant of English.
> 
> Ok yeah that is a nasty problem unfortunately. Maybe this should use the
> same split design that libreoffice does, where libreoffice depends on
> libreoffice-l10n and the latter can be freely rebuilt and only installs
> some small static files.
> 
> Did you try submitting a bug report?

Nope. I might, now that you've suggested a solution.

--->8

> The index of packages comes from the same website as the package files
> themselves, so in theory they should always be consistent with each
> other. https://bugs.gentoo.org/936288 may be relevant.

I don't think that's my problem; rather, the whole site is some time behind 
the master - of course.

> If you aren't dealing with a broken mirror (bytemark was outright
> dropping some directories via rsync!) then it's just a matter of time.
> The binhost starts up at around 5am each day, runs for a bit, and
> publishes all packages for the day. I usually sync around noon. More
> details at
> 
> https://wiki.gentoo.org/wiki/Gentoo_Binary_Host_Quickstart#I_used_to_have_a_
> binary_package_but_not_anymore.2C_what_happened.3F

I'm sure that works for you, but I'd have to wait until the afternoon to use 
the same approach.  ;-(

-- 
Regards,
Peter.

Re: [gentoo-user] Prebuilt packages

[Eli Schwartz]

[-- Attachment #1.1: Type: text/plain, Size: 1443 bytes --]

On 2/11/25 6:04 PM, Mickaël Bucas wrote:
> I think Firefox is currently a special case, because there's also
> www-client/firefox-bin which obviously is precompiled.
> So prebuilt packages probably aren't built for www-client/firefox
> 
> There're a few like this, compiled by Gentoo :
> app-office/libreoffice-bin
> dev-lang/rust-bin
> mail-client/thunderbird-bin
> 
> I found about 200 packages ending with "-bin", I didn't check further
> if non-bin packages are available and then if prebuilt packages for
> them exist, but it seems logical to me to not compile twice.
> However if you want the binary version, you have to switch manually,
> because emerge sees them as different packages.


As you noted downthread, both exist.

The ebuild names with "-bin" in them long predate the gentoo binhost
project. In the case of firefox, those are direct prebuilt binaries
compiled and hosted by Mozilla. In the case of rust, some of those are
the Rust project's prebuilt binaries (the same ones downloaded by rustup
into $HOME) and for architectures that the Rust project doesn't cover,
Arthur builds a standalone rust and uploads it.

libreoffice-bin is entirely gentoo-built for the original purpose of
being a stopgap measure instead of a binhost. It may eventually go away.
The others are arguably useful inasmuch as users may want to test the
official Mozilla etc production binaries.


-- 
Eli Schwartz

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 236 bytes --]

Re: [gentoo-user] Prebuilt packages

[Eli Schwartz]

[-- Attachment #1.1: Type: text/plain, Size: 1451 bytes --]

On 2/11/25 7:37 PM, Peter Humphrey wrote:
>> If you aren't dealing with a broken mirror (bytemark was outright
>> dropping some directories via rsync!) then it's just a matter of time.
>> The binhost starts up at around 5am each day, runs for a bit, and
>> publishes all packages for the day. I usually sync around noon. More
>> details at
>>
>> https://wiki.gentoo.org/wiki/Gentoo_Binary_Host_Quickstart#I_used_to_have_a_
>> binary_package_but_not_anymore.2C_what_happened.3F
> 
> I'm sure that works for you, but I'd have to wait until the afternoon to use 
> the same approach.  ;-(


There is also ongoing work at https://bugs.gentoo.org/924772 to allow
portage to detect the version of ::gentoo that was used to produce the
binhost and artificially lock `emerge --sync` to that revision. The idea
is that then, the packages available to emerge are always corresponding
to what the binhost has up-to-date coverage for.

Some building blocks for this are already implemented e.g. there is a
header in the binhost that identifies the REPO_REVISIONS it supports. No
client tooling yet, though you could probably hack together some kind of
shell script if you're sufficiently motivated.

Really though, I'm fine with emerge --sync and then emerge -puDU @world
and just going away and coming back later IFF I notice it is asking to
build packages from source that I don't expect to build from source.


-- 
Eli Schwartz

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 236 bytes --]

Re: [gentoo-user] Prebuilt packages

[Peter Humphrey]

On Tuesday 11 February 2025 23:04:49 Greenwich Mean Time Mickaël Bucas wrote:

> I recently converted a fully locally *slowly* built machine to a mixed
> prebuilt/local build
> Of the 1857 packages on this machine, 1056 are now using prebuilt
> packages and the others are local builds
> USE flags and ~amd64 prevent use of prebuilt packages
> But I adjusted USE flags until dev-qt/qtwebengine:6 found a prebuilt package!

Good result!

> And I switched manually to www-client/firefox-bin and dev-lang/rust-bin

I tried firefox-bin, but I couldn't get it to pick up settings and plugins from 
the original firefox. Rust has been -bin for years here.

-- 
Regards,
Peter.

Re: [gentoo-user] Prebuilt packages

[Jay Faulkner]

[-- Attachment #1.1.1: Type: text/plain, Size: 1060 bytes --]


On 2/12/25 4:07 AM, Peter Humphrey wrote:
> On Tuesday 11 February 2025 23:04:49 Greenwich Mean Time Mickaël Bucas wrote:
>
>> I recently converted a fully locally *slowly* built machine to a mixed
>> prebuilt/local build
>> Of the 1857 packages on this machine, 1056 are now using prebuilt
>> packages and the others are local builds
>> USE flags and ~amd64 prevent use of prebuilt packages
>> But I adjusted USE flags until dev-qt/qtwebengine:6 found a prebuilt package!
> Good result!
>
>> And I switched manually to www-client/firefox-bin and dev-lang/rust-bin
> I tried firefox-bin, but I couldn't get it to pick up settings and plugins from
> the original firefox. Rust has been -bin for years here.

The firefox-bin ebuild uses different paths for profiles so they can be 
used/installed simultaneously with firefox (not-bin). I've successfully 
done the (unsupported) operation of copying the profile from one place 
to the other in order to get my -bin profile in my source installed 
version.

Good luck,

Jay Faulkner



[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 3191 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 840 bytes --]