[gentoo-user] Migrating existing Gentoo to binpkg

[gentoo-user] Migrating existing Gentoo to binpkg

[byte.size226]

[-- Attachment #1: Type: text/plain, Size: 2055 bytes --]

Hello,

I'm sure this has been asked before but, clearly, my archive search 
skills are sub-par. I'm hoping it would be a quick "yes/no" kind of answer.

Long story short, I decided to try out the Gentoo setup with binary 
package host. I've been making my way through the relevant news item [1] 
and guides [2,3].

I already have "/etc/portage/binrepos.conf" and "/etc/portage/gnupg" set 
up correctly - no issues there. In make.conf I also added:

     FEATURES="getbinpkg binpkg-request-signature"

My only question that remains is whether I should change the existing 
value for CFLAGS (I presume so). Currently, I have:

     COMMON_FLAGS="-march=znver4 -O2 -pipe"

(yes CFLAGS etc are set to use $COMMON_FLAGS).

Since I'm planning to use binary packages from x86-64-v3, I presume this 
should be changed to:

     COMMON_FLAGS="-march=x86-64-v3 -O2 -pipe"

or, perhaps:

     COMMON_FLAGS="-march=x86-64-v3 -mtune=znver4 -O2 -pipe" ?

I also have a hefty $CPU_FLA
GS_X86 (also added to $USE) from 
"cpuid2cpuflags" but am not worried about this as packages that don't 
fit will simply be built from source as usual.

Anyway, in both cases running my usual:

     emerge -aqvND --update --keep-going --with-bdeps=y @world

doesn't yield any changes to the system. I presume because, as of now, 
everything is up to date.

In contrast, adding "--rebuilt-binaries" shows a lot of binary packages 
being pulled in and "--emptytree", as expected, shows a full rebuild 
with a lot of binary packages being pulled in.

So, to the actual questions:

1) What would be the preferred CFLAGS configuration or, since 
"-march=znver4" is a newer subset of x86-64-v3, can I simply keep the 
existing one?

2) To reinstall the current source based packages with their binary 
equivalent, "--rebuilt-binaries" sufficient or should I just go for 
"--emptytree @world"?

Thanks,
Victor

[1] https://www.gentoo.org/news/2023/12/29/Gentoo-binary.html
[2] h
ttps://wiki.gentoo.org/wiki/Gentoo_Binary_Host_Quickstart
[3] https://wiki.gentoo.org/wiki/Binary_package_guide

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 249 bytes --]

Re: [gentoo-user] Migrating existing Gentoo to binpkg

[Jacques Montier]

[-- Attachment #1: Type: text/plain, Size: 2439 bytes --]

Le ven. 15 nov. 2024 à 13:42, <byte.size226@simplelogin.com> a écrit :

> Hello,
>
> I'm sure this has been asked before but, clearly, my archive search
> skills are sub-par. I'm hoping it would be a quick "yes/no" kind of answer.
>
> Long story short, I decided to try out the Gentoo setup with binary
> package host. I've been making my way through the relevant news item [1]
> and guides [2,3].
>
> I already have "/etc/portage/binrepos.conf" and "/etc/portage/gnupg" set
> up correctly - no issues there. In make.conf I also added:
>
>      FEATURES="getbinpkg binpkg-request-signature"
>
> My only question that remains is whether I should change the existing
> value for CFLAGS (I presume so). Currently, I have:
>
>      COMMON_FLAGS="-march=znver4 -O2 -pipe"
>
> (yes CFLAGS etc are set to use $COMMON_FLAGS).
>
> Since I'm planning to use binary packages from x86-64-v3, I presume this
> should be changed to:
>
>      COMMON_FLAGS="-march=x86-64-v3 -O2 -pipe"
>
> or, perhaps:
>
>      COMMON_FLAGS="-march=x86-64-v3 -mtune=znver4 -O2 -pipe" ?
>
> I also have a hefty $CPU_FLA
> GS_X86 (also added to $USE) from
> "cpuid2cpuflags" but am not worried about this as packages that don't
> fit will simply be built from source as usual.
>
> Anyway, in both cases running my usual:
>
>      emerge -aqvND --update --keep-going --with-bdeps=y @world
>
> doesn't yield any changes to the system. I presume because, as of now,
> everything is up to date.
>
> In contrast, adding "--rebuilt-binaries" shows a lot of binary packages
> being pulled in and "--emptytree", as expected, shows a full rebuild
> with a lot of binary packages being pulled in.
>
> So, to the actual questions:
>
> 1) What would be the preferred CFLAGS configuration or, since
> "-march=znver4" is a newer subset of x86-64-v3, can I simply keep the
> existing one?
>
> 2) To reinstall the current source based packages with their binary
> equivalent, "--rebuilt-binaries" sufficient or should I just go for
> "--emptytree @world"?
>
> Thanks,
> Victor
>
> [1] https://www.gentoo.org/news/2023/12/29/Gentoo-binary.html
> [2] h
> ttps://wiki.gentoo.org/wiki/Gentoo_Binary_Host_Quickstart
> [3] https://wiki.gentoo.org/wiki/Binary_package_guide



Hello,

What if you try this :
emerge -auvDN --getbinpkgonly --with-bdeps=y --binpkg-respect-use=y
--keep-going world

Cheers,

--
Jacques

[-- Attachment #2: Type: text/html, Size: 3814 bytes --]

Re: [gentoo-user] Migrating existing Gentoo to binpkg

[Matt Jolly]

Hi,

> Since I'm planning to use binary packages from x86-64-v3, I presume this 
> should be changed to:
> 
>      COMMON_FLAGS="-march=x86-64-v3 -O2 -pipe"
> 
> or, perhaps:
> 
>      COMMON_FLAGS="-march=x86-64-v3 -mtune=znver4 -O2 -pipe" ?

You want to match the binhost flags:

https://wiki.gentoo.org/wiki/Gentoo_Binary_Host_Quickstart#x86-64-v3_variant


> I also have a hefty $CPU_FLA
> GS_X86 (also added to $USE) from "cpuid2cpuflags" but am not worried 
> about this as packages that don't fit will simply be built from source 
> as usual.

You could also match with the flags the binhost has if you want to avoid
compilation altogether. See above.

> 1) What would be the preferred CFLAGS configuration

See above.

> 2) To reinstall the current source based packages with their binary 
> equivalent, "--rebuilt-binaries" sufficient or should I just go for "-- 
> emptytree @world"?
> 

If you're currently using x86-64-v4 there's no benefit to replacing the
existing packages. Just let them age out and be replaced over time as
as new versions are released.

Cheers,

Matt

Re: [gentoo-user] Migrating existing Gentoo to binpkg

[byte.size226]

[-- Attachment #1: Type: text/plain, Size: 2205 bytes --]

Thank you, both!


On 15/11/2024 14:05, Jacques Montier wrote:
 > What if you try this :
 > emerge -auvDN --getbinpkgonly --with-bdeps=y --binpkg-respect-use=y --
 > keep-going world

This does indeed suggest to replace existing builds with the upstream 
binary ones. I traced this to --getbinpkgonly which seems to force the 
use of the binary packages over source builds where possible.

There are, however, minor differences between this and my earlier 
example with --rebuilt-binaries. Using "--getbinpkgonly" suggests a few 
downgrades, including to sys-kernel/gentoo-sources.

"--binpkg-respect-use=y" makes no difference as emerge(1) suggests this 
is the default.

On 15/11/2024 14:15, Matt Jolly wrote:
> You want to match the binhost flags:
> 
> https://wiki.gentoo.org/wiki/Gentoo_Binary_Host_Quickstart#x86-64-v3_variant
> 
That was my assumption as well. But doing so and running a rebuild of 
@world with --emptytree did not yield any differences in which binar
y 
packages are used between "-march=x86-64-v3" and "-march=znver4".

What caught my attention was a passage from the detailed guide:

"""
* The builder and client architecture and CHOST must match.
* The CFLAGS and CXXFLAGS variables used to build the binary packages 
must be compatible with all clients.

...

Portage can not validate if these requirements match. In case of doubt, 
it is the responsibility of the system administrator to guard these 
settings.
"""

This made me wonder if CFLAGS need to match exactly in a situation, such 
as mine, where -march=znver4, being newer, is compatible and anything 
built with has "-march=x86-64-v3" /should/ run fine.

If my understanding is correct, then the local value for CFLAGS should 
only affect the source builds.

 > If you're currently using x86-64-v4 there's no benefit to replacing the
 > existing packages. Just let them age out and be replaced over time as
 > as new versions are released.

Indeed, that's my
 understanding too. Combined with being the same 
version, I suspect this is why emerge is not suggesting to replace the 
existing source builds with the binary builds, but is happy to do so 
with the nuclear "--emptytree" approach.


Best Regards,
Victor

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 249 bytes --]

Re: [gentoo-user] Migrating existing Gentoo to binpkg

[Eli Schwartz]

[-- Attachment #1.1: Type: text/plain, Size: 4333 bytes --]

On 11/15/24 7:42 AM, byte.size226@simplelogin.com wrote:
> My only question that remains is whether I should change the existing
> value for CFLAGS (I presume so). Currently, I have:
> 
>     COMMON_FLAGS="-march=znver4 -O2 -pipe"
> 
> (yes CFLAGS etc are set to use $COMMON_FLAGS).
> 
> Since I'm planning to use binary packages from x86-64-v3, I presume this
> should be changed to:
> 
>     COMMON_FLAGS="-march=x86-64-v3 -O2 -pipe"
> 
> or, perhaps:
> 
>     COMMON_FLAGS="-march=x86-64-v3 -mtune=znver4 -O2 -pipe" ?


Your CFLAGS are irrelevant here. CFLAGS on the binhost server must be
compatible with your local machine, or you will successfully install
packages that then abort with SIGILL when you try to run the programs.

The reverse is not true -- CFLAGS on your local machine just need to be
compatible with, well, your local machine (where they get installed),
not the binhost, where they don't get installed. :)

It's fully possible to change your CFLAGS in make.conf whenever you
want, run emerge to do world updates etc, and portage doesn't care and
effectively doesn't know you changed it since all installed packages are
"compatible" with that make.conf -- this is extremely unlike USE flags,
which are actually treated as match specifiers and if you change USE,
then emerge insists on rebuilding / reinstalling. Which brings us to...


> I also have a hefty $CPU_FLA
> GS_X86 (also added to $USE) from "cpuid2cpuflags" but am not worried
> about this as packages that don't fit will simply be built from source
> as usual.


Yup. CPU_FLAGS_X86 don't need to be added to $USE because they already
*are* USE. And USE flags will force rebuilds where necessary, or prevent
seeing binary packages as compatible. The binhost already builds with
those USE flags as appropriate for v3 systems, but you may have a couple
more enabled so packages which specifically can take advantage of that,
will be built locally.


> Anyway, in both cases running my usual:
> 
>     emerge -aqvND --update --keep-going --with-bdeps=y @world
> 
> doesn't yield any changes to the system. I presume because, as of now,
> everything is up to date.
> 
> In contrast, adding "--rebuilt-binaries" shows a lot of binary packages
> being pulled in and "--emptytree", as expected, shows a full rebuild
> with a lot of binary packages being pulled in.
> 
> So, to the actual questions:
> 
> 1) What would be the preferred CFLAGS configuration or, since "-
> march=znver4" is a newer subset of x86-64-v3, can I simply keep the
> existing one?


Keep your existing CFLAGS that you were using before enabling the
binhost. It's a free optimization for any packages that you ended up
building from source anyway.


> 2) To reinstall the current source based packages with their binary
> equivalent, "--rebuilt-binaries" sufficient or should I just go for "--
> emptytree @world"?


--rebuilt-binaries will update all packages with their binary
equivalents, if the binary on the server has a BUILD_TIME newer than the
time you rebuilt it locally. This isn't really the same as reinstalling
all source based packages with their binary equivalent.

--emptytree will update all packages with their binary equivalents, and
also update all other packages by building them from source, which isn't
actually what you asked for, though it does what you asked for as a side
effect. Some people encourage running weekly --emptytree anyways,
though. Usually the idea is doing it as an experiment to see whether all
packages still build and run successfully, and report bugs / commit
fixes to gentoo.git whenever issues are encountered. It may also go well
with running ACCEPT_KEYWORDS="**" for sys-devel/gcc and reporting
regressions to the GCC bugtracker. ;)


It's not actually strictly necessary to reinstall all source based
packages with their binary equivalent. If you want to do that though,
you can do that by looping over directories in /var/db/pkg/*/* and
checking which ones have a file called BINPKGMD5 in them. Those were
installed from a binpackage. Try installing all the rest with --ask,
check if it says [binary] or [ebuild], and repeat after manually
narrowing the list to just the ones that offer binaries.



-- 
Eli Schwartz

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 236 bytes --]

Re: [gentoo-user] Migrating existing Gentoo to binpkg

[Eli Schwartz]

[-- Attachment #1.1: Type: text/plain, Size: 1377 bytes --]

On 11/15/24 9:52 AM, byte.size226@simplelogin.com wrote:
> Thank you, both!
> 
> 
> On 15/11/2024 14:05, Jacques Montier wrote:
>> What if you try this :
>> emerge -auvDN --getbinpkgonly --with-bdeps=y --binpkg-respect-use=y --
>> keep-going world
> 
> This does indeed suggest to replace existing builds with the upstream
> binary ones. I traced this to --getbinpkgonly which seems to force the
> use of the binary packages over source builds where possible.
> 
> There are, however, minor differences between this and my earlier
> example with --rebuilt-binaries. Using "--getbinpkgonly" suggests a few
> downgrades, including to sys-kernel/gentoo-sources.


gentoo-sources is not part of the guaranteed package set for the
binhost. Any package *may* end up built on the binhost, either as a
dependency for another package that stops being a dependency because the
other package has changed, or due to the fact that we run a few
different build configurations, including choosing a couple packages by
lottery each day. Those may be available only for an older version in
the ::gentoo tree if they don't get lucky a second time.


> "--binpkg-respect-use=y" makes no difference as emerge(1) suggests this
> is the default.


It is only the default when using --getbinpkg, not when you are instead
using --getbinpkgonly.


-- 
Eli Schwartz

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 236 bytes --]

Re: [gentoo-user] Migrating existing Gentoo to binpkg

[byte.size226]

[-- Attachment #1: Type: text/plain, Size: 2399 bytes --]


On 15/11/2024 15:41, Eli Schwartz wrote:
> gentoo-sources is not part of the guaranteed package set for the
> binhost. Any package*may* end up built on the binhost, either as a
> dependency for another package that stops being a dependency because the
> other package has changed, or due to the fact that we run a few
> different build configurations, including choosing a couple packages by
> lottery each day. Those may be available only for an older version in
> the ::gentoo tree if they don't get lucky a second time.

Indeed, also why shortly after my response I also added 
"--usepkg-exclude 'sys-kernel/gentoo-sources virtual/*" to 
EMERGE_DEFAULT_OPTS, as was recommended by one of the guides anyway :)

I did suspect packages may or may not be built from source anyway 
depending on the current state of dependencies and what's available on 
the upstream binhost. This is already the case for about 300 or so 
packages that I have.

I was mostly interested in the binary builds due to usual suspects that 
take forever to build. But also out of general curiosity, having 
previously dabbled a bit into having a local binhost a few years back. I 
have to admit, the flexibility is pretty sweet.

> It is only the default when using --getbinpkg, not when you are instead
> using --getbinpkgonly.

Ah! Of course, you're right. Should have calibrated my eyes better while 
reading the man page.

> Your CFLAGS are irrelevant here. CFLAGS on the binhost server must be
> compatible with your local machine, or you will successfully install
> packages that then abort with SIGILL when you try to run the programs.
> 
> The reverse is not true -- CFLAGS on your local machine just need to be
> compatible with, well, your local machine (where they get installed),
> not the binhost, where they don't get installed. 🙂
> ...
> Keep your existing CFLAGS that you were using before enabling the
> binhost. It's a free optimization for any packages that you ended up
> building from source anyway.

Just what I suspected 🙂

All working perfectly like a charm.

Thanks again to everyone for the responses! That, and having kept an eye 
on some other discussions, the Gentoo community never seizes to 
positively amaze. Unlike the respective communities of some other... 
popular distros, that I shall refrain from mentioning.

Cheers,
Victor

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 249 bytes --]

Re: [gentoo-user] Migrating existing Gentoo to binpkg

[Dr Rainer Woitok]

Victor,

On Sun, 17 Nov 2024 14:44:40 +0000 you wrote:

> ...
>                                            I also added 
> "--usepkg-exclude 'sys-kernel/gentoo-sources virtual/*" to 
> EMERGE_DEFAULT_OPTS, as was recommended by one of the guides anyway :)

Hm, I can't see any logic in Gentoo developers wasting electrons and key
strokes to provide at least  one "sys-kernel/gentoo-sources"  and a ple-
thora of  "virtual/*" packages  on the binhost  and then advising to ex-
clude them from being used.  Which guide recommended that?

Sincerely,
  Rainer

Re: [gentoo-user] Migrating existing Gentoo to binpkg

[byte.size226]

[-- Attachment #1: Type: text/plain, Size: 1559 bytes --]

On 18/11/2024 09:21, Dr Rainer Woitok wrote:
>>                                             I also added
>> "--usepkg-exclude 'sys-kernel/gentoo-sources virtual/*" to
>> EMERGE_DEFAULT_OPTS, as was recommended by one of the guides anyway :)
> 
> Hm, I can't see any logic in Gentoo developers wasting electrons and key
> strokes to provide at least  one "sys-kernel/gentoo-sources"  and a ple-
> thora of  "virtual/*" packages  on the binhost  and then advising to ex-
> clude them from being used.  Which guide recommended that?
> 

The in-depth "Binary package guide" [1], section 3.7 "Additional client 
settings".

I guess I misspoke by referring to it as "recommendation" but the 
example given, which excludes "sys-kernel/gentoo-sources", can be seen 
as de facto recommendation as it is quite sound.

It makes sense as the "sys-kernel/gentoo-sources" package does not have 
a "binary" package in the sense of a running kernel, which is provided 
by "sys-kernel/gentoo-kerne
l-bin".

Emerging gentoo-source simply shows it as an "[ebuild]" which is what I 
would expect for a package that is supposed to only contain source code. 
Meanwhile, gentoo-kernel-bin is correctly being pulled from the binhost 
as "[binary]".

As for "virtual/", well they don't really contain anything so there's no 
binary to provide. I'm not sure if there's anything "[binary]" on the 
binhost for them as I would struggle to see what that would be other 
than a label for a given... version/revision?

Regards,
Victor

[1] https://wiki.gentoo.org/wiki/Binary_package_guide

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 249 bytes --]