[gentoo-dev] Profile 23.0 testing with stages and binhost (part 2 of 2)

[gentoo-dev] Profile 23.0 testing with stages and binhost (part 2 of 2)

[Andreas K. Huettel]

[-- Attachment #1: Type: text/plain, Size: 2178 bytes --]

Hi all, 

the 23.0 profiles are ready for testing, including stage downloads,
binary packages, and update instructions for existing installations,
for all arches.

IMPORTANT Exception IMPORTANT
** musl on (32bit) arm and x86 does NOT work yet (gcc build failure) **

IMPORTANT Update instructions IMPORTANT
https://wiki.gentoo.org/wiki/Project:Toolchain/23.0_update_instructions

Stage downloads (temporarily, for all arches):
[preferably] https://distfiles.gentoo.org/experimental/x86/23.0_stages/
[direct/osu] https://gentoo.osuosl.org/experimental/x86/23.0_stages/

The changes can be seen here
https://wiki.gentoo.org/wiki/Project:Toolchain/23.0_profile_transition

and the timeline so far here
https://wiki.gentoo.org/wiki/Project:Toolchain/23.0_profile_timeline

The update instructions also double as the news item that should be
published max. 1-2 weeks from now. They are mostly unchanged compared to 
my last e-mail, just some wording has been clarified.

Note 1: The next steps are, now really in 1-2 weeks max:
* make 23.0 profiles the same stability level as 17.x profiles,
* degrade 17.x profiles all to exp (so the CI doesn't explode)
* publish news item
* replace stage downloads with 23.0 version (in situ)

Note 2: While there are 23.0 split-usr profiles, the *stage* downloads
are *all* of the merged-usr type. Why? Not because I'm a big fan of that, 
but because we should try to unify and standardize a bit again - to 
avoid too many different build configurations leading to too many Heisenbugs.

Note 3: amd64 now has CET turned on by default.
https://docs.kernel.org/next/x86/shstk.html
If you have already used the unannounced 23.0 profiles, you should wipe
your package cache and emerge -ev world now.

Note 4: arm64 does *not* have its equivalent turned on yet since we
encountered last-minute problems (guess what, gcc build failure).

Note 5: There are no hppa builds yet since our machine is still busy.
One gcc build takes about a week there.

Cheers & have fun,
Andreas

-- 
Andreas K. Hüttel
dilfridge@gentoo.org
Gentoo Linux developer
(council, toolchain, base-system, perl, libreoffice)

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-dev] Profile 23.0 testing with stages and binhost (part 2 of 2)

[Sam James]

[-- Attachment #1: Type: text/plain, Size: 1143 bytes --]

"Andreas K. Huettel" <dilfridge@gentoo.org> writes:

> Hi all, 
>
> the 23.0 profiles are ready for testing, including stage downloads,
> binary packages, and update instructions for existing installations,
> for all arches.
>
> [...]
>
> Note 2: While there are 23.0 split-usr profiles, the *stage* downloads
> are *all* of the merged-usr type. Why? Not because I'm a big fan of that, 
> but because we should try to unify and standardize a bit again - to 
> avoid too many different build configurations leading to too many Heisenbugs.

I don't think this is a good idea.

We've promised people that they can keep unmerged-usr if they want, but
not having stages means new installs aren't doable, and it also makes
testing a pain because you can't easily unmerge.

You can easily merge, but you can't easily unmerge.

What you can do is provide a limited number of non-merged-usr variants
given it's just about saving people rebuilds.

(I also think it's the wrong way to do such a change anyway - the releng
part should be last after decision-making, not first.)

> [...]
> Cheers & have fun,

thanks,
sam

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 377 bytes --]

Re: [gentoo-dev] Profile 23.0 testing with stages and binhost (part 2 of 2)

[Andreas K. Huettel]

[-- Attachment #1: Type: text/plain, Size: 1805 bytes --]

> > Note 2: While there are 23.0 split-usr profiles, the *stage* downloads
> > are *all* of the merged-usr type. Why? Not because I'm a big fan of that, 
> > but because we should try to unify and standardize a bit again - to 
> > avoid too many different build configurations leading to too many Heisenbugs.
> 
> I don't think this is a good idea.
> 
> We've promised people that they can keep unmerged-usr if they want, 

And they can.

[However, I don't see the point for it. Apart from ideological considerations,
there is no obvious advantage to the split-usr layout anymore.]

> but not having stages means new installs aren't doable,

Yes.

> and it also makes testing a pain because you can't easily unmerge.
> You can easily merge, but you can't easily unmerge.

That is the imho more important and valid point, maintaining the remaining 
split-usr installs will get harder.

> What you can do is provide a limited number of non-merged-usr variants
> given it's just about saving people rebuilds.

For amd64 and arm64 that's doable (since builds are cheap there).
I would very much discourage using these variants for new installs though.

[And yes I would prefer to deprecate the split-usr profiles and remove them
at some point in the not-so-far future. That is however a topic that needs
separate debate.]

> (I also think it's the wrong way to do such a change anyway - the releng
> part should be last after decision-making, not first.)

The decision where this is going has been made long ago... just not by us
because we've been lagging behind. But I get what you mean.


-- 
Andreas K. Hüttel
dilfridge@gentoo.org
Gentoo Linux developer 
(council, comrel, toolchain, base-system, perl, libreoffice)
https://wiki.gentoo.org/wiki/User:Dilfridge

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[gentoo-dev] Re: Profile 23.0 testing with stages and binhost (part 2 of 2)

[Duncan]

Andreas K. Huettel posted on Fri, 15 Mar 2024 19:12:54 +0100 as excerpted:

> Note 3: amd64 now has CET turned on by default.
> https://docs.kernel.org/next/x86/shstk.html If you have already used the
> unannounced 23.0 profiles, you should wipe your package cache and emerge
> -ev world now.

There's not much about CET in any of the links.  While the kernel.org link 
describes what it does (in a line, "yese": yet another security 
enhancement) a bit, it doesn't say how to actually find whether your 
hardware supports it, and the gentoo wiki and bug links say even less -- 
in particular, unless I missed it, the changes and update instructions 
links don't appear to mention CET or shadow-stacks AT ALL.

What I ended up doing here after some DDG googling, was emerging cpuid, 
then doing:

$$ cpuid -1 | grep -i 'cet\|shadow'
CET_SS: CET shadow stack                 = false
CET_IBT: CET indirect branch tracking    = false
CET_U user state                     = false
CET_S supervisor state               = false
supervisor shadow stack                 = false

With all of those false it would seem CET can't work here in any case so 
there's no point rebuilding again, which is what I already suspected but 
wanted to /know/.  (I've been on a 23.0 merged-usr profile[1] for some 
time now as I already had much of what it does already enabled before the 
new profiles were announced here, so it /would/ be "rebuilding again" to 
get that, but as it seems it won't do anything useful anyway...)

Clearer instructions for finding that out (and preferably what actually 
has to be true, I still don't know that for sure) so others don't have to 
google it, could be useful.

---
[1] Already on a merged-usr profile:  Of course including developing an 
auto-applied-on-update patch to do s:[[ ! -h "${EROOT%/}/bin" ]]:false: to 
the profile bashrc after that test was added, because I am indeed usr-
merged (on systemd) here but that test fails because the operating symlink 
is /usr -> . instead, aka reverse-usrmerge.  Tho making the canonical 
path /realbin and doing /bin -> /realbin  would appear to satisfy the test 
too, and would allow me to avoid patching the profile bashrc, but at least 
here, having /bin be the system's real bin location is part of the _point_ 
of a reverse-usrmerge.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-dev] Re: Profile 23.0 testing with stages and binhost (part 2 of 2)

[Andreas K. Huettel]

[-- Attachment #1: Type: text/plain, Size: 1159 bytes --]

Am Samstag, 16. März 2024, 13:12:04 CET schrieb Duncan:
> Andreas K. Huettel posted on Fri, 15 Mar 2024 19:12:54 +0100 as excerpted:
> 
> > Note 3: amd64 now has CET turned on by default.
> > https://docs.kernel.org/next/x86/shstk.html If you have already used the
> > unannounced 23.0 profiles, you should wipe your package cache and emerge
> > -ev world now.
> 
> There's not much about CET in any of the links.  While the kernel.org link 
> describes what it does (in a line, "yese": yet another security 
> enhancement) a bit, it doesn't say how to actually find whether your 
> hardware supports it, and the gentoo wiki and bug links say even less -- 
> in particular, unless I missed it, the changes and update instructions 
> links don't appear to mention CET or shadow-stacks AT ALL.

That's because it was a last-minute addition, and not particularly well
thought through. :|

Ignore Note 3. The part about emerge -ev world is just plain wrong for now.

-- 
Andreas K. Hüttel
dilfridge@gentoo.org
Gentoo Linux developer 
(council, comrel, toolchain, base-system, perl, libreoffice)
https://wiki.gentoo.org/wiki/User:Dilfridge

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 833 bytes --]