[gentoo-dev] Update on the 23.0 profiles

[gentoo-dev] Update on the 23.0 profiles

[Andreas K. Huettel]

[-- Attachment #1: Type: text/plain, Size: 1565 bytes --]

Hi all, 

so here's a small update on the state of the 23.0 profiles:

* For all arches, the 23.0 profiles are now marked at the same stability status
  (mostly for the CI and pkgcheck) as before the 17.x profiles. Most 17.x profiles
  have been downgraded to "exp".

* All stage downloads (with the exception of hppa, where our builder is a tad slow)
  are now based on 23.0.

* For all ISO / other boot media, the specs have been changed as of today, so that
  the next succeeding build will also be based on 23.0. 
  The new builds need testing, so if you happen to have suitable hardware around...

* With that we can nail down the timetable for the next steps:
  2024-06-06 (in 2 months): deprecation of the 17.x profiles; binpkg builders for
     the 17.x profiles are stopped
  2025-06-06 (1 year later): removal of the 17.x profiles from profiles.desc
  ????-??-?? (when infra has moved :o): removal of the 17.x profile trees in gentoo.git

Comments and feedback welcome. I would really prefer to *not* extend the time until
deprecation though, since maintenance (and CPU time) goes down as soon as we don't
build twice the amount of packages...

In general, our installation ISO need in my opinion some review regarding the package
set and the detailed build instructions. As long as they work that is not really
urgent though.

Cheers,
Andreas


-- 
Andreas K. Hüttel
dilfridge@gentoo.org
Gentoo Linux developer 
(council, comrel, toolchain, base-system, perl, libreoffice)
https://wiki.gentoo.org/wiki/User:Dilfridge

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-dev] Update on the 23.0 profiles

[Michael Orlitzky]

On Sat, 2024-04-06 at 17:06 +0200, Andreas K. Huettel wrote:
> Hi all, 
> 
> so here's a small update on the state of the 23.0 profiles:
> 

Why was this silently added to make.defaults for all 23.0 profiles?

> # This just makes sense nowadays, if only for distfiles...          
> USE="lzma zstd"

It doesn't help with distfiles in any way, wasn't discussed here,
doesn't belong there, and creates a mess on systems where they should
be disabled. Use per-package defaults if they're important for certain
packages.

Re: [gentoo-dev] Update on the 23.0 profiles

[Andreas K. Huettel]

[-- Attachment #1: Type: text/plain, Size: 1189 bytes --]

Am Sonntag, 7. April 2024, 04:03:01 CEST schrieb Michael Orlitzky:
> On Sat, 2024-04-06 at 17:06 +0200, Andreas K. Huettel wrote:
> > Hi all, 
> > 
> > so here's a small update on the state of the 23.0 profiles:
> > 
> 
> Why was this silently added to make.defaults for all 23.0 profiles?
> 
> > # This just makes sense nowadays, if only for distfiles...          
> > USE="lzma zstd"

Uhh, I dont really remember, I think some Chinese-sounding guy asked
me for it... (j/k) 

Jokes aside, we did have bz2 in the default useflags for ages, and 
at the time this made a lot of sense since xz/lzma and zstd were
steadily becoming the most prevalent compression algorithms.

And for anyone interested in the timeline, this was one of the first
additions.

commit 99a7cb9e0b1728ca75242ddfee6357dc008bd1cd
Author:     Andreas K. Hüttel <dilfridge@gentoo.org>
AuthorDate: Sun Nov 13 19:26:40 2022 +0100
Commit:     Andreas K. Hüttel <dilfridge@gentoo.org>
CommitDate: Sun Nov 13 19:27:36 2022 +0100

    profiles: Set USE="xz zstd" in 23.0


-- 
Andreas K. Hüttel
dilfridge@gentoo.org
Gentoo Linux developer
(council, toolchain, base-system, perl, libreoffice)

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 981 bytes --]

Re: [gentoo-dev] Update on the 23.0 profiles

[Michael Orlitzky]

On Sun, 2024-04-07 at 14:35 +0200, Andreas K. Huettel wrote:
> 
> Uhh, I dont really remember, I think some Chinese-sounding guy asked
> me for it... (j/k) 

It is remarkably bad timing. How it looks: Gentoo's response to the xz
incident is to have me rebuild my entire system with everything that
could possibly be linked to liblzma, linked to liblzma. Even on the
hardened profiles, and with no easy way to prevent it.



> Jokes aside, we did have bz2 in the default useflags for ages, and 
> at the time this made a lot of sense since xz/lzma and zstd were
> steadily becoming the most prevalent compression algorithms.

The flags that predate IUSE defaults can of course be forgiven.

What is improved by having these flags in every profile, vs only (say)
the desktop profiles, or in IUSE defaults?

Here's what is made worse: I can't undo it. To maintain the status quo
(I was quite happy to not have 100 packages pointlessly linked to
liblzma last week), what I want to do is set USE="-lzma -zstd". But if
I do that, then it overrides the IUSE defaults for the few packages
whose maintainers are doing the right thing, and setting IUSE="+lzma
+zstd" where it is important. For example, sys-apps/kmod, where the
defaults ensure that your system isn't made unbootable by accident.

The other option is for me to perpetually maintain a list of everything
that uses lzma/zstd inside my package.use. And to avoid the problem
above, I now have to read the ebuilds to see which ones have defaults
and which ones don't. This is also not a great way to spend my time.

tl;dr can we turn them back off in the profile? In any scenario where
they are beneficial, there's a better place to put them.

Re: [gentoo-dev] Update on the 23.0 profiles

[Andreas K. Huettel]

[-- Attachment #1: Type: text/plain, Size: 1012 bytes --]

Am Sonntag, 7. April 2024, 14:51:55 CEST schrieb Michael Orlitzky:
> On Sun, 2024-04-07 at 14:35 +0200, Andreas K. Huettel wrote:
> > 
> > Uhh, I dont really remember, I think some Chinese-sounding guy asked
> > me for it... (j/k) 
> 
> It is remarkably bad timing. How it looks: Gentoo's response to the xz
> incident is to have me rebuild my entire system with everything that
> could possibly be linked to liblzma, linked to liblzma. Even on the
> hardened profiles, and with no easy way to prevent it.

Well, we're now working with the best-audited compression library ever, 
I guess.

> tl;dr can we turn them back off in the profile? In any scenario where
> they are beneficial, there's a better place to put them.

Easily doable with lzma, if there is consensus for it.

Slightly more complex for zstd since this affects gcc and binutils.
Still doable though.

-- 
Andreas K. Hüttel
dilfridge@gentoo.org
Gentoo Linux developer
(council, toolchain, base-system, perl, libreoffice)

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 981 bytes --]

[gentoo-dev] Re: Update on the 23.0 profiles

[Duncan]

Andreas K. Huettel posted on Sun, 07 Apr 2024 15:07:01 +0200 as excerpted:

> Am Sonntag, 7. April 2024, 14:51:55 CEST schrieb Michael Orlitzky:

[USE="lzma zstd" in 23.0 profiles]

>> [R]emarkably bad timing. How it looks: Gentoo's response to the xz
>> incident is to have me rebuild my entire system with everything that
>> could possibly be linked to liblzma, linked to liblzma. Even on the
>> hardened profiles, and with no easy way to prevent it.

Agreed.  Timing is ... unfortunate, making for absolutely terrible 
appearances.  Tho for better or worse Gentoo will likely avoid the bad 
press Arch or the the big guys would get for such a play as we're simply 
not mainline enough any longer (Arch having eclipsed us as "the techie 
distro" in the press years ago now).

> Well, we're now working with the best-audited compression library ever,
> I guess.

Also agreed...

>> tl;dr can we turn them back off in the profile? In any scenario where
>> they are beneficial, there's a better place to put them.

That's the core operational debate.  Perhaps better to debate zstd and 
lzma separately due to timing and relative ease (see below).

> Easily doable with lzma, if there is consensus for it.

Given lzma's easy, I'd vote for the revert, if only due to the unfortunate 
timing.  It can always be reconsidered later, even if for pragmatic 
reasons "later" ends up being the /next/ profile upgrade, presumably some 
years away.

But with the 17 downgrade to exp (if not deprecated yet), if we're 
changing it (and not temporarily reverting the 17 exp) it should be ASAP!

> Slightly more complex for zstd since this affects gcc and binutils.
> Still doable though.

For zstad I'd keep as-is because it's both more difficult and lacks the 
direct timing issues.

TLDR stop!

FWIW, no effect either way here/personally, because I configured portage 
to ignore profile USE flags (as well as IUSE-defaults) years ago, in large 
part precisely /because/ of the undesired USE-flag churn.  And in general 
it has made me a /much/ happier gentooer, because USE-flag no longer 
blindly toggle without my having to go unduly out of my way to find out 
why.  (I already review the git log when a USE flag suddenly (dis)?
appears, unless it's blindingly obvious why without checking the log.)

The one thing I wish I had was an indication of IUSE-defaults for changes 
on upgrades and for new packages.  Sure I can (and do) grep for IUSE if I 
have reason to wonder (more frequently for new packages if I don't know 
whether I want something enabled or not), but I imagine I miss most of the 
on-upgrade ISUE-default-changes as unlike the flag entirely (dis)?
appearing or (un)masking (which is still active from the profile) there's 
nothing alerting me to IUSE-default changes.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-dev] Update on the 23.0 profiles

[Michael Orlitzky]

On Sun, 2024-04-07 at 15:07 +0200, Andreas K. Huettel wrote:
> > tl;dr can we turn them back off in the profile? In any scenario where
> > they are beneficial, there's a better place to put them.
> 
> Easily doable with lzma, if there is consensus for it.
> 
> Slightly more complex for zstd since this affects gcc and binutils.
> Still doable though.
> 

Thanks:

* https://bugs.gentoo.org/928932
* https://bugs.gentoo.org/928933

Re: [gentoo-dev] Update on the 23.0 profiles

[Eddie Chapman]

Michael Orlitzky wrote:
> On Sun, 2024-04-07 at 15:07 +0200, Andreas K. Huettel wrote:
> 
>>> tl;dr can we turn them back off in the profile? In any scenario where
>>>  they are beneficial, there's a better place to put them.
>> 
>> Easily doable with lzma, if there is consensus for it. 
>> 
>> Slightly more complex for zstd since this affects gcc and binutils.
>> Still doable though.
> 
> Thanks:
> 
> * https://bugs.gentoo.org/928932
> * https://bugs.gentoo.org/928933

I know this thread is only for people actually involved in Gentoo 
decision making, but I'll add my 2c anyway.

I'm sure nobody is surprised that I support Michael Orlitzky here 100%.

My personal "dream" is to have a Gentoo in the future where *all* 
compression is optional, only enabled by those who want it, not forced 
on anybody.

In my opinion the importance of compression in general diminishes every 
year that goes by as naturally the trend in storage space has to be that 
it increases. So compression will increasingly become a) an extra 
undesirable security risk (it's quite complex to write and maintain 
which only increases rather than decreases the likelihood of security 
issues) and b) a cpu cycle waster (cpu resources will likely remain more 
precious than storage).

I'd love to eventually see a Gentoo where most upstream source is pulled 
in untouched and uncompressed by default and if people want compression 
they can enable it. So I would hope that as each new profile release 
comes, Gentoo becomes less chained to any particular compression 
libraries than it was before, not more. But I'm aware this is 
unrealistic today from the pov of Gentoo infra. Still, I'm allowed to 
dream right?

P.S. This is not a demand, just 2c, and yes, I do know ultimately the 
ones who role their sleeves up and submit patches will decide these things.

Eddie

Re: [gentoo-dev] Update on the 23.0 profiles

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 800 bytes --]

On Sun, 2024-04-07 at 08:51 -0400, Michael Orlitzky wrote:
> On Sun, 2024-04-07 at 14:35 +0200, Andreas K. Huettel wrote:
> > 
> > Uhh, I dont really remember, I think some Chinese-sounding guy asked
> > me for it... (j/k) 
> 
> It is remarkably bad timing. How it looks: Gentoo's response to the xz
> incident is to have me rebuild my entire system with everything that
> could possibly be linked to liblzma, linked to liblzma. Even on the
> hardened profiles, and with no easy way to prevent it.

So, what you're basically saying, is that the best Gentoo response right
now would be to frantically remove LZMA support everywhere?  I'm sure
that would be so much better than our response of masking vulnerable
versions and issuing a statement.

-- 
Best regards,
Michał Górny


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 512 bytes --]

Re: [gentoo-dev] Update on the 23.0 profiles

[Michael Orlitzky]

On Sun, 2024-04-07 at 16:48 +0200, Michał Górny wrote:
> 
> So, what you're basically saying, is that the best Gentoo response right
> now would be to frantically remove LZMA support everywhere?  I'm sure
> that would be so much better than our response of masking vulnerable
> versions and issuing a statement.
> 

Only in the sense that people who park their cars in the bike lane are
basically Hitler. This really has nothing to do with the xz thing. The
timing was funny, that's all.

What I am saying is that I want the freedom to not have things
pointlessly enabled on my systems, because similar problems (and worse)
happen all day every day. The less exposure I have, the better. The
liblzma backdoor was timely because it will prevent most people from
telling me I'm being paranoid, but it could have been USE=anything on
any other day. Moving the defaults out of the high-level profiles will
give control back to the user, hence my complaint about it.

Re: [gentoo-dev] Update on the 23.0 profiles

[Alex Boag-Munroe]

On Sun, 7 Apr 2024 at 22:09, Michael Orlitzky <mjo@gentoo.org> wrote:
<snip>
> What I am saying is that I want the freedom to not have things
> pointlessly enabled on my systems, because similar problems (and worse)
> happen all day every day. The less exposure I have, the better. The
> liblzma backdoor was timely because it will prevent most people from
> telling me I'm being paranoid, but it could have been USE=anything on
> any other day. Moving the defaults out of the high-level profiles will
> give control back to the user, hence my complaint about it.
>

I agree, to be honest. The spirit of profiles has always felt like it
switches on safe/sane defaults that you'd expect for the name (a
desktop plasma profile switches on all the useful desktop USE flags, a
basic profile enables the bare minimum for a bootable system, etc),
giving an expected functionality in the resulting outcome of a
re-merge of world.

Outside of this, preferred compression tools, preferred editors
etc...should be up to the user, or implied in the profile name if it's
going to be switched on in the profile defaults. I don't use zstd
myself, I prefer xz or lz4 depending on my purpose. It's on my system
because some things I chose to have required it. It feels un-Gentoo
for me to have zstd around _just because_, which the profile default
would bring into play.

--
Ninpo

Re: [gentoo-dev] Update on the 23.0 profiles

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 2257 bytes --]

On Mon, 2024-04-08 at 01:22 +0100, Alex Boag-Munroe wrote:
> On Sun, 7 Apr 2024 at 22:09, Michael Orlitzky <mjo@gentoo.org> wrote:
> <snip>
> > What I am saying is that I want the freedom to not have things
> > pointlessly enabled on my systems, because similar problems (and worse)
> > happen all day every day. The less exposure I have, the better. The
> > liblzma backdoor was timely because it will prevent most people from
> > telling me I'm being paranoid, but it could have been USE=anything on
> > any other day. Moving the defaults out of the high-level profiles will
> > give control back to the user, hence my complaint about it.
> > 
> 
> I agree, to be honest. The spirit of profiles has always felt like it
> switches on safe/sane defaults that you'd expect for the name (a
> desktop plasma profile switches on all the useful desktop USE flags, a
> basic profile enables the bare minimum for a bootable system, etc),
> giving an expected functionality in the resulting outcome of a
> re-merge of world.

Precisely.

> Outside of this, preferred compression tools, preferred editors
> etc...should be up to the user, or implied in the profile name if it's
> going to be switched on in the profile defaults. I don't use zstd
> myself, I prefer xz or lz4 depending on my purpose. It's on my system
> because some things I chose to have required it. It feels un-Gentoo
> for me to have zstd around _just because_, which the profile default
> would bring into play.
> 

It's not a "preferred compression tool".  "Preferred compression tool"
is selected via adding the package to your @world set.  The flag is used
for enable specific functionality on packages.  This function may be
limited to being able to optionally compress something.  But it could
e.g. also be responsible for being able to, say, open a specific file
format (and I'm not talking of explicitly .xz compressed files)
or a database, or receive proper interoperability elsewhere.

The cost of enabling support for a compression library that's already
installed by default (because you need it to unpack distfiles) is very
little compared to the cost of suddenly discovering that things don't
work.

-- 
Best regards,
Michał Górny


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 512 bytes --]

Re: [gentoo-dev] Update on the 23.0 profiles

[Florian Schmaus]

[-- Attachment #1.1.1: Type: text/plain, Size: 731 bytes --]

On 06/04/2024 17.06, Andreas K. Huettel wrote:
> Hi all,
> 
> so here's a small update on the state of the 23.0 profiles:

Thanks for the update and the work on the 23.0 profiles. :)


> Most 17.x profiles have been downgraded to "exp".

I could imagine there is a reason to downgrade those back to 'exp', 
could you elaborate a bit on that?

Isn't it bit strange that a 'stable' profiles gets downgraded back to 
'exp'? Then again, I am not sure about the implications of this nor 
about the rationale behind it.

However, I also notice that there is a outstanding PR that reverts that 
[1]. Maybe we should introduce a new state 'oldstable' or so?

- Flow


1: https://github.com/gentoo/gentoo/pull/35871

[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 17797 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 618 bytes --]

[gentoo-dev] Re: Update on the 23.0 profiles

[Madhu]

> Thanks for the update and the work on the 23.0 profiles. :)
>> Most 17.x profiles have been downgraded to "exp".
> I could imagine there is a reason to downgrade those back to 'exp',
> could you elaborate a bit on that?
>
> Isn't it bit strange that a 'stable' profiles gets downgraded back to
> 'exp'? Then again, I am not sure about the implications of this nor
> about the rationale behind it.
>
> However, I also notice that there is a outstanding PR that reverts
> that [1]. Maybe we should introduce a new state 'oldstable' or so?

I see no way of migrating to 23.0 profile because of not-recompilable
packages that are installed (over 4 years) which block --emptytree,
and do not wish to be forced to migrate to merged-usr on an openrc box
without a compelling need (on principle).

Will patching back the 17.0 profile files into the portage tree if and
when they are removed work?

Are there any options at all for this situation (like freezing the the
last supported tree protecting it from emerge-syncs, and using an
overlay for further updates?)

Re: [gentoo-dev] Re: Update on the 23.0 profiles

[Andreas K. Huettel]

[-- Attachment #1: Type: text/plain, Size: 878 bytes --]

> I see no way of migrating to 23.0 profile because of not-recompilable
> packages that are installed (over 4 years) which block --emptytree,
> and do not wish to be forced to migrate to merged-usr on an openrc box
> without a compelling need (on principle).

That sounds a bit like self-inflicted pain.

> Will patching back the 17.0 profile files into the portage tree if and
> when they are removed work?

Unknown. 

> Are there any options at all for this situation (like freezing the the
> last supported tree protecting it from emerge-syncs, and using an
> overlay for further updates?)

You can try to just skip these packages (with --exclude) during the
"emerge --emptytree ..." step. It should work, but no guarantees given.

-- 
Andreas K. Hüttel
dilfridge@gentoo.org
Gentoo Linux developer
(council, toolchain, base-system, perl, libreoffice)

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 981 bytes --]

[gentoo-dev] Re: Re: Update on the 23.0 profiles

[Madhu]

* "Andreas K. Huettel" <2862978.mvXUDI8C0e @pinacolada> :
Wrote on Sun, 07 Apr 2024 15:27:42 +0200:

>> I see no way of migrating to 23.0 profile because of not-recompilable
>> packages that are installed (over 4 years) which block --emptytree,
>> and do not wish to be forced to migrate to merged-usr on an openrc box
>> without a compelling need (on principle).
> That sounds a bit like self-inflicted pain.
>> Will patching back the 17.0 profile files into the portage tree if and
>> when they are removed work?
> Unknown.
>
>> Are there any options at all for this situation (like freezing the the
>> last supported tree protecting it from emerge-syncs, and using an
>> overlay for further updates?)
>
> You can try to just skip these packages (with --exclude) during the
> "emerge --emptytree ..." step. It should work, but no guarantees given.

I switched the make.conf symlink (from
portage/profiles/default/linux/amd64/17.1 to
portage/profiles/default/linux/amd64/23.0/split-usr ) and the only
difference in the emerge --info output is that LDFLAGS now
additionally has "-Wl,-z,pack-relative-relocs"

My use pattern is I'm only emerging packages by hand and setting
useflags on a case by case basis. Also I have binpkgs going back to 5
years that I don't want to lose (by going to merged-usr, right now I
can unpack and test these in a pinch). I also have various other
packages outside the gentoo tree which depend on stuff which is not in
gentoo portage anymore, which are really not rebuildable, but because
of past portage flexibility multiple installed work fine and can be
tested at the same time.

The question is: Now if I don't attempt to do a rebuild and just
update libtool and do further upgrades and installs on a case by case
basis, it will will eventually pick up the new profile defaults. Is
there any foreseeable downside to just doing this?

Re: [gentoo-dev] Update on the 23.0 profiles

[Andreas K. Huettel]

[-- Attachment #1: Type: text/plain, Size: 878 bytes --]

> > Most 17.x profiles have been downgraded to "exp".
> 
> I could imagine there is a reason to downgrade those back to 'exp', 
> could you elaborate a bit on that?
> 
> Isn't it bit strange that a 'stable' profiles gets downgraded back to 
> 'exp'? Then again, I am not sure about the implications of this nor 
> about the rationale behind it.

Mostly so the load on the CI does not suddenly double. 

There's no real reason why we can't keep a few of the profiles stable.
Also not much reason to do that though...

> 
> However, I also notice that there is a outstanding PR that reverts that 
> [1]. Maybe we should introduce a new state 'oldstable' or so?
> 
> - Flow
> 
> 
> 1: https://github.com/gentoo/gentoo/pull/35871
> 


-- 
Andreas K. Hüttel
dilfridge@gentoo.org
Gentoo Linux developer
(council, toolchain, base-system, perl, libreoffice)

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 981 bytes --]