[gentoo-dev] Current unavoidable use of xz utils in Gentoo

[gentoo-dev] Current unavoidable use of xz utils in Gentoo

[Eddie Chapman]

Given what we've learnt in the last 24hrs about xz utilities, you could
forgive a paranoid person for seriously considering getting rid entirely
of them from their systems, especially since there are suitable
alternatives available.  Some might say that's a bit extreme, xz-utils
will get a thorough audit and it will all be fine. But when a malicious
actor has been a key maintainer of something as complex as a decompression
utility for years, I'm not sure I could ever trust that codebase again.
Maybe a complete rewrite will emerge, but I'm personally unwilling to
continue using xz utils in the meantime for uncompressing anything on my
systems, even if it is done by an unprivileged process.

I see that many system package ebuilds unconditionally expect
app-arch/xz-utils to be installed simply to be able to decompress the
source archive in SRC_URI. So simply specifying -lzma on your system isn't
going to get rid of it.

No one could have been expected to foresee what's happened with xz-utils,
but now that it's here, perhaps Gentoo (and other projects that do) should
consider not relying on a single decompression algorithm for source
archives, even just as an insurance against some other yet unknown
disaster with one algorithm or another in future?

And yes I'm sure there will be individual packages that currently
absolutely need xz-utils installed during the build process, and one or
two that absolutely have to have it available at runtime, but those
bridges can be crossed as and when.

Eddie

Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo

[orbea]

On Sat, 30 Mar 2024 03:07:13 -0000
"Eddie Chapman" <eddie@ehuk.net> wrote:

> Given what we've learnt in the last 24hrs about xz utilities, you
> could forgive a paranoid person for seriously considering getting rid
> entirely of them from their systems, especially since there are
> suitable alternatives available.  Some might say that's a bit
> extreme, xz-utils will get a thorough audit and it will all be fine.
> But when a malicious actor has been a key maintainer of something as
> complex as a decompression utility for years, I'm not sure I could
> ever trust that codebase again. Maybe a complete rewrite will emerge,
> but I'm personally unwilling to continue using xz utils in the
> meantime for uncompressing anything on my systems, even if it is done
> by an unprivileged process.
> 
> I see that many system package ebuilds unconditionally expect
> app-arch/xz-utils to be installed simply to be able to decompress the
> source archive in SRC_URI. So simply specifying -lzma on your system
> isn't going to get rid of it.
> 
> No one could have been expected to foresee what's happened with
> xz-utils, but now that it's here, perhaps Gentoo (and other projects
> that do) should consider not relying on a single decompression
> algorithm for source archives, even just as an insurance against some
> other yet unknown disaster with one algorithm or another in future?
> 
> And yes I'm sure there will be individual packages that currently
> absolutely need xz-utils installed during the build process, and one
> or two that absolutely have to have it available at runtime, but those
> bridges can be crossed as and when.
> 
> Eddie
> 
> 

I think this is an overreaction and we should wait for the dust to
settle before making drastic disruptive changes.

Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo

[Dale]

orbea wrote:
> On Sat, 30 Mar 2024 03:07:13 -0000
> "Eddie Chapman" <eddie@ehuk.net> wrote:
>
>> Given what we've learnt in the last 24hrs about xz utilities, you
>> could forgive a paranoid person for seriously considering getting rid
>> entirely of them from their systems, especially since there are
>> suitable alternatives available.  Some might say that's a bit
>> extreme, xz-utils will get a thorough audit and it will all be fine.
>> But when a malicious actor has been a key maintainer of something as
>> complex as a decompression utility for years, I'm not sure I could
>> ever trust that codebase again. Maybe a complete rewrite will emerge,
>> but I'm personally unwilling to continue using xz utils in the
>> meantime for uncompressing anything on my systems, even if it is done
>> by an unprivileged process.
>>
>> I see that many system package ebuilds unconditionally expect
>> app-arch/xz-utils to be installed simply to be able to decompress the
>> source archive in SRC_URI. So simply specifying -lzma on your system
>> isn't going to get rid of it.
>>
>> No one could have been expected to foresee what's happened with
>> xz-utils, but now that it's here, perhaps Gentoo (and other projects
>> that do) should consider not relying on a single decompression
>> algorithm for source archives, even just as an insurance against some
>> other yet unknown disaster with one algorithm or another in future?
>>
>> And yes I'm sure there will be individual packages that currently
>> absolutely need xz-utils installed during the build process, and one
>> or two that absolutely have to have it available at runtime, but those
>> bridges can be crossed as and when.
>>
>> Eddie
>>
>>
> I think this is an overreaction and we should wait for the dust to
> settle before making drastic disruptive changes.
>
>


From the news item email: 


"Impact
======

Our current understanding of the backdoor is that is does not affect
Gentoo systems, because

1. the backdoor only appears to be included on specific systems and
Gentoo does not qualify;
2. the backdoor as it is currently understood targets OpenSSH patched to
work with systemd-notify support. Gentoo does not support or include
these patches;

Analysis is still ongoing, however, and additional vectors may still be
identified. For this reason we are still issuing this advisory as if
that will be the case."


When I started reading it, I was concerned as well as I know it is used on my system.  However, when I got to the part about it not likely to affect Gentoo, my level of concern dropped significantly.  If this is still true, there's no need to be concerned.  If things has changed and it does affect Gentoo, I'm sure there will be changes made that will either fix the issue for good or at least provide a workaround until a solution is found.  Gentoo has some awesome devs. Someone will find a solution.  I notice that it has already been changed in the tree to a version that does not have the malicious code.  That alone should be a solution until a new plan is made.  

While I'm a little concerned and hope for a proper solution, I'm not to worried.  I certainly don't think we should overreact this early.  Give the devs and upstream time to work this out.  

Just a users opinion.  

Dale 

:-)  :-)  

[gentoo-dev] Re: Current unavoidable use of xz utils in Gentoo

[Duncan]

Dale posted on Sat, 30 Mar 2024 02:06:26 -0500 as excerpted:

> Gentoo has some awesome devs.

Agreed with the whole thing and the above is a bit of an aside from the 
thread, but it's worth repeating!

Thanks devs!  (And security contributors, infra providers, testers, 
tinder-box runners, bug reporters and wranglers, docs/wiki/lists/forums/
chat contributors, and all of the above for our upstreams too... it takes 
a big community to make a full distro and we all have our part! =:^)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo

[Rich Freeman]

On Sat, Mar 30, 2024 at 3:06 AM Dale <rdalek1967@gmail.com> wrote:
>
> when I got to the part about it not likely to affect Gentoo, my level of concern dropped significantly.  If this is still true, there's no need to be concerned.

"not likely" is the best way to characterize this.  The exploit has
not been fully analyzed, and it could have additional malicious
behavior, either designed by its author, or perhaps even unintended by
its author.

I just wanted to toss in that caveat, but agree that the defaults
deployed in Gentoo seem the most sensible for general use.  There is
nothing magical about xz - ANY widely-used library could have
something like this embedded in it, and the attacker exploited what
they had access to in order to go after a configuration that was going
to be widely deployed and reachable (xz+deb/rpm+systemd+openssh).  If
the attacker had an intended target that used gentoo+openrc and access
to something in our supply chain, this could have been a vulnerability
that only impacted Gentoo.

I think the big lesson here is that FOSS continues to suffer from core
dependencies that are challenged for resources, and that efforts to
fix this have to constantly keep up with the changing landscape.  xz
is going on 15 years old, but I don't think it was nearly as commonly
used until fairly recently.

libz has been a pretty well-known source of security flaws for ages
(granted, usually not intentional like this).  It isn't too surprising
that in both cases compression libraries were targeted, as these are
so widely depended on.

This is getting tangential, but part of me wonders if there is a
better way to do authentication.  Programs like ssh tend to run as
root so that they can authenticate users and then fork and suid to the
appropriate user.  Could some OS-level facility be created to allow
unprivileged processes to run the daemons and then as part of the
authentication process they can have the OS accept a controlled and
minimal set of data to create the process as the new user and hand
over the connection?  PAM already has a large amount of control over
the authentication process, so it seems like we just need to change
the security context that this runs in.  That's just
brainstorming-level thinking though - there could be obvious issues
with this that just haven't occurred to me.

-- 
Rich

Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo

[Eddie Chapman]

Rich, Duncan, Dale, orbea, you have to admit the situation with xz-utils
is nothing like the typical scenario people usually worry about, where a
bad actor manages to compromise a project and slip something into a widely
used piece of software. No, this is the the bad actor *themselves* being a
principal author of the software, working stealthily and in very
sophisticated ways for years, to manoeuvrer themselves and their software
into a position of trust in the ecosystem whereby they were almost able to
pull off the mother of all security nightmares for the world.  And many
very smart people reviewed what they did and were fooled by them (which is
no reflection on those people, it was just because the bad actor did a
very, very good job of fooling them). I have to ask, if you still trust a
codebase to be right at the heart of your system after that, what on earth
would it take for you to start to feel uncomfortable??!!

Sometimes, it's good when you're inside the house that is on fire, to
*not* stand there and say to yourself "well the engineers who built this
place must have built it to withstand a fire, I'm sure it will stop
burning soon. And anyway, the fire brigade will be here soon, I'm sure it
will all be fine". I'm not saying the world of OSS & Linux is on fire, of
course not. This is a very isolated and rare situation with just 1 piece
of software. No, I'm just using probably a probably bad analogy to make
the following point: while almost all of the time a reasoned, "lets just
calm down and think about this" approach is right, in some rare situations
it is important to see a situation as serious as it and act accordingly.

In this case, if I weigh up the benefits of using this piece of software
versus another (relatively small gains in file size reduction, some gains
in resource usage) against the risks of continuing to use it (and lets be
realistic about those risks please rather than "I'm sure it will all be
fine"), the risks are far greater.

Note, I'm not advocating ripping xz-utils out of tree, all I'm saying is
wouldn't it be nice if there were at least 2 alternatives to choose from?
That doesn't have to be disruptive in any way, people who wish to continue
using and trusting xz-utils should be able to continue to do so without
any friction whatsoever.

Rich Freeman wrote:
> On Sat, Mar 30, 2024 at 3:06 AM Dale <rdalek1967@gmail.com> wrote:
>
>>
>> when I got to the part about it not likely to affect Gentoo, my level
>> of concern dropped significantly.  If this is still true, there's no
>> need to be concerned.
>
> "not likely" is the best way to characterize this.  The exploit has
> not been fully analyzed, and it could have additional malicious behavior,
> either designed by its author, or perhaps even unintended by its author.
>
> I just wanted to toss in that caveat, but agree that the defaults
> deployed in Gentoo seem the most sensible for general use.  There is
> nothing magical about xz - ANY widely-used library could have something
> like this embedded in it, and the attacker exploited what they had access
> to in order to go after a configuration that was going to be widely
> deployed and reachable (xz+deb/rpm+systemd+openssh).  If the attacker had
> an intended target that used gentoo+openrc and access to something in our
> supply chain, this could have been a vulnerability that only impacted
> Gentoo.
>
>
> I think the big lesson here is that FOSS continues to suffer from core
> dependencies that are challenged for resources, and that efforts to fix
> this have to constantly keep up with the changing landscape.  xz is going
> on 15 years old, but I don't think it was nearly as commonly used until
> fairly recently.
>
> libz has been a pretty well-known source of security flaws for ages
> (granted, usually not intentional like this).  It isn't too surprising
> that in both cases compression libraries were targeted, as these are so
> widely depended on.
>
> This is getting tangential, but part of me wonders if there is a
> better way to do authentication.  Programs like ssh tend to run as root so
> that they can authenticate users and then fork and suid to the appropriate
> user.  Could some OS-level facility be created to allow unprivileged
> processes to run the daemons and then as part of the authentication
> process they can have the OS accept a controlled and minimal set of data
> to create the process as the new user and hand over the connection?  PAM
> already has a large amount of control over the authentication process, so
> it seems like we just need to change the security context that this runs
> in.  That's just brainstorming-level thinking though - there could be
> obvious issues with this that just haven't occurred to me.
>
> --
> Rich
>
>
>
>

Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 640 bytes --]

On Sat, 2024-03-30 at 14:57 +0000, Eddie Chapman wrote:
> Note, I'm not advocating ripping xz-utils out of tree, all I'm saying is
> wouldn't it be nice if there were at least 2 alternatives to choose from?
> That doesn't have to be disruptive in any way, people who wish to continue
> using and trusting xz-utils should be able to continue to do so without
> any friction whatsoever.

So, you're basically saying we should go out of our way, recompress all
distfiles using two alternative compression formats, increase mirror
load four times and add a lot of complexity to ebuilds, right?

-- 
Best regards,
Michał Górny


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 512 bytes --]

Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo

[Rich Freeman]

On Sat, Mar 30, 2024 at 10:57 AM Eddie Chapman <eddie@ehuk.net> wrote:
>
> No, this is the the bad actor *themselves* being a
> principal author of the software, working stealthily and in very
> sophisticated ways for years, to manoeuvrer themselves and their software
> into a position of trust in the ecosystem whereby they were almost able to
> pull off the mother of all security nightmares for the world.

This is entirely speculative at this point.  It isn't certain that the
author is the one behind the exploit, and if they were, it is not
known for how long their intentions were malicious, or even what their
motivations were.  It is also unclear what pseudonymous accounts with
what projects are associated with the attacker.

You could end up being right, but it probably makes sense to at least
give things a few days for more facts to become available, before
making decisions to retool the entire distro.

I think the bigger challenge is what could have been done to prevent
this sort of problem in the first place.  There are so many projects
that end up with code running as root that have one or two people
taking care of them, and if somebody does the work to become one of
those maintainers, there aren't many people looking out for problems.

I think one thing that would help here is for distros to have better
ways to ensure that the code in the scm matches the code in the
tarball.  It is pretty common for releases to be manipulated in some
way (even if only to gpg sign them, but often to switch from commit
IDs to version numbers and so on), and that can be a place where stuff
gets added.  That still says nothing about obfuscated code, which this
also involved.

-- 
Rich

Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo

[Eddie Chapman]

Michał Górny wrote:
> On Sat, 2024-03-30 at 14:57 +0000, Eddie Chapman wrote:
>
>> Note, I'm not advocating ripping xz-utils out of tree, all I'm saying
>> is wouldn't it be nice if there were at least 2 alternatives to choose
>> from? That doesn't have to be disruptive in any way, people who wish to
>> continue using and trusting xz-utils should be able to continue to do so
>> without any friction whatsoever.
>
> So, you're basically saying we should go out of our way, recompress all
> distfiles using two alternative compression formats, increase mirror load
> four times and add a lot of complexity to ebuilds, right?
>
> --
> Best regards,
> Michał Górny
>

Yes that's a very good point, that was something I was wondering in
weighing up both sides, what the costs would be practically, as I don't
know the realities of running Gentoo infrastructure. And maybe the costs
is just too high of a price to pay.

I wonder if increased use of git repos rather than distributed tarballs
could be part of a solution to those issues, although that could put quite
a storage burden on every user. Unless they were all shallow git pulls and
the user could optionally choose to tar up the git directory after clone
with compression.  But yes granted then there is even more ebuild
complexity.

Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo

[orbea]

On Sat, 30 Mar 2024 16:02:25 +0100
Michał Górny <mgorny@gentoo.org> wrote:

> On Sat, 2024-03-30 at 14:57 +0000, Eddie Chapman wrote:
> > Note, I'm not advocating ripping xz-utils out of tree, all I'm
> > saying is wouldn't it be nice if there were at least 2 alternatives
> > to choose from? That doesn't have to be disruptive in any way,
> > people who wish to continue using and trusting xz-utils should be
> > able to continue to do so without any friction whatsoever.  
> 
> So, you're basically saying we should go out of our way, recompress
> all distfiles using two alternative compression formats, increase
> mirror load four times and add a lot of complexity to ebuilds, right?
> 

How would Gentoo even recompress all .xz distfiles if xz-utils is not
even in the repo? And this will certainly be a reoccurring theme...

Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1603 bytes --]

On Sat, 2024-03-30 at 15:17 +0000, Eddie Chapman wrote:
> Michał Górny wrote:
> > On Sat, 2024-03-30 at 14:57 +0000, Eddie Chapman wrote:
> > 
> > > Note, I'm not advocating ripping xz-utils out of tree, all I'm saying
> > > is wouldn't it be nice if there were at least 2 alternatives to choose
> > > from? That doesn't have to be disruptive in any way, people who wish to
> > > continue using and trusting xz-utils should be able to continue to do so
> > > without any friction whatsoever.
> > 
> > So, you're basically saying we should go out of our way, recompress all
> > distfiles using two alternative compression formats, increase mirror load
> > four times and add a lot of complexity to ebuilds, right?
> > 
> > --
> > Best regards,
> > Michał Górny
> > 
> 
> Yes that's a very good point, that was something I was wondering in
> weighing up both sides, what the costs would be practically, as I don't
> know the realities of running Gentoo infrastructure. And maybe the costs
> is just too high of a price to pay.
> 
> I wonder if increased use of git repos rather than distributed tarballs
> could be part of a solution to those issues, although that could put quite
> a storage burden on every user. Unless they were all shallow git pulls and
> the user could optionally choose to tar up the git directory after clone
> with compression.  But yes granted then there is even more ebuild
> complexity.
> 

Should we convert git repositories to Mercurial and Bazaar too, to avoid
relying too much on a single tool?

-- 
Best regards,
Michał Górny


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 512 bytes --]

Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo

[Eddie Chapman]

Michał Górny wrote:
> On Sat, 2024-03-30 at 15:17 +0000, Eddie Chapman wrote:
>
>> Michał Górny wrote:
>>
>>> On Sat, 2024-03-30 at 14:57 +0000, Eddie Chapman wrote:
>>>
>>>
>>>> Note, I'm not advocating ripping xz-utils out of tree, all I'm
>>>> saying is wouldn't it be nice if there were at least 2 alternatives
>>>> to choose from? That doesn't have to be disruptive in any way,
>>>> people who wish to continue using and trusting xz-utils should be
>>>> able to continue to do so without any friction whatsoever.
>>>
>>> So, you're basically saying we should go out of our way, recompress
>>> all distfiles using two alternative compression formats, increase
>>> mirror load four times and add a lot of complexity to ebuilds, right?
>>>
>>> --
>>> Best regards,
>>> Michał Górny
>>>
>>>
>>
>> Yes that's a very good point, that was something I was wondering in
>> weighing up both sides, what the costs would be practically, as I don't
>> know the realities of running Gentoo infrastructure. And maybe the
>> costs is just too high of a price to pay.
>>
>> I wonder if increased use of git repos rather than distributed tarballs
>>  could be part of a solution to those issues, although that could put
>> quite a storage burden on every user. Unless they were all shallow git
>> pulls and the user could optionally choose to tar up the git directory
>> after clone with compression.  But yes granted then there is even more
>> ebuild complexity.
>>
>
> Should we convert git repositories to Mercurial and Bazaar too, to avoid
> relying too much on a single tool?
>
> --
> Best regards,
> Michał Górny
>

I sense that question may have been slightly in jest :-) At least I hope
so as it could also be interpreted as an attempt at ridicule. I'll take it
as the former. In case you are seriously asking; of course not, that's
totally unnecessary. The objective is simply to obtain the upstream source
code intact. We don't need whatever version control of their source they
are using, which of course is the whole point of fetching distributed
tarballs. My suggestion of git pulls is just to address your point of
resource usage on gentoo infra, it reduces the need to store binary dist
files. I've also heard some argue that relying on distributed tarballs is
part of the overall problem and what the bad actor was taking advantage
of. They may have a point.

Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo

[Dale]

Eddie Chapman wrote:
> Michał Górny wrote:
>> On Sat, 2024-03-30 at 14:57 +0000, Eddie Chapman wrote:
>>
>>> Note, I'm not advocating ripping xz-utils out of tree, all I'm saying
>>> is wouldn't it be nice if there were at least 2 alternatives to choose
>>> from? That doesn't have to be disruptive in any way, people who wish to
>>> continue using and trusting xz-utils should be able to continue to do so
>>> without any friction whatsoever.
>> So, you're basically saying we should go out of our way, recompress all
>> distfiles using two alternative compression formats, increase mirror load
>> four times and add a lot of complexity to ebuilds, right?
>>
>> --
>> Best regards,
>> Michał Górny
>>
> Yes that's a very good point, that was something I was wondering in
> weighing up both sides, what the costs would be practically, as I don't
> know the realities of running Gentoo infrastructure. And maybe the costs
> is just too high of a price to pay.
>
> I wonder if increased use of git repos rather than distributed tarballs
> could be part of a solution to those issues, although that could put quite
> a storage burden on every user. Unless they were all shallow git pulls and
> the user could optionally choose to tar up the git directory after clone
> with compression.  But yes granted then there is even more ebuild
> complexity.
>
>
> .
>

There is a lot of unknowns out there.  From what I've read, the person
responsible for writing the code inserted this hack.  There may be no
way to prevent this.  Basically, the person that should have been
trusted with this code violated that trust.  Why is unknown but I'm as
curious about that as anything.  It's like when someone goes to a
grocery store to buy a tomato.  They want organic and there is a organic
sticker on the tomato.  You either trust that sticker, and the
person/company who put it on there, or you don't trust that sticker at
all and avoid buying all tomatoes.  The trust starts with the
person/company that puts that sticker on the tomato.  The person who was
trusted with that code, broke that trust.  There is likely hundreds of
packages out there in the exact same position.  Any package that has few
or only one person writing the code can do the same thing. 

While this should be analyzed as more info comes in, right now, we
should let the devs get us back to as safe a place as possible.  Since
it appears to affect systemd users who don't use Gentoo, which is a huge
target, they certainly need to react as quickly as they can to the devs
actions.  Let's just not overreact just yet.  The devs has rolled back
to a safe, safer, version.  Let time and more info sort this out. If it
is needed, xz will go away, which shouldn't come as a surprise.  I'm
sure the person who did this will never get that trust back. 

Long term, this is going to be interesting to see what all gets
revealed.  The why is one thing.  Another is how to prevent if it can be
at all. 

I'm going back to my hole now. 

Dale

:-)  :-) 

P. S.  Links that some may want to follow, instead of a -dev thread. 

https://bugs.gentoo.org/928134

https://forums.gentoo.org/viewtopic.php?p=8821925

Re[2]: [gentoo-dev] Current unavoidable use of xz utils in Gentoo

[Stefan Schmiedl]

[-- Attachment #1: Type: text/plain, Size: 1916 bytes --]

------ Original Message ------
From "Eddie Chapman" <eddie@ehuk.net>
To gentoo-dev@lists.gentoo.org
Date 30.03.2024 16:17:19
Subject Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo

>Michał Górny wrote:
>>  On Sat, 2024-03-30 at 14:57 +0000, Eddie Chapman wrote:
>>
>>>  Note, I'm not advocating ripping xz-utils out of tree, all I'm saying
>>>  is wouldn't it be nice if there were at least 2 alternatives to choose
>>>  from? That doesn't have to be disruptive in any way, people who wish to
>>>  continue using and trusting xz-utils should be able to continue to do so
>>>  without any friction whatsoever.
>>
>>  So, you're basically saying we should go out of our way, recompress all
>>  distfiles using two alternative compression formats, increase mirror load
>>  four times and add a lot of complexity to ebuilds, right?
>>
>>  --
>>  Best regards,
>>  Michał Górny
>>
>
>Yes that's a very good point, that was something I was wondering in
>weighing up both sides, what the costs would be practically, as I don't
>know the realities of running Gentoo infrastructure. And maybe the costs
>is just too high of a price to pay.
>
>I wonder if increased use of git repos rather than distributed tarballs
>could be part of a solution to those issues, although that could put quite
>a storage burden on every user. Unless they were all shallow git pulls and
>the user could optionally choose to tar up the git directory after clone
>with compression.  But yes granted then there is even more ebuild
>complexity.
>
Huh ... I read your original message as

"wouldn't it be nice to have at least 2 alternative [implementations of 
xz-utils]
to choose from"

As long as the file format itself is not inherently messed up, the 
archives
could stay as .xz, only a "minimal" unxz (similar to unrar) would be 
required
to access the contents.

Regards,
s.

[-- Attachment #2: Type: text/html, Size: 4875 bytes --]

Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo

[Eddie Chapman]

Rich Freeman wrote:
> On Sat, Mar 30, 2024 at 10:57 AM Eddie Chapman <eddie@ehuk.net> wrote:
>
>> No, this is the the bad actor *themselves* being a
>> principal author of the software, working stealthily and in very
>> sophisticated ways for years, to manoeuvrer themselves and their
>> software into a position of trust in the ecosystem whereby they were
>> almost able to pull off the mother of all security nightmares for the
>> world.
>
> This is entirely speculative at this point.  It isn't certain that the
> author is the one behind the exploit, and if they were, it is not known for
> how long their intentions were malicious, or even what their motivations
> were.  It is also unclear what pseudonymous accounts with what projects
> are associated with the attacker.

For the purposes of this discussion I'm not speculating nor interested in
*who* is behind this, or whether or whoever committed commits was a victim
of account takeover. Certain key actions that have been taken over time by
whoever is/was behind this do not require any speculation, they speak for
themselves, and are clearly malicious. There is no need to wait for
anything more to be revealed to be able to plainly see how bad it is.

While we wait and see, huge numbers of people might be suffering real and
serious consequences of continued use of xz-utils. The consequences may be
completely hidden, if you go by how well the bad actor here has managed to
hide what they have done. If you are following developments you can see
right now with your own eyes how incredibly difficult it is for our
smartest people to unravel and pick through what this actor has done. To
have faith that everything malicious that the perpetrator has done will be
unravelled over time by our collective smart minds by going over the
codebase with a fine tooth-comb puts far too much faith in human beings
and takes unnecessary risks for something that is not worth that risk when
there are alternatives. If you were looking for a compression tool for a
new project, why would anyone sane take such risks for such little gain?
You just wouldn't. Of course the reason there is hesitancy is because xz
has become so deeply entrenched in our world, it's become almost too hard
to extrapolate ourselves from it. I dare say the attacker realised this
and probably sought to take advantage of that fact.

However, I do acknowledge and realise the significant practical
difficulties that would be involved in making xz-utils something optional
within Gentoo.

Re: Re[2]: [gentoo-dev] Current unavoidable use of xz utils in Gentoo

[Eddie Chapman]

Stefan Schmiedl wrote:
> ------ Original Message ------
>
>> From "Eddie Chapman" <eddie@ehuk.net>
>>
> To gentoo-dev@lists.gentoo.org
> Date 30.03.2024 16:17:19
> Subject Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo
>
>> Michał Górny wrote:
>>
>>> On Sat, 2024-03-30 at 14:57 +0000, Eddie Chapman wrote:
>>>
>>>
>>>> Note, I'm not advocating ripping xz-utils out of tree, all I'm
>>>> saying is wouldn't it be nice if there were at least 2 alternatives
>>>> to choose from? That doesn't have to be disruptive in any way,
>>>> people who wish to continue using and trusting xz-utils should be
>>>> able to continue to do so without any friction whatsoever.
>>>
>>> So, you're basically saying we should go out of our way, recompress
>>> all distfiles using two alternative compression formats, increase
>>> mirror load four times and add a lot of complexity to ebuilds, right?
>>>
>>> --
>>> Best regards,
>>> Michał Górny
>>
>> Yes that's a very good point, that was something I was wondering in
>> weighing up both sides, what the costs would be practically, as I don't
>> know the realities of running Gentoo infrastructure. And maybe the
>> costs is just too high of a price to pay.
>>
>> I wonder if increased use of git repos rather than distributed tarballs
>>  could be part of a solution to those issues, although that could put
>> quite a storage burden on every user. Unless they were all shallow git
>> pulls and the user could optionally choose to tar up the git directory
>> after clone with compression.  But yes granted then there is even more
>> ebuild complexity.
>>
> Huh ... I read your original message as
>
> "wouldn't it be nice to have at least 2 alternative [implementations of
> xz-utils] to choose from"
>
> As long as the file format itself is not inherently messed up, the
> archives could stay as .xz, only a "minimal" unxz (similar to unrar) would
> be required to access the contents.
>
> Regards,
> s.

I see, no, I originally meant to have two compression/decompression
formats; LZMA (xz) and another. But yes the way you understood it is
interesting. Initially I dismissed this idea as would take too long for a
new tool to reach stability. But yes what you suggest is it could be a
very simple implementation that only does decompression so perhaps more
realistically achievable. Still a tall order. I wonder if any general
purpose languages (python, perl, etc) have developed their own built in
LZMA decompression implementation? I doubt it, would have thought they'd
just link against liblzma.so and not re-invent the wheel.

Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo

[Eddie Chapman]

Eddie Chapman wrote:
> Michał Górny wrote:
>
>> On Sat, 2024-03-30 at 14:57 +0000, Eddie Chapman wrote:
>>
>>
>>> Note, I'm not advocating ripping xz-utils out of tree, all I'm saying
>>>  is wouldn't it be nice if there were at least 2 alternatives to
>>> choose from? That doesn't have to be disruptive in any way, people who
>>> wish to continue using and trusting xz-utils should be able to
>>> continue to do so without any friction whatsoever.
>>
>> So, you're basically saying we should go out of our way, recompress all
>>  distfiles using two alternative compression formats, increase mirror
>> load four times and add a lot of complexity to ebuilds, right?
>>
>> --
>> Best regards,
>> Michał Górny
>>
> Yes that's a very good point, that was something I was wondering in
> weighing up both sides, what the costs would be practically, as I don't
> know the realities of running Gentoo infrastructure. And maybe the costs
> is just too high of a price to pay.
>
> I wonder if increased use of git repos rather than distributed tarballs
> could be part of a solution to those issues, although that could put quite
>  a storage burden on every user. Unless they were all shallow git pulls
> and the user could optionally choose to tar up the git directory after
> clone with compression.  But yes granted then there is even more ebuild
> complexity.
>

I've been thinking a little about how Gentoo without
compression/decompression of distfiles could work, as a feature, without
any impact on the existing world order, and no increased stress on Gentoo
infra. I was wondering how palatable the following idea might be to others
...

The basis of the idea is to add a feature to Portage which would let a
person optionally indicate in make.conf that whenever a path in SRC_URI
resolves to a file with a compression extension (.gz, .bz2, .xz, etc),
that Portage should attempt to fetch it without the compression extension.

So as an example, lets take sys-apps/pciutils, which currently has:
SRC_URI="https://mj.ucw.cz/download/linux/pci/${P}.tar.gz"

the feature would tell portage to simply translate this to:
SRC_URI="https://mj.ucw.cz/download/linux/pci/${P}.tar"

So perhaps it could be a flag that goes in FEATURES= called something like
"strip_dist_comp" or something similar, or maybe someone has a better idea
about that.

Now, of course, I'm not proposing that Gentoo infra keeps uncompressed
versions of distfiles. So by default Portage would encounter a 404 error
when it tries to fetch the uncompressed file from Gentoo mirrors.

However, this feature would then pave the way for a person to then
configure Portage to fetch distfiles from their own server as well as
Gentoo mirrors, and that person could then keep their own uncompressed
versions of distfiles on their  server, for however many and whichever
distfiles they might wish to keep there, as the compressed version would
get fetched from a Gentoo mirror if the uncompressed version is not there.
Such a person would then have to obtain or create their own uncompressed
distfile independently.

A caveat of this solution would be that one would have to disable checksum
verification (and gpg checks?) for this to work, as of course there would
be no checksum for the uncompressed version in the Manifest, and Gentoo
infra certainly should not be expected to especially uncompress each
distfile once in order to generate an extra checksum for the Manifest. In
fact I'd consider than undesirable, as anyone paranoid enough to want to
do this would not trust such a checksum anyway, since it would be a
checksum of a file that has been compressed at source and then
decompressed on Gentoo infra, potentially introducing vulnerabilities.
However, the lack of checksum is not a problem for someone who wants to
keep distfiles on their own server, as such a person can also be
responsible themselves for first verifying whatever they put on there, and
for keeping said server secured from tampering.

This seems to me to be something that would probably be relatively
straightforward to implement within Portage, maybe with just a few lines
around the python code that fetches the SRC_URI, and zero extra work or
resources required from Gentoo infra.

I'd consider it a feature for anyone who wants to eliminate a whole
potential class of vulnerabilities that may or may not be present either
now or in future in compression algorithm tools. Surely that would be a
nice feature to have for some folk?

Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo

[Sam James]

"Eddie Chapman" <eddie@ehuk.net> writes:

> Given what we've learnt in the last 24hrs about xz utilities, you could
> forgive a paranoid person for seriously considering getting rid entirely
> of them from their systems, especially since there are suitable
> alternatives available.  Some might say that's a bit extreme, xz-utils
> will get a thorough audit and it will all be fine. But when a malicious
> actor has been a key maintainer of something as complex as a decompression
> utility for years, I'm not sure I could ever trust that codebase again.
> Maybe a complete rewrite will emerge, but I'm personally unwilling to
> continue using xz utils in the meantime for uncompressing anything on my
> systems, even if it is done by an unprivileged process.

My own view is that there'll be a time for introspection, reflection,
and discussion of changes once the crisis is over. We're not there yet.

But I don't think us fetching several variants of compression formats
and testing & verifying all of them is feasible.

I also think it's (and I don't mean this derogatorily towards you) naive
for people in general to suggest that this is really specific to xz at
all. Unfortunately, there's many. many projects this could've happened to.

>
> I see that many system package ebuilds unconditionally expect
> app-arch/xz-utils to be installed simply to be able to decompress the
> source archive in SRC_URI. So simply specifying -lzma on your system isn't
> going to get rid of it.
>
> No one could have been expected to foresee what's happened with xz-utils,
> but now that it's here, perhaps Gentoo (and other projects that do) should
> consider not relying on a single decompression algorithm for source
> archives, even just as an insurance against some other yet unknown
> disaster with one algorithm or another in future?

I think there's real discussions to be had about relying on dist
tarballs and such but I don't really see how we could try to avoid
compression algorithms.

>
> And yes I'm sure there will be individual packages that currently
> absolutely need xz-utils installed during the build process, and one or
> two that absolutely have to have it available at runtime, but those
> bridges can be crossed as and when.
>
> Eddie

thanks,
sam

Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo

[Eli Schwartz]

[-- Attachment #1.1.1: Type: text/plain, Size: 1047 bytes --]

On 3/29/24 11:07 PM, Eddie Chapman wrote:
> Given what we've learnt in the last 24hrs about xz utilities, you could
> forgive a paranoid person for seriously considering getting rid entirely
> of them from their systems, especially since there are suitable
> alternatives available.  Some might say that's a bit extreme, xz-utils
> will get a thorough audit and it will all be fine. But when a malicious
> actor has been a key maintainer of something as complex as a decompression
> utility for years, I'm not sure I could ever trust that codebase again.
> Maybe a complete rewrite will emerge, but I'm personally unwilling to
> continue using xz utils in the meantime for uncompressing anything on my
> systems, even if it is done by an unprivileged process.


It suffices to downgrade to the version of xz before a social
engineering attack by a malicious actor to gain maintainership of the xz
project.

Have you been linked to this yet?
https://www.mail-archive.com/xz-devel@tukaani.org/msg00571.html


-- 
Eli Schwartz

[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 18399 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo

[Eli Schwartz]

[-- Attachment #1.1.1: Type: text/plain, Size: 1107 bytes --]

On 3/30/24 11:17 AM, Eddie Chapman wrote:
> Yes that's a very good point, that was something I was wondering in
> weighing up both sides, what the costs would be practically, as I don't
> know the realities of running Gentoo infrastructure. And maybe the costs
> is just too high of a price to pay.
> 
> I wonder if increased use of git repos rather than distributed tarballs
> could be part of a solution to those issues, although that could put quite
> a storage burden on every user. Unless they were all shallow git pulls and
> the user could optionally choose to tar up the git directory after clone
> with compression.  But yes granted then there is even more ebuild
> complexity.


Live ebuilds cannot have keywords, so using git repos is not a valid
option. There's not really much to discuss here.

Recompressing all distfiles in gentoo-specific ways is... definitely a
decision. It's a decision that Debian has made, mind you, so it's not
like Gentoo would be breaking new ground here, but frankly I don't
really regard that as fundamentally palatable.


-- 
Eli Schwartz

[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 18399 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Re[2]: [gentoo-dev] Current unavoidable use of xz utils in Gentoo

[Thomas Gall]

[-- Attachment #1: Type: text/plain, Size: 2956 bytes --]

A decompression implementation all in rust it would seem.

https://github.com/gendx/lzma-rs



On Sat, Mar 30, 2024 at 12:36 PM Eddie Chapman <eddie@ehuk.net> wrote:

> Stefan Schmiedl wrote:
> > ------ Original Message ------
> >
> >> From "Eddie Chapman" <eddie@ehuk.net>
> >>
> > To gentoo-dev@lists.gentoo.org
> > Date 30.03.2024 16:17:19
> > Subject Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo
> >
> >> Michał Górny wrote:
> >>
> >>> On Sat, 2024-03-30 at 14:57 +0000, Eddie Chapman wrote:
> >>>
> >>>
> >>>> Note, I'm not advocating ripping xz-utils out of tree, all I'm
> >>>> saying is wouldn't it be nice if there were at least 2 alternatives
> >>>> to choose from? That doesn't have to be disruptive in any way,
> >>>> people who wish to continue using and trusting xz-utils should be
> >>>> able to continue to do so without any friction whatsoever.
> >>>
> >>> So, you're basically saying we should go out of our way, recompress
> >>> all distfiles using two alternative compression formats, increase
> >>> mirror load four times and add a lot of complexity to ebuilds, right?
> >>>
> >>> --
> >>> Best regards,
> >>> Michał Górny
> >>
> >> Yes that's a very good point, that was something I was wondering in
> >> weighing up both sides, what the costs would be practically, as I don't
> >> know the realities of running Gentoo infrastructure. And maybe the
> >> costs is just too high of a price to pay.
> >>
> >> I wonder if increased use of git repos rather than distributed tarballs
> >>  could be part of a solution to those issues, although that could put
> >> quite a storage burden on every user. Unless they were all shallow git
> >> pulls and the user could optionally choose to tar up the git directory
> >> after clone with compression.  But yes granted then there is even more
> >> ebuild complexity.
> >>
> > Huh ... I read your original message as
> >
> > "wouldn't it be nice to have at least 2 alternative [implementations of
> > xz-utils] to choose from"
> >
> > As long as the file format itself is not inherently messed up, the
> > archives could stay as .xz, only a "minimal" unxz (similar to unrar)
> would
> > be required to access the contents.
> >
> > Regards,
> > s.
>
> I see, no, I originally meant to have two compression/decompression
> formats; LZMA (xz) and another. But yes the way you understood it is
> interesting. Initially I dismissed this idea as would take too long for a
> new tool to reach stability. But yes what you suggest is it could be a
> very simple implementation that only does decompression so perhaps more
> realistically achievable. Still a tall order. I wonder if any general
> purpose languages (python, perl, etc) have developed their own built in
> LZMA decompression implementation? I doubt it, would have thought they'd
> just link against liblzma.so and not re-invent the wheel.
>
>
>

[-- Attachment #2: Type: text/html, Size: 4034 bytes --]

Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo

[Eddie Chapman]

Eli Schwartz wrote:
> On 3/29/24 11:07 PM, Eddie Chapman wrote:
>
>> Given what we've learnt in the last 24hrs about xz utilities, you could
>>  forgive a paranoid person for seriously considering getting rid
>> entirely of them from their systems, especially since there are suitable
>>  alternatives available.  Some might say that's a bit extreme, xz-utils
>>  will get a thorough audit and it will all be fine. But when a
>> malicious actor has been a key maintainer of something as complex as a
>> decompression utility for years, I'm not sure I could ever trust that
>> codebase again. Maybe a complete rewrite will emerge, but I'm personally
>> unwilling to continue using xz utils in the meantime for uncompressing
>> anything on my systems, even if it is done by an unprivileged process.
>
>
> It suffices to downgrade to the version of xz before a social
> engineering attack by a malicious actor to gain maintainership of the xz
> project.
>
> Have you been linked to this yet?
> https://www.mail-archive.com/xz-devel@tukaani.org/msg00571.html
>
> --
> Eli Schwartz
>

Yes I saw that yesterday. It only increased my level of concern about the
project ten-fold rather than decreased it, particularly because of "he has
been helping a lot off-list and is practically a co-maintainer already".
It's not possible to just downgrade to before the bad actor's commits and
then feel fine about things because they have been heavily involved
offline even before commit access. We'll never know how much and when
because I also cannot trust what the apparently innocent maintainer (who
is most likely a victim here as well) might say about that now. Not
because of anything about them (I don't know them or anything about them),
just because of what has happened, there is too much of an incentive for
that person to now downplay the involvement of the bad actor.  I'm sorry
if that may seem harsh but, in my view, this situation is so severe it
warrants it. The world is facing threats from very sophisticated and
capable bad actors, mostly criminal organisations. If people here want to
run systems that are actually secure and also have other people trust
their stewardship, then things need to be taken seriously and high
standards need to be maintained. Especially where it is a tool that is not
super essential (it has just become heavily entrenched) and where there
are great alternatives, there should be no hesitancy to jettison a project
that has been infiltrated to such an extent as we have seen here (this is
far beyond just some devs workstation got compromised and there was a few
bad commits made it into the repo). At the moment there is far too much of
a cavalier attitude about the whole thing being shown by too many,
including here I'm sad to see.

Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo

[stefan11111]

On 2024-03-31 01:33, Eli Schwartz wrote:
> On 3/29/24 11:07 PM, Eddie Chapman wrote:
>> Given what we've learnt in the last 24hrs about xz utilities, you 
>> could
>> forgive a paranoid person for seriously considering getting rid 
>> entirely
>> of them from their systems, especially since there are suitable
>> alternatives available.  Some might say that's a bit extreme, xz-utils
>> will get a thorough audit and it will all be fine. But when a 
>> malicious
>> actor has been a key maintainer of something as complex as a 
>> decompression
>> utility for years, I'm not sure I could ever trust that codebase 
>> again.
>> Maybe a complete rewrite will emerge, but I'm personally unwilling to
>> continue using xz utils in the meantime for uncompressing anything on 
>> my
>> systems, even if it is done by an unprivileged process.
> 
> 
> It suffices to downgrade to the version of xz before a social
> engineering attack by a malicious actor to gain maintainership of the 
> xz
> project.
> 
> Have you been linked to this yet?
> https://www.mail-archive.com/xz-devel@tukaani.org/msg00571.html

> Wed, 29 Jun 2022 13:07:07 -0700

This is 2 years ago.

Had I seen someone say that a bad actor would spend years gaining the 
trust of FOSS
project maintainers in order to gain commit access and introduce such 
sophisticated
back doors, I would have told them to take their meds.
This is insane.

Not even this seems impossible anymore:
https://01.me/en/2014/11/insert-backdoor-into-compiler/

If this happened to something like firefox, I don't think anyone would 
have found out.
No one bats an eye if a website loads 0.5s longer.

-- 
Linux-gentoo-x86_64-Intel-R-_Core-TM-_i5-7400_CPU_@_3.00GHz

COMMON_FLAGS="-O3 -pipe -march=native -fno-stack-protector 
-ftree-vectorize -ffast-math -funswitch-loops -fuse-linker-plugin -flto 
-fdevirtualize-at-ltrans -fno-plt -fno-semantic-interposition 
-falign-functions=64 -fgraphite-identity -floop-nest-optimize"

USE="-* git verify-sig rsync-verify man alsa X grub ssl ipv6 lto 
libressl olde-gentoo asm native-symlinks threads jit jumbo-build minimal 
strip system-man"

INSTALL_MASK="/etc/systemd /lib/systemd /usr/lib/systemd 
/usr/lib/modules-load.d /usr/lib/tmpfiles.d *tmpfiles* /var/lib/dbus 
/lib/udev /usr/share/icons /usr/share/applications 
/usr/share/gtk-3.0/emoji"

Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo

[Matt Jolly]

Hi Eddie,

On 31/3/24 21:13, Eddie Chapman wrote:
> At the moment there is far too much of
> a cavalier attitude about the whole thing being shown by too many,
> including here I'm sad to see.

It's obvious that this is something that you are very worried about, but 
I think that you need to take a deep breath and relax a little. I have 
not seen a cavalier attitude towards this issue.

What I see instead are developers who have made an initial assessment 
and disclosure, have taken some actions to mitigate the severity of the 
issue, and are carefully continuing their investigations (over a major 
holiday in large parts of the world) so that they can issue sane 
responses to actual threats, and not a knee-jerk response like 'Gentoo 
should re-encode all xzs in other formats' which, as was discussed 
above, adds significant complexity for no real benefit.

I've seen from your previous emails to the list that you know what 
paragraphs are and how to use them to break up your content into 
digestible chunks. Please continue using them - it makes it 
significantly easier to respond to your ideas and gives off an aura of 
professionalism that you will need if you want your concerns to be taken 
seriously and addressed directly.

Cheers,

Matt

Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo

[Eddie Chapman]

Matt Jolly wrote:
> Hi Eddie,
>
> On 31/3/24 21:13, Eddie Chapman wrote:
>
>> At the moment there is far too much of
>> a cavalier attitude about the whole thing being shown by too many,
>> including here I'm sad to see.
>
> It's obvious that this is something that you are very worried about, but
> I think that you need to take a deep breath and relax a little. I have
> not seen a cavalier attitude towards this issue.

No, I don't need to do that. I don't appreciate suggestions to "just calm
down", especially when I'm not being hysterical.  Your comment to me just
reinforces what I mean when I say there is far too much of a cavalier
attitude.

> What I see instead are developers who have made an initial assessment
> and disclosure, have taken some actions to mitigate the severity of the
> issue, and are carefully continuing their investigations (over a major
> holiday in large parts of the world) so that they can issue sane responses
> to actual threats, and not a knee-jerk response like 'Gentoo should
> re-encode all xzs in other formats' which, as was discussed above, adds
> significant complexity for no real benefit.

I stand by and reiterate my view that there is far too much of a cavalier
attitude towards the matter in general out there including here in Gentoo.
But not in particular here, it is everywhere where this is being discussed
at the moment.

But please think a little about what I mean when I say a "cavalier
attitude", and what it does NOT mean. It does not mean that a lot of
people are not working very hard to analyse and learn about this issue and
taking steps to try to mitigate it. It does not mean people are not well
intentioned, everyone wants to fix this. I have great appreciation and
admiration for a lot of fantastic work I see going on including by people
involved in Gentoo. But I believe it will only really be beneficial in the
far future, not right now.

How are people in general being cavalier? By trying desperately to salvage
the situation with xz-utils above all else, by focussing too much on how
the original author of xz-utils and rallying round them (absolutely a
great thing to do but has absolutely nothing to do with what is good or
not good for users as a whole right now), there is too much clouded
judgment. There is more I could argue about why I use that word, but I
know by now that I am going against the grain of what the majority want
and it's not what people want to hear so I'm done, this discussion is now
a waste of everyone's time here including mine.

> I've seen from your previous emails to the list that you know what
> paragraphs are and how to use them to break up your content into digestible
> chunks. Please continue using them - it makes it significantly easier to
> respond to your ideas and gives off an aura of professionalism that you
> will need if you want your concerns to be taken seriously and addressed
> directly.

Yes you are right, I do apologise for not using paragraphs in my last
message, I slipped there, thanks for pointing it out. I've tried to do so
in this message.

> Cheers,
>
> Matt

And I should make more of an effort to sign off, it's a little more
friendly :-)

Regards,
Eddie

Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo

[Eli Schwartz]

[-- Attachment #1.1.1: Type: text/plain, Size: 4985 bytes --]

On 4/1/24 3:57 AM, Eddie Chapman wrote:
> No, I don't need to do that. I don't appreciate suggestions to "just calm
> down", especially when I'm not being hysterical.  Your comment to me just
> reinforces what I mean when I say there is far too much of a cavalier
> attitude.


I think you're making a big mistake by confusing "approach the issue
with a calm and clearheaded approach, be methodical about how you
analyze and react to trouble spots" with

"everyone is being cavalier".

But also, please keep in mind that 98% of all people on the internet can
do whatever they want and it simply doesn't matter. They are public
commentators at a three-ring circus and their cavalier or panicked
attitudes change nothing.

Well, they change one thing. It's hard for the security professionals at
work to deal with things when they are constantly having to respond to
the three-ring circus.


> I stand by and reiterate my view that there is far too much of a cavalier
> attitude towards the matter in general out there including here in Gentoo.
> But not in particular here, it is everywhere where this is being discussed
> at the moment.


I don't care where this is being "discussed", scare quotes intentional.


> But please think a little about what I mean when I say a "cavalier
> attitude", and what it does NOT mean. It does not mean that a lot of
> people are not working very hard to analyse and learn about this issue and
> taking steps to try to mitigate it. It does not mean people are not well
> intentioned, everyone wants to fix this. I have great appreciation and
> admiration for a lot of fantastic work I see going on including by people
> involved in Gentoo. But I believe it will only really be beneficial in the
> far future, not right now.


Please stop insulting the work of the people who are working very hard
to analyze and learn about this issue and taking steps to try to
mitigate it...



> How are people in general being cavalier? By trying desperately to salvage
> the situation with xz-utils above all else, by focussing too much on how
> the original author of xz-utils and rallying round them (absolutely a
> great thing to do but has absolutely nothing to do with what is good or
> not good for users as a whole right now), there is too much clouded
> judgment. There is more I could argue about why I use that word, but I
> know by now that I am going against the grain of what the majority want
> and it's not what people want to hear so I'm done, this discussion is now
> a waste of everyone's time here including mine.


... by implying that people who are NOT part of that process "rallying
around the original author" (an act of human compassion!!! which you
admit is a good thing) is, somehow, detrimental to the process of
working very hard to analyze and learn about this issue and taking steps
to try to mitigate it.

What does one have to do with the other? Why is it necessary to claim
that based on some sort of vibe check "there is too much compassion
going around in our communities, and this must mean that not enough
effort is being expended on the technical and cleanup aspects"?

...

Reading in between the lines, e.g. "trying desperately to salvage the
situation with xz-utils", I suspect you are trying to subtly suggest
that any second of time where gentoo hasn't yet removed xz-utils from
gentoo as a dead end is "cavalier".

Considering the fact that xz-utils is widely used and on the critpath
for people to actually get work done, including to actually acquire
extremely important software that already exists and must somehow be
dealt with, I do indeed think that the situation needs salvaging and the
community needs some form of xz decompressor. Fortunately, as you've
agreed, we know the original xz-utils circa 2020 and before is
trustworthy, so using that is viable and under discussion.


I understand that you are passionate about your suggestion to make
portage not validate distfile hashes. But I don't understand how you
think it's a solution to the xz-utils problem. For a wide variety of
reasons, but the simplest one is that your proposal has zero plan of
action for solving this at the community level and is entirely designed
to allow "lone wolf" users to use throwaway systems performing
security-sensitive actions (decompressing and hosting distfiles) in a
networked environment that has the xz-utils installed, to feed into
other security-sensitive systems (daily drivers etc.) that don't, but do
have to trust the artifacts produced by the former.

It's not being cavalier when zero portage developers responded by saying
"good idea I'll drop everything so I can get right on this and implement
it".

But if you are absolutely positive this is the right solution, I have an
offer for you: implement this yourself, submit patches, and then we'll
have something to talk about.



-- 
Eli Schwartz

[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 18399 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 873 bytes --]

On Mon, 2024-04-01 at 08:57 +0100, Eddie Chapman wrote:
> I stand by and reiterate my view that there is far too much of a cavalier
> attitude towards the matter in general out there including here in Gentoo.
> But not in particular here, it is everywhere where this is being discussed
> at the moment.

I would like to point out that the xz/sshd issue was primarily a social
one, not a technical one.

The primary problem in open source today isn't bad code.  It's projects
relying on overburdened, burned out maintainers.  And on top of that,
users who are complaining, demanding, outright hostile or primarily
contributing by walls of text on a mailing lists, that bring nothing to
discussion except for furthering the burnout of open source developers
who are actually trying to do something.

Think about that.

-- 
Best regards,
Michał Górny


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 512 bytes --]

Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo

[Azamat Hackimov]

сб, 30 мар. 2024 г. в 06:07, Eddie Chapman <eddie@ehuk.net>:
>
> Given what we've learnt in the last 24hrs about xz utilities, you could
> forgive a paranoid person for seriously considering getting rid entirely
> of them from their systems, especially since there are suitable
> alternatives available.  Some might say that's a bit extreme, xz-utils
> will get a thorough audit and it will all be fine. But when a malicious
> actor has been a key maintainer of something as complex as a decompression
> utility for years, I'm not sure I could ever trust that codebase again.
> Maybe a complete rewrite will emerge, but I'm personally unwilling to
> continue using xz utils in the meantime for uncompressing anything on my
> systems, even if it is done by an unprivileged process.
>
> I see that many system package ebuilds unconditionally expect
> app-arch/xz-utils to be installed simply to be able to decompress the
> source archive in SRC_URI. So simply specifying -lzma on your system isn't
> going to get rid of it.
>
> No one could have been expected to foresee what's happened with xz-utils,
> but now that it's here, perhaps Gentoo (and other projects that do) should
> consider not relying on a single decompression algorithm for source
> archives, even just as an insurance against some other yet unknown
> disaster with one algorithm or another in future?
>
> And yes I'm sure there will be individual packages that currently
> absolutely need xz-utils installed during the build process, and one or
> two that absolutely have to have it available at runtime, but those
> bridges can be crossed as and when.
>
> Eddie
>
>

There is no problem in the XZ/LZMA format itself as the reference
algorithm is not compromised. It's all about trust between developers
of application and developers of distribution. If you lost trust to
xz-utils's developers, you may use alternatives like app-arch/pxz or
app-arch/pixz. I don't see reasons why we should change format instead
of changing a tool.


--
From Siberia with Love!

Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo

[Kenton Groombridge]

[-- Attachment #1: Type: text/plain, Size: 2573 bytes --]

On 24/03/31 12:13PM, Eddie Chapman wrote:
> Eli Schwartz wrote:
> > On 3/29/24 11:07 PM, Eddie Chapman wrote:
> >
> >> Given what we've learnt in the last 24hrs about xz utilities, you could
> >>  forgive a paranoid person for seriously considering getting rid
> >> entirely of them from their systems, especially since there are suitable
> >>  alternatives available.  Some might say that's a bit extreme, xz-utils
> >>  will get a thorough audit and it will all be fine. But when a
> >> malicious actor has been a key maintainer of something as complex as a
> >> decompression utility for years, I'm not sure I could ever trust that
> >> codebase again. Maybe a complete rewrite will emerge, but I'm personally
> >> unwilling to continue using xz utils in the meantime for uncompressing
> >> anything on my systems, even if it is done by an unprivileged process.
> >
> >
> > It suffices to downgrade to the version of xz before a social
> > engineering attack by a malicious actor to gain maintainership of the xz
> > project.
> >
> > Have you been linked to this yet?
> > https://www.mail-archive.com/xz-devel@tukaani.org/msg00571.html
> >
> > --
> > Eli Schwartz
> >
> 
> Yes I saw that yesterday. It only increased my level of concern about the
> project ten-fold rather than decreased it, particularly because of "he has
> been helping a lot off-list and is practically a co-maintainer already".
> 
> 

I think it's important to realize that this could have potentially
happened to any number of various open source projects, not just
xz-utils. Simply ripping it out and replacing it is not enough to
prevent these kinds of issues from happening in the future.

There is a major shifting of perspectives as a result of this
unfortunate compromise. Right now there are serious considerations about
banning (or otherwise auditing) binary blobs in some projects. There are
talks about banning the use of older build systems like autotools in
favor of ones more easily readable and auditable. Ultimately what is
happening is a reflection on how we audit critical system components and
contributions made to them. Change is not going to happen over night.

We saw a similar shift with OpenSSL's heartbleed, which ultimately led
to positive changes in code quality and improving their vulnerability
reporting process.

There is some good to come of this event, but it's important to
recognize what went wrong and how open source can improve as a whole.

-- 
Kenton Groombridge
Gentoo Linux Developer, SELinux Project

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 963 bytes --]

Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo

[orbea]

On Mon, 1 Apr 2024 11:14:15 -0400
Kenton Groombridge <concord@gentoo.org> wrote:

> On 24/03/31 12:13PM, Eddie Chapman wrote:
> > Eli Schwartz wrote:  
> > > On 3/29/24 11:07 PM, Eddie Chapman wrote:
> > >  
> > >> Given what we've learnt in the last 24hrs about xz utilities,
> > >> you could forgive a paranoid person for seriously considering
> > >> getting rid entirely of them from their systems, especially
> > >> since there are suitable alternatives available.  Some might say
> > >> that's a bit extreme, xz-utils will get a thorough audit and it
> > >> will all be fine. But when a malicious actor has been a key
> > >> maintainer of something as complex as a decompression utility
> > >> for years, I'm not sure I could ever trust that codebase again.
> > >> Maybe a complete rewrite will emerge, but I'm personally
> > >> unwilling to continue using xz utils in the meantime for
> > >> uncompressing anything on my systems, even if it is done by an
> > >> unprivileged process.  
> > >
> > >
> > > It suffices to downgrade to the version of xz before a social
> > > engineering attack by a malicious actor to gain maintainership of
> > > the xz project.
> > >
> > > Have you been linked to this yet?
> > > https://www.mail-archive.com/xz-devel@tukaani.org/msg00571.html
> > >
> > > --
> > > Eli Schwartz
> > >  
> > 
> > Yes I saw that yesterday. It only increased my level of concern
> > about the project ten-fold rather than decreased it, particularly
> > because of "he has been helping a lot off-list and is practically a
> > co-maintainer already".
> > 
> >   
> 
> I think it's important to realize that this could have potentially
> happened to any number of various open source projects, not just
> xz-utils. Simply ripping it out and replacing it is not enough to
> prevent these kinds of issues from happening in the future.
> 
> There is a major shifting of perspectives as a result of this
> unfortunate compromise. Right now there are serious considerations
> about banning (or otherwise auditing) binary blobs in some projects.
> There are talks about banning the use of older build systems like
> autotools in favor of ones more easily readable and auditable.

Talk about throwing the baby out with the bathwater...

Its fully possible to write autotools build systems which are simple
and easy to audit. Depending on what blob does it may be far from
trivial or advisable to get rid of it.

This attack as already has been clearly stated is social, not
technical. If xz-utils used meson or cmake instead it would of not
changed the situation.

> Ultimately what is happening is a reflection on how we audit critical
> system components and contributions made to them. Change is not going
> to happen over night.
> 
> We saw a similar shift with OpenSSL's heartbleed, which ultimately led
> to positive changes in code quality and improving their vulnerability
> reporting process.
> 
> There is some good to come of this event, but it's important to
> recognize what went wrong and how open source can improve as a whole.
> 

Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo

[Kenton Groombridge]

[-- Attachment #1: Type: text/plain, Size: 3635 bytes --]

On 24/04/01 08:40AM, orbea wrote:
> On Mon, 1 Apr 2024 11:14:15 -0400
> Kenton Groombridge <concord@gentoo.org> wrote:
> 
> > On 24/03/31 12:13PM, Eddie Chapman wrote:
> > > Eli Schwartz wrote:  
> > > > On 3/29/24 11:07 PM, Eddie Chapman wrote:
> > > >  
> > > >> Given what we've learnt in the last 24hrs about xz utilities,
> > > >> you could forgive a paranoid person for seriously considering
> > > >> getting rid entirely of them from their systems, especially
> > > >> since there are suitable alternatives available.  Some might say
> > > >> that's a bit extreme, xz-utils will get a thorough audit and it
> > > >> will all be fine. But when a malicious actor has been a key
> > > >> maintainer of something as complex as a decompression utility
> > > >> for years, I'm not sure I could ever trust that codebase again.
> > > >> Maybe a complete rewrite will emerge, but I'm personally
> > > >> unwilling to continue using xz utils in the meantime for
> > > >> uncompressing anything on my systems, even if it is done by an
> > > >> unprivileged process.  
> > > >
> > > >
> > > > It suffices to downgrade to the version of xz before a social
> > > > engineering attack by a malicious actor to gain maintainership of
> > > > the xz project.
> > > >
> > > > Have you been linked to this yet?
> > > > https://www.mail-archive.com/xz-devel@tukaani.org/msg00571.html
> > > >
> > > > --
> > > > Eli Schwartz
> > > >  
> > > 
> > > Yes I saw that yesterday. It only increased my level of concern
> > > about the project ten-fold rather than decreased it, particularly
> > > because of "he has been helping a lot off-list and is practically a
> > > co-maintainer already".
> > > 
> > >   
> > 
> > I think it's important to realize that this could have potentially
> > happened to any number of various open source projects, not just
> > xz-utils. Simply ripping it out and replacing it is not enough to
> > prevent these kinds of issues from happening in the future.
> > 
> > There is a major shifting of perspectives as a result of this
> > unfortunate compromise. Right now there are serious considerations
> > about banning (or otherwise auditing) binary blobs in some projects.
> > There are talks about banning the use of older build systems like
> > autotools in favor of ones more easily readable and auditable.
> 
> Talk about throwing the baby out with the bathwater...
> 

Let's not shoot the messenger here. :)

I cited this specific example to highlight the shared intent behind
positive changes to auditing code not just in the program but also its
build system. I didn't mean to imply that this was a great solution.

> Its fully possible to write autotools build systems which are simple
> and easy to audit. Depending on what blob does it may be far from
> trivial or advisable to get rid of it.
> 
> This attack as already has been clearly stated is social, not
> technical. If xz-utils used meson or cmake instead it would of not
> changed the situation.
> 
> > Ultimately what is happening is a reflection on how we audit critical
> > system components and contributions made to them. Change is not going
> > to happen over night.
> > 
> > We saw a similar shift with OpenSSL's heartbleed, which ultimately led
> > to positive changes in code quality and improving their vulnerability
> > reporting process.
> > 
> > There is some good to come of this event, but it's important to
> > recognize what went wrong and how open source can improve as a whole.
> > 
> 
> 

-- 
Kenton Groombridge
Gentoo Linux Developer, SELinux Project

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 963 bytes --]

Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo

[orbea]

On Mon, 1 Apr 2024 12:01:13 -0400
Kenton Groombridge <concord@gentoo.org> wrote:

> On 24/04/01 08:40AM, orbea wrote:
> > On Mon, 1 Apr 2024 11:14:15 -0400
> > Kenton Groombridge <concord@gentoo.org> wrote:
> >   
> > > On 24/03/31 12:13PM, Eddie Chapman wrote:  
> > > > Eli Schwartz wrote:    
> > > > > On 3/29/24 11:07 PM, Eddie Chapman wrote:
> > > > >    
> > > > >> Given what we've learnt in the last 24hrs about xz utilities,
> > > > >> you could forgive a paranoid person for seriously considering
> > > > >> getting rid entirely of them from their systems, especially
> > > > >> since there are suitable alternatives available.  Some might
> > > > >> say that's a bit extreme, xz-utils will get a thorough audit
> > > > >> and it will all be fine. But when a malicious actor has been
> > > > >> a key maintainer of something as complex as a decompression
> > > > >> utility for years, I'm not sure I could ever trust that
> > > > >> codebase again. Maybe a complete rewrite will emerge, but
> > > > >> I'm personally unwilling to continue using xz utils in the
> > > > >> meantime for uncompressing anything on my systems, even if
> > > > >> it is done by an unprivileged process.    
> > > > >
> > > > >
> > > > > It suffices to downgrade to the version of xz before a social
> > > > > engineering attack by a malicious actor to gain
> > > > > maintainership of the xz project.
> > > > >
> > > > > Have you been linked to this yet?
> > > > > https://www.mail-archive.com/xz-devel@tukaani.org/msg00571.html
> > > > >
> > > > > --
> > > > > Eli Schwartz
> > > > >    
> > > > 
> > > > Yes I saw that yesterday. It only increased my level of concern
> > > > about the project ten-fold rather than decreased it,
> > > > particularly because of "he has been helping a lot off-list and
> > > > is practically a co-maintainer already".
> > > > 
> > > >     
> > > 
> > > I think it's important to realize that this could have potentially
> > > happened to any number of various open source projects, not just
> > > xz-utils. Simply ripping it out and replacing it is not enough to
> > > prevent these kinds of issues from happening in the future.
> > > 
> > > There is a major shifting of perspectives as a result of this
> > > unfortunate compromise. Right now there are serious considerations
> > > about banning (or otherwise auditing) binary blobs in some
> > > projects. There are talks about banning the use of older build
> > > systems like autotools in favor of ones more easily readable and
> > > auditable.  
> > 
> > Talk about throwing the baby out with the bathwater...
> >   
> 
> Let's not shoot the messenger here. :)
> 
> I cited this specific example to highlight the shared intent behind
> positive changes to auditing code not just in the program but also its
> build system. I didn't mean to imply that this was a great solution.

Thanks for clarifying that, it wasn't clear to me when I read the
earlier e-mail.

Personally I think the long term solution is to identify critical code
bases that have a low bus factor before the bad actors do and make a
concentrated community effort to help audit and maintain these code
bases.

> 
> > Its fully possible to write autotools build systems which are simple
> > and easy to audit. Depending on what blob does it may be far from
> > trivial or advisable to get rid of it.
> > 
> > This attack as already has been clearly stated is social, not
> > technical. If xz-utils used meson or cmake instead it would of not
> > changed the situation.
> >   
> > > Ultimately what is happening is a reflection on how we audit
> > > critical system components and contributions made to them. Change
> > > is not going to happen over night.
> > > 
> > > We saw a similar shift with OpenSSL's heartbleed, which
> > > ultimately led to positive changes in code quality and improving
> > > their vulnerability reporting process.
> > > 
> > > There is some good to come of this event, but it's important to
> > > recognize what went wrong and how open source can improve as a
> > > whole. 
> > 
> >   
> 

Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo

[Kévin GASPARD DE RENEFORT]

> Thanks for clarifying that, it wasn't clear to me when I read the
> earlier e-mail.
>
> Personally I think the long term solution is to identify critical code
> bases that have a low bus factor before the bad actors do and make a
> concentrated community effort to help audit and maintain these code
> bases.

Hi,

I hope this is not a stupid suggestion, that is also my first mail here 
so if something does not suits habits feel free to tell me please, but 
after reading the whole topic here I did not find this suggestion.

It’s merely a proposition out of my mind, also something I know very 
little about.

---

I read Linus T. speaking about usage of AI nowadays, in the IT field and 
stating that is an awful idea to write code with it (at least, for now)… 
But not to ask an AI to read the code and try to found by this way 
security holes, bad habits, bugs and such.

Again, my skill and knowledge about AI, specially nowadays, is very 
small. But would take it lot of works to sets an AI to simple «read» 
codes to look for undesired stuff ? That won’t even modify anything, 
merely says : «Ah, found something weird, **here**.». Maybe, properly 
configured, it would have detected this social-hacking. Maybe not.

Since programming is a very hard works, specially when it’s about 
security and bug, I also have very poor programing skill, but since the 
whole purpose of a computer and it’s set of software is to do what an 
human could NOT do properly (like being attentives while reading dozens 
of hundreds line of code…) and automate stuff, it *seems* to perfectly 
suits this need.

I guess the process on Gentoo side while it’s about "packaging" is 
writing the good ebuild that download source code, compressed (and that 
is the whole problem here if I understand) and then unpack it, compile 
it, etc…

Could an AI reading the code could be a step somewhere ?

On other distribution I would say it needs to act **before** the package 
is made, while building it I guess, for Gentoo I do not know.

But that is not the job of Gentoo’s ebuild writer to check other 
projects code, that would be a non-sense ! Right ?

I’m curious of what an AI could bring in this subject.

If it’s a stupid suggestion, well, will keep reading this topic, very 
interesting. And sorry for the noise.

PS: Thanks for the works behind libre software, open-source and here, 
Gentoo. I trust you since I do not have knowledge to judge properly the 
works, but Gentoo is indeed one of the best Linux available, if not the 
best in some field. Don’t let burn-out takes you and keep your real 
priority among everything, even Gentoo or libre software. We are humans, 
not machines.

Regards,
GASPARD DE RENEFORT Kévin

Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo

[James Le Cuirot]

[-- Attachment #1: Type: text/plain, Size: 3585 bytes --]

On Mon, 2024-04-01 at 20:51 +0200, Kévin GASPARD DE RENEFORT wrote:
> > Thanks for clarifying that, it wasn't clear to me when I read the
> > earlier e-mail.
> > 
> > Personally I think the long term solution is to identify critical code
> > bases that have a low bus factor before the bad actors do and make a
> > concentrated community effort to help audit and maintain these code
> > bases.
> 
> Hi,
> 
> I hope this is not a stupid suggestion, that is also my first mail here 
> so if something does not suits habits feel free to tell me please, but 
> after reading the whole topic here I did not find this suggestion.
> 
> It’s merely a proposition out of my mind, also something I know very 
> little about.
> 
> ---
> 
> I read Linus T. speaking about usage of AI nowadays, in the IT field and 
> stating that is an awful idea to write code with it (at least, for now)… 
> But not to ask an AI to read the code and try to found by this way 
> security holes, bad habits, bugs and such.
> 
> Again, my skill and knowledge about AI, specially nowadays, is very 
> small. But would take it lot of works to sets an AI to simple «read» 
> codes to look for undesired stuff ? That won’t even modify anything, 
> merely says : «Ah, found something weird, **here**.». Maybe, properly 
> configured, it would have detected this social-hacking. Maybe not.
> 
> Since programming is a very hard works, specially when it’s about 
> security and bug, I also have very poor programing skill, but since the 
> whole purpose of a computer and it’s set of software is to do what an 
> human could NOT do properly (like being attentives while reading dozens 
> of hundreds line of code…) and automate stuff, it *seems* to perfectly 
> suits this need.
> 
> I guess the process on Gentoo side while it’s about "packaging" is 
> writing the good ebuild that download source code, compressed (and that 
> is the whole problem here if I understand) and then unpack it, compile 
> it, etc…
> 
> Could an AI reading the code could be a step somewhere ?
> 
> On other distribution I would say it needs to act **before** the package 
> is made, while building it I guess, for Gentoo I do not know.
> 
> But that is not the job of Gentoo’s ebuild writer to check other 
> projects code, that would be a non-sense ! Right ?
> 
> I’m curious of what an AI could bring in this subject.
> 
> If it’s a stupid suggestion, well, will keep reading this topic, very 
> interesting. And sorry for the noise.
> 
> PS: Thanks for the works behind libre software, open-source and here, 
> Gentoo. I trust you since I do not have knowledge to judge properly the 
> works, but Gentoo is indeed one of the best Linux available, if not the 
> best in some field. Don’t let burn-out takes you and keep your real 
> priority among everything, even Gentoo or libre software. We are humans, 
> not machines.
> 
> Regards,
> GASPARD DE RENEFORT Kévin

That's not stupid at all, I'd been thinking exactly the same thing. I raised
this whole issue during a discussion at FOSDEM 2019, where I admitted that I
didn't check the code changes for packages I was bumping, knowing that few to
none of the other people in the room did so either. Despite speaking up then,
I still didn't do it because it's a heavy a burden and I'm not paid to do it.
Now I'm thinking I really should, but I could really use some help. I'll raise
this idea at work. You could say that we specialise in these things. :)

Regards,
Chewi

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 858 bytes --]

Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo

[Joonas Niilola]

[-- Attachment #1.1: Type: text/plain, Size: 937 bytes --]

On 1.4.2024 23.07, James Le Cuirot wrote:
> 
> That's not stupid at all, I'd been thinking exactly the same thing. I raised
> this whole issue during a discussion at FOSDEM 2019, where I admitted that I
> didn't check the code changes for packages I was bumping, knowing that few to
> none of the other people in the room did so either. Despite speaking up then,
> I still didn't do it because it's a heavy a burden and I'm not paid to do it.
> Now I'm thinking I really should, but I could really use some help. I'll raise
> this idea at work. You could say that we specialise in these things. :)
> 
> Regards,
> Chewi

Offtopic but I'll just throw this out there: "pkgdiff-mg -b" from
mgorny-dev-scripts does wonders when bumping packages. Not everyone
knows about this so posting for awareness.

(Maybe slightly related after all since it would've shown the suspicious
CmakeLists.txt change at least)

-- juippis

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 618 bytes --]

Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo

[Eddie Chapman]

OK, I said I was done and this is a waste of time for everyone, but if
people want to keep the discussion going I'll bite :-)

Eli Schwartz wrote:
> But also, please keep in mind that 98% of all people on the internet can
> do whatever they want and it simply doesn't matter. They are public
> commentators at a three-ring circus and their cavalier or panicked
> attitudes change nothing.

I disagree, think it is very important to have discussions about what the
oss/linux community thinks, not just what they do. And I think those
discussions do significantly influence what is actually done, whether the
"doers" actually realise it or not.

> Well, they change one thing. It's hard for the security professionals at
> work to deal with things when they are constantly having to respond to the
> three-ring circus.

This is a complaint I hear very often from the people working at the heart
of things. Stop making noise, shut up, we're overworked here and dealing
with your "complaints" just adds to our stress. I do understand and
sympathise with those feelings, believe me I do, I feel them myself in
other contexts.

But I hope you understand this is not finding things to nitpick about for
the sake of it. Does the Gentoo dev community want people on the "outside"
to raise their concerns on their mailing list if those persons feel like
said community have got something very wrong, yes or no? If not then put a
note on the mailing list page saying "please don't bother us, we're too
overworked, just post patches" or something to that effect.

> Please stop insulting the work of the people who are working very hard
> to analyze and learn about this issue and taking steps to try to mitigate
> it...

I'm certainly not trying to insult anyone. I've expressed a lot of
appreciation for their work. I have *criticised* the prevailing view
though.

> What does one have to do with the other? Why is it necessary to claim
> that based on some sort of vibe check "there is too much compassion going
> around in our communities, and this must mean that not enough effort is
> being expended on the technical and cleanup aspects"?

I have not made such a claim, I've said I see lots of technical and
cleanup aspects. I've only stated the things that *are* happening versus
what is not happening at all (literally zilch) and which should be
happening, which is efforts towards a solution *not* involving the xz
utilities.

> Reading in between the lines, e.g. "trying desperately to salvage the
> situation with xz-utils", I suspect you are trying to subtly suggest that
> any second of time where gentoo hasn't yet removed xz-utils from gentoo as
> a dead end is "cavalier".

Not quite, I've never advocated removing xz-utils at all, more than happy
for it to remain for whoever wants to use it. The only reason I started
this thread is I'm very unhappy about that fact that it is currently
impossible to NOT execute xz utilities on the Gentoo systems I'm
responsible for, without heavy customisation.

I'm also not demanding anything, let alone demanding anything instantly.
If I have please point out where.

> I understand that you are passionate about your suggestion to make
> portage not validate distfile hashes.

That's incorrect, I've never suggested Portage should not validate
distfile hashes. The current behaviour is that validating distfile hashes
is something that can be disabled if a user wishes to, and I have no
problem with that at all, would not change a thing. I've said that, in
order to implement what I have suggested, a user would have to disable it,
which is not ideal, but acceptable if the user controls the distfile
distribution. And I only suggested that in order to try and make the idea
more acceptable by not requiring impractical infra changes that would be
needed to generate uncompressed hashes for the Manifests).

> But I don't understand how you think
> it's a solution to the xz-utils problem. For a wide variety of reasons,
> but the simplest one is that your proposal has zero plan of action for
> solving this at the community level and is entirely designed to allow
> "lone wolf" users to use throwaway systems performing
> security-sensitive actions (decompressing and hosting distfiles) in a
> networked environment that has the xz-utils installed, to feed into other
> security-sensitive systems (daily drivers etc.) that don't, but do have to
> trust the artifacts produced by the former.

I'm not entirely clear what you're trying to say in this paragraph. But
what I will say is I've tried very hard in any suggestions I've made to
only suggest things which will NOT change any default behaviour or require
big changes. The average user would not see any change from my revised
suggestions at all. I accepted after the first responses in this thread
that there was no appetite here to stop using xz utils. I then asked the
list about an idea I had just to see how palatable it might be. It was not
supposed to be a concrete plan, I was seeking discussion about how it
might be possible in practise for someone to use Gentoo without
compression and decompression of distfiles. I tried to suggest a solution
that could be an optional feature people could enable if they wanted it.

> It's not being cavalier when zero portage developers responded by saying
> "good idea I'll drop everything so I can get right on this and implement
> it".

I'll just point out that I've never expected nor asked for anyone to
unquestionably accept anything I've said, let alone in the way you have
characterised there that I might have done. I do think that the oss/linux
community as a whole including Gentoo developers should seriously consider
changing direction on this though. And I still think it is cavalier,
simply because by deciding on the current direction that is being taken,
very big (not an exaggeration) risks on behalf of all users are being
taken, while a much safer path for everyone is available but being
completely ignored.  I do acknowledge, though, as I have said before, that
this is far from easy in practise.

> But if you are absolutely positive this is the right solution, I have an
> offer for you: implement this yourself, submit patches, and then we'll have
> something to talk about.

That was always my ultimate intention, but only if I saw there was at
least some appetite for anything that might remotely look like what I was
suggesting. I don't see the point in developing and submitting anything
concrete to a community that has no desire for it in the first place.

Thanks,
Eddie

P.S. I've done a certain amount of "snipping" in my reply to try and
reduce the "wall of text" effect somewhat at least, apologies if you feel
I've taken anything out that I should not have, please let me know if so.

Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo

[Eddie Chapman]

Michał Górny wrote:
> On Mon, 2024-04-01 at 08:57 +0100, Eddie Chapman wrote:
>
>> I stand by and reiterate my view that there is far too much of a
>> cavalier attitude towards the matter in general out there including here
>> in Gentoo. But not in particular here, it is everywhere where this is
>> being discussed at the moment.
>
> I would like to point out that the xz/sshd issue was primarily a social
> one, not a technical one.
>
> The primary problem in open source today isn't bad code.  It's projects
> relying on overburdened, burned out maintainers.  And on top of that, users
> who are complaining, demanding, outright hostile or primarily contributing
> by walls of text on a mailing lists, that bring nothing to discussion
> except for furthering the burnout of open source developers who are
> actually trying to do something.
>
> Think about that.
>
>
> --
> Best regards,
> Michał Górny
>

I'm sorry for having contributed to your burnout. I have a lot of respect
for you personally Michał, the quality of your contributions to Gentoo are
outstanding, and have to admit I've often felt a little worried for you
with the amount of work you do. I don't know you at all, I hope you don't
mind me saying that. Don't worry I think it's quite unlikely I'll bring
any new concerns to this list again in future, I'll certainly think twice
about it.

regards,
Eddie

Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo

[Eddie Chapman]

On 01/04/2024 15:56, Azamat Hackimov wrote:
> There is no problem in the XZ/LZMA format itself as the reference
> algorithm is not compromised. It's all about trust between developers
> of application and developers of distribution. If you lost trust to
> xz-utils's developers, you may use alternatives like app-arch/pxz or
> app-arch/pixz. I don't see reasons why we should change format instead
> of changing a tool.
> 

Hello Azamat,

Yes, I have no issue with the format at all, just with the xz utils 
project.  But I was suggesting to have support for two compression 
algorithms as an improvement for the future, in case of some unknown 
other major problem with one of them emerges in future. I suppose kind 
of a similar reasoning, but not quite the same, that we have BLAKE2B and 
SHA512 hashes. But there are severe practical problems for Gentoo infra 
resources with having two of course.

Thank you for the pointer to app-arch/pxz and app-arch/pixz. I've had a 
close look at them both but sadly they are not suitable as they both 
rely on the xz utils project to do the main work. Once calls the xz exe 
and the other links against liblzma.

I have been looking around a bit since Friday at what true alternatives 
(no relying on liblzma) there are to just decompress existing XZ/LZMA 
binaries. There is p7zip which is a command line fork of 7zip that's 
been around a good while. However in the years since that fork was 
created the 7zip project themselves have begun doing source code 
releases with build instructions, with the command line tool apparently 
working fine.

On balance the upstream 7zip actually looks like a better option than 
p7zip now since p7zip maintenance has stagnated somewhat. On the one 
hand 7zip is actively developed, of course because of the large Windows 
userbase, and security fixes would be available immediately when a new 
release comes about (there were sec issues fixed in 7zip last year for 
example which didn't make it into p7zip in a timely fashion).  But on 
the other hand most distros have used p7zip and I've only seen Arch and 
Debian that currently package the 7zip releases, so the latest 7zip 
releases have had only minimum real world testing and code scrutiny in 
the Linux world (although it's likely much of the code will still be the 
same as what it was when p7zip was forked, so in that sense at least a 
significant portion of the code has had wider testing, in a manner of 
speaking). Still, I'm not sure about 7zip, doesn't seem ideal.

Thomas Gall, elsewhere in this thread, pointed out a pure Rust 
implementation which is interesting.
https://github.com/gendx/lzma-rs

The GH page says it supports decompression of "LZMA, LZMA2 and a subset 
of the .xz file format".

If anyone else knows of any other true alternatives please do let me 
know. I'm currently looking into the feasibility of hacking my Gentoo 
installations so that .xz distfiles are decompressed during the ebuild 
process using an alternative implementation, allowing me to get rid of 
xz utils.

Thanks,
Eddie

Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo

[Eli Schwartz]

[-- Attachment #1.1.1: Type: text/plain, Size: 10521 bytes --]

On 4/2/24 4:43 AM, Eddie Chapman wrote:
>> Well, they change one thing. It's hard for the security professionals at
>> work to deal with things when they are constantly having to respond to the
>> three-ring circus.
> 
> This is a complaint I hear very often from the people working at the heart
> of things. Stop making noise, shut up, we're overworked here and dealing
> with your "complaints" just adds to our stress. I do understand and
> sympathise with those feelings, believe me I do, I feel them myself in
> other contexts.
> 
> But I hope you understand this is not finding things to nitpick about for
> the sake of it. Does the Gentoo dev community want people on the "outside"
> to raise their concerns on their mailing list if those persons feel like
> said community have got something very wrong, yes or no? If not then put a
> note on the mailing list page saying "please don't bother us, we're too
> overworked, just post patches" or something to that effect.


I would be delighted to hear reasonable concerns. That's not what I'm
referring to by "three-ring circus".


>> What does one have to do with the other? Why is it necessary to claim
>> that based on some sort of vibe check "there is too much compassion going
>> around in our communities, and this must mean that not enough effort is
>> being expended on the technical and cleanup aspects"?
> 
> I have not made such a claim, I've said I see lots of technical and
> cleanup aspects. I've only stated the things that *are* happening versus
> what is not happening at all (literally zilch) and which should be
> happening, which is efforts towards a solution *not* involving the xz
> utilities.


You say that as though a solution *not* involving the xz utilities is a
reasonable takeaway from this scenario.

In order to demonstrate that such efforts deserve discussion at all, let
alone efforts towards solving it, you first need to prove that:

- the xz utilities as shipped by Gentoo are something that should be
  moved away from

- the xz utilities as released in 2022, when the backdoor author had
  just as much access as you or I -- that is, none, aside for the right
  to submit patches as suggestions -- are something that should be moved
  away from

You have made no effort to justify either approach aside for claiming
that for unidentified reasons you believe this scenario demonstrates
that the "apparently innocent maintainer" now has an incentive to
"downplay the involvement of the bad actor".

If you had, I would be infinitely more interested in what you have to
say on the topic.

Also, if you had started with such.


>> Reading in between the lines, e.g. "trying desperately to salvage the
>> situation with xz-utils", I suspect you are trying to subtly suggest that
>> any second of time where gentoo hasn't yet removed xz-utils from gentoo as
>> a dead end is "cavalier".
> 
> Not quite, I've never advocated removing xz-utils at all, more than happy
> for it to remain for whoever wants to use it. The only reason I started
> this thread is I'm very unhappy about that fact that it is currently
> impossible to NOT execute xz utilities on the Gentoo systems I'm
> responsible for, without heavy customisation.
> 
> I'm also not demanding anything, let alone demanding anything instantly.
> If I have please point out where.


Thank you for clarifying.

I am now just as convinced as I was yesterday, that you are trying to
subtly suggest it, but I'm newly convinced that you're also being
mendacious about it.

"It is cavalier for Gentoo to allow xz-utils as a package in the @system
set" is not meaningfully distinct from "it is cavalier for Gentoo to not
work to allow me to depclean xz-utils".


>> I understand that you are passionate about your suggestion to make
>> portage not validate distfile hashes.
> 
> That's incorrect, I've never suggested Portage should not validate
> distfile hashes. The current behaviour is that validating distfile hashes
> is something that can be disabled if a user wishes to, and I have no
> problem with that at all, would not change a thing. I've said that, in
> order to implement what I have suggested, a user would have to disable it,
> which is not ideal, but acceptable if the user controls the distfile
> distribution. And I only suggested that in order to try and make the idea
> more acceptable by not requiring impractical infra changes that would be
> needed to generate uncompressed hashes for the Manifests).


In other words, you didn't care to find a robust solution, you just want
something that you personally can use, and which requires being less
secure than using xz-utils.

But it's okay! It's not harassing portage devs with unreasonable
demands! Because it's *optional*, and by default people would just...
use xz-utils.

Even though the ***entire premise*** of changing anything here is that
xz-utils as shipped by Gentoo is somehow dangerous and users have a
valid reason to want to avoid it entirely.

If you're going to recommend a solution for users who consider xz-utils
to be dangerous, then that solution should, you know, be better than
using xz-utils.

Not worse and less secure.


>> But I don't understand how you think
>> it's a solution to the xz-utils problem. For a wide variety of reasons,
>> but the simplest one is that your proposal has zero plan of action for
>> solving this at the community level and is entirely designed to allow
>> "lone wolf" users to use throwaway systems performing
>> security-sensitive actions (decompressing and hosting distfiles) in a
>> networked environment that has the xz-utils installed, to feed into other
>> security-sensitive systems (daily drivers etc.) that don't, but do have to
>> trust the artifacts produced by the former.
> 
> I'm not entirely clear what you're trying to say in this paragraph.


I am sarcastically saying that your proposal makes things worse and less
secure for you, and doesn't even stop you from having to use xz-utils in
a context where a malicious xz-utils has the ability to inject attack
code into the uncompressed source code .tar files your proposal depends on.


> But
> what I will say is I've tried very hard in any suggestions I've made to
> only suggest things which will NOT change any default behaviour or require
> big changes. The average user would not see any change from my revised
> suggestions at all. I accepted after the first responses in this thread
> that there was no appetite here to stop using xz utils. I then asked the
> list about an idea I had just to see how palatable it might be. It was not
> supposed to be a concrete plan, I was seeking discussion about how it
> might be possible in practise for someone to use Gentoo without
> compression and decompression of distfiles. I tried to suggest a solution
> that could be an optional feature people could enable if they wanted it.


But you should be proposing a change in default behavior! If it's not a
change in default behavior then the change helps no one and there's no
point in making it.

If the change is good enough to make, it should be good enough to
propose its use by default, or at least as a recommended change people
should be encouraged to make while data is collected for rolling out the
change as a switched default.

If you want a change that only applies to your personal system, Gentoo
already has a solution: make an overlay that contains slightly modified
versions of every ebuild in gentoo that has a SRC_URI mentioning *.xz files.

No changes to portage needed. Host your own tarballs on your own server
in SRC_URI. Validate them with checksums, even.

...

Or you could suggest patches to portage that improve the sandbox used to
invoke decompression and archive extraction, so that it's "safe" to have
xz as a statically linked executable with no accompanying library
installed for the purpose of untrusted distfile extraction.


>> It's not being cavalier when zero portage developers responded by saying
>> "good idea I'll drop everything so I can get right on this and implement
>> it".
> 
> I'll just point out that I've never expected nor asked for anyone to
> unquestionably accept anything I've said, let alone in the way you have
> characterised there that I might have done. I do think that the oss/linux
> community as a whole including Gentoo developers should seriously consider
> changing direction on this though. And I still think it is cavalier,
> simply because by deciding on the current direction that is being taken,
> very big (not an exaggeration) risks on behalf of all users are being
> taken, while a much safer path for everyone is available but being
> completely ignored.  I do acknowledge, though, as I have said before, that
> this is far from easy in practise.


Sorry, but no.

When you say that the community's response is "cavalier" because the
community is not accepting what you've said, you are inherently working
off the belief that the community's failure to accept what you've said
is because the community is *wrong* to question your suggestions.

If you think the community is wrong to question your suggestions then
you should be prepared to defend that point of view, particularly when
they are busy trying to coordinate a security response together with
security professionals from a wide variety of other distros, commercial
vendors, OSS communities, etc. and have limited time to explain to
*hundreds* of people who each have their own badly thought out ideas
about what the FOSS community should do to solve the problem.

Your suggestion is only one such badly thought out idea. However, it
stands out from the rest because your suggestion has something that the
rest by and large lack: it has an accusation that distros, in this case
Gentoo, are being cavalier about security.


This attitude of "Gentoo is being cavalier about security" is
disproportionately worse than the average user interaction and, as has
been noted, is the reason why FOSS maintainers suffer burnout.

It has nothing to do with bringing up concerns. It has everything to do
with "if you don't agree with me you're being cavalier about MY security
as a Gentoo user".

Seriously. Please learn to bring up suggestions as suggestions, not as
demands. It makes all the difference in the world.


-- 
Eli Schwartz

[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 18399 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo

[Eddie Chapman]

On 02/04/2024 20:46, Eli Schwartz wrote:
> On 4/2/24 4:43 AM, Eddie Chapman wrote:
>>> Well, they change one thing. It's hard for the security professionals at
>>> work to deal with things when they are constantly having to respond to the
>>> three-ring circus.
>>
>> This is a complaint I hear very often from the people working at the heart
>> of things. Stop making noise, shut up, we're overworked here and dealing
>> with your "complaints" just adds to our stress. I do understand and
>> sympathise with those feelings, believe me I do, I feel them myself in
>> other contexts.
>>
>> But I hope you understand this is not finding things to nitpick about for
>> the sake of it. Does the Gentoo dev community want people on the "outside"
>> to raise their concerns on their mailing list if those persons feel like
>> said community have got something very wrong, yes or no? If not then put a
>> note on the mailing list page saying "please don't bother us, we're too
>> overworked, just post patches" or something to that effect.
> 
> 
> I would be delighted to hear reasonable concerns. That's not what I'm
> referring to by "three-ring circus".
> 
> 
>>> What does one have to do with the other? Why is it necessary to claim
>>> that based on some sort of vibe check "there is too much compassion going
>>> around in our communities, and this must mean that not enough effort is
>>> being expended on the technical and cleanup aspects"?
>>
>> I have not made such a claim, I've said I see lots of technical and
>> cleanup aspects. I've only stated the things that *are* happening versus
>> what is not happening at all (literally zilch) and which should be
>> happening, which is efforts towards a solution *not* involving the xz
>> utilities.
> 
> 
> You say that as though a solution *not* involving the xz utilities is a
> reasonable takeaway from this scenario.
> 
> In order to demonstrate that such efforts deserve discussion at all, let
> alone efforts towards solving it, you first need to prove that:
> 
> - the xz utilities as shipped by Gentoo are something that should be
>    moved away from
> 
> - the xz utilities as released in 2022, when the backdoor author had
>    just as much access as you or I -- that is, none, aside for the right
>    to submit patches as suggestions -- are something that should be moved
>    away from
> 
> You have made no effort to justify either approach aside for claiming
> that for unidentified reasons you believe this scenario demonstrates
> that the "apparently innocent maintainer" now has an incentive to
> "downplay the involvement of the bad actor".
> 
> If you had, I would be infinitely more interested in what you have to
> say on the topic.
> 
> Also, if you had started with such.
> 
> 
>>> Reading in between the lines, e.g. "trying desperately to salvage the
>>> situation with xz-utils", I suspect you are trying to subtly suggest that
>>> any second of time where gentoo hasn't yet removed xz-utils from gentoo as
>>> a dead end is "cavalier".
>>
>> Not quite, I've never advocated removing xz-utils at all, more than happy
>> for it to remain for whoever wants to use it. The only reason I started
>> this thread is I'm very unhappy about that fact that it is currently
>> impossible to NOT execute xz utilities on the Gentoo systems I'm
>> responsible for, without heavy customisation.
>>
>> I'm also not demanding anything, let alone demanding anything instantly.
>> If I have please point out where.
> 
> 
> Thank you for clarifying.
> 
> I am now just as convinced as I was yesterday, that you are trying to
> subtly suggest it, but I'm newly convinced that you're also being
> mendacious about it.
> 
> "It is cavalier for Gentoo to allow xz-utils as a package in the @system
> set" is not meaningfully distinct from "it is cavalier for Gentoo to not
> work to allow me to depclean xz-utils".
> 
> 
>>> I understand that you are passionate about your suggestion to make
>>> portage not validate distfile hashes.
>>
>> That's incorrect, I've never suggested Portage should not validate
>> distfile hashes. The current behaviour is that validating distfile hashes
>> is something that can be disabled if a user wishes to, and I have no
>> problem with that at all, would not change a thing. I've said that, in
>> order to implement what I have suggested, a user would have to disable it,
>> which is not ideal, but acceptable if the user controls the distfile
>> distribution. And I only suggested that in order to try and make the idea
>> more acceptable by not requiring impractical infra changes that would be
>> needed to generate uncompressed hashes for the Manifests).
> 
> 
> In other words, you didn't care to find a robust solution, you just want
> something that you personally can use, and which requires being less
> secure than using xz-utils.
> 
> But it's okay! It's not harassing portage devs with unreasonable
> demands! Because it's *optional*, and by default people would just...
> use xz-utils.
> 
> Even though the ***entire premise*** of changing anything here is that
> xz-utils as shipped by Gentoo is somehow dangerous and users have a
> valid reason to want to avoid it entirely.
> 
> If you're going to recommend a solution for users who consider xz-utils
> to be dangerous, then that solution should, you know, be better than
> using xz-utils.
> 
> Not worse and less secure.
> 
> 
>>> But I don't understand how you think
>>> it's a solution to the xz-utils problem. For a wide variety of reasons,
>>> but the simplest one is that your proposal has zero plan of action for
>>> solving this at the community level and is entirely designed to allow
>>> "lone wolf" users to use throwaway systems performing
>>> security-sensitive actions (decompressing and hosting distfiles) in a
>>> networked environment that has the xz-utils installed, to feed into other
>>> security-sensitive systems (daily drivers etc.) that don't, but do have to
>>> trust the artifacts produced by the former.
>>
>> I'm not entirely clear what you're trying to say in this paragraph.
> 
> 
> I am sarcastically saying that your proposal makes things worse and less
> secure for you, and doesn't even stop you from having to use xz-utils in
> a context where a malicious xz-utils has the ability to inject attack
> code into the uncompressed source code .tar files your proposal depends on.
> 
> 
>> But
>> what I will say is I've tried very hard in any suggestions I've made to
>> only suggest things which will NOT change any default behaviour or require
>> big changes. The average user would not see any change from my revised
>> suggestions at all. I accepted after the first responses in this thread
>> that there was no appetite here to stop using xz utils. I then asked the
>> list about an idea I had just to see how palatable it might be. It was not
>> supposed to be a concrete plan, I was seeking discussion about how it
>> might be possible in practise for someone to use Gentoo without
>> compression and decompression of distfiles. I tried to suggest a solution
>> that could be an optional feature people could enable if they wanted it.
> 
> 
> But you should be proposing a change in default behavior! If it's not a
> change in default behavior then the change helps no one and there's no
> point in making it.
> 
> If the change is good enough to make, it should be good enough to
> propose its use by default, or at least as a recommended change people
> should be encouraged to make while data is collected for rolling out the
> change as a switched default.
> 
> If you want a change that only applies to your personal system, Gentoo
> already has a solution: make an overlay that contains slightly modified
> versions of every ebuild in gentoo that has a SRC_URI mentioning *.xz files.
> 
> No changes to portage needed. Host your own tarballs on your own server
> in SRC_URI. Validate them with checksums, even.
> 
> ...
> 
> Or you could suggest patches to portage that improve the sandbox used to
> invoke decompression and archive extraction, so that it's "safe" to have
> xz as a statically linked executable with no accompanying library
> installed for the purpose of untrusted distfile extraction.
> 
> 
>>> It's not being cavalier when zero portage developers responded by saying
>>> "good idea I'll drop everything so I can get right on this and implement
>>> it".
>>
>> I'll just point out that I've never expected nor asked for anyone to
>> unquestionably accept anything I've said, let alone in the way you have
>> characterised there that I might have done. I do think that the oss/linux
>> community as a whole including Gentoo developers should seriously consider
>> changing direction on this though. And I still think it is cavalier,
>> simply because by deciding on the current direction that is being taken,
>> very big (not an exaggeration) risks on behalf of all users are being
>> taken, while a much safer path for everyone is available but being
>> completely ignored.  I do acknowledge, though, as I have said before, that
>> this is far from easy in practise.
> 
> 
> Sorry, but no.
> 
> When you say that the community's response is "cavalier" because the
> community is not accepting what you've said, you are inherently working
> off the belief that the community's failure to accept what you've said
> is because the community is *wrong* to question your suggestions.
> 
> If you think the community is wrong to question your suggestions then
> you should be prepared to defend that point of view, particularly when
> they are busy trying to coordinate a security response together with
> security professionals from a wide variety of other distros, commercial
> vendors, OSS communities, etc. and have limited time to explain to
> *hundreds* of people who each have their own badly thought out ideas
> about what the FOSS community should do to solve the problem.
> 
> Your suggestion is only one such badly thought out idea. However, it
> stands out from the rest because your suggestion has something that the
> rest by and large lack: it has an accusation that distros, in this case
> Gentoo, are being cavalier about security.
> 
> This attitude of "Gentoo is being cavalier about security" is
> disproportionately worse than the average user interaction and, as has
> been noted, is the reason why FOSS maintainers suffer burnout.
> 
> It has nothing to do with bringing up concerns. It has everything to do
> with "if you don't agree with me you're being cavalier about MY security
> as a Gentoo user".
> 
> Seriously. Please learn to bring up suggestions as suggestions, not as
> demands. It makes all the difference in the world.

I'll just reply only to let you know I'm not going to take part in some 
sort of battle. I've been courteous with you and I have not aimed my 
opinions or arguments at any person in particular.  However, your reply 
here is venturing beyond healthy debate into some sort of twisted, 
completely out of order, nastiness towards me. It is completely 
fruitless, so I won't be engaging with you any further. It's a shame as 
some your arguments, if you take away the poison from them, really, 
really need to be challenged and I would enjoy doing so. But like this, 
no, definitely not, I'm done here.

[gentoo-dev] Re: Current unavoidable use of xz utils in Gentoo

[Duncan]

Eddie Chapman posted on Tue, 2 Apr 2024 20:32:41 +0100 as excerpted:

> Yes, I have no issue with the format at all, just with the xz utils
> project.

FWIW, feel free to do that bug-fix or package-bump if you'd rather instead 
of reading this long thing! I won't complain! =:^)

IMO...

The thing is, the actual problem isn't the xz-utils project.  Roll the 
dice.  The attack could have happened to any one (or more) of a number of 
projects... and a few years ago, before the Linux Foundation sponsored a 
project to coordinate helping out various one-person upstreams with 
security sponsorships and etc, it could have been quite a few more.  xz-
utils just happens to be the one one with the bad luck to be attacked and 
where we actually discovered the attack... that wasn't yet covered by the 
LF security support program as its "core Linux infra" connection via 
systemd is relatively new and only developed more or less in parallel with 
that program, so it had until now fallen through the cracks.

Now that the attack on xz-utils has been exposed, we've already rolled 
back to before the attacker got involved.  So we see the (symptomatic/
surface) problem there and have already addressed it, tho as I said the 
real problem isn't xz-utils at all.

Sure we could go to more extreme lengths to allow xz-utils to be taken out 
of the picture, but what would that do?  We've already reverted to before 
the attacker was in the picture, and it's simply impossible to 
alternative-force /all/ of the currently at-risk packages, some of which 
might already be compromised only we don't know it yet, actually putting 
them in a WORSE position, or there'd likely not be enough left to make a 
working distro!

In fact, the attacker (or one of the near-zero-history posters that urged 
his xz-utils releases be accepted, I don't remember the specifics as I 
read a lot of articles and a lot of followup discussion over the weekend) 
had apparently also made a commit to libarchive, tho it's quite possible 
it was of the "gain their trust with an innocent commit first" kind or 
that they abandoned that effort when the xz-utils effort appeared to be 
working better.  Of course by the time I read about that over the weekend 
people were already scouring that and looking for others.

Are we going to kill libarchive too, when it's just one commit and it's 
now known and scoured?  How many other packages are going to have low-
history/bad-history commits that look iffy now?  How may of them can we 
really find alternatives for that don't have the same or worse problems?  

So we can't throw the baby out with the bath-water, which is where we'd 
end up if we took the same approach for all the packages in a similar 
situation if not worse because we can't fix what we haven't discovered 
yet!

Rather, the (IMO) more reasonable approach is what people are already 
doing, addressing the systemic issues.

One of these systemic issues is that for not until now examined historical 
reasons, it's considered reasonable for release tarballs to differ from 
the tarball created from the pure repo release-tag checkout.  In some 
cases that's at least presently needed due to the way autotools and 
traditional release processes work, but that's being reexamined now, with 
some packages already able to switch to release-tag-corresponding 
tarballs, while others can't yet, but are high-priority examining changes 
in procedure and/or release tooling so they can.  So that's likely to 
change for many packages within a release or two, and people will be 
pressuring the ones that don't and/or considering switching to 
alternatives.

*That* is actually a (IMO) /reasonable/ alternative-consideration -- over 
the next months, examine packages that aren't already switching to tag-
corresponding release tarballs and consider alternatives to /them/!

Another of the systemic issues is the number of Linux-core-infra packages 
with a "bus-factor" of one or even two (consider that until this happened, 
xz-utils seemed to now have two, nobody realizing the one was actually an 
attacker).  Now that xz-utils had this known attack AND it's now known to 
be core-critical (for distros with that systemd integration... which of 
course includes both two big names and two enterprise products with real 
money behind them), I'm sure the original author has all sorts of offers 
of help, both for simple maintenance, and scouring for any other security 
issues.  We really shouldn't have to worry about it any longer.  And I've 
already mentioned the LF security help program which helps for many.  But 
there's likely a dozen (more?) other packages that are either relatively 
newly core-integrated or that have fallen through the cracks until now for 
other reasons.  There's going to be real-money efforts to find these and 
add them to the security-help list now, because there's real-money 
products at stake!

Yet another issue, once tarballs can be verified against release tags, is 
that a lot of distro maintainers don't actually verify the code changes.  
Some simply don't have the necesary skills.  Others have the skills, but 
still don't verify, because the tooling isn't there to make it fast/simple 
enough for them and there's always more packages to bump then time to 
actually do it.

Now, due to the xz-utils attack revealing the problem, there's already 
community efforts to improve the tooling to make it easier for distro 
maintainers to not only look at the commit logs, but go a bit deeper than 
that and better show the changed code.  And many maintainers are 
redoubling their efforts to make routine at least minimal change-audits 
with the existing tooling in the mean time.


Helping with any of these three would certainly be reasonable.  But 
demanding a *LOT* of work to alternative-force an already attack-reverted 
package, when we actually KNOW about that one, it's reverted to pre-attack 
and there's likely to be no more mischief there /because/ everybody's 
looking at it now, when it could have been any of a number of packages, 
some of which might already be compromised and we just didn't happen to 
find it, IMO really doesn't make much sense.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-dev] Re: Current unavoidable use of xz utils in Gentoo

[Sam James]

Duncan <1i5t5.duncan@cox.net> writes:

> Eddie Chapman posted on Tue, 2 Apr 2024 20:32:41 +0100 as excerpted:
>
>> Yes, I have no issue with the format at all, just with the xz utils
>> project.
>
> FWIW, feel free to do that bug-fix or package-bump if you'd rather instead 
> of reading this long thing! I won't complain! =:^)

Something's wrong - we're agreeing ;)

You're spot on. Tangible real changes and efforts are useful, not
"please boil the ocean".

> [...]

thanks,
sam

Re: [gentoo-dev] Re: Current unavoidable use of xz utils in Gentoo

[Kévin GASPARD DE RENEFORT]

> Helping with any of these three would certainly be reasonable.  But
> demanding a *LOT* of work to alternative-force an already attack-reverted
> package, when we actually KNOW about that one, it's reverted to pre-attack
> and there's likely to be no more mischief there /because/ everybody's
> looking at it now, when it could have been any of a number of packages,
> some of which might already be compromised and we just didn't happen to
> find it, IMO really doesn't make much sense.

Hello,

After so much reading and seeing almost a dead-end to this talk and from 
this citation above I had an idea for OP.

1/ OP is sure that Gentoo and others distro *should* avoid using 
xz-utils, at all cost.

(IMHO that is a respectable choice, *IF* it's possible without adding 
tremendous of works while Gentoo's dev could works on something else… 
Like being sure xz-utils is now safe to use…)

2/ Gentoo's dev stating that it's:

     a) Non-required, to not say useless.

     b) Would ask a lot of money to extend the infrastructure of Gentoo 
(two times the compressed file and the new non-xz would take like +30% 
in size…) and some works in addition for the systems administrators. As 
someone that had this job for some years, that is not always easy as it 
looks like and having more works is never fun while you already have 
some cooking… specially when you are not paid for this.

     c) Would ask a *LOT* of works for Gentoo's devs, ebuild mainteneurs…

     d) For, from Gentoos's dev opinion, something that only a very few 
users will actually use, without speaking about adding a layer of 
complexity in every process, from installing Gentoo or maintaining the 
packages. Looks like an awful jobs to be honest.

If OP is really that sure that Gentoo's dev are having a cavalier 
attitude, non-thinking enough about security in this subject, while 
(sorry but that's true) not paying much respect to the works into the 
community (Gentoo and free software in general)… Well:

Fork Gentoo, or any other distros, start a LFS…

I mean, this is *free software* (as in freedom), what makes you not 
starting your own project with peoples sharing your point-of-view ?

Some debian's user didn't liked the coming of SystemD, some made Devian 
(not even know if it's still around, but that is a simple example). 
Don't some *BSD distribution were borne for technical different 
point-of-view ? Yes, some did and are still here, since decades.

I think, IMHO, you should try to see if peoples around are having the 
same philosophy as you, if you find a bunch of peoples having times and 
willing to do it.

I suppose you have some knowledge, but I can only assume, maybe you 
don't have enough, could take years even if you have already these. Even 
more if you start from 0.

If you are alone, you have two choices:

1/ Do like Slackware, create as a lone-wolf your own distribution.

2/ Accept the idea that your idea is maybe not true, or good.

When a lot of peoples state that you are wrong, it doesn't means you are 
all the time. But at the same time, you were explained more than once 
that it's not a good idea, a really better way or they (Gentoo's dev) 
have other matter to take care of. Maybe Gentoo's dev are wrong. But in 
my case, I'll keep my side for the peoples that has proven theirs skills 
by their works. For more than 20 years, now.

That is just my opinion. You don't like it ? Fork it, find an 
alternative OR accept your faith. Or change for a distribution sharing 
your opinion about that.

PS : Sorry for my English.

Regards,
GASPARD DE RENEFORT Kévin

Re: [gentoo-dev] Re: Current unavoidable use of xz utils in Gentoo

[Kévin GASPARD DE RENEFORT]

Sorry but I wanted to add something to what is written below:

I'll insist as other did before: An other alternative would be to start 
your own overlay, push something to help Gentoo's dev, anything, because 
saying more or less "Do that because actually it's bad" is something 
rarely appreciated from peoples already giving their free time (as in 
free beer), for zero money at the end of the month for it, mostly. Here 
we only see someone that is 100% sure he's right, when responsible*S* 
peoples says he's not. At least, start something, share, propose by 
actions. Then they will listen you a bit further and taking you way more 
seriously, I'm pretty sure of that.

Please, respect that.

Sorry again for my English.

>
> Hello,
>
> After so much reading and seeing almost a dead-end to this talk and 
> from this citation above I had an idea for OP.
>
> 1/ OP is sure that Gentoo and others distro *should* avoid using 
> xz-utils, at all cost.
>
> (IMHO that is a respectable choice, *IF* it's possible without adding 
> tremendous of works while Gentoo's dev could works on something else… 
> Like being sure xz-utils is now safe to use…)
>
> 2/ Gentoo's dev stating that it's:
>
>     a) Non-required, to not say useless.
>
>     b) Would ask a lot of money to extend the infrastructure of Gentoo 
> (two times the compressed file and the new non-xz would take like +30% 
> in size…) and some works in addition for the systems administrators. 
> As someone that had this job for some years, that is not always easy 
> as it looks like and having more works is never fun while you already 
> have some cooking… specially when you are not paid for this.
>
>     c) Would ask a *LOT* of works for Gentoo's devs, ebuild mainteneurs…
>
>     d) For, from Gentoos's dev opinion, something that only a very few 
> users will actually use, without speaking about adding a layer of 
> complexity in every process, from installing Gentoo or maintaining the 
> packages. Looks like an awful jobs to be honest.
>
> If OP is really that sure that Gentoo's dev are having a cavalier 
> attitude, non-thinking enough about security in this subject, while 
> (sorry but that's true) not paying much respect to the works into the 
> community (Gentoo and free software in general)… Well:
>
> Fork Gentoo, or any other distros, start a LFS…
>
> I mean, this is *free software* (as in freedom), what makes you not 
> starting your own project with peoples sharing your point-of-view ?
>
> Some debian's user didn't liked the coming of SystemD, some made 
> Devian (not even know if it's still around, but that is a simple 
> example). Don't some *BSD distribution were borne for technical 
> different point-of-view ? Yes, some did and are still here, since 
> decades.
>
> I think, IMHO, you should try to see if peoples around are having the 
> same philosophy as you, if you find a bunch of peoples having times 
> and willing to do it.
>
> I suppose you have some knowledge, but I can only assume, maybe you 
> don't have enough, could take years even if you have already these. 
> Even more if you start from 0.
>
> If you are alone, you have two choices:
>
> 1/ Do like Slackware, create as a lone-wolf your own distribution.
>
> 2/ Accept the idea that your idea is maybe not true, or good.
>
> When a lot of peoples state that you are wrong, it doesn't means you 
> are all the time. But at the same time, you were explained more than 
> once that it's not a good idea, a really better way or they (Gentoo's 
> dev) have other matter to take care of. Maybe Gentoo's dev are wrong. 
> But in my case, I'll keep my side for the peoples that has proven 
> theirs skills by their works. For more than 20 years, now.
>
> That is just my opinion. You don't like it ? Fork it, find an 
> alternative OR accept your faith. Or change for a distribution sharing 
> your opinion about that.
>
> PS : Sorry for my English.
>
> Regards,
> GASPARD DE RENEFORT Kévin
>
>

Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo

[Eddie Chapman]

Just to report I've been able to remove app-arch/xz-utils from my own
workstation, with 2412 packages installed and running kde. I'm going to
roll it out to my other gentoo systems which have a lot less stuff on them
so am confident will be fine. It's not completely trivial but not as
difficult as I imagined it to be, certainly something an advance Gentoo
user could do if they wanted, with instructions. It does involve a
relatively small hack and functionality previously provided by xz-utils is
replaced by app-arch/p7zip.

I haven't had to give up distfile checksum verifications, everything
builds, boots and is working fine, I don't miss it at all. There is some
small (at least small for me) functionality you'll lose, but nothing I'll
miss.  I also had to uninstall 2 desktop packages which I can live
without, in order to get there, but I believe even they can be later
coaxed into working with some persuasion. I would imagine the majority of
Gentoo installations would not miss it unless they do a lot of work daily
directly using xz-utils and they particularly like all the different
permutations of how it can be run and inserted here and there.

Also, it goes without saying if you run any sort of application that
absolutely refuses to run without liblzma.so being present and the
requirement cannot be compiled out, or you have scripts which rely on
liblzma.so provided functionality inside the language they use, then of
course this cannot be done.

If anyone wants to know the details of how to do it no problem just ask. 
I won't post if no one asks, to not get on people's nerves here with and
this mail getting longer and longer, especially when most here are not
interested in doing this and don't believe it is necessary.

For all the overworked Gentoo developers I'd like to compliment you for
once rather than irritating you; it's only realistically possible to do
because of the the powerful distribution you have created, particularly
the great choices you've made along the way, that allows the Linux user to
do advanced stuff that would just not be realistically achievable to users
on the vast majority of other distros.

Eddie

Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo

[Michael Orlitzky]

On Wed, 2024-04-03 at 16:30 +0100, Eddie Chapman wrote:
> It does involve a
> relatively small hack and functionality previously provided by xz-utils is
> replaced by app-arch/p7zip.

I did the same thing with app-arch/unzip a long time ago. You caught a
lot of shit for your post, but I don't think it was out of line.

Worst case? You spent a lot of time building a fragile solution to a
non-problem that everyone said you were crazy for wanting in the first
place. Hi, this is Gentoo, glad to have you.

[gentoo-dev] Re: Current unavoidable use of xz utils in Gentoo

[Duncan]

Kévin GASPARD DE RENEFORT posted on Wed, 3 Apr 2024 14:22:18 +0200 as
excerpted:

> Fork Gentoo, or any other distros, start a LFS…

In fact, Gentoo has been forked in this way at least three times.  The 
first time was over 20 years ago, before 2004 as I remember researching it 
before I switched to Gentoo myself.  IDR what it was called but it had 
already by then pretty much fizzled out.

#2 and #3 are I believe still around and there's still some healthy 
interaction between them and Gentoo.  Funtoo is one, created by Gentoo's 
original founder.  It still uses portage and some of portage's features 
are primarily used there.  As a result, their contributions back to 
portage have continued to make it better for all users.  The other IDR the 
name (maybe Herb...?, with paludis as the package-manager) but PMS, the 
package-management-specification, that defines a portage- and package--
manager independent spec and the EAPI that packages use, is one of the 
major efforts to come of it as they split off.  And while most gentooers 
still use portage, pms is the reason other package managers can really 
work at all, and the thing much of the automated CI testing uses now, 
making it a BIG benefit to Gentoo.

So forking is a legitimate and respected if not necessarily pleasant while 
it's happening route to go, and often, some contributions from the process 
continue to be useful to and benefit both forks for many years after the 
fork.  While forks do generally mean duplication of effort in some areas 
and are often unpleasant to go through, the results aren't necessarily all 
bad, particularly when viewed with an appreciation that people often 
aren't paid for their work and could simply quite contributing entirely, 
which means the duplication of effort isn't all negative if it still means 
more contributions to the community than would happen if they just quit.

So if Gentoo's not doing it for you, in addition to the option of creating 
your own fork being a not necessarily entirely bad option, there's two 
other existing forks you might wish to look into, in case they're a better 
fit for you than Gentoo.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

[gentoo-dev] Re: Current unavoidable use of xz utils in Gentoo

[Duncan]

Michael Orlitzky posted on Wed, 03 Apr 2024 12:40:26 -0400 as excerpted:

> On Wed, 2024-04-03 at 16:30 +0100, Eddie Chapman wrote:
>> It does involve a relatively small hack and functionality previously
>> provided by xz-utils is replaced by app-arch/p7zip.
> 
> I did the same thing with app-arch/unzip a long time ago. You caught a
> lot of shit for your post, but I don't think it was out of line.
> 
> Worst case? You spent a lot of time building a fragile solution to a
> non-problem that everyone said you were crazy for wanting in the first
> place. Hi, this is Gentoo, glad to have you.

Gentoo as "meta-distro":

Yes.  I suspect many, perhaps most, Gentooers (individually or at the 
company level for corporate deployments) eventually end up doing their own 
thing to some degree or another.  I haven't seen the term used much 
recently, but Gentoo can legitimately lay claim to "meta-distro", that is, 
a distro that makes it reasonably easy to do your own thing, creating a 
"mini-distro" for your own use.  In fact it's reasonable to argue that (at 
least before the gentoo-mainstream binary packages became a thing) the 
relative costs of building it yourself likely ultimately lead most users 
who do /not/ need the meta-distro aspect to switch back to a more binary-
inclined distro, perhaps arch if they still want a lot of flexibility, 
which means the ones that stick around on Gentoo for say a decade or 
longer tend to do so /because/ they ended up using that meta-distro 
aspect.

In my own case my reverse-usrmerge ( /usr -> .) is certainly my biggest 
current meta-distro level divergence, tho historically, keeping
USE=-semantic-desktop functionality alive locally during the period that 
the gentoo/kde project dropped it was an equally major divergence... but 
equally doable due to Gentoo's meta-distro aspect.

Tho both would be rather harder were it not for git; I may not have done 
either one if git hadn't happened and svn was still king.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo

[Eli Schwartz]

[-- Attachment #1.1.1: Type: text/plain, Size: 1943 bytes --]

On 4/3/24 11:30 AM, Eddie Chapman wrote:
> Just to report I've been able to remove app-arch/xz-utils from my own
> workstation, with 2412 packages installed and running kde. I'm going to
> roll it out to my other gentoo systems which have a lot less stuff on them
> so am confident will be fine. It's not completely trivial but not as
> difficult as I imagined it to be, certainly something an advance Gentoo
> user could do if they wanted, with instructions. It does involve a
> relatively small hack and functionality previously provided by xz-utils is
> replaced by app-arch/p7zip.


I'd just like to clarify my previous posts: what you're describing here
is neat and productive and valid to my eyes. Actually, I wish this had
been the topic of the *first* post in this thread. :)

Replacing implementations has several great uses. There's some prior art
in make.conf, but it doesn't go far enough:

PORTAGE_BZIP2_COMMAND
BINPKG_COMPRESS
BINPKG_COMPRESS_FLAGS

Disregarding the security component entirely, one might wish to use pixz
or pigz instead of the default programs. Why not 7zip as well?

In terms of security, this suggests an easy and simple way both to allow
users to depclean xz-utils without sacrificing the ability to install
packages using *.tar.xz sources, and for Gentoo to roll out an update
that would do this distribution-wide if necessary via a trivial
configuration change.


https://dev.gentoo.org/~ulm/pms/head/pms.html#section-12.3.15 may need
updating to allow this. But it seems very valid to propose doing exactly
that. I am not sure why it specifies e.g. "must ensure that GNU gzip"
with heavy ties to implementations, when it doesn't specify such for
compression.

I'm guessing what you did was override/hook the unpack phase helper
function and divert it to 7zip instead. ;) It would be interesting to
have actual hooks for that instead.





-- 
Eli Schwartz

[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 18399 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo

[Sam James]

Eli Schwartz <eschwartz93@gmail.com> writes:

> On 4/3/24 11:30 AM, Eddie Chapman wrote:
>> Just to report I've been able to remove app-arch/xz-utils from my own
>> workstation, with 2412 packages installed and running kde. I'm going to
>> roll it out to my other gentoo systems which have a lot less stuff on them
>> so am confident will be fine. It's not completely trivial but not as
>> difficult as I imagined it to be, certainly something an advance Gentoo
>> user could do if they wanted, with instructions. It does involve a
>> relatively small hack and functionality previously provided by xz-utils is
>> replaced by app-arch/p7zip.
>
>
> I'd just like to clarify my previous posts: what you're describing here
> is neat and productive and valid to my eyes. Actually, I wish this had
> been the topic of the *first* post in this thread. :)

Completely agreed. We just prefer shorter text and focusing on technical
changes.

This sounds fun!

> [...]

thanks,
sam

Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo

[Kévin GASPARD DE RENEFORT]

If that’s working, it could at least be on an user personnal page on the 
wiki as well.

Le 04/04/2024 à 10:32, Sam James a écrit :
> Eli Schwartz <eschwartz93@gmail.com> writes:
>
>> On 4/3/24 11:30 AM, Eddie Chapman wrote:
>>> Just to report I've been able to remove app-arch/xz-utils from my own
>>> workstation, with 2412 packages installed and running kde. I'm going to
>>> roll it out to my other gentoo systems which have a lot less stuff on them
>>> so am confident will be fine. It's not completely trivial but not as
>>> difficult as I imagined it to be, certainly something an advance Gentoo
>>> user could do if they wanted, with instructions. It does involve a
>>> relatively small hack and functionality previously provided by xz-utils is
>>> replaced by app-arch/p7zip.
>>
>> I'd just like to clarify my previous posts: what you're describing here
>> is neat and productive and valid to my eyes. Actually, I wish this had
>> been the topic of the *first* post in this thread. :)
> Completely agreed. We just prefer shorter text and focusing on technical
> changes.
>
> This sounds fun!
>
>> [...]
> thanks,
> sam
>

Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo

[Eddie Chapman]

Eli Schwartz wrote:
> On 4/3/24 11:30 AM, Eddie Chapman wrote:
>
>> Just to report I've been able to remove app-arch/xz-utils from my own
>> workstation, with 2412 packages installed and running kde. I'm going to
>> roll it out to my other gentoo systems which have a lot less stuff on
>> them so am confident will be fine. It's not completely trivial but not
>> as difficult as I imagined it to be, certainly something an advance
>> Gentoo
>> user could do if they wanted, with instructions. It does involve a
>> relatively small hack and functionality previously provided by xz-utils
>> is replaced by app-arch/p7zip.
>
>
> I'd just like to clarify my previous posts: what you're describing here
> is neat and productive and valid to my eyes. Actually, I wish this had been
> the topic of the *first* post in this thread. :)
>
> Replacing implementations has several great uses. There's some prior art
> in make.conf, but it doesn't go far enough:
>
> PORTAGE_BZIP2_COMMAND
> BINPKG_COMPRESS
> BINPKG_COMPRESS_FLAGS
>
>
> Disregarding the security component entirely, one might wish to use pixz
> or pigz instead of the default programs. Why not 7zip as well?

One of my emails elsewhere in this thread (easy to miss in a long thread,
I know) I discussed pixz, pigz and 7zip. The former two were not suitable
for me as both rely on xz utils. However I will probably switch from p7zip
to the latest upstream 7zip in the near future, for reasons discussed in
that email.

> In terms of security, this suggests an easy and simple way both to allow
> users to depclean xz-utils without sacrificing the ability to install
> packages using *.tar.xz sources, and for Gentoo to roll out an update that
> would do this distribution-wide if necessary via a trivial configuration
> change.
>
> https://dev.gentoo.org/~ulm/pms/head/pms.html#section-12.3.15 may need
> updating to allow this. But it seems very valid to propose doing exactly
> that. I am not sure why it specifies e.g. "must ensure that GNU gzip" with
> heavy ties to implementations, when it doesn't specify such for
> compression.

That would certainly be a nice improvement for all users if it were ever
to come to pass.

> I'm guessing what you did was override/hook the unpack phase helper
> function and divert it to 7zip instead. ;) It would be interesting to have
> actual hooks for that instead.

Yes it is in the unpack phase where emerge calls /usr/bin/xz mostly. In
fact I didn't have to touch emerge/portage, it was more crude, I
uninstalled app-arch/xz-utils (and put it in
/etc/portage/profile/package.provided) and replaced /usr/bin/xz with a
bash script to behave like what the unpack phase was expecting, but using
/usr/lib64/p7zip/7za to do the decompression.

However, I need to do some more work on this "wrapper" (though it's more
than a wrapper) as I found one package where xz is called from the install
phase and my script doesn't handle that yet it just throws an error for
anything other than the unpack phase case (which is 99.9 percent of
packages).

But ultimately doing something along the lines of what you suggest instead
would of course be much better than this dirty hack (though it works just
fine for me for now).

Since there appears to be some interest I'll put together a single email
to the list later today detailing everything, as I needed to do more
things overall in addition to replacing /usr/bin/xz.

Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo

[Eddie Chapman]

Sam James wrote:
> Eli Schwartz <eschwartz93@gmail.com> writes:
>
>> On 4/3/24 11:30 AM, Eddie Chapman wrote:
>>
>>> Just to report I've been able to remove app-arch/xz-utils from my own
>>>  workstation, with 2412 packages installed and running kde. I'm going
>>> to roll it out to my other gentoo systems which have a lot less stuff
>>> on them so am confident will be fine. It's not completely trivial but
>>> not as difficult as I imagined it to be, certainly something an
>>> advance Gentoo user could do if they wanted, with instructions. It
>>> does involve a relatively small hack and functionality previously
>>> provided by xz-utils is replaced by app-arch/p7zip.
>>
>> I'd just like to clarify my previous posts: what you're describing here
>> is neat and productive and valid to my eyes. Actually, I wish this had
>> been the topic of the *first* post in this thread. :)
>
> Completely agreed. We just prefer shorter text and focusing on technical
> changes.
>
> This sounds fun!
>
>
>> [...]
>>
>
> thanks, sam

Well, I didn't think my first post was so bad, but OK, I'll take that
criticism onboard and in future will think more about how I bring
something to this list, and will take into account what you and Eli have
said. Thanks for sharing your thoughts above, both of you, in a
constructive way.

regards,
Eddie

Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo

[Eddie Chapman]

[-- Attachment #1: Type: text/plain, Size: 18716 bytes --]

On 04/04/2024 15:24, Eddie Chapman wrote:
> Since there appears to be some interest I'll put together a single email
> to the list later today detailing everything, as I needed to do more
> things overall in addition to replacing /usr/bin/xz.

Below is a guide I've written to removing app-arch/xz-utils in case 
anyone else wants to do so.  Attached is the current version of the Bash 
wrapper script I now use in place of /usr/bin/xz

Comments, corrections on anything technical in the guide or script are 
welcome, apart from flames about how this is ridiculous and unnecessary :-).

Best wishes,
Eddie


==== Guide to removing xz utils on a Gentoo system ====

=== Introduction ===

This guide is for people who wish to remove xz utils (app-arch/xz-utils) 
from a Gentoo system.

I've been able to remove xz utils from two Gentoo workstations with 2412 
packages and KDE 5.x as the desktop, and it has not been painful at all. 
I've gone on to remove it from several Gentoo server systems without any 
pain. These are all SElinux systems.

In this guide we replace app-arch/xz-utils with app-arch/p7zip which 
will do all the work of uncompressing xz distfiles for Portage going 
forward. It works perfectly fine for that right now.

I've written a bash wrapper script which is designed to be installed as 
/usr/bin/xz, which is referred to in the instructions below. It is 
attached to this email as xz.txt. It tries to takes care of 
decompressing .xz files transparently whenever Portage runs /usr/bin/xz, 
by behaving like it but using app-arch/p7zip in the background. You will 
need it if you want to get rid of app-arch/xz-utils. But don't blindly 
use it, check it yourself first of course. If you don't like it you will 
either need to write your own script, or hack emerge/Portage in various 
places to use something else to decompress xz files.

You're mileage may vary with any of this, proceed at your own risk, 
don't blame me if you break your system or lose data.


=== Warnings / Caveats / Breakages ===

Before you do this, you should identify whether you have applications or 
scripts which use the Tukaani xz utils, or that link against 
liblzma.so.5. This could include non-Gentoo apps or scripts you run 
which call any of the xz utils (xz, unxz, xzgrep|xzegrep|xzfgrep, xzcat, 
xzcmp, xzdec, xzdiff, lzma, unlzma, lzgrep|lzegrep|lzfgrep, lzmainfo, 
lzmadec, lzcmp, lzdiff, lzcat). Those programs will all be gone, so you 
should not do this if you want or need them and cannot use alternatives.

99% of packages in Gentoo work fine without xz utils, it's just that 
some might optionally link against liblzma.so.5 in order to provide 
support for xz (de)compression along with other algorithms. We will 
rebuild those packages so they don't link against liblzma.so.5 anymore.

xx utils is a relative newcomer to the Linux/OSS/GNU world so you will 
find there aren't any low level system packages that absolutely need it 
to do their main job. You are highly unlikely to render your system 
completely unbootable doing this.

But removing it does carry some risk. You might discover along the way 
there is some application you have installed that cannot function 
without xz utils. You might just have to uninstall it and find an 
alternative, if the situation cannot be resolved by creating your own 
custom ebuild and tweaking configure/meson options. But worst case if 
you have to uninstall a package and other packages depend on it, you 
might have to remove them too, and I'm sure you know how that remove 
list can potentially turn into a long one once all deps are worked out.

You will lose some things. I've had to uninstall the following two 
packages for now:

media-gfx/gimp
kde-apps/ark (and kde-apps/kdeutils-meta which depends on it)

(I'll probably figure out later how to coax them into working without 
xz. There might even be upstream updates soon that make xz optional, who 
knows. I'll also need to add to my world file at some point everything 
that was in kde-apps/kdeutils-meta.)

If you run another desktop (e.g. Gnome) I've no idea what might or might 
not need xz utils. The situation with your desktop environment may be 
worse, more painful, or impossible.

You will lose lzma support in the core Python language (dev-lang/python) 
in 3.x versions and higher (not sure when exactly support was introduced 
but 2.7 does not have it, 3.11 & 3.12 do), so if you have python scripts 
that happen to need that, well, they will definitely throw a big error 
after this :-) But I was able to rebuild the 179 dev-python packages on 
my workstations and everything in app-portage and none of them 
complained. I've been able to go on and do plenty of rebuilding with 
Portage after this without any problem, so core Python functionality in 
Gentoo is fine (although see next paragraph about Gemato).

There is one significant thing that breaks, which is Gemato 
(app-portage/gemato). Gemato requires lzma support in core python in 
order to do GPG signature verification. This means you will have to say 
goodbye (for now) to verifying upstream GPG signatures on distfiles, and 
verification of Portage metadata after doing an emerge --sync. These 
features have been added to Portage relatively recently (2022?) so are 
"nice to have", without them your system is just less hardened, but 
still with the very high level of security that Gentoo systems have has 
always had prior to these features, in my opinion. Personally I can live 
without them for now. Verifying hashes in Manifest files still works 
fine and that's the main thing. You may disagree in which case, well, 
don't do this then. I'm going to figure out an alternative way I can 
verify Portage metadata soon, as there are other ways if you are creative.

In practise this means you have to use USE="-verify-sig" for every 
emerge with a package that has a corresponding sec-keys package, and you 
have to set:

sync-rsync-verify-metamanifest = no

in files in /etc/portage/repos.conf/

But after doing that all works fine.

Here's some other very minor things you might lose if you are currently 
using them:

- KDE users will lose xz compression support from KArchive 
(kde-frameworks/karchive). AFAICT this has NOT had any impact on my own 
KDE experience, I've not seen any errors and everything I use works fine 
in my KDE sessions. KArchive will still support GZip, BZip2 and Zstd, 
just not xz. I suspect nothing that uses KArchive is using xz by 
default, but I'm not completely sure. All I know is my KDE sessions are 
running fine without it, and I can do everything in KDE I did before 
(apart from use Ark of course, see above). I don't know anything about 
KArchive. Full details of compression support in KArchive are at 
https://api.kde.org/frameworks/karchive/html/classKCompressionDevice.html

- Portage binary packages: You cannot use xz compression if you create 
Portage binary packages. You will need to use one of bzip2, gzip, lz4, 
lzip, lzop, or zstd in BINPKG_COMPRESS in make.conf instead of xz (if 
that is what you were using, or is it the default?). I have always used 
gzip so no probs for me, creating binary packages works fine, I've 
already updated several Gentoo systems from many binary packages I've 
created using gzip without xz utils installed.

- Grub bootloader: If you happened to have been using the optional, not 
used by default, --compress argument for grub-install, and you happen to 
have chosen xz, well you can't anymore. You will have to use gz or lzo 
instead, or stop using --compress if you don't like either of those two. 
Grub still builds, installs, works fine without xz utils for almost 
everyone. But if you did happen to previously use --compress=xz with 
grub-install before, make sure you check out fully what you might or 
might not have to do before next rebooting (I have no idea, I have never 
used this feature, Grub has continued working fine for me after 
rebuilding it without xz-utils and running grub-install again on my boot 
drives).

- Dovecot: net-mail/dovecot links liblzma.so.5 in order to support it's 
optional Zlib plugin ( 
https://doc.dovecot.org/configuration_manual/zlib_plugin/ ) for 
reading/writing compressed mail files. Despite the plugin being called 
"Zlib" it supports several different compression algorithms. At one time 
they supported xz, but in recent Dovecot releases they decided to 
deprecate it. They still support reading (not writing) xz compressed 
files, so when net-mail/dovecot is built, if it finds liblzma.so.5 it 
will use it, if it doesn't find it, it wont, and then you just have no 
support for xz in the Zlib plugin (again, only *if* you are using that 
plugin, which is not default). From what I can gather, if you use this 
plugin you should migrate away from using xz compressed mail files (to 
another supported compression). So you should do that before you do 
this, if that applies to you. I use Dovecot but never enabled mail file 
compression so this did not affect me, Dovecot has continued working 
fine with the mail stores I look after.

- Mariadb: If you happen to make use of the optional InnoDB Page 
Compression feature in Mariadb ( 
https://mariadb.com/kb/en/innodb-page-compression/ ), and if you happen 
to have chosen lzma compression for that feature (not the default) 
rather than one of the other 5 algorithms, then that is very unlucky, 
you will need to change that in your MariaDB installation in order to 
use one of the other 5 compression algorithms instead. dev-db/mariadb 
during build will automatically pick up support for the compression 
algorithms you have installed on the system, you don't currently specify 
anything in the ebuild that affects that. So if you have dev-db/mariadb 
installed you will have to rebuild it after removing xz utils as it 
links against liblzma.so.5 for this feature, and on rebuilding it you 
will lose support for lzma in InnoDB Page Compression. If you don't know 
if you are using it, this sql query will tell you:

SHOW GLOBAL VARIABLES LIKE 'innodb_compression_algorithm';

On my MariaDB 10.6 server it returned:

+------------------------------+-------+
| Variable_name                | Value |
+------------------------------+-------+
| innodb_compression_algorithm | zlib  |
+------------------------------+-------+

So I was not using lzma and was not affected. Tested: my MariaDB 10.6 
server is now using rebuilt dev-db/mariadb without liblzma.so.5 and is 
running with no problems.

- sys-apps/fwupd might stop working properly (though it will still build 
fine) due to what you have to change with dev-libs/libxmlb below. I'm 
not sure as I haven't checked yet, I just suspect it will. So bear that 
in mind if you need to rely on sys-apps/fwupd at the moment. But this 
"might" is temporary, upstream has now decided to make lzma optional, so 
this will trickle down to Gentoo soon.

- app-arch/rpm will probably not be able to extract some rpm archives if 
they are compressed with xz, but I haven't checked that yet. Though it 
will still build fine. This does not affect building Gentoo packages 
which come with .rpm distfiles (e.g. libreoffice), Portage uses rpm2tgz 
for that and my script takes care of the rest.


=== The instructions ===

Follow them in order.

1.  Do an emerge --sync and @world update first to make sure any
     upgrades/updates have been applied. Makes it easier for the
     things you need to do after you remove xz utils.

2.  Install p7zip: emerge app-arch/p7zip

3.  Add -lzma to USE flags in make.conf

4.  Rebuild @world. This will rebuild only a few packages which
     respect -lzma

5.  Copy the bash wrapper script to somewhere on the machine you
     are doing this on (but NOT to /usr/bin yet)

6.  Prepare the script to be installed. Rename it to "xz" (with
     no extension), set permissions to 0755, owned by root:root.

7.  On an SElinux installation, set the SElinux context of the
     script to whatever the current /usr/bin/xz binary is set to.

8.  Remove xz utils, ignoring the warning about it being part
     of system: emerge --unmerge app-arch/xz-utils
     Once it is removed Portage will tell you that it preserved
     liblzma.so.5. More on that below.

9.  Install the bash wrapper script to /usr/bin/xz

10. Add the following line:
     app-arch/xz-utils-5.4.2
     to /etc/portage/profile/package.provided

11. Remove kde-apps/kdeutils-meta and kde-apps/ark if you use
     KDE, and media-gfx/gimp if you use it:
     emerge --unmerge kde-apps/kdeutils-meta kde-apps/ark media-gfx/gimp

12. (optional) Add -verify-sig to USE flags in make.conf. If you
     do you will soon have to rebuild all packages that rely on it.
     If you don't, you can just add USE="-verify-sig" in front of
     every emerge command you have to do from now on, or add to
     individual packages in your package.use file.

13. Now you will need to rebuild all packages with files that rely
     on the preserved liblzma.so.5 library. See below for further
     notes about that.

14. set:
     sync-rsync-verify-metamanifest = no
     in applicable files in /etc/portage/repos.conf/ before you do
     your next emerge --sync

15. Eventually, you will have to rebuild all packages that have
     corresponding signatures in sec-keys.

That's all, enjoy life without app-arch/xz-utils! But read on for more 
info about step 13.


=== Notes about Step 13 ===

These are the packages that I needed to rebuild on my systems before the 
preserved liblzma.so.5 library was finally removed by Portage:

app-arch/libarchive
app-arch/rpm
sys-boot/grub
dev-db/mariadb
dev-lang/python:2.7
kde-frameworks/karchive
dev-lang/python:3.11 (needs custom ebuild, see below)
dev-lang/python:3.12 (needs custom ebuild, see below)
net-mail/dovecot (needs custom ebuild, see below)
dev-libs/libxmlb (needs custom ebuild, see last note at the bottom of 
this guide)

There might be others on your system. In most cases just rebuilding them 
will be enough. Some you might be able to clone the ebuild to your local 
repo and tweak configure/meson options so that the package does not link 
against liblzma.so.5. There may be packages with issues too difficult to 
resolve so you might have to just uninstall them if you can't live 
without them :-(  (or resign yourself to rolling back and having to live 
with xz utils)

Remember you will need to specify USE="-verify-sig" for any packages 
that rely on that, in whichever is your preferred way.

 From my list I had to clone the following 3 packages to my local 
ebuilds directory with small modification to each in order to get them 
to build without linking against liblzma.so.5:
net-mail/dovecot
dev-lang/python:3.11
dev-lang/python:3.12

Here are 3 diffs showing what I had to change:

--- /usr/portage/net-mail/dovecot/dovecot-2.3.21-r1.ebuild
+++ /usr/local/portage/net-mail/dovecot/dovecot-2.3.21-r1.ebuild
@@ -43,7 +43,6 @@

  DEPEND="
         app-arch/bzip2
-       app-arch/xz-utils
         dev-libs/icu:=
         dev-libs/openssl:0=
         sys-libs/zlib:=
@@ -126,7 +125,7 @@
                 --disable-rpath \
                 --with-bzlib \
                 --without-libbsd \
-               --with-lzma \
+               --without-lzma \
                 --with-icu \
                 --with-ssl \
                 --with-zlib \

--- /usr/portage/dev-lang/python/python-3.11.8_p1.ebuild
+++ /usr/local/portage/dev-lang/python/python-3.11.8_p1.ebuild
@@ -179,6 +179,7 @@
         # Avoid as many dependencies as possible for the cross build.
         cat >> Makefile <<-EOF || die
                 MODULE_NIS_STATE=disabled
+               MODULE__LZMA_STATE=disabled
                 MODULE__DBM_STATE=disabled
                 MODULE__GDBM_STATE=disabled
                 MODULE__DBM_STATE=disabled
@@ -328,7 +329,7 @@
         fi

         # force-disable modules we don't want built
-       local disable_modules=( NIS )
+       local disable_modules=( NIS _LZMA )
         use gdbm || disable_modules+=( _GDBM _DBM )
         use sqlite || disable_modules+=( _SQLITE3 )
         use ssl || disable_modules+=( _HASHLIB _SSL )


--- /usr/portage/dev-lang/python/python-3.12.2_p1.ebuild
+++ /usr/local/portage/dev-lang/python/python-3.12.2_p1.ebuild
@@ -177,6 +177,7 @@
         cat > Modules/Setup.local <<-EOF || die
                 *disabled*
                 nis
+               _lzma
                 _dbm _gdbm
                 _sqlite3
                 _hashlib _ssl
@@ -299,6 +300,7 @@
         cat > Modules/Setup.local <<-EOF || die
                 *disabled*
                 nis
+               _lzma
                 $(usev !gdbm '_gdbm _dbm')
                 $(usev !sqlite '_sqlite3')
                 $(usev !ssl '_hashlib _ssl')


Lastly, I needed to create a custom dev-libs/libxmlb ebuild in order to 
upgrade it from 0.3.14 (latest in Gentoo at time of writing) to 0.3.15.

I also needed to apply a very recent patch from upstream, from this 
commit, which makes LZMA support optional:
https://github.com/hughsie/libxmlb/commit/bdf845510fbed40b88465b2272ccad9e93656639

and I needed to make some small changes to the ebuild.

So this is what you need to do at the time of writing (6th April 2024):

1. Copy the in-tree /usr/portage/dev-libs/libxmlb ebuild directory into 
your local ebuilds directory.

2. Rename the ebuild file from libxmlb-0.3.14.ebuild to 
libxmlb-0.3.15.ebuild

3. Download the raw patch, you can use this link:
 
https://github.com/hughsie/libxmlb/commit/bdf845510fbed40b88465b2272ccad9e93656639.patch
    rename it to:
    libxmlb-0.3.15-make_lzma_optional.patch
    and place it in the local "files" directory.

4. Modify the new ebuild according to the diff below. Then just rebuild it.

--- /usr/portage/dev-libs/libxmlb/libxmlb-0.3.14.ebuild
+++ /usr/local/portage/dev-libs/libxmlb/libxmlb-0.3.15.ebuild
@@ -14,15 +14,15 @@
  SLOT="0/2" # libxmlb.so version

  KEYWORDS="amd64 ~arm arm64 ~loong ppc ppc64 ~riscv x86"
-IUSE="doc introspection stemmer test +zstd"
+IUSE="doc introspection -lzma stemmer test +zstd"

  RESTRICT="!test? ( test )"

  RDEPEND="
-       app-arch/xz-utils
         dev-libs/glib:2
         sys-apps/util-linux
         stemmer? ( dev-libs/snowball-stemmer:= )
+       lzma? ( app-arch/xz-utils:= )
         zstd? ( app-arch/zstd:= )
  "

@@ -43,6 +43,7 @@

  PATCHES=(
         "${FILESDIR}"/${PN}-0.3.12-no_installed_tests.patch
+       "${FILESDIR}"/${PN}-0.3.15-make_lzma_optional.patch
  )

  python_check_deps() {
@@ -60,6 +61,7 @@
                 $(meson_use stemmer)
                 $(meson_use test tests)
                 $(meson_use zstd)
+               $(meson_feature lzma)
         )
         meson_src_configure
  }

[-- Attachment #2: xz.txt --]
[-- Type: text/plain, Size: 10962 bytes --]

#!/usr/bin/env bash

# SPDX-License-Identifier: GPL-2.0-only
# SPDX-FileCopyrightText: 2024 Eddie Chapman <eddie@ehuk.net>

# WARNING: this script is currently not a full replacement for xz, it just mimicks 
# some of the decompression functionality of xz. It is only designed at the
# moment to be called by Portage and even then it does not yet cover all cases of 
# that.

# Some places in portage where xz is called:
# - /usr/lib/portage/python3.11/phase-helpers.sh
#      This is where 99% of calls to xz happen, from the line:
#      __unpack_tar "xz -T$(___makeopts_jobs) -d"
#      in the unpack phase.
#      This results in a call to xz inside __unpack_tar() where the -c arg is added (for stdout)
#      and the filename is added as an argument.
# - /usr/bin/deb2targz
#      Some packages e.g. google-chrome have deb distfiles which can contain a data.tar.xz
#      file so deb2targz launches xz -dc to decompress that.
# - /usr/bin/rpm2tar
#      Some packages e.g. libreoffice have rpm distfiles compressed with xz so rpm2tar launches 
#      xz -dc to decompress them
# - /usr/portage/eclass/llvm.org.eclass
#      xz is not called directly but tar -x -J is run (and tar then runs "xz -d" with piped 
#      in/out, with no file as argument)

LOGGER=$(command -v logger)

if [ ! -x "${LOGGER}" ]; then
echo "(wrapper): Fatal error: logger command does not appear to exist!"
exit 1
fi

LOG_PREFIX="(wrapper):"

# /usr/bin/7za is just a wrapper that executes this
SEVEN_ZA="/usr/lib64/p7zip/7za"

DATE_CMD=$(command -v date)
MKTEMP_CMD=$(command -v mktemp)
PS_CMD=$(command -v ps)
CAT_CMD=$(command -v cat)
CHMOD=$(command -v chmod)
WHOAMI=$(command -v whoami)
GREP=$(command -v grep)
FILE_CMD=$(command -v file)
READLINK=$(command -v readlink)

for EXE_F in ${SEVEN_ZA} ${DATE_CMD} ${MKTEMP_CMD} ${PS_CMD} ${CAT_CMD} ${CHMOD} \
${WHOAMI} ${GREP} ${FILE_CMD} ${READLINK}; do

	if [ ! -x "${EXE_F}" ]; then
	MSG="${LOG_PREFIX} Fatal Error: ${EXE_F} does not exist or is not an exe!"
	${LOGGER} -p syslog.err -t "${0}" "${MSG}"
	exit 1
	fi

done

DECOMPRESS_REQUESTED=N
STDOUT_REQUESTED=N

for myarg in "${@}"; do

	# Look for the 3 forms of xz's decompress argument when it is by itself.
	# TO-DO: collapse these into one grep command, improve the horrible regex.
	echo "${myarg}" | ${GREP} -Eq '^[-]d$' 
	retA=$?
	echo "${myarg}" | ${GREP} -Eq '^[-][-]decompress$'
	retB=$?
	echo "${myarg}" | ${GREP} -Eq '^[-][-]uncompress$'
	retC=$?

	if [ ${retA} -eq 0 ] || [ ${retB} -eq 0 ] || [ ${retC} -eq 0 ]; then
	DECOMPRESS_REQUESTED=Y
	fi

	# Look for the 3 forms of xz's stdout argument when it is by itself.
	# TO-DO: collapse these into one grep command, improve the horrible regex.
	echo "${myarg}" | ${GREP} -Eq '^[-]c$' 
	retA=$?
	echo "${myarg}" | ${GREP} -Eq '^[-][-]to[-]stdout$'
	retB=$?
	echo "${myarg}" | ${GREP} -Eq '^[-][-]stdout$'
	retC=$?

	if [ ${retA} -eq 0 ] || [ ${retB} -eq 0 ] || [ ${retC} -eq 0 ]; then
	STDOUT_REQUESTED=Y
	fi

	# and look for both together as -dc or -cd
	# TO-DO: collapse these into one grep command, improve the horrible regex.
	echo "${myarg}" | ${GREP} -Eq '^[-]dc$' 
	retA=$?
	echo "${myarg}" | ${GREP} -Eq '^[-]cd$' 
	retB=$?

	if [ ${retA} -eq 0 ] || [ ${retB} -eq 0 ]; then
	DECOMPRESS_REQUESTED=Y
	STDOUT_REQUESTED=Y
	fi

done

# This script only tries to decompress. No compress functionaility at all
# at this stage in its development.
if [ "${DECOMPRESS_REQUESTED}" = "N" ]; then

	MSG="${LOG_PREFIX} Fatal Error: no (d|decompress|uncompress) option on the command line. Sorry, this wrapper script only supports decompression."
	#echo "$MSG"
	${LOGGER} -p syslog.err -t "${0}" "${MSG}"
	exit 1

fi

# DEBUG
#MSG="${LOG_PREFIX} (DEBUG) stdout requested? ${STDOUT_REQUESTED}"
#${LOGGER} -p syslog.info -t "${0}" "${MSG}"

WHO_CALLED=$(${WHOAMI})

# get the parent command, very useful for debugging
PARENT_CMD=$(${PS_CMD} -o args= ${PPID})

# DEBUG, avoid leaving enabled long term as potential for future security problem, 
# due to unescaped attacker controlled info being passed to logger
#MSG="${LOG_PREFIX} (DEBUG) U: ${WHO_CALLED}, PARENT: ${PARENT_CMD}, ARGS: ${@}"
#${LOGGER} -p syslog.info -t "${0}" "${MSG}"

f_passed_to_script=

# loop over args again to see if any file has been passed as an arg
# TO-DO there will be a better way of doing this.
for myarg in "${@}"; do

	# TO-DO, are there other possible extensions for xz. Also theoretically possible we
	# could be passed one with extension in caps.
	echo "${myarg}" | ${GREP} -Eq '[.]xz$' 
	r=$?

	if [ ${r} -eq 0 ]; then
	f_passed_to_script="${myarg}"
	break
	fi

done


function do_uncompress {

	# remember return numbers can only be btw 0 - 255

	# some sanity checks follow ...

	if [ -z "${1}" ]; then
	MSG_TO_SHOW="function requires 1 argument; a filename with our without leading path"
	return 184
	fi

	realf=$(${READLINK} -e "${1}" 2>/dev/null)

	if [ ! -f "${realf}" ]; then
	MSG_TO_SHOW="argument supplied either is not a file or, if it is a file, I cannot find it."
	return 194
	fi

	if ! test -s "${realf}"; then
	MSG_TO_SHOW="file supplied is empty!"
	return 204
	fi

	if [ ! -r "${realf}" ]; then
	MSG_TO_SHOW="file supplied cannot be read!"
	return 214
	fi

	# 7z does not like the file path being passed to it inside quotes. So lets just make 
	# sure not to pass it anything that contains any characters NOT in our sane list 
	# in our regex below (so nothing needs quoting as no shell metachars).
	# TO-DO: there will be a better way of dealing with this, prob by converting to an 
	# escaped string. But for now (2024) haven't come across any distfiles with weird chars 
	# in them thankfully.
	echo "${realf}" | ${GREP} -Evq '[A-Za-z0-9_:@%+/.-]'
	r=$?

	if [ ${r} -eq 0 ]; then
	MSG_TO_SHOW="found unsupported characters in the (real) file path."
	return 224
	fi

	# Make sure we have been given an xz file.
	# the -e arguments exclude tests we're not interested in, hopefully some tiny perf gain
	# but more importantly reduce attack surface.
	# Also we hae it output the mime type rather than a human readable string, more reliable
	${FILE_CMD} -e ascii -e cdf -e apptype -e csv -e elf -e json -e simh -e tar --mime ${realf} 2>/dev/null | ${GREP} -q 'application/x-xz;'
	r=$?

	if [ ${r} -ne 0 ]; then
	MSG_TO_SHOW="file supplied does not appear to be an xz file, according to the file command"
	return 234
	fi

	# initialise this string for the 7za stdout option (-so). Empty (no stdout) by default.
	STDOUT_OPT_STR=''
	# and this string to, by default, redirect 7za output to /dev/null, as it is somewhat
	# chatty when decompressing.
	STDOUT_REDIR_STR='>/dev/null'

	# If the uncompressed data should be sent to stdout then the above vars need to be changed.
	if [ "${STDOUT_REQUESTED}" = "Y" ]; then
	STDOUT_OPT_STR='-so'
	STDOUT_REDIR_STR=''
	fi

	# we currently set stderr to redirect to /dev/null always.
	STDERR_REDIR_STR='2>/dev/null'

	SEVEN_ZA_FULL_CMD="${SEVEN_ZA} e ${realf} ${STDOUT_OPT_STR} -bd ${STDOUT_REDIR_STR} ${STDERR_REDIR_STR}"

	MSG="${LOG_PREFIX} About to run: ${SEVEN_ZA_FULL_CMD}"
	${LOGGER} -p syslog.info -t "${0}" "${MSG}"

	# In 99% of cases we will redirect 7za decompressed output to stdout (-so).
	# So make really sure nothing gets output to stdout by this script after this point!
	# Log messages all only to logger.
	eval "${SEVEN_ZA_FULL_CMD}"
	return $?

}

last_r=0
LAST_ERR_MSG=

# if no file was detected we assume we will get compressed data via stdin
if [ -z "${f_passed_to_script}" ]; then

	# We create a temp file to save the stdin compressed data into as the current p7zip provided 
	# 7za does not work properly if fed data via stdin, though according to docs it
	# should work, so probably a bug.
	# Also note within sandbox tmp space is not the system /tmp AFAICT.
	# Unfortunately this means we need to make sure we have enough space for the uncompressed
	# data both in /tmp as well as whichever filesystem we set portage to do builds on.

	mytf=`${MKTEMP_CMD}`
	sleep 0.3

	# sanity
	if [ -e "${mytf}" ]; then

		${CHMOD} 0600 "${mytf}"

	else

		MSG="${LOG_PREFIX} Fatal Error: something has gone very wrong, no temp file exists!"
		${LOGGER} -p syslog.err -t "${0}" "${MSG}"
		exit 1

	fi

	# save stdin to our tmpfile. quotes shld not be needed but, what the hell, might as well.
	cat > "${mytf}"
	r=$?

	# sanity
	if [ ${r} -ne 0 ]; then

		MSG="${LOG_PREFIX} Fatal Error: cat returned non-zero code of ${r} when redirecting to ${mytf}!"
		${LOGGER} -p syslog.err -t "${0}" "${MSG}"
		exit 1

	else

		# Even if stdout was NOT requested using a command line argument (and thus STDOUT_REQUESTED will be set to N),
		# xz assumes that you *do* want the uncompressed stream to go to stdout if no file was given on the command line (naturally).
		# So we need to force this to Y here to make sure that happens.
		STDOUT_REQUESTED=Y

		# Having saved stdin to the tmp file above we can now have 7za decompress said tmp file.
		# Remember from here on we run 7za which will in most cases output binary data to stdout.
		# So make REALLY sure nothing else gets output after this!
		# Log messages all only to logger.
		do_uncompress "${mytf}"
		last_r=$?

		# error message numbers inside do_uncompress(), hopefully none clash with those used by 7za
		if [ ${last_r} = 184 ] || [ ${last_r} = 194 ] || [ ${last_r} = 204 ] || [ ${last_r} = 214 ] || [ ${last_r} = 224 ] || [ ${last_r} = 234 ]; then

			LAST_ERR_MSG="do_uncompress(): ${MSG_TO_SHOW}"

		elif [ ${last_r} -ne 0 ]; then

			LAST_ERR_MSG="7za returned non-zero code of ${last_r} when trying to decompress stdin!"

		fi

	fi
	
	# this is our created temp file rather than one supplied to the script so shld be deleted
	rm -f "${mytf}"

elif [ -e "${f_passed_to_script}" ]; then

	# Remember from here on we run 7za which will in most cases output binary data to stdout.
	# So make REALLY sure nothing else gets output after this!
	# Log messages all only to logger.
	do_uncompress "${f_passed_to_script}"
	last_r=$?

	# error message numbers inside do_uncompress(), hopefully none clash with those used by 7za
	if [ ${last_r} = 184 ] || [ ${last_r} = 194 ] || [ ${last_r} = 204 ] || [ ${last_r} = 214 ] || [ ${last_r} = 224 ] || [ ${last_r} = 234 ]; then

		LAST_ERR_MSG="do_uncompress(): ${MSG_TO_SHOW}"

	elif [ ${last_r} -ne 0 ]; then

		LAST_ERR_MSG="7za returned non-zero code of ${last_r} when trying to decompress the file!"

	fi

	# 7za does not delete the original file by default but xz does
	# If stdout was not requested then they will be expecting us to delete it so do that.
	# TO-DO: catch --keep argument and do not delete if it is passed
	if [ "${STDOUT_REQUESTED}" = "N" ]; then
	rm -f "${f_passed_to_script}"
	fi

else

	last_r=1
	LAST_ERR_MSG="no valid file was found in supplied arguments and stdin was not an xz stream!"

fi

if [ ${last_r} -ne 0 ]; then

	${LOGGER} -p syslog.err -t "${0}" "${LOG_PREFIX} Fatal Error: ${LAST_ERR_MSG}" >/dev/null 2>&1
	exit 1

else

	exit 0

fi

Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo

[Ulrich Mueller]

>>>>> On Sat, 06 Apr 2024, Eddie Chapman wrote:

> [...] this is ridiculous and unnecessary :-).

Indeed.

SCNR,
Ulrich

Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo

[Roy Bamford]

[-- Attachment #1: Type: text/plain, Size: 894 bytes --]

On 2024.04.06 12:57, Eddie Chapman wrote:
> On 04/04/2024 15:24, Eddie Chapman wrote:
> > Since there appears to be some interest I'll put together a single
> email
> > to the list later today detailing everything, as I needed to do more
> > things overall in addition to replacing /usr/bin/xz.
> 
> Below is a guide I've written to removing app-arch/xz-utils in case 
> anyone else wants to do so.  Attached is the current version of the
> Bash 
> wrapper script I now use in place of /usr/bin/xz
> 
> Comments, corrections on anything technical in the guide or script are
> 
> welcome, apart from flames about how this is ridiculous and
> unnecessary :-).
> 
> Best wishes,
> Eddie
> 
>[snip method]

"Because I can" is a good enough reason to do anything with Gentoo.

-- 
Regards,

Roy Bamford
(Neddyseagoon) a member of
elections
gentoo-ops
forum-mods
arm64

[-- Attachment #2: Type: application/pgp-signature, Size: 488 bytes --]

Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo

[Fabian Groffen]

[-- Attachment #1: Type: text/plain, Size: 1206 bytes --]

On 06-04-2024 12:57:23 +0100, Eddie Chapman wrote:
> There is one significant thing that breaks, which is Gemato 
> (app-portage/gemato). Gemato requires lzma support in core python in 
> order to do GPG signature verification. This means you will have to say 
> goodbye (for now) to verifying upstream GPG signatures on distfiles, and 
> verification of Portage metadata after doing an emerge --sync.  These 
> features have been added to Portage relatively recently (2022?) so are 
> "nice to have", without them your system is just less hardened, but 
> still with the very high level of security that Gentoo systems have has 
> always had prior to these features, in my opinion. Personally I can live 
> without them for now. Verifying hashes in Manifest files still works 
> fine and that's the main thing. You may disagree in which case, well, 
> don't do this then. I'm going to figure out an alternative way I can 
> verify Portage metadata soon, as there are other ways if you are creative.

If you just want to verify signatures and manifests after sync,
qmanifest from portage-utils can help you do this.

Thanks,
Fabian


-- 
Fabian Groffen
Gentoo on a different level

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo

[Sam James]

Eddie Chapman <eddie@ehuk.net> writes:

> On 04/04/2024 15:24, Eddie Chapman wrote:
>> Since there appears to be some interest I'll put together a single email
>> to the list later today detailing everything, as I needed to do more
>> things overall in addition to replacing /usr/bin/xz.
>
> Below is a guide I've written to removing app-arch/xz-utils in case
> anyone else wants to do so.  Attached is the current version of the
> Bash wrapper script I now use in place of /usr/bin/xz
>
> Comments, corrections on anything technical in the guide or script are
> welcome, apart from flames about how this is ridiculous and
> unnecessary :-).

For an experiment I'm doing (distinct from trying to purge xz-utils,
just verification work), I've packaged the following:
* app-arch/gxz (pure Go impl.)
* app-arch/7zip (7zip upstream are supporting Linux now, app-arch/p7zip
was an unofficial port)

You might find those useful too.

At a glance, it appears https://github.com/fpgaminer/rust-lzma and
https://github.com/gendx/lzma-rs don't provide executables - just a
library - so I didn't bother looking further.

>
> Best wishes,
> Eddie
>
>
> ==== Guide to removing xz utils on a Gentoo system ====
>
> [...]
> There is one significant thing that breaks, which is Gemato
> (app-portage/gemato). Gemato requires lzma support in core python in
> order to do GPG signature verification. This means you will have to
> say goodbye (for now) to verifying upstream GPG signatures on
> distfiles, and verification of Portage metadata after doing an emerge
> --sync. These features have been added to Portage relatively recently
> (2022?) so are "nice to have", without them your system is just less

No.. much older. It was introduced around the time of the github mirror
being hacked. It's not just theatre!

Like, this is very much NOT hypothetical.

It's not just about metadata, it's about the ebuilds if using rsync, or
the whole git checkout if using git.

> hardened, but still with the very high level of security that Gentoo
> systems have has always had prior to these features, in my
> opinion. Personally I can live without them for now. Verifying hashes
> in Manifest files still works fine and that's the main thing. You may
> disagree in which case, well, don't do this then. I'm going to figure
> out an alternative way I can verify Portage metadata soon, as there
> are other ways if you are creative.

See grobian's reply which should help.

> [...]
> - Portage binary packages: You cannot use xz compression if you create
>   Portage binary packages. You will need to use one of bzip2, gzip,
>   lz4, lzip, lzop, or zstd in BINPKG_COMPRESS in make.conf instead of
>   xz (if that is what you were using, or is it the default?). I have

zstd is the default for "new" installs (since a few years ago), yeah.

> [...]
> - sys-apps/fwupd might stop working properly (though it will still
>   build fine) due to what you have to change with dev-libs/libxmlb
>   below. I'm not sure as I haven't checked yet, I just suspect it
>   will. So bear that in mind if you need to rely on sys-apps/fwupd at
>   the moment. But this "might" is temporary, upstream has now decided
>   to make lzma optional, so this will trickle down to Gentoo soon.

Just for completeness, this is
https://blogs.gnome.org/hughsie/2024/04/03/fwupd-and-xz-metadata/.

> [...]

Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo

[Eddie Chapman]

Fabian Groffen wrote:
> If you just want to verify signatures and manifests after sync,
> qmanifest from portage-utils can help you do this.
>
> Thanks,
> Fabian

Thanks for the pointer, and I see you are one of the authors, thanks for
writing a very useful tool!

Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo

[Eddie Chapman]

Sam James wrote:
> Eddie Chapman <eddie@ehuk.net> writes:
>> Below is a guide I've written to removing app-arch/xz-utils in case
>> anyone else wants to do so.  Attached is the current version of the Bash
>> wrapper script I now use in place of /usr/bin/xz
>>
>> Comments, corrections on anything technical in the guide or script are
>> welcome, apart from flames about how this is ridiculous and unnecessary
>> :-).
>
> For an experiment I'm doing (distinct from trying to purge xz-utils,
> just verification work), I've packaged the following: * app-arch/gxz (pure
> Go impl.)
> * app-arch/7zip (7zip upstream are supporting Linux now, app-arch/p7zip
> was an unofficial port)
>
> You might find those useful too.

That's fantastic. I wrote about p7zip vs. upstream 7zip in another mail in
this thread and was intending to create a local ebuild for 7zip soon but
won't have to know it's in tree :-)

> At a glance, it appears https://github.com/fpgaminer/rust-lzma and
> https://github.com/gendx/lzma-rs don't provide executables - just a
> library - so I didn't bother looking further.
>>
>> ==== Guide to removing xz utils on a Gentoo system ====
>>
>> [...]
>> There is one significant thing that breaks, which is Gemato
>> (app-portage/gemato). Gemato requires lzma support in core python in
>> order to do GPG signature verification. This means you will have to say
>> goodbye (for now) to verifying upstream GPG signatures on distfiles, and
>> verification of Portage metadata after doing an emerge --sync. These
>> features have been added to Portage relatively recently (2022?) so are
>> "nice to have", without them your system is just less
>
> No.. much older. It was introduced around the time of the github mirror
> being hacked. It's not just theatre!
>
> Like, this is very much NOT hypothetical.

Thanks, couldn't remember when it was.

> It's not just about metadata, it's about the ebuilds if using rsync, or
> the whole git checkout if using git.

Completely agree with you that this was a great feature to be added from a
security point of view. Without it there was still a level of trust,
however small, that could be placed in the choice of mirror. But there's
no doubt gpg sigs of repo data are order of magnitudes better, so yes it
was a little unjust to describe it as only "nice to have".

But in the current situation I personally consider it so critically
important to get rid of xz utils from my systems that a short, temporary
period of not having this while switching to another method of
verification I consider an acceptable tradeoff (side note to anyone
reading: yes I know how at odds I am with the rest of the world on this,
it has now been argued to death in this thread so for anyone thinking
about replying about that, maybe lets do everyone a favour, agree to
disagree, and move on :-) )

>> hardened, but still with the very high level of security that Gentoo
>> systems have has always had prior to these features, in my opinion.
>> Personally I can live without them for now. Verifying hashes
>> in Manifest files still works fine and that's the main thing. You may
>> disagree in which case, well, don't do this then. I'm going to figure
>> out an alternative way I can verify Portage metadata soon, as there are
>> other ways if you are creative.
>
> See grobian's reply which should help.
>
>
>> [...]
>> - Portage binary packages: You cannot use xz compression if you create
>> Portage binary packages. You will need to use one of bzip2, gzip,
>> lz4, lzip, lzop, or zstd in BINPKG_COMPRESS in make.conf instead of xz
>> (if that is what you were using, or is it the default?). I have
>>
>
> zstd is the default for "new" installs (since a few years ago), yeah.
>
>> [...]
>> - sys-apps/fwupd might stop working properly (though it will still
>> build fine) due to what you have to change with dev-libs/libxmlb below.
>> I'm not sure as I haven't checked yet, I just suspect it
>> will. So bear that in mind if you need to rely on sys-apps/fwupd at the
>> moment. But this "might" is temporary, upstream has now decided to make
>> lzma optional, so this will trickle down to Gentoo soon.
>
> Just for completeness, this is
> https://blogs.gnome.org/hughsie/2024/04/03/fwupd-and-xz-metadata/.
>

Thanks for all the useful additions of info :-)

cheers,
Eddie

Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo

[Joonas Niilola]

[-- Attachment #1.1: Type: text/plain, Size: 5325 bytes --]

On 6.4.2024 14.57, Eddie Chapman wrote:
> 
> --- /usr/portage/net-mail/dovecot/dovecot-2.3.21-r1.ebuild
> +++ /usr/local/portage/net-mail/dovecot/dovecot-2.3.21-r1.ebuild
> @@ -43,7 +43,6 @@
> 
>  DEPEND="
>         app-arch/bzip2
> -       app-arch/xz-utils
>         dev-libs/icu:=
>         dev-libs/openssl:0=
>         sys-libs/zlib:=
> @@ -126,7 +125,7 @@
>                 --disable-rpath \
>                 --with-bzlib \
>                 --without-libbsd \
> -               --with-lzma \
> +               --without-lzma \
>                 --with-icu \
>                 --with-ssl \
>                 --with-zlib \
> 
> --- /usr/portage/dev-lang/python/python-3.11.8_p1.ebuild
> +++ /usr/local/portage/dev-lang/python/python-3.11.8_p1.ebuild
> @@ -179,6 +179,7 @@
>         # Avoid as many dependencies as possible for the cross build.
>         cat >> Makefile <<-EOF || die
>                 MODULE_NIS_STATE=disabled
> +               MODULE__LZMA_STATE=disabled
>                 MODULE__DBM_STATE=disabled
>                 MODULE__GDBM_STATE=disabled
>                 MODULE__DBM_STATE=disabled
> @@ -328,7 +329,7 @@
>         fi
> 
>         # force-disable modules we don't want built
> -       local disable_modules=( NIS )
> +       local disable_modules=( NIS _LZMA )
>         use gdbm || disable_modules+=( _GDBM _DBM )
>         use sqlite || disable_modules+=( _SQLITE3 )
>         use ssl || disable_modules+=( _HASHLIB _SSL )
> 
> 
> --- /usr/portage/dev-lang/python/python-3.12.2_p1.ebuild
> +++ /usr/local/portage/dev-lang/python/python-3.12.2_p1.ebuild
> @@ -177,6 +177,7 @@
>         cat > Modules/Setup.local <<-EOF || die
>                 *disabled*
>                 nis
> +               _lzma
>                 _dbm _gdbm
>                 _sqlite3
>                 _hashlib _ssl
> @@ -299,6 +300,7 @@
>         cat > Modules/Setup.local <<-EOF || die
>                 *disabled*
>                 nis
> +               _lzma
>                 $(usev !gdbm '_gdbm _dbm')
>                 $(usev !sqlite '_sqlite3')
>                 $(usev !ssl '_hashlib _ssl')
> 
> 
> Lastly, I needed to create a custom dev-libs/libxmlb ebuild in order to
> upgrade it from 0.3.14 (latest in Gentoo at time of writing) to 0.3.15.
> 
> I also needed to apply a very recent patch from upstream, from this
> commit, which makes LZMA support optional:
> https://github.com/hughsie/libxmlb/commit/bdf845510fbed40b88465b2272ccad9e93656639
> 
> and I needed to make some small changes to the ebuild.
> 
> So this is what you need to do at the time of writing (6th April 2024):
> 
> 1. Copy the in-tree /usr/portage/dev-libs/libxmlb ebuild directory into
> your local ebuilds directory.
> 
> 2. Rename the ebuild file from libxmlb-0.3.14.ebuild to
> libxmlb-0.3.15.ebuild
> 
> 3. Download the raw patch, you can use this link:
> 
> https://github.com/hughsie/libxmlb/commit/bdf845510fbed40b88465b2272ccad9e93656639.patch
>    rename it to:
>    libxmlb-0.3.15-make_lzma_optional.patch
>    and place it in the local "files" directory.
> 
> 4. Modify the new ebuild according to the diff below. Then just rebuild it.
> 
> --- /usr/portage/dev-libs/libxmlb/libxmlb-0.3.14.ebuild
> +++ /usr/local/portage/dev-libs/libxmlb/libxmlb-0.3.15.ebuild
> @@ -14,15 +14,15 @@
>  SLOT="0/2" # libxmlb.so version
> 
>  KEYWORDS="amd64 ~arm arm64 ~loong ppc ppc64 ~riscv x86"
> -IUSE="doc introspection stemmer test +zstd"
> +IUSE="doc introspection -lzma stemmer test +zstd"
> 
>  RESTRICT="!test? ( test )"
> 
>  RDEPEND="
> -       app-arch/xz-utils
>         dev-libs/glib:2
>         sys-apps/util-linux
>         stemmer? ( dev-libs/snowball-stemmer:= )
> +       lzma? ( app-arch/xz-utils:= )
>         zstd? ( app-arch/zstd:= )
>  "
> 
> @@ -43,6 +43,7 @@
> 
>  PATCHES=(
>         "${FILESDIR}"/${PN}-0.3.12-no_installed_tests.patch
> +       "${FILESDIR}"/${PN}-0.3.15-make_lzma_optional.patch
>  )
> 
>  python_check_deps() {
> @@ -60,6 +61,7 @@
>                 $(meson_use stemmer)
>                 $(meson_use test tests)
>                 $(meson_use zstd)
> +               $(meson_feature lzma)
>         )
>         meson_src_configure
>  }

Hey,

I'll admit I didn't read everything, but I just want to point out you
may not have to edit ebuilds at all. If xz-utils is package.provided
portage should ignore the dependency without you removing the dep from
an ebuild. Then you can utilize /etc/portage/patches to apply any
patches and finally try using EXTRA_ECONF and MYMESONARGS to override
configure options via package.env.

-- juippis

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 618 bytes --]

[gentoo-dev] Re: Current unavoidable use of xz utils in Gentoo

[Duncan]

Joonas Niilola posted on Thu, 11 Apr 2024 08:21:39 +0300 as excerpted:

> I just want to point out you
> may not have to edit ebuilds at all. If xz-utils is package.provided
> portage should ignore the dependency without you removing the dep from
> an ebuild.

package.provided:

YMMV, but here rather than doing package.provided I create a "null-ebuild" 
for the package in my overlay.  Much like virtual/* packages from which I 
took the idea except of course that they're named as the category/package 
they're replacing instead of virtual/*, null-ebuilds install no files but 
allow detailed control of IUSE, slot, etc, in case some of the revdeps 
need that.

For versioning, my convention is -999 or -n.999 to imply a virtual (tho I 
do have a real perl bigint package v 1.999.842 installed...), much like 
the -9999/-n.9999 variants imply a live-package, with similar effect in 
terms of preferring it to any reasonable real version number as well.   So 
for xz-utils it'd be app-arch/xz-utils-999 as it's not slotted, or app-
arch/xz-utils-5.999 if it were or if something wants 5.x specifically.  Or 
use five-nines or six-nines or ten nines...

A null-pkg I actually use here? sys-fs/udisks-2.999:2 (slot 2 dep actually 
required by some of its rev-deps).  udisks itself is a script so doesn't 
provide headers necessary to build other things and should be a runtime-
only dep.  As a script the installation itself would be too trivial to 
bother with, were it not for its absolutely wicked pulled-in deps for 
functionality I'm not going to use and don't have turned on for my kernel 
in any case.  Luckily kde/solid/kio/etc degrade functionality gracefully 
if their attempted udisks calls return command-not-found, making it an 
ideal candidate for null-pkging.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo

[Eddie Chapman]

Joonas Niilola wrote:
> Hey,
> 
> I'll admit I didn't read everything, but I just want to point out you
> may not have to edit ebuilds at all. If xz-utils is package.provided 
> portage should ignore the dependency without you removing the dep from an
> ebuild. Then you can utilize /etc/portage/patches to apply any patches and
> finally try using EXTRA_ECONF and MYMESONARGS to override configure
> options via package.env.
> 
> -- juippis

Hi Joonas,

The local ebuilds in the guide were not created because of the xz-utils 
dep. If you search through ebuilds in the tree there are hundreds of 
packages that specify xz-utils as a hard dep, so yes, as you say, 
package.provided takes care of all of then.

No, the ebuilds were needed for various customisations to build 
arguments. However, the dev-libs/libxmlb ebuild is no longer needed as, 
since I wrote the guide, libxmlb 0.3.17, which makes liblzma.so.5 dep 
optional, is now in Gentoo, thanks whoever added that :-)

You might be able to dispense with the need for the separate 
net-mail/dovecot ebuild by using EXTRA_ECONF, as you say. However, 
AFAICS local dev-lang/python ebuilds are unavoidable, unfortunately, 
you'll see why if you look at the diffs for them in my guide. It would 
be wonderful if dev-lang/python made its liblzma dep optional. It would 
be a simple change to the ebuild. However, I suspect the developers 
might feel that *not* depending on liblzma.so.5 is unsupported because 
it results in Gemato failing due to lack of support in core python for 
liblzma. The only way around that issue I can see is for Gemato to 
instead use /usr/bin/xz like the rest of Portage does. If that were to 
happen then dev-lang/python could be modified to respect -lzma and I 
can't see that anything significant in Gentoo would miss it. Then if 
there are any dev-python packages that need liblzma in core python 
either presently or in future (I've not encountered any yet) then of 
course they would just need a hard dependency on dev-lang/python with 
lzma USE flag set.

I now have many Gentoo systems running without xz-utils installed (using 
my wrapper script from the guide) and I've not had a single issue 
anywhere, everything working perfectly, so I'm delighted that it has 
been possible :-)

Eddie