Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo

["Eddie Chapman" <eddie@ehuk.net>]

Eddie Chapman wrote:
> Michał Górny wrote:
>
>> On Sat, 2024-03-30 at 14:57 +0000, Eddie Chapman wrote:
>>
>>
>>> Note, I'm not advocating ripping xz-utils out of tree, all I'm saying
>>>  is wouldn't it be nice if there were at least 2 alternatives to
>>> choose from? That doesn't have to be disruptive in any way, people who
>>> wish to continue using and trusting xz-utils should be able to
>>> continue to do so without any friction whatsoever.
>>
>> So, you're basically saying we should go out of our way, recompress all
>>  distfiles using two alternative compression formats, increase mirror
>> load four times and add a lot of complexity to ebuilds, right?
>>
>> --
>> Best regards,
>> Michał Górny
>>
> Yes that's a very good point, that was something I was wondering in
> weighing up both sides, what the costs would be practically, as I don't
> know the realities of running Gentoo infrastructure. And maybe the costs
> is just too high of a price to pay.
>
> I wonder if increased use of git repos rather than distributed tarballs
> could be part of a solution to those issues, although that could put quite
>  a storage burden on every user. Unless they were all shallow git pulls
> and the user could optionally choose to tar up the git directory after
> clone with compression.  But yes granted then there is even more ebuild
> complexity.
>

I've been thinking a little about how Gentoo without
compression/decompression of distfiles could work, as a feature, without
any impact on the existing world order, and no increased stress on Gentoo
infra. I was wondering how palatable the following idea might be to others
...

The basis of the idea is to add a feature to Portage which would let a
person optionally indicate in make.conf that whenever a path in SRC_URI
resolves to a file with a compression extension (.gz, .bz2, .xz, etc),
that Portage should attempt to fetch it without the compression extension.

So as an example, lets take sys-apps/pciutils, which currently has:
SRC_URI="https://mj.ucw.cz/download/linux/pci/${P}.tar.gz"

the feature would tell portage to simply translate this to:
SRC_URI="https://mj.ucw.cz/download/linux/pci/${P}.tar"

So perhaps it could be a flag that goes in FEATURES= called something like
"strip_dist_comp" or something similar, or maybe someone has a better idea
about that.

Now, of course, I'm not proposing that Gentoo infra keeps uncompressed
versions of distfiles. So by default Portage would encounter a 404 error
when it tries to fetch the uncompressed file from Gentoo mirrors.

However, this feature would then pave the way for a person to then
configure Portage to fetch distfiles from their own server as well as
Gentoo mirrors, and that person could then keep their own uncompressed
versions of distfiles on their  server, for however many and whichever
distfiles they might wish to keep there, as the compressed version would
get fetched from a Gentoo mirror if the uncompressed version is not there.
Such a person would then have to obtain or create their own uncompressed
distfile independently.

A caveat of this solution would be that one would have to disable checksum
verification (and gpg checks?) for this to work, as of course there would
be no checksum for the uncompressed version in the Manifest, and Gentoo
infra certainly should not be expected to especially uncompress each
distfile once in order to generate an extra checksum for the Manifest. In
fact I'd consider than undesirable, as anyone paranoid enough to want to
do this would not trust such a checksum anyway, since it would be a
checksum of a file that has been compressed at source and then
decompressed on Gentoo infra, potentially introducing vulnerabilities.
However, the lack of checksum is not a problem for someone who wants to
keep distfiles on their own server, as such a person can also be
responsible themselves for first verifying whatever they put on there, and
for keeping said server secured from tampering.

This seems to me to be something that would probably be relatively
straightforward to implement within Portage, maybe with just a few lines
around the python code that fetches the SRC_URI, and zero extra work or
resources required from Gentoo infra.

I'd consider it a feature for anyone who wants to eliminate a whole
potential class of vulnerabilities that may or may not be present either
now or in future in compression algorithm tools. Surely that would be a
nice feature to have for some folk?