Fwd: Re: [gentoo-dev] [pre-GLEP] Gentoo binary package container format [gentoo@jonesmz.com]

[Roy Bamford <neddyseagoon@gentoo.org>]

[-- Attachment #1.1: Type: text/plain, Size: 169 bytes --]

See attached.

Replying off list because I am not on the whitelist ...

-- 
Regards,

Roy Bamford
(Neddyseagoon) a member of
elections
gentoo-ops
forum-mods

[-- Attachment #1.2: Type: message/rfc822, Size: 12975 bytes --]

[-- Attachment #1.2.1.1: Type: text/plain, Size: 4245 bytes --]

On Sun, Nov 18, 2018 at 5:04 AM Roy Bamford <neddyseagoon@gentoo.org> wrote:

> On 2018.11.18 09:38, Michał Górny wrote:
> > On Sun, 2018-11-18 at 10:16 +0100, Fabian Groffen wrote:
> > > On 17-11-2018 12:21:40 +0100, Michał Górny wrote:
> > > > Problems with the current binary package format
>
> [snip]
>
> > > > 2. **The format relies on obscure compressor feature of ignoring
> > > >    trailing garbage**.  While this behavior is traditionally
> > implemented
> > > >    by many compressors, the original reasons for it have become
> > long
> > > >    irrelevant and it is not surprising that new compressors do not
> > > >    support it.  In particular, Portage already hit this problem
> > twice:
> > > >    once when users replaced bzip2 with parallel-capable pbzip2
> > > >    implementation [#PBZIP2]_, and the second time when support for
> > zstd
> > > >    compressor was added [#ZSTD]_.
> > >
> > > I think this is actually the result of a rather opportunistic
> > > implementation.  The fault is that we chose to use an extension that
> > > suggests the file is a regular compressed tarball.
> > > When one detects that a file is xpak padded, it is trivial to feed
> > the
> > > decompressor just the relevant part of the datastream.  The format
> > > itself isn't bad, and doesn't rely on obscure behaviour.
> >
> > Except if you don't have the proper tools installed.  In which case
> > the 'opportunistic' behavior made it possible to extract the contents
> > without special tools... except when it actually happens not to work
> > anymore.  Roy's reply indicates that there is actually interest in
> > this
> > design feature.
> >
> [snip]
>
> Team,
>
> I use to post something like https://wiki.gentoo.org/wiki/Fix_My_Gentoo
> with a link to Patricks binhost on the forums every three or four months.
> It made it worth writing that wiki page anyway.
>
> We still get users removing elements of their toolchain or glbc from time
> to time.  The requirement that I didn't express very well, is that it
> shall
> be possible to install binary packages without the use of any Gentoo
> specific tooling.
>
> The current tarball of tarballs proposal would satisfy that requirement.
>
> Its unlikely that a custom binary format would.  Of course, this being
> Gentoo someone would write a run anywhere script that did the
> unpicking, We already have deb2targz and rpm2targz. We have the
> opportunity to design out binpgk2targz before it exists.
>
> --
> Regards,
>
> Roy Bamford
> (Neddyseagoon) a member of
> elections
> gentoo-ops
> forum-mods
>


Replying off list because I am not on the whitelist.

Please also consider my use case:

I have a cluster file system, cephfs, which all of my gentoo machines mount
for access to various shared file resources.

I want to have all of them mount a cephfs path to the folder which portage
is configured to look for binary packages.

This works great if all of the machines have identical portage
configurations, but breaks down as soon as one machine uses a different use
flag.

The reason for this is that the package file names do not encode anything
other than the package name and version number. So if a binpkg already
exists in my binpkg repository, and another machine builds with different
use flags, the binpkg gets overwritten, potentially while a third machine
is reading the binpkg file.

The filename also does not represent compile time dependencies, or any
number of other possible points of differentiation

This issue could be (at least partially) solved at least 3 ways.

1) append a uuid to each filename. Generated when the bin package file is
generated.
2) encode the hostname of the machine that generated the file
3) encode the use flags in the filename.

Perhaps a fuller solution is to respect an environment variable
"BINARY_PKG_FILENAME_FORMAT" that accepts a series of variable
substitutions to append after the package name and version number?

This variable would be used only when generating the binary package.
Portage would still use any binary package that it found that matched its
needs, regardless of suffix.

Thanks for your time.

[-- Attachment #1.2.1.2: Type: text/html, Size: 5590 bytes --]

[-- Attachment #2: Type: application/pgp-signature, Size: 488 bytes --]