From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by finch.gentoo.org (Postfix) with ESMTPS id B213F138334 for ; Sun, 18 Nov 2018 21:10:44 +0000 (UTC) Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id 5E9E4E0AB4; Sun, 18 Nov 2018 21:10:40 +0000 (UTC) Received: from smarthost01c.mail.zen.net.uk (smarthost01c.mail.zen.net.uk [212.23.1.5]) (using TLSv1.2 with cipher DHE-RSA-AES128-SHA256 (128/128 bits)) (No client certificate requested) by pigeon.gentoo.org (Postfix) with ESMTPS id D1492E0A9C for ; Sun, 18 Nov 2018 21:10:39 +0000 (UTC) Received: from [62.3.120.142] (helo=NeddySeagoon_Static) by smarthost01c.mail.zen.net.uk with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.80) (envelope-from ) id 1gOUL0-0008Mr-7B for gentoo-dev@lists.gentoo.org; Sun, 18 Nov 2018 21:10:38 +0000 Date: Sun, 18 Nov 2018 21:10:03 +0000 From: Roy Bamford Subject: Fwd: Re: [gentoo-dev] [pre-GLEP] Gentoo binary package container format [gentoo@jonesmz.com] To: gentoo-dev@lists.gentoo.org X-Mailer: Balsa 2.5.3 Message-Id: <8wbjQMoEQy/EntGTUihxxc@IqujqQJNQp+Tbney0Ttn8> Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-dev@lists.gentoo.org Reply-to: gentoo-dev@lists.gentoo.org MIME-Version: 1.0 Content-Type: multipart/signed; micalg=PGP-SHA256; protocol="application/pgp-signature"; boundary="=-hlPXkjdMTBcR5NJRLkde" X-Originating-smarthost01c-IP: [62.3.120.142] Feedback-ID: 62.3.120.142 X-Archives-Salt: 7aa6b41a-8e72-4b74-8daf-1a9c0d8031f6 X-Archives-Hash: 28eddf4511a904c4aaa76c6a14bf090e --=-hlPXkjdMTBcR5NJRLkde Content-Type: multipart/mixed; boundary="=-9706ftanGeddlkdbyhdR" --=-9706ftanGeddlkdbyhdR Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable See attached. Replying off list because I am not on the whitelist ... --=20 Regards, Roy Bamford (Neddyseagoon) a member of elections gentoo-ops forum-mods = --=-9706ftanGeddlkdbyhdR Content-Type: message/rfc822 Return-Path: X-Original-To: neddyseagoon@gentoo.org Delivered-To: neddyseagoon@gentoo.org Received: from localhost (localhost [127.0.0.1]) by smtp.gentoo.org (Postfix) with ESMTP id 0C520335C5D for ; Sun, 18 Nov 2018 20:58:06 +0000 (UTC) X-Virus-Scanned: by amavisd-new using ClamAV at gentoo.org X-Spam-Flag: NO X-Spam-Score: -101.9 X-Spam-Level: X-Spam-Status: No, score=-101.9 tagged_above=-9999 required=5.5 tests=[BAYES_00=-1.9, SHORTCIRCUIT=-100] autolearn=disabled Received: from smtp.gentoo.org ([127.0.0.1]) by localhost (smtp.gentoo.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id dg2iLG8J4FST for ; Sun, 18 Nov 2018 20:58:05 +0000 (UTC) Received: from mail-qk1-f196.google.com (mail-qk1-f196.google.com [209.85.222.196]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.gentoo.org (Postfix) with ESMTPS id 9C039335C39 for ; Sun, 18 Nov 2018 20:58:02 +0000 (UTC) Received: by mail-qk1-f196.google.com with SMTP id 189so45664016qkj.8 for ; Sun, 18 Nov 2018 12:58:02 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=NcwqtM0MdOrpIW3JPPKkUOknHrHbeE+73p5XVq8lqos=; b=ItMy+JXnnXeAl9gVCoacrxaQgUf4/VmufftPgpIU8Tg8o1iI/AeBif+ulbURZe5Fky Zs050bzi5dfLypDXtaPP50KthgeEbVWgv8sNsp73pBaDWgDgEnILVAqKfKap5I7EQQLt NsjcbyGeamvAA/Zn6Rr4s9vWZA7H9BLBZqpNBG5OV7L2HfWIc6M2p4+tQaeSaQGghYIE pGEQxjs7oSn3pwVsFqrdqNBAwTG4YZZVzLCm1tn8yBOVJEP6TaJaqU2S6SjOEi0bLVsw FQOe3efAsEeeNAlwDMRNIwICqOFYDWr57KWdTkGeeo0RIclyiWlGq4X5uo8AfWLneEVP Pi7w== X-Gm-Message-State: AGRZ1gIYRoyr0QvarZ+rTMD6Ic2XGQAuQ67iLVKVbJfR2aIYUVzg+mG3 ddqSPLzV6cQs3Poptpu+1JDGdf911F7ah7FUD89RLw== X-Google-Smtp-Source: AJdET5c+z2AgZ6uOjcS/fiz9PsxwELSiFTJKiAkFnmN2AvFPUe2V845615j4ytVhNQ2SsHj5yV2Rk PBzcjytJt6KmN0= X-Received: by 2002:ac8:2906:: with SMTP id y6mr18179258qty.42.1542574681085; Sun, 18 Nov 2018 12:58:01 -0800 (PST) MIME-Version: 1.0 References: <1542533931.1293.23.camel@gentoo.org> In-Reply-To: From: Michael Jones Date: Sun, 18 Nov 2018 14:57:49 -0600 Message-ID: Subject: Re: [gentoo-dev] [pre-GLEP] Gentoo binary package container format To: gentoo-dev@lists.gentoo.org, neddyseagoon@gentoo.org, grobian@gentoo.org Status: R X-Status: Content-Type: multipart/alternative; boundary=000000000000e7eec8057af6a79f --000000000000e7eec8057af6a79f Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable On Sun, Nov 18, 2018 at 5:04 AM Roy Bamford wrote= : > On 2018.11.18 09:38, Micha=C5=82 G=C3=B3rny wrote: > > On Sun, 2018-11-18 at 10:16 +0100, Fabian Groffen wrote: > > > On 17-11-2018 12:21:40 +0100, Micha=C5=82 G=C3=B3rny wrote: > > > > Problems with the current binary package format > > [snip] > > > > > 2. **The format relies on obscure compressor feature of ignoring > > > > trailing garbage**. While this behavior is traditionally > > implemented > > > > by many compressors, the original reasons for it have become > > long > > > > irrelevant and it is not surprising that new compressors do not > > > > support it. In particular, Portage already hit this problem > > twice: > > > > once when users replaced bzip2 with parallel-capable pbzip2 > > > > implementation [#PBZIP2]_, and the second time when support for > > zstd > > > > compressor was added [#ZSTD]_. > > > > > > I think this is actually the result of a rather opportunistic > > > implementation. The fault is that we chose to use an extension that > > > suggests the file is a regular compressed tarball. > > > When one detects that a file is xpak padded, it is trivial to feed > > the > > > decompressor just the relevant part of the datastream. The format > > > itself isn't bad, and doesn't rely on obscure behaviour. > > > > Except if you don't have the proper tools installed. In which case > > the 'opportunistic' behavior made it possible to extract the contents > > without special tools... except when it actually happens not to work > > anymore. Roy's reply indicates that there is actually interest in > > this > > design feature. > > > [snip] > > Team, > > I use to post something like https://wiki.gentoo.org/wiki/Fix_My_Gentoo > with a link to Patricks binhost on the forums every three or four months. > It made it worth writing that wiki page anyway. > > We still get users removing elements of their toolchain or glbc from time > to time. The requirement that I didn't express very well, is that it > shall > be possible to install binary packages without the use of any Gentoo > specific tooling. > > The current tarball of tarballs proposal would satisfy that requirement. > > Its unlikely that a custom binary format would. Of course, this being > Gentoo someone would write a run anywhere script that did the > unpicking, We already have deb2targz and rpm2targz. We have the > opportunity to design out binpgk2targz before it exists. > > -- > Regards, > > Roy Bamford > (Neddyseagoon) a member of > elections > gentoo-ops > forum-mods > Replying off list because I am not on the whitelist. Please also consider my use case: I have a cluster file system, cephfs, which all of my gentoo machines mount for access to various shared file resources. I want to have all of them mount a cephfs path to the folder which portage is configured to look for binary packages. This works great if all of the machines have identical portage configurations, but breaks down as soon as one machine uses a different use flag. The reason for this is that the package file names do not encode anything other than the package name and version number. So if a binpkg already exists in my binpkg repository, and another machine builds with different use flags, the binpkg gets overwritten, potentially while a third machine is reading the binpkg file. The filename also does not represent compile time dependencies, or any number of other possible points of differentiation This issue could be (at least partially) solved at least 3 ways. 1) append a uuid to each filename. Generated when the bin package file is generated. 2) encode the hostname of the machine that generated the file 3) encode the use flags in the filename. Perhaps a fuller solution is to respect an environment variable "BINARY_PKG_FILENAME_FORMAT" that accepts a series of variable substitutions to append after the package name and version number? This variable would be used only when generating the binary package. Portage would still use any binary package that it found that matched its needs, regardless of suffix. Thanks for your time. --000000000000e7eec8057af6a79f Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
On Sun, Nov 18= , 2018 at 5:04 AM Roy Bamford <neddyseagoon@gentoo.org> wrote:
On 2018.11.18 09:38, Micha=C5=82 G=C3=B3rny wrote:
> On Sun, 2018-11-18 at 10:16 +0100, Fabian Groffen wrote:
> > On 17-11-2018 12:21:40 +0100, Micha=C5=82 G=C3=B3rny wrote:
> > > Problems with the current binary package format

[snip]

> > > 2. **The format relies on obscure compressor feature of igno= ring
> > >=C2=A0 =C2=A0 trailing garbage**.=C2=A0 While this behavior i= s traditionally
> implemented
> > >=C2=A0 =C2=A0 by many compressors, the original reasons for i= t have become
> long
> > >=C2=A0 =C2=A0 irrelevant and it is not surprising that new co= mpressors do not
> > >=C2=A0 =C2=A0 support it.=C2=A0 In particular, Portage alread= y hit this problem
> twice:
> > >=C2=A0 =C2=A0 once when users replaced bzip2 with parallel-ca= pable pbzip2
> > >=C2=A0 =C2=A0 implementation [#PBZIP2]_, and the second time = when support for
> zstd
> > >=C2=A0 =C2=A0 compressor was added [#ZSTD]_.
> >
> > I think this is actually the result of a rather opportunistic
> > implementation.=C2=A0 The fault is that we chose to use an extens= ion that
> > suggests the file is a regular compressed tarball.
> > When one detects that a file is xpak padded, it is trivial to fee= d
> the
> > decompressor just the relevant part of the datastream.=C2=A0 The = format
> > itself isn't bad, and doesn't rely on obscure behaviour.<= br> >
> Except if you don't have the proper tools installed.=C2=A0 In whic= h case
> the 'opportunistic' behavior made it possible to extract the c= ontents
> without special tools... except when it actually happens not to work > anymore.=C2=A0 Roy's reply indicates that there is actually intere= st in
> this
> design feature.
>
[snip]

Team,

I use to post something like https://wiki.gentoo.org/wiki/= Fix_My_Gentoo
with a link to Patricks binhost on the forums every three or four months. <= br> It made it worth writing that wiki page anyway.

We still get users removing elements of their toolchain or glbc from time to time.=C2=A0 The requirement that I didn't express very well, is that= it shall
be possible to install binary packages without the use of any Gentoo
specific tooling.

The current tarball of tarballs proposal would satisfy that requirement.
Its unlikely that a custom binary format would.=C2=A0 Of course, this being=
Gentoo someone would write a run anywhere script that did the
unpicking, We already have deb2targz and rpm2targz. We have the
opportunity to design out binpgk2targz before it exists.

--
Regards,

Roy Bamford
(Neddyseagoon) a member of
elections
gentoo-ops
forum-mods


Replying off list = because I am not on the whitelist.

Please also consider my use case:

I have a cluster file system, cephfs, which all of my gentoo= machines mount for access to various shared file resources.

I want to have all of them mount a ce= phfs path to the folder which portage is configured to look for binary pack= ages.

This works great i= f all of the machines have identical portage configurations, but breaks dow= n as soon as one machine uses a different use flag.=C2=A0

The reason for this is that the package f= ile names do not encode anything other than the package name and version nu= mber. So if a binpkg already exists in my binpkg repository, and another ma= chine builds with different use flags, the binpkg gets overwritten, potenti= ally while a third machine is reading the binpkg file.

The filename also does not represent compile= time dependencies, or any number of other possible points of differentiati= on

This issue could be (= at least partially) solved at least 3 ways.

1) append a uuid to each filename. Generated when the b= in package file is generated.=C2=A0
2) encode the ho= stname of the machine that generated the file
3) enc= ode the use flags in the filename.

Perhaps a fuller solution is to respect an environment variable = "BINARY_PKG_FILENAME_FORMAT" that accepts a series of variable su= bstitutions to append after the package name and version number?

This variable would be used only w= hen generating the binary package. Portage would still use any binary packa= ge that it found that matched its needs, regardless of suffix.

Thanks for your time.



=C2=A0
--000000000000e7eec8057af6a79f-- --=-9706ftanGeddlkdbyhdR-- --=-hlPXkjdMTBcR5NJRLkde Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- iQEzBAABCAAdFiEEsOrcx0gZrrCMwJzo/xJODTqpeT4FAlvx1SsACgkQ/xJODTqp eT5/YAgArobVdqWRNAX8LhuF0sOQJmr/goyfWxQ0+QFLCFlwBVs24PTOrD+hghpA +joYJD6teZJ6FGlew3ApGcBVHbhBZNIu5j1xT4uRi6Ff+xEEjR4NchnHHPtlCKSB BXgZP61l6V+RjpWKpHtcLGJWaB0QDB8/uMtjrMA+rnIFIwfIp1xupSEbPve+GAVi IdqoKMHmBf8IkkEiyD0HqsvVEzD+vsD49PX4VaOOZiiFVvZ96Uhym7gT4oqJfX/I i1Fltzmfep5BEZHlRggA9LEUijFaHlrBeXtR1MUn+e0h5KStX7QYxNdOkqk4Urov x9g9OrOoIhvWSAUPL2d0Q7Yh6bSLHw== =CRPu -----END PGP SIGNATURE----- --=-hlPXkjdMTBcR5NJRLkde--