Re: [gentoo-portage-dev] [RFC] New file layout for PKGDIR and binhosts

[Matthew Thode <prometheanfire@gentoo.org>]

On 12/23/2014 07:51 PM, Zac Medico wrote:
> Hi,
> 
> As discussed in bug 150031 [1], it would be useful if PKGDIR could
> accommodate multiple binary packages built from the same source ebuild.
> Use cases for preserving multiple builds typically involve supporting
> multiple clients (with partially compatible configurations) from a
> single unified binhost. In this context, some of the reasons to retain
> multiple builds are:
> 
> * Different USE flag combinations enabled (--newuse/--binpkg-respect-use
> needed)
> 
> * Different versions of installed dependencies (EAPI 5 slot := operators
> needed)
> 
> * Different repositories/overlays, with variance in the time of the last
> sync (--changed-deps/--binpkg-changed-deps needed if dependencies change
> due to eclass changes or ebuild modifications without revbump)
> 
> Given the above variety of reasons to retain previous builds, a simple
> counter (1, 2, 3,...) seems like a reasonable means to generate unique
> file names.
> 
> In order to avoid having too many files in a directory, we can use a
> separate directory for each ${CATEGORY}/${PN}, like we do for the source
> ebuild repositories.
> 
> In order to avoid having to deal with multiple file extensions for
> different compression types, we can simply use .xpak for the file
> extension [2], since that's the name of the format that we use to append
> metadata to our existing tbz2 files. We can simply probe the first few
> bytes of the file in order to determine the compression type:
> 
>       gzip: 1f 8b
>      bzip2: 42 5a 68 39
>         xz: fd 37 7a 58 5a 00
> 
> Users will be able change their compression settings at any time, but
> the .xpak file extension will remain constant regardless of that
> setting. It won't matter if they have a mixture of files compressed with
> different compressors.
> 
> A tool like eclean-pkg will be needed to clean up old binary packages
> based on user preferences. We might also provide a variety of on-the-fly
> garbage collection settings.
> 
> Based on the above discussion, the location of any particular binary
> package can be expressed as follows:
> 
> 	${PKGDIR}/${CATEGORY}/${PN}/${PF}-${COUNTER}.xpak
> 
> The existing format of the ${PKGDIR}/Packages index will work fine,
> since it allows each package to specify a PATH attribute which
> corresponds to the path of the file relative to the base directory. If
> the .xpak files use bzip2 compression, it will even be compatible with
> existing clients (though they won't be able to intelligently choose
> between multiple packages of the same version). If all the packages of a
> given version are ordered by ${COUNTER}, then existing clients will
> simply download the latest build.
> 
> [1] https://bugs.gentoo.org/show_bug.cgi?id=150031
> [2] http://dev.gentoo.org/~zmedico/portage/doc/man/xpak.5.html
> 
I like this (and it has been a long time coming).  What format are we
going to store the metadata of the use flag combinations and the rest?
I guess that's already stored since portage knows not to use binpkgs if
those change.

Also, would this change be a good time to change to store that metadata
externally?  Running portage over NFS with binpkgs takes forever, I
don't think a binhost makes it faster either.  If there were some way to
get all the info for the binpkgs into one file (so it could be run on
cron or something), this could mean that I'd only have to do one file
request for all that metadata and would be much quicker than inspecting
all those files.

-- 
-- Matthew Thode (prometheanfire)