public inbox for gentoo-portage-dev@lists.gentoo.org
 help / color / mirror / Atom feed
From: Zac Medico <zmedico@gentoo.org>
To: gentoo-portage-dev@lists.gentoo.org
Subject: [gentoo-portage-dev] [RFC] New file layout for PKGDIR and binhosts
Date: Tue, 23 Dec 2014 17:51:45 -0800	[thread overview]
Message-ID: <549A1C31.8040500@gentoo.org> (raw)

Hi,

As discussed in bug 150031 [1], it would be useful if PKGDIR could
accommodate multiple binary packages built from the same source ebuild.
Use cases for preserving multiple builds typically involve supporting
multiple clients (with partially compatible configurations) from a
single unified binhost. In this context, some of the reasons to retain
multiple builds are:

* Different USE flag combinations enabled (--newuse/--binpkg-respect-use
needed)

* Different versions of installed dependencies (EAPI 5 slot := operators
needed)

* Different repositories/overlays, with variance in the time of the last
sync (--changed-deps/--binpkg-changed-deps needed if dependencies change
due to eclass changes or ebuild modifications without revbump)

Given the above variety of reasons to retain previous builds, a simple
counter (1, 2, 3,...) seems like a reasonable means to generate unique
file names.

In order to avoid having too many files in a directory, we can use a
separate directory for each ${CATEGORY}/${PN}, like we do for the source
ebuild repositories.

In order to avoid having to deal with multiple file extensions for
different compression types, we can simply use .xpak for the file
extension [2], since that's the name of the format that we use to append
metadata to our existing tbz2 files. We can simply probe the first few
bytes of the file in order to determine the compression type:

      gzip: 1f 8b
     bzip2: 42 5a 68 39
        xz: fd 37 7a 58 5a 00

Users will be able change their compression settings at any time, but
the .xpak file extension will remain constant regardless of that
setting. It won't matter if they have a mixture of files compressed with
different compressors.

A tool like eclean-pkg will be needed to clean up old binary packages
based on user preferences. We might also provide a variety of on-the-fly
garbage collection settings.

Based on the above discussion, the location of any particular binary
package can be expressed as follows:

	${PKGDIR}/${CATEGORY}/${PN}/${PF}-${COUNTER}.xpak

The existing format of the ${PKGDIR}/Packages index will work fine,
since it allows each package to specify a PATH attribute which
corresponds to the path of the file relative to the base directory. If
the .xpak files use bzip2 compression, it will even be compatible with
existing clients (though they won't be able to intelligently choose
between multiple packages of the same version). If all the packages of a
given version are ordered by ${COUNTER}, then existing clients will
simply download the latest build.

[1] https://bugs.gentoo.org/show_bug.cgi?id=150031
[2] http://dev.gentoo.org/~zmedico/portage/doc/man/xpak.5.html
-- 
Thanks,
Zac


             reply	other threads:[~2014-12-24  1:51 UTC|newest]

Thread overview: 13+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2014-12-24  1:51 Zac Medico [this message]
2014-12-24  5:16 ` [gentoo-portage-dev] [RFC] New file layout for PKGDIR and binhosts Matthew Thode
2014-12-24  8:13   ` Zac Medico
2014-12-24 12:01     ` vivo75
2014-12-24 16:07       ` Zac Medico
2014-12-24 18:36         ` vivo75
2014-12-24 19:17           ` Zac Medico
2014-12-25 10:03     ` [gentoo-portage-dev] " Duncan
2014-12-25 11:04       ` Zac Medico
2014-12-26  5:20         ` Duncan
2015-01-07  5:32       ` Brian Dolbec
2014-12-27 14:25 ` [gentoo-portage-dev] " Rick "Zero_Chaos" Farina
2014-12-29  3:53   ` Zac Medico

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=549A1C31.8040500@gentoo.org \
    --to=zmedico@gentoo.org \
    --cc=gentoo-portage-dev@lists.gentoo.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox