From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80]) by finch.gentoo.org (Postfix) with ESMTP id 1C5061389E2 for ; Wed, 24 Dec 2014 05:14:49 +0000 (UTC) Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id 1CD6BE08D5; Wed, 24 Dec 2014 05:14:47 +0000 (UTC) Received: from mx1.mthode.org (216-82-208-22.static.grandenetworks.net [216.82.208.22]) by pigeon.gentoo.org (Postfix) with ESMTP id 72178E08D2 for ; Wed, 24 Dec 2014 05:14:46 +0000 (UTC) Received: from [IPv6:2001:470:e1cc:2:a75f:95a1:31c1:5f8] (unknown [IPv6:2001:470:e1cc:2:a75f:95a1:31c1:5f8]) (using TLSv1 with cipher ECDHE-RSA-AES128-SHA (128/128 bits)) (No client certificate requested) by mx1.mthode.org (Postfix) with ESMTPSA id 9DAD81B6E2 for ; Wed, 24 Dec 2014 00:15:52 -0500 (EST) Message-ID: <549A4C25.9050000@gentoo.org> Date: Tue, 23 Dec 2014 23:16:21 -0600 From: Matthew Thode User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.3.0 Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-portage-dev@lists.gentoo.org Reply-to: gentoo-portage-dev@lists.gentoo.org MIME-Version: 1.0 To: gentoo-portage-dev@lists.gentoo.org Subject: Re: [gentoo-portage-dev] [RFC] New file layout for PKGDIR and binhosts References: <549A1C31.8040500@gentoo.org> In-Reply-To: <549A1C31.8040500@gentoo.org> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Archives-Salt: 6ba47091-92c8-4498-b9c3-0c66cb80644a X-Archives-Hash: 48a14b09a49a5f9d1b300b0a826c82d8 On 12/23/2014 07:51 PM, Zac Medico wrote: > Hi, > > As discussed in bug 150031 [1], it would be useful if PKGDIR could > accommodate multiple binary packages built from the same source ebuild. > Use cases for preserving multiple builds typically involve supporting > multiple clients (with partially compatible configurations) from a > single unified binhost. In this context, some of the reasons to retain > multiple builds are: > > * Different USE flag combinations enabled (--newuse/--binpkg-respect-use > needed) > > * Different versions of installed dependencies (EAPI 5 slot := operators > needed) > > * Different repositories/overlays, with variance in the time of the last > sync (--changed-deps/--binpkg-changed-deps needed if dependencies change > due to eclass changes or ebuild modifications without revbump) > > Given the above variety of reasons to retain previous builds, a simple > counter (1, 2, 3,...) seems like a reasonable means to generate unique > file names. > > In order to avoid having too many files in a directory, we can use a > separate directory for each ${CATEGORY}/${PN}, like we do for the source > ebuild repositories. > > In order to avoid having to deal with multiple file extensions for > different compression types, we can simply use .xpak for the file > extension [2], since that's the name of the format that we use to append > metadata to our existing tbz2 files. We can simply probe the first few > bytes of the file in order to determine the compression type: > > gzip: 1f 8b > bzip2: 42 5a 68 39 > xz: fd 37 7a 58 5a 00 > > Users will be able change their compression settings at any time, but > the .xpak file extension will remain constant regardless of that > setting. It won't matter if they have a mixture of files compressed with > different compressors. > > A tool like eclean-pkg will be needed to clean up old binary packages > based on user preferences. We might also provide a variety of on-the-fly > garbage collection settings. > > Based on the above discussion, the location of any particular binary > package can be expressed as follows: > > ${PKGDIR}/${CATEGORY}/${PN}/${PF}-${COUNTER}.xpak > > The existing format of the ${PKGDIR}/Packages index will work fine, > since it allows each package to specify a PATH attribute which > corresponds to the path of the file relative to the base directory. If > the .xpak files use bzip2 compression, it will even be compatible with > existing clients (though they won't be able to intelligently choose > between multiple packages of the same version). If all the packages of a > given version are ordered by ${COUNTER}, then existing clients will > simply download the latest build. > > [1] https://bugs.gentoo.org/show_bug.cgi?id=150031 > [2] http://dev.gentoo.org/~zmedico/portage/doc/man/xpak.5.html > I like this (and it has been a long time coming). What format are we going to store the metadata of the use flag combinations and the rest? I guess that's already stored since portage knows not to use binpkgs if those change. Also, would this change be a good time to change to store that metadata externally? Running portage over NFS with binpkgs takes forever, I don't think a binhost makes it faster either. If there were some way to get all the info for the binpkgs into one file (so it could be run on cron or something), this could mean that I'd only have to do one file request for all that metadata and would be much quicker than inspecting all those files. -- -- Matthew Thode (prometheanfire)