From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80]) by finch.gentoo.org (Postfix) with ESMTP id E2F771389E2 for ; Wed, 24 Dec 2014 01:51:54 +0000 (UTC) Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id 41C64E07ED; Wed, 24 Dec 2014 01:51:53 +0000 (UTC) Received: from smtp.gentoo.org (smtp.gentoo.org [140.211.166.183]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by pigeon.gentoo.org (Postfix) with ESMTPS id 89512E07DF for ; Wed, 24 Dec 2014 01:51:52 +0000 (UTC) Received: from [10.0.31.246] (unknown [100.42.98.7]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) (Authenticated sender: zmedico) by smtp.gentoo.org (Postfix) with ESMTPSA id DA6193405E0 for ; Wed, 24 Dec 2014 01:51:51 +0000 (UTC) Message-ID: <549A1C31.8040500@gentoo.org> Date: Tue, 23 Dec 2014 17:51:45 -0800 From: Zac Medico User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.3.0 Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-portage-dev@lists.gentoo.org Reply-to: gentoo-portage-dev@lists.gentoo.org MIME-Version: 1.0 To: gentoo-portage-dev@lists.gentoo.org Subject: [gentoo-portage-dev] [RFC] New file layout for PKGDIR and binhosts Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Archives-Salt: a2c8275a-b563-45df-8570-cf943e9c1331 X-Archives-Hash: e5ab624c5563a10e36c7c2259f8d903b Hi, As discussed in bug 150031 [1], it would be useful if PKGDIR could accommodate multiple binary packages built from the same source ebuild. Use cases for preserving multiple builds typically involve supporting multiple clients (with partially compatible configurations) from a single unified binhost. In this context, some of the reasons to retain multiple builds are: * Different USE flag combinations enabled (--newuse/--binpkg-respect-use needed) * Different versions of installed dependencies (EAPI 5 slot := operators needed) * Different repositories/overlays, with variance in the time of the last sync (--changed-deps/--binpkg-changed-deps needed if dependencies change due to eclass changes or ebuild modifications without revbump) Given the above variety of reasons to retain previous builds, a simple counter (1, 2, 3,...) seems like a reasonable means to generate unique file names. In order to avoid having too many files in a directory, we can use a separate directory for each ${CATEGORY}/${PN}, like we do for the source ebuild repositories. In order to avoid having to deal with multiple file extensions for different compression types, we can simply use .xpak for the file extension [2], since that's the name of the format that we use to append metadata to our existing tbz2 files. We can simply probe the first few bytes of the file in order to determine the compression type: gzip: 1f 8b bzip2: 42 5a 68 39 xz: fd 37 7a 58 5a 00 Users will be able change their compression settings at any time, but the .xpak file extension will remain constant regardless of that setting. It won't matter if they have a mixture of files compressed with different compressors. A tool like eclean-pkg will be needed to clean up old binary packages based on user preferences. We might also provide a variety of on-the-fly garbage collection settings. Based on the above discussion, the location of any particular binary package can be expressed as follows: ${PKGDIR}/${CATEGORY}/${PN}/${PF}-${COUNTER}.xpak The existing format of the ${PKGDIR}/Packages index will work fine, since it allows each package to specify a PATH attribute which corresponds to the path of the file relative to the base directory. If the .xpak files use bzip2 compression, it will even be compatible with existing clients (though they won't be able to intelligently choose between multiple packages of the same version). If all the packages of a given version are ordered by ${COUNTER}, then existing clients will simply download the latest build. [1] https://bugs.gentoo.org/show_bug.cgi?id=150031 [2] http://dev.gentoo.org/~zmedico/portage/doc/man/xpak.5.html -- Thanks, Zac