From mboxrd@z Thu Jan 1 00:00:00 1970
Return-Path: <gentoo-portage-dev+bounces-4344-garchives=archives.gentoo.org@lists.gentoo.org>
Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80])
by finch.gentoo.org (Postfix) with ESMTP id 636241389E2
for <garchives@archives.gentoo.org>; Wed, 24 Dec 2014 19:17:37 +0000 (UTC)
Received: from pigeon.gentoo.org (localhost [127.0.0.1])
by pigeon.gentoo.org (Postfix) with SMTP id D6B8CE087E;
Wed, 24 Dec 2014 19:17:35 +0000 (UTC)
Received: from smtp.gentoo.org (smtp.gentoo.org [140.211.166.183])
(using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits))
(No client certificate requested)
by pigeon.gentoo.org (Postfix) with ESMTPS id 55080E087D
for <gentoo-portage-dev@lists.gentoo.org>; Wed, 24 Dec 2014 19:17:35 +0000 (UTC)
Received: from [10.0.31.246] (unknown [100.42.98.7])
(using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits))
(No client certificate requested)
(Authenticated sender: zmedico)
by smtp.gentoo.org (Postfix) with ESMTPSA id B923F3403C1
for <gentoo-portage-dev@lists.gentoo.org>; Wed, 24 Dec 2014 19:17:34 +0000 (UTC)
Message-ID: <549B1147.4060007@gentoo.org>
Date: Wed, 24 Dec 2014 11:17:27 -0800
From: Zac Medico <zmedico@gentoo.org>
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.3.0
Precedence: bulk
List-Post: <mailto:gentoo-portage-dev@lists.gentoo.org>
List-Help: <mailto:gentoo-portage-dev+help@lists.gentoo.org>
List-Unsubscribe: <mailto:gentoo-portage-dev+unsubscribe@lists.gentoo.org>
List-Subscribe: <mailto:gentoo-portage-dev+subscribe@lists.gentoo.org>
List-Id: Gentoo Linux mail <gentoo-portage-dev.gentoo.org>
X-BeenThere: gentoo-portage-dev@lists.gentoo.org
Reply-to: gentoo-portage-dev@lists.gentoo.org
MIME-Version: 1.0
To: gentoo-portage-dev@lists.gentoo.org
Subject: Re: [gentoo-portage-dev] [RFC] New file layout for PKGDIR and binhosts
References: <549A1C31.8040500@gentoo.org> <549A4C25.9050000@gentoo.org> <549A75C5.70600@gentoo.org> <549AAB08.2050908@gmail.com> <549AE4C4.3080306@gentoo.org> <549B07C9.9090006@gmail.com>
In-Reply-To: <549B07C9.9090006@gmail.com>
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8bit
X-Archives-Salt: e8192bc1-a4c0-4050-8313-c1165d583fc9
X-Archives-Hash: d9d98cc37e506fcf906c682038d9298f
On 12/24/2014 10:36 AM, vivo75@gmail.com wrote:
> Il 24/12/2014 17:07, Zac Medico ha scritto:
>> On 12/24/2014 04:01 AM, vivo75@gmail.com wrote:
>>> Il 24/12/2014 09:13, Zac Medico ha scritto:
>>>>> I like this (and it has been a long time coming). What format are we
>>>>>> going to store the metadata of the use flag combinations and the rest?
>>>> The current approach is to store the data in an xpak segment that is
>>>> appended to the end of the tbz2 file. The $PKGDIR/Packages files serves
>>>> as a cache for the essential parts of the xpak data that are used in
>>>> dependency calculations.
>>>>
>>> I'd like to see the xpak data being put in it's own file at the
>>> _beginning_ of the tar file.
>>>
>>> tar -Jcf \
>>> ${PKGDIR}/${CATEGORY}/${PN}/${PF}-${COUNTER}.xpak \
>>> tmp/${CATEGORY}:${PN}:${PF}-${COUNTER}.xpak \
>>> *all_the_other_stuff*
>>>
>>> this way reading it could be faster on some media and filesystem and it
>>> would not deviate from the standard tar.
>> There wouldn't be any benefit, because the data is practically always
>> read from the $PKGDIR/Packages cache anyway. The cache is generated when
>> the package is built, and the rate-limiting step there is the building
>> of the package.
> ack, and what about emerge on destination host?
The destination host uses a downloaded copy of $PKGDIR/Packages for
dependency calculations.
Also, I suspect that you're drastically over-estimating the cost of
accessing the xpak metadata, since otherwise you wouldn't be asking this
question.
>>> Being in /tmp/ is only for commodity but the place is debatable.
>>> Instead the fact it _must_ be the first file it's not, in a sequential
>>> archive file like tar some things depend on it.
>> With the current approach, the xpak segment is not part of the tar file.
>> The tar file is compressed, and the xpak segment is appended to the end
>> of the resulting bzip2 file.
>>
>>> seem to be the right time to do the change, since tool need to be
>>> rewritten anyway, but I'll leave to you analyze the fallout of this change.
>> There will be zero benefits from doing that.
> I see at least two however admittedly not too big
>
> 1) I see having a canonical tarball as an advantage, opposed to be
> forced to use /usr/bin/{qtbz2,qxpak}
You can already use tar to unpack our existing tbz2/xpak files. You only
need special tools if you want to access the package metadata, and I
think it's reasonable to expect anyone who wants to access the metadata
to have the appropriate tools.
> 2) there is a small benefit in space (which increase using xz) for
> bigger packages it would be smaller in percentage.
>
> -rw-r--r-- 1 root root 8720 24 dic 18.25 linuxtv-dvb-headers-5.8.tar.bz2
> -rw-r--r-- 1 root root 9691 24 dic 18.07 linuxtv-dvb-headers-5.8.tbz2
It think this savings is negligible. The environment.bz2 file accounts
for most of the space consumed by the xpak segment, and it is compressed
in order to save space.
> this has been obtained unpacking the xpak _and_ environment.bz2 (to
> avoid double compression) and putting them in a subdirectory
>
> ls -l tmp/linuxtv-dvb-headers-5.8/ usr
> tmp/linuxtv-dvb-headers-5.8/:
> total 80
> -rw-r--r-- 1 root root 11 Dec 24 18:20 BUILD_TIME
> -rw-r--r-- 1 root root 8 Dec 24 18:20 CATEGORY
> -rw-r--r-- 1 root root 2 Dec 24 18:20 DEFINED_PHASES
> -rw-r--r-- 1 root root 52 Dec 24 18:20 DESCRIPTION
> -rw-r--r-- 1 root root 2 Dec 24 18:20 EAPI
> -rw-r--r-- 1 root root 394 Dec 24 18:20 FEATURES
> -rw-r--r-- 1 root root 1 Dec 24 18:20 IUSE
> -rw-r--r-- 1 root root 30 Dec 24 18:20 KEYWORDS
> -rw-r--r-- 1 root root 24 Dec 24 18:20 PF
> -rw-r--r-- 1 root root 31 Dec 24 18:20 RDEPEND
> -rw-r--r-- 1 root root 2 Dec 24 18:20 SIZE
> -rw-r--r-- 1 root root 2 Dec 24 18:20 SLOT
> -rw-r--r-- 1 root root 65 Dec 24 18:20 USE
> -rw-r--r-- 1 root root 22859 Dec 24 18:21 environment
> -rw-r--r-- 1 root root 443 Dec 24 18:20 linuxtv-dvb-headers-5.8.ebuild
> -rw-r--r-- 1 root root 7 Dec 24 18:20 repository
We could save some more space in the xpak segment if we omitted the
ebuild, since environment.bz2 contains everything that we need from the
ebuild. We could also omit FEATURES, since it's non-essential and it's
included in environment.bz2 anyway. However, these are trivial
micro-optimizations.
--
Thanks,
Zac