[gentoo-dev] Multi hash support in portage - status

[gentoo-dev] Multi hash support in portage - status

[Marius Mauch]

[-- Attachment #1: Type: text/plain, Size: 3810 bytes --]

So, along with the gpg signing stuff came along again the question to
have multiple hash formats in digests and manifests.

Current status is that portage only generates MD5 checksums and can
verify both MD5 and SHA1 checksums. Creation of SHA1 is also possible
but has so far been disabled as older portage versions would break if
they found a non-MD5 line in digest files (this was fixed somewhere
last year in the .51 series).

Ok I have three modifications that are pending to go into portage:
- The first simply enables creation of SHA1 checksums (and others if
implemented like with the second mod), if you want to try it yourself
see the attached patch.
- Another thing that has been requested often is to offer even more
hashing functions. Earlier today I sent a patch to the
gentoo-portage-dev list that adds optional support for SHA256 and
RMD160 if dev-python/pycrypto is installed on the system.
- The last and most intrusive change is support for a new enhanced
Manifest format (called Manifest2 for now). Don't worry, there will be
GLEP and more info before this gets added, I just list here for
reference below

The first two changes are ready to be added and deployed (in .54), so
without looking at the third one it would be a no brainer. But of
course there is a drawback: The current Manifest/digest format is quite
inefficient wrt storing multiple checksums as it repeats the filename
and filesize for every checksum added, it looks like this:

MD5 82e806ed62f0596fb7bef493d225712f metadata.xml 269
RMD160 39d775de55f9963f8946feaf088aa0324770bacb metadata.xml 269
SHA1 4fd7b285049d0e587f89e86becf06c0fd77bae6d metadata.xml 269
SHA256 3787959f4a775b1e787b35ff8380949d8f68bd67b81c2cf5a748733c9740cb94
metadata.xml 269

The Manifest2 format solves this problem (and has some other benefits)
by listing all checksums on one line:

MISCFILE metadata.xml 269 RMD160
39d775de55f9963f8946feaf088aa0324770bacb SHA1
4fd7b285049d0e587f89e86becf06c0fd77bae6d SHA256
3787959f4a775b1e787b35ff8380949d8f68bd67b81c2cf5a748733c9740cb94 MD5
82e806ed62f0596fb7bef493d225712f

Not much of a difference you might say, but this is just looking at a
pure Manifest2 entry. To keep compability with existing portage
versions we have to list both the old format and the new format in the
Manifest (digests are handled differently with Manifest2, but the
concept also applies to them), which can potentially increase the tree
size by ~10% (at a guess). I'm talking about actual data size here, not
required discspace.
And before you ask "why manifest2 if it adds this overhead?",
the main point isn't the new format but a long term reduction of the
tree size by removing the digest files (but wait for the GLEP to discuss
this).

So much for background information, now to the actual question:
Would you rather have now the ability to create multi-hash digests and
Manifests with the result of a short and mid-term larger portage tree
(in the long term the format will be phased out hopefully) or rather
wait for Manifest2 support (which will definitely include multi hash
support)?

Basically just getting some feedback before adding it and later getting
the complaints about bloating the tree ;)

Note that this is (technically) completely unrelated to gpg signing of
Manifests, so any gpg related bitching doesn't belong here.
Generally only reply here if you're replying to the question I posted
(implementation discussions belong on gentoo-portage-dev, and the
quota for whining/trolling/flaming for this month was already exceeded).

Marius

-- 
Public Key at http://www.genone.de/info/gpg-key.pub

In the beginning, there was nothing. And God said, 'Let there be
Light.' And there was still nothing, but you could see a bit better.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Multi hash support in portage - status

[Marius Mauch]

[-- Attachment #1.1: Type: text/plain, Size: 635 bytes --]

On Thu, 24 Nov 2005 01:04:32 +0100
Marius Mauch <genone@gentoo.org> wrote:

> Ok I have three modifications that are pending to go into portage:
> - The first simply enables creation of SHA1 checksums (and others if
> implemented like with the second mod), if you want to try it yourself
> see the attached patch.

Bah, really have to port my old Outlook mods so I don't keep forgetting
the attachments all the time.

Marius

-- 
Public Key at http://www.genone.de/info/gpg-key.pub

In the beginning, there was nothing. And God said, 'Let there be
Light.' And there was still nothing, but you could see a bit better.

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1.2: sha1-enable.diff --]
[-- Type: text/x-patch; name=sha1-enable.diff, Size: 537 bytes --]

Index: pym/portage.py
===================================================================
--- pym/portage.py	(revision 2316)
+++ pym/portage.py	(working copy)
@@ -2095,11 +2095,6 @@
 			myline += " "+mysum
 			myline += " "+myarchive
 			myline += " "+str(mysize)
-			if sumName != "MD5":
-				# XXXXXXXXXXXXXXXX This cannot be used!
-				# Older portage make very dumb assumptions about the formats.
-				# We need a lead-in period before we break everything.
-				continue
 			mylines.append(myline)
 	return mylines
 

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Multi hash support in portage - status

[Jason Stubbs]

On Thursday 24 November 2005 09:32, Marius Mauch wrote:
> On Thu, 24 Nov 2005 01:04:32 +0100
>
> Marius Mauch <genone@gentoo.org> wrote:
> > Ok I have three modifications that are pending to go into portage:
> > - The first simply enables creation of SHA1 checksums (and others if
> > implemented like with the second mod), if you want to try it yourself
> > see the attached patch.

Looking through CVS, this was supported in at least portage-2.0.51_rc10 right? 
This implies that the only versions that will have problems are 2.0.50-r11 
and under? If so, they've already got the cascaded profile problem so 
breaking things a little more won't hurt much. ;)

Seriously though, those that can't handle the new format would have to do 
what? Regenerate digests for sandbox and portage and then emerge each of them 
with --oneshot? Am I missing anything else there?

--
Jason Stubbs
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Multi hash support in portage - status

[Marius Mauch]

[-- Attachment #1: Type: text/plain, Size: 1411 bytes --]

On Thu, 24 Nov 2005 09:49:20 +0900
Jason Stubbs <jstubbs@gentoo.org> wrote:

> On Thursday 24 November 2005 09:32, Marius Mauch wrote:
> > On Thu, 24 Nov 2005 01:04:32 +0100
> >
> > Marius Mauch <genone@gentoo.org> wrote:
> > > Ok I have three modifications that are pending to go into portage:
> > > - The first simply enables creation of SHA1 checksums (and others
> > > if implemented like with the second mod), if you want to try it
> > > yourself see the attached patch.
> 
> Looking through CVS, this was supported in at least
> portage-2.0.51_rc10 right? This implies that the only versions that
> will have problems are 2.0.50-r11 and under? If so, they've already
> got the cascaded profile problem so breaking things a little more
> won't hurt much. ;)
> 
> Seriously though, those that can't handle the new format would have
> to do what? Regenerate digests for sandbox and portage and then
> emerge each of them with --oneshot? Am I missing anything else there?

Nope, not missing anything. Thought I said it, compability isn't a
reason to hold this up anymore, only asking if people want multi-hashes
now at the expense of a bigger tree when Manifest2 comes along.

Marius

-- 
Public Key at http://www.genone.de/info/gpg-key.pub

In the beginning, there was nothing. And God said, 'Let there be
Light.' And there was still nothing, but you could see a bit better.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Multi hash support in portage - status

[Marc Hildebrand]

Marius Mauch wrote:
[..]
> So much for background information, now to the actual question:
> Would you rather have now the ability to create multi-hash digests and
> Manifests with the result of a short and mid-term larger portage tree
> (in the long term the format will be phased out hopefully) or rather
> wait for Manifest2 support (which will definitely include multi hash
> support)?

I'd rather wait for Manifest2 support.
What is the ETA for the GLEP and the implementation after i?

Cheers,

Marc.
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Multi hash support in portage - status

[Marius Mauch]

[-- Attachment #1: Type: text/plain, Size: 1290 bytes --]

On Thu, 24 Nov 2005 07:33:34 +0100
Marc Hildebrand <zypher@gentoo.org> wrote:

> Marius Mauch wrote:
> [..]
> > So much for background information, now to the actual question:
> > Would you rather have now the ability to create multi-hash digests
> > and Manifests with the result of a short and mid-term larger
> > portage tree (in the long term the format will be phased out
> > hopefully) or rather wait for Manifest2 support (which will
> > definitely include multi hash support)?
> 
> I'd rather wait for Manifest2 support.
> What is the ETA for the GLEP and the implementation after i?

GLEP I still have to start writing (mostly a reformatting of a mail I
sent a long time ago), there is already a prototype implementation
(doesn't cover everything yet but works generally), target is
for when current trunk will be released (still have to settle on a
version for it), which should hopefully be after 2.0.54 gets out
(which should be in the next few weeks). At a guess I'd say 4 months
till stable (but really, that's just a guess, see the 2.1 fiasko).

Marius

-- 
Public Key at http://www.genone.de/info/gpg-key.pub

In the beginning, there was nothing. And God said, 'Let there be
Light.' And there was still nothing, but you could see a bit better.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Multi hash support in portage - status

[Jason Stubbs]

On Thursday 24 November 2005 10:07, Marius Mauch wrote:
> On Thu, 24 Nov 2005 09:49:20 +0900
>
> Jason Stubbs <jstubbs@gentoo.org> wrote:
> > On Thursday 24 November 2005 09:32, Marius Mauch wrote:
> > > On Thu, 24 Nov 2005 01:04:32 +0100
> > >
> > > Marius Mauch <genone@gentoo.org> wrote:
> > > > Ok I have three modifications that are pending to go into portage:
> > > > - The first simply enables creation of SHA1 checksums (and others
> > > > if implemented like with the second mod), if you want to try it
> > > > yourself see the attached patch.
> >
> > Looking through CVS, this was supported in at least
> > portage-2.0.51_rc10 right? This implies that the only versions that
> > will have problems are 2.0.50-r11 and under? If so, they've already
> > got the cascaded profile problem so breaking things a little more
> > won't hurt much. ;)
> >
> > Seriously though, those that can't handle the new format would have
> > to do what? Regenerate digests for sandbox and portage and then
> > emerge each of them with --oneshot? Am I missing anything else there?
>
> Nope, not missing anything. Thought I said it, compability isn't a
> reason to hold this up anymore, only asking if people want multi-hashes
> now at the expense of a bigger tree when Manifest2 comes along.

I'm referring to portage-2.0.50 and below. What exactly needs to be done by 
those few that are still using it to upgrade to a better portage after it 
dies on finding SHA1 sums in portage's digest?

--
Jason Stubbs
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Multi hash support in portage - status

[Grant Goodyear]

[-- Attachment #1: Type: text/plain, Size: 886 bytes --]

Marius Mauch wrote: [Thu Nov 24 2005, 04:38:44AM CST]
> GLEP I still have to start writing (mostly a reformatting of a mail I
> sent a long time ago), there is already a prototype implementation
> (doesn't cover everything yet but works generally), target is
> for when current trunk will be released (still have to settle on a
> version for it), which should hopefully be after 2.0.54 gets out
> (which should be in the next few weeks). At a guess I'd say 4 months
> till stable (but really, that's just a guess, see the 2.1 fiasko).

Personally, I'd much rather see the nascent support go in, even at the
cost of expanding the tree a tad, rather than push it off into the
fairly distant future.

-g2boojum-
-- 
Grant Goodyear	
Gentoo Developer
g2boojum@gentoo.org
http://www.gentoo.org/~g2boojum
GPG Fingerprint: D706 9802 1663 DEF5 81B0  9573 A6DC 7152 E0F6 5B76

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Multi hash support in portage - status

[Chris Gianelloni]

[-- Attachment #1: Type: text/plain, Size: 1007 bytes --]

On Thu, 2005-11-24 at 11:38 +0100, Marius Mauch wrote:
> > I'd rather wait for Manifest2 support.
> > What is the ETA for the GLEP and the implementation after i?
> 
> GLEP I still have to start writing (mostly a reformatting of a mail I
> sent a long time ago), there is already a prototype implementation
> (doesn't cover everything yet but works generally), target is
> for when current trunk will be released (still have to settle on a
> version for it), which should hopefully be after 2.0.54 gets out
> (which should be in the next few weeks). At a guess I'd say 4 months
> till stable (but really, that's just a guess, see the 2.1 fiasko).

I think I would rather see the multi-hash support now, rather than
later, as it seems to be something our users are asking about and it
seems like all the work is really there for an implementation sooner
rather than later.

-- 
Chris Gianelloni
Release Engineering - Strategic Lead
x86 Architecture Team
Games - Developer
Gentoo Linux

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Multi hash support in portage - status

[Chris Gianelloni]

[-- Attachment #1: Type: text/plain, Size: 913 bytes --]

On Thu, 2005-11-24 at 20:57 +0900, Jason Stubbs wrote:
> > Nope, not missing anything. Thought I said it, compability isn't a
> > reason to hold this up anymore, only asking if people want multi-hashes
> > now at the expense of a bigger tree when Manifest2 comes along.
> 
> I'm referring to portage-2.0.50 and below. What exactly needs to be done by 
> those few that are still using it to upgrade to a better portage after it 
> dies on finding SHA1 sums in portage's digest?

They would need to redigest portage/sandbox with their version of
portage, as you said.  They would need to create a symlink
from /etc/make.profile to /usr/portage/profiles/obsolete/$arch, then
they would need to emerge --oneshot sandbox portage.  After that, they
can upgrade to a supported profile.

-- 
Chris Gianelloni
Release Engineering - Strategic Lead
x86 Architecture Team
Games - Developer
Gentoo Linux

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Multi hash support in portage - status

[Marius Mauch]

[-- Attachment #1: Type: text/plain, Size: 810 bytes --]

On Thu, 24 Nov 2005 01:04:32 +0100
Marius Mauch <genone@gentoo.org> wrote:

> Would you rather have now the ability to create multi-hash digests and
> Manifests with the result of a short and mid-term larger portage tree
> (in the long term the format will be phased out hopefully) or rather
> wait for Manifest2 support (which will definitely include multi hash
> support)?

Ok, so far two votes for now and one vote for later, unless this
changes significantly I'll ask the council to add the decision to the
agenda for its next meeting (sorry, just don't want to be the bad guy
here ;)

Marius

-- 
Public Key at http://www.genone.de/info/gpg-key.pub

In the beginning, there was nothing. And God said, 'Let there be
Light.' And there was still nothing, but you could see a bit better.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Multi hash support in portage - status

[Jason Stubbs]

On Saturday 26 November 2005 11:12, Marius Mauch wrote:
> On Thu, 24 Nov 2005 01:04:32 +0100
>
> Marius Mauch <genone@gentoo.org> wrote:
> > Would you rather have now the ability to create multi-hash digests and
> > Manifests with the result of a short and mid-term larger portage tree
> > (in the long term the format will be phased out hopefully) or rather
> > wait for Manifest2 support (which will definitely include multi hash
> > support)?
>
> Ok, so far two votes for now and one vote for later, unless this
> changes significantly I'll ask the council to add the decision to the
> agenda for its next meeting (sorry, just don't want to be the bad guy
> here ;)

/me adds a vote for later to even it up.

--
Jason Stubbs
-- 
gentoo-dev@gentoo.org mailing list