[gentoo-portage-dev] [GLEP59v2 0/5] GLEP59: Manifest2 hash types

[gentoo-portage-dev] [GLEP59v2 0/5] GLEP59: Manifest2 hash types

[Robin H. Johnson]

Respun now with the help of ferringb. Cleans up the implementation and catches
a few bug and improvements:
- mhash priority moved lower than pycrypto/hashlib because mhash holds GIL
  while the other implementations don't.
- hashlib does offer whirlpool if it was built against openssl 1.0.

1/5: Refactor RMD160 hashlib code for less-hardcoding
2/5: Manifest2 hash: Whirlpool
3/5: Manifest2 hash: SHA512
4/5: Manifest2 hash backend provider: mhash
5/5: GLEP59: Change live Manifest2 hashes to SHA256,

[gentoo-portage-dev] [GLEP59v2 1/5] Refactor RMD160 hashlib code for less-hardcoding

[Robin H. Johnson]

From: "Robin H. Johnson" <robbat2@gentoo.org>

To be used shortly for WHIRLPOOL as well as RMD160.

Signed-off-by: Robin H. Johnson <robbat2@gentoo.org>
---
 pym/portage/checksum.py |   21 ++++++++++++---------
 1 files changed, 12 insertions(+), 9 deletions(-)

diff --git a/pym/portage/checksum.py b/pym/portage/checksum.py
index 9e7e455..e5455fa 100644
--- a/pym/portage/checksum.py
+++ b/pym/portage/checksum.py
@@ -82,19 +82,22 @@ except ImportError as e:
 # Use hashlib from python-2.5 if available and prefer it over pycrypto and internal fallbacks.
 # Need special handling for RMD160 as it may not always be provided by hashlib.
 try:
-	import hashlib
+	import hashlib, functools
 	
 	md5hash = _generate_hash_function("MD5", hashlib.md5, origin="hashlib")
 	sha1hash = _generate_hash_function("SHA1", hashlib.sha1, origin="hashlib")
 	sha256hash = _generate_hash_function("SHA256", hashlib.sha256, origin="hashlib")
-	try:
-		hashlib.new('ripemd160')
-	except ValueError:
-		pass
-	else:
-		def rmd160():
-			return hashlib.new('ripemd160')
-		rmd160hash = _generate_hash_function("RMD160", rmd160, origin="hashlib")
+	for local_name, hash_name in (("rmd160", "ripemd160"), ):
+		try:
+			hashlib.new(hash_name)
+		except ValueError:
+			pass
+		else:
+			globals()['%shash' % local_name] = \
+				_generate_hash_function(local_name.upper(), \
+				functools.partial(hashlib.new, hash_name), \
+				origin='hashlib')
+
 except ImportError as e:
 	pass
 	
-- 
1.7.7

[gentoo-portage-dev] [GLEP59v2 2/5] Manifest2 hash: Whirlpool

[Robin H. Johnson]

From: "Robin H. Johnson" <robbat2@gentoo.org>

Provide public-domain implementation of the Whirlpool hash algorithm to
be used as new Manifest2 hash.

Signed-off-by: Robin H. Johnson <robbat2@gentoo.org>
---
 pym/portage/checksum.py |    8 ++++++--
 1 files changed, 6 insertions(+), 2 deletions(-)

diff --git a/pym/portage/checksum.py b/pym/portage/checksum.py
index e5455fa..3593686 100644
--- a/pym/portage/checksum.py
+++ b/pym/portage/checksum.py
@@ -71,6 +71,10 @@ except ImportError:
 
 sha1hash = _generate_hash_function("SHA1", _new_sha1, origin="internal")
 
+# Bundled WHIRLPOOL implementation
+from portage.util.whirlpool import new as _new_whirlpool
+whirlpoolhash = _generate_hash_function("WHIRLPOOL", _new_whirlpool, origin="bundled")
+
 # Use pycrypto when available, prefer it over the internal fallbacks
 try:
 	from Crypto.Hash import SHA256, RIPEMD
@@ -80,14 +84,14 @@ except ImportError as e:
 	pass
 
 # Use hashlib from python-2.5 if available and prefer it over pycrypto and internal fallbacks.
-# Need special handling for RMD160 as it may not always be provided by hashlib.
+# Need special handling for RMD160/WHIRLPOOL as they may not always be provided by hashlib.
 try:
 	import hashlib, functools
 	
 	md5hash = _generate_hash_function("MD5", hashlib.md5, origin="hashlib")
 	sha1hash = _generate_hash_function("SHA1", hashlib.sha1, origin="hashlib")
 	sha256hash = _generate_hash_function("SHA256", hashlib.sha256, origin="hashlib")
-	for local_name, hash_name in (("rmd160", "ripemd160"), ):
+	for local_name, hash_name in (("rmd160", "ripemd160"), ("whirlpool", "whirlpool")):
 		try:
 			hashlib.new(hash_name)
 		except ValueError:
-- 
1.7.7

Re: [gentoo-portage-dev] [GLEP59v2 2/5] Manifest2 hash: Whirlpool

[Brian Harring]

On Sat, Oct 01, 2011 at 07:40:52AM +0000, Robin H. Johnson wrote:
> From: "Robin H. Johnson" <robbat2@gentoo.org>
> 
> Provide public-domain implementation of the Whirlpool hash algorithm to
> be used as new Manifest2 hash.
> 
> Signed-off-by: Robin H. Johnson <robbat2@gentoo.org>
> ---
>  pym/portage/checksum.py |    8 ++++++--
>  1 files changed, 6 insertions(+), 2 deletions(-)
> 
> diff --git a/pym/portage/checksum.py b/pym/portage/checksum.py
> index e5455fa..3593686 100644
> --- a/pym/portage/checksum.py
> +++ b/pym/portage/checksum.py
> @@ -71,6 +71,10 @@ except ImportError:
>  
>  sha1hash = _generate_hash_function("SHA1", _new_sha1, origin="internal")
>  
> +# Bundled WHIRLPOOL implementation
> +from portage.util.whirlpool import new as _new_whirlpool
> +whirlpoolhash = _generate_hash_function("WHIRLPOOL", _new_whirlpool, origin="bundled")
> +

Likely should shift this to a trailing check if no whirlpool 
implementation was found; via this, we can avoid the import unless 
it's needed.
~brian

>  # Use pycrypto when available, prefer it over the internal fallbacks
>  try:
>  	from Crypto.Hash import SHA256, RIPEMD
> @@ -80,14 +84,14 @@ except ImportError as e:
>  	pass
>  
>  # Use hashlib from python-2.5 if available and prefer it over pycrypto and internal fallbacks.
> -# Need special handling for RMD160 as it may not always be provided by hashlib.
> +# Need special handling for RMD160/WHIRLPOOL as they may not always be provided by hashlib.
>  try:
>  	import hashlib, functools
>  	
>  	md5hash = _generate_hash_function("MD5", hashlib.md5, origin="hashlib")
>  	sha1hash = _generate_hash_function("SHA1", hashlib.sha1, origin="hashlib")
>  	sha256hash = _generate_hash_function("SHA256", hashlib.sha256, origin="hashlib")
> -	for local_name, hash_name in (("rmd160", "ripemd160"), ):
> +	for local_name, hash_name in (("rmd160", "ripemd160"), ("whirlpool", "whirlpool")):
>  		try:
>  			hashlib.new(hash_name)
>  		except ValueError:
> -- 
> 1.7.7
> 

Re: [gentoo-portage-dev] [GLEP59v2 2/5] Manifest2 hash: Whirlpool

[Zac Medico]

On 10/01/2011 02:41 AM, Brian Harring wrote:
> On Sat, Oct 01, 2011 at 07:40:52AM +0000, Robin H. Johnson wrote:
>> From: "Robin H. Johnson" <robbat2@gentoo.org>
>>
>> Provide public-domain implementation of the Whirlpool hash algorithm to
>> be used as new Manifest2 hash.
>>
>> Signed-off-by: Robin H. Johnson <robbat2@gentoo.org>
>> ---
>>  pym/portage/checksum.py |    8 ++++++--
>>  1 files changed, 6 insertions(+), 2 deletions(-)
>>
>> diff --git a/pym/portage/checksum.py b/pym/portage/checksum.py
>> index e5455fa..3593686 100644
>> --- a/pym/portage/checksum.py
>> +++ b/pym/portage/checksum.py
>> @@ -71,6 +71,10 @@ except ImportError:
>>  
>>  sha1hash = _generate_hash_function("SHA1", _new_sha1, origin="internal")
>>  
>> +# Bundled WHIRLPOOL implementation
>> +from portage.util.whirlpool import new as _new_whirlpool
>> +whirlpoolhash = _generate_hash_function("WHIRLPOOL", _new_whirlpool, origin="bundled")
>> +
> 
> Likely should shift this to a trailing check if no whirlpool 
> implementation was found; via this, we can avoid the import unless 
> it's needed.
> ~brian

Thanks, that's done now:

http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=commit;h=06ad8911b5790a2ed963fe1b981751ab0a2be8d5

-- 
Thanks,
Zac

[gentoo-portage-dev] [GLEP59v2 3/5] Manifest2 hash: SHA512

[Robin H. Johnson]

From: "Robin H. Johnson" <robbat2@gentoo.org>

Provide SHA512 hash algorithm to be used as new Manifest2 hash.

Signed-off-by: Robin H. Johnson <robbat2@gentoo.org>
---
 pym/portage/checksum.py |    1 +
 1 files changed, 1 insertions(+), 0 deletions(-)

diff --git a/pym/portage/checksum.py b/pym/portage/checksum.py
index 3593686..40ae836 100644
--- a/pym/portage/checksum.py
+++ b/pym/portage/checksum.py
@@ -91,6 +91,7 @@ try:
 	md5hash = _generate_hash_function("MD5", hashlib.md5, origin="hashlib")
 	sha1hash = _generate_hash_function("SHA1", hashlib.sha1, origin="hashlib")
 	sha256hash = _generate_hash_function("SHA256", hashlib.sha256, origin="hashlib")
+	sha512hash = _generate_hash_function("SHA512", hashlib.sha512, origin="hashlib")
 	for local_name, hash_name in (("rmd160", "ripemd160"), ("whirlpool", "whirlpool")):
 		try:
 			hashlib.new(hash_name)
-- 
1.7.7

[gentoo-portage-dev] [GLEP59v2 4/5] Manifest2 hash backend provider: mhash

[Robin H. Johnson]

From: "Robin H. Johnson" <robbat2@gentoo.org>

Offer mhash as a provider for Manifest2 hash generation and validation.
This is important as either of pycrypto or fchksum offer an accelerated
Whirlpool implementation, and hashlib might not offer it. Additionally,
the mhash implementation is accelerated and ships with a rigorious
testsuite.

Signed-off-by: Robin H. Johnson <robbat2@gentoo.org>
---
 pym/portage/checksum.py |   19 +++++++++++++++++++
 1 files changed, 19 insertions(+), 0 deletions(-)

diff --git a/pym/portage/checksum.py b/pym/portage/checksum.py
index 40ae836..c0c7c04 100644
--- a/pym/portage/checksum.py
+++ b/pym/portage/checksum.py
@@ -75,6 +75,25 @@ sha1hash = _generate_hash_function("SHA1", _new_sha1, origin="internal")
 from portage.util.whirlpool import new as _new_whirlpool
 whirlpoolhash = _generate_hash_function("WHIRLPOOL", _new_whirlpool, origin="bundled")
 
+# Try to use mhash if available
+# mhash causes GIL presently, so it gets less priority than hashlib and
+# pycrypto. However, it might be the only accelerated implementation of
+# WHIRLPOOL available.
+try:
+	import mhash, functools
+	md5hash = _generate_hash_function("MD5", functools.partial(mhash.MHASH, mhash.MHASH_MD5), origin="mhash")
+	sha1hash = _generate_hash_function("SHA1", functools.partial(mhash.MHASH, mhash.MHASH_SHA1), origin="mhash")
+	sha256hash = _generate_hash_function("SHA256", functools.partial(mhash.MHASH, mhash.MHASH_SHA256), origin="mhash")
+	sha512hash = _generate_hash_function("SHA512", functools.partial(mhash.MHASH, mhash.MHASH_SHA512), origin="mhash")
+	for local_name, hash_name in (("rmd160", "ripemd160"), ("whirlpool", "whirlpool")):
+		if hasattr(mhash, 'MHASH_%s' % local_name.upper()):
+			globals()['%shash' % local_name] = \
+				_generate_hash_function(local_name.upper(), \
+				functools.partial(mhash.MHASH, getattr(mhash, 'MHASH_%s' % s.upper())), \
+				origin='mhash')
+except ImportError as e:
+	pass
+
 # Use pycrypto when available, prefer it over the internal fallbacks
 try:
 	from Crypto.Hash import SHA256, RIPEMD
-- 
1.7.7

[gentoo-portage-dev] [GLEP59v2 5/5] GLEP59: Change live Manifest2 hashes to SHA256, SHA512, WHIRLPOOL

[Robin H. Johnson]

From: "Robin H. Johnson" <robbat2@gentoo.org>

Change Manifest2 hashes to a more secure set as approved in GLEP59.
SHA512 and WHIRLPOOL are added, SHA1 and RMD160 are dropped.

SHA256 is now the lowest security hash, and must remain in Manifest
files for at least 1 year, otherwise older Portage installs will
complain that they do not support any of the hashes in the Manifest
files.

Future events:
After 2012/10/01:
- Change MANIFEST2_REQUIRED_HASH to WHIRLPOOL.
- Remove SHA256 from MANIFEST2_HASH_FUNCTIONS.
After SHA-3 is approved:
- Add new hashes to MANIFEST2_HASH_FUNCTIONS.

Signed-off-by: Robin H. Johnson <robbat2@gentoo.org>
---
 pym/portage/const.py |    6 ++++--
 1 files changed, 4 insertions(+), 2 deletions(-)

diff --git a/pym/portage/const.py b/pym/portage/const.py
index 8b5f4ac..a42ebe8 100644
--- a/pym/portage/const.py
+++ b/pym/portage/const.py
@@ -109,10 +109,12 @@ EAPI                     = 4
 
 HASHING_BLOCKSIZE        = 32768
 MANIFEST1_HASH_FUNCTIONS = ("MD5", "SHA256", "RMD160")
-MANIFEST2_HASH_FUNCTIONS = ("SHA1", "SHA256", "RMD160")
+MANIFEST2_HASH_FUNCTIONS = ("SHA256", "SHA512", "WHIRLPOOL") 
+# FUTURE: Add SHA-3 when available; remove SHA256 after 2012/10/01
 
 MANIFEST1_REQUIRED_HASH  = "MD5"
-MANIFEST2_REQUIRED_HASH  = "SHA1"
+MANIFEST2_REQUIRED_HASH  = "SHA256" 
+# FUTURE: Change to WHIRLPOOL after 2012/10/01
 
 MANIFEST2_IDENTIFIERS    = ("AUX", "MISC", "DIST", "EBUILD")
 # ===========================================================================
-- 
1.7.7

Re: [gentoo-portage-dev] [GLEP59v2 5/5] GLEP59: Change live Manifest2 hashes to SHA256, SHA512, WHIRLPOOL

[Zac Medico]

On 10/01/2011 12:40 AM, Robin H. Johnson wrote:
> diff --git a/pym/portage/const.py b/pym/portage/const.py
> index 8b5f4ac..a42ebe8 100644
> --- a/pym/portage/const.py
> +++ b/pym/portage/const.py
> @@ -109,10 +109,12 @@ EAPI                     = 4
>  
>  HASHING_BLOCKSIZE        = 32768
>  MANIFEST1_HASH_FUNCTIONS = ("MD5", "SHA256", "RMD160")
> -MANIFEST2_HASH_FUNCTIONS = ("SHA1", "SHA256", "RMD160")
> +MANIFEST2_HASH_FUNCTIONS = ("SHA256", "SHA512", "WHIRLPOOL") 
> +# FUTURE: Add SHA-3 when available; remove SHA256 after 2012/10/01
>  
>  MANIFEST1_REQUIRED_HASH  = "MD5"
> -MANIFEST2_REQUIRED_HASH  = "SHA1"
> +MANIFEST2_REQUIRED_HASH  = "SHA256" 
> +# FUTURE: Change to WHIRLPOOL after 2012/10/01
>  
>  MANIFEST2_IDENTIFIERS    = ("AUX", "MISC", "DIST", "EBUILD")
>  # ===========================================================================

If we control these hashes via metadata/layout.conf, then we can toggle
it atomically for all commiters. Otherwise, we'll have an annoying
period of time where different committers are committing different sets
of hashes, depending on their portage version.
-- 
Thanks,
Zac

Re: [gentoo-portage-dev] [GLEP59v2 5/5] GLEP59: Change live Manifest2 hashes to SHA256, SHA512, WHIRLPOOL

[Zac Medico]

On 10/01/2011 09:40 PM, Zac Medico wrote:
> On 10/01/2011 12:40 AM, Robin H. Johnson wrote:
>> diff --git a/pym/portage/const.py b/pym/portage/const.py
>> index 8b5f4ac..a42ebe8 100644
>> --- a/pym/portage/const.py
>> +++ b/pym/portage/const.py
>> @@ -109,10 +109,12 @@ EAPI                     = 4
>>  
>>  HASHING_BLOCKSIZE        = 32768
>>  MANIFEST1_HASH_FUNCTIONS = ("MD5", "SHA256", "RMD160")
>> -MANIFEST2_HASH_FUNCTIONS = ("SHA1", "SHA256", "RMD160")
>> +MANIFEST2_HASH_FUNCTIONS = ("SHA256", "SHA512", "WHIRLPOOL") 
>> +# FUTURE: Add SHA-3 when available; remove SHA256 after 2012/10/01
>>  
>>  MANIFEST1_REQUIRED_HASH  = "MD5"
>> -MANIFEST2_REQUIRED_HASH  = "SHA1"
>> +MANIFEST2_REQUIRED_HASH  = "SHA256" 
>> +# FUTURE: Change to WHIRLPOOL after 2012/10/01
>>  
>>  MANIFEST2_IDENTIFIERS    = ("AUX", "MISC", "DIST", "EBUILD")
>>  # ===========================================================================
> 
> If we control these hashes via metadata/layout.conf, then we can toggle
> it atomically for all commiters. Otherwise, we'll have an annoying
> period of time where different committers are committing different sets
> of hashes, depending on their portage version.

I've applied the whole series, except for 5/5:

http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=commit;h=f27473d04e6dee44983d1e5ac32ea9d4d375b5a2
http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=commit;h=f3b05d6eed63e19cdfa7f645cf0190ee8019dd90
http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=commit;h=8ac29097395f24ad331602d8e87fdf105ebd972b
http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=commit;h=faf87ba9877e3b5a7866c6649f956f15950e789a

-- 
Thanks,
Zac

Re: [gentoo-portage-dev] [GLEP59v2 5/5] GLEP59: Change live Manifest2 hashes to SHA256, SHA512, WHIRLPOOL

[Robin H. Johnson]

On Sat, Oct 01, 2011 at 09:40:13PM -0700, Zac Medico wrote:
> If we control these hashes via metadata/layout.conf, then we can toggle
> it atomically for all commiters. Otherwise, we'll have an annoying
> period of time where different committers are committing different sets
> of hashes, depending on their portage version.
How do you suggest doing it via layout.conf? I've kept SHA256 in both
sets for now, but if you could enforce new signatures including both
WHIRLPOOL and SHA256, that would be great.

-- 
Robin Hugh Johnson
Gentoo Linux: Developer, Trustee & Infrastructure Lead
E-Mail     : robbat2@gentoo.org
GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85

Re: [gentoo-portage-dev] [GLEP59v2 5/5] GLEP59: Change live Manifest2 hashes to SHA256, SHA512, WHIRLPOOL

[Zac Medico]

On 10/02/2011 05:46 AM, Robin H. Johnson wrote:
> On Sat, Oct 01, 2011 at 09:40:13PM -0700, Zac Medico wrote:
>> If we control these hashes via metadata/layout.conf, then we can toggle
>> it atomically for all commiters. Otherwise, we'll have an annoying
>> period of time where different committers are committing different sets
>> of hashes, depending on their portage version.
> How do you suggest doing it via layout.conf? I've kept SHA256 in both
> sets for now, but if you could enforce new signatures including both
> WHIRLPOOL and SHA256, that would be great.

How about if we put something like this in
gentoo-x86/metadata/layout.conf now:

   manifest2-sha1 = true
   manifest2-whirlpool = false

Then we'll patch portage so that by default it will disable SHA1 and
enable WHIRLPOOL, and the above settings will override the defaults.
After the patched portage is marked stable in a month or so, we'll send
an announcement to gentoo-announce, and remove the above settings from
layout.conf.
-- 
Thanks,
Zac

Re: [gentoo-portage-dev] [GLEP59v2 5/5] GLEP59: Change live Manifest2 hashes to SHA256, SHA512, WHIRLPOOL

[Alec Warner]

On Sun, Oct 2, 2011 at 1:39 PM, Zac Medico <zmedico@gentoo.org> wrote:
> On 10/02/2011 05:46 AM, Robin H. Johnson wrote:
>> On Sat, Oct 01, 2011 at 09:40:13PM -0700, Zac Medico wrote:
>>> If we control these hashes via metadata/layout.conf, then we can toggle
>>> it atomically for all commiters. Otherwise, we'll have an annoying
>>> period of time where different committers are committing different sets
>>> of hashes, depending on their portage version.
>> How do you suggest doing it via layout.conf? I've kept SHA256 in both
>> sets for now, but if you could enforce new signatures including both
>> WHIRLPOOL and SHA256, that would be great.
>
> How about if we put something like this in
> gentoo-x86/metadata/layout.conf now:

Reminds me, I was going to do an analysis on -commit mails to track
portage versions; I'll do that now.

>
>   manifest2-sha1 = true
>   manifest2-whirlpool = false
>
> Then we'll patch portage so that by default it will disable SHA1 and
> enable WHIRLPOOL, and the above settings will override the defaults.
> After the patched portage is marked stable in a month or so, we'll send
> an announcement to gentoo-announce, and remove the above settings from
> layout.conf.
> --
> Thanks,
> Zac
>
>

Re: [gentoo-portage-dev] [GLEP59v2 5/5] GLEP59: Change live Manifest2 hashes to SHA256, SHA512, WHIRLPOOL

[Robin H. Johnson]

On Sun, Oct 02, 2011 at 01:39:41PM -0700, Zac Medico wrote:
> On 10/02/2011 05:46 AM, Robin H. Johnson wrote:
> > On Sat, Oct 01, 2011 at 09:40:13PM -0700, Zac Medico wrote:
> >> If we control these hashes via metadata/layout.conf, then we can toggle
> >> it atomically for all commiters. Otherwise, we'll have an annoying
> >> period of time where different committers are committing different sets
> >> of hashes, depending on their portage version.
> > How do you suggest doing it via layout.conf? I've kept SHA256 in both
> > sets for now, but if you could enforce new signatures including both
> > WHIRLPOOL and SHA256, that would be great.
> How about if we put something like this in
> gentoo-x86/metadata/layout.conf now:
Did you mean profiles/layout.conf? I just want to make sure no scripts
that pull from CVS and expect that dir to not exist don't break.

>    manifest2-sha1 = true
>    manifest2-whirlpool = false
Bikeshedding slightly, but can we figure something like a list or dict
instead? (Also gives us a chance to make the required hashes a list).
manifest2-hashes = ['SHA1', 'SHA256', 'RMD160']

> Then we'll patch portage so that by default it will disable SHA1 and
> enable WHIRLPOOL, and the above settings will override the defaults.
> After the patched portage is marked stable in a month or so, we'll send
> an announcement to gentoo-announce, and remove the above settings from
> layout.conf.
Sounds good to me. Hopefully I'll have more of the MetaManifest
prototype code in the next few days to go live around the same time.

-- 
Robin Hugh Johnson
Gentoo Linux: Developer, Trustee & Infrastructure Lead
E-Mail     : robbat2@gentoo.org
GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85

Re: [gentoo-portage-dev] [GLEP59v2 5/5] GLEP59: Change live Manifest2 hashes to SHA256, SHA512, WHIRLPOOL

[Zac Medico]

On 10/02/2011 01:54 PM, Robin H. Johnson wrote:
> On Sun, Oct 02, 2011 at 01:39:41PM -0700, Zac Medico wrote:
>> On 10/02/2011 05:46 AM, Robin H. Johnson wrote:
>>> On Sat, Oct 01, 2011 at 09:40:13PM -0700, Zac Medico wrote:
>>>> If we control these hashes via metadata/layout.conf, then we can toggle
>>>> it atomically for all commiters. Otherwise, we'll have an annoying
>>>> period of time where different committers are committing different sets
>>>> of hashes, depending on their portage version.
>>> How do you suggest doing it via layout.conf? I've kept SHA256 in both
>>> sets for now, but if you could enforce new signatures including both
>>> WHIRLPOOL and SHA256, that would be great.
>> How about if we put something like this in
>> gentoo-x86/metadata/layout.conf now:
> Did you mean profiles/layout.conf? I just want to make sure no scripts
> that pull from CVS and expect that dir to not exist don't break.

No, it's metadata/layout.conf. I didn't choose the location. We actually
inherited it from paludis about 1.5 years ago:


http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=commit;h=f16aee82cefa95e9903fa46f448d30f6d4350f64

We're also using it to control thin-manifest support, among other things
now:

  https://bugs.gentoo.org/show_bug.cgi?id=333691

>>    manifest2-sha1 = true
>>    manifest2-whirlpool = false
> Bikeshedding slightly, but can we figure something like a list or dict
> instead? (Also gives us a chance to make the required hashes a list).
> manifest2-hashes = ['SHA1', 'SHA256', 'RMD160']

Well, booleans are simpler. Also, note that I designed them to be
removed from layout.conf eventually, which means that we will accumulate
less bloat in layout.conf over time.

>> Then we'll patch portage so that by default it will disable SHA1 and
>> enable WHIRLPOOL, and the above settings will override the defaults.
>> After the patched portage is marked stable in a month or so, we'll send
>> an announcement to gentoo-announce, and remove the above settings from
>> layout.conf.
> Sounds good to me. Hopefully I'll have more of the MetaManifest
> prototype code in the next few days to go live around the same time.

I'll see if I can get a layout.conf patch done today.
-- 
Thanks,
Zac

Re: [gentoo-portage-dev] [GLEP59v2 5/5] GLEP59: Change live Manifest2 hashes to SHA256, SHA512, WHIRLPOOL

[Brian Harring]

On Sun, Oct 02, 2011 at 02:10:09PM -0700, Zac Medico wrote:
> On 10/02/2011 01:54 PM, Robin H. Johnson wrote:
> > On Sun, Oct 02, 2011 at 01:39:41PM -0700, Zac Medico wrote:
> >> On 10/02/2011 05:46 AM, Robin H. Johnson wrote:
> >>> On Sat, Oct 01, 2011 at 09:40:13PM -0700, Zac Medico wrote:
> >>>> If we control these hashes via metadata/layout.conf, then we can toggle
> >>>> it atomically for all commiters. Otherwise, we'll have an annoying
> >>>> period of time where different committers are committing different sets
> >>>> of hashes, depending on their portage version.
> >>> How do you suggest doing it via layout.conf? I've kept SHA256 in both
> >>> sets for now, but if you could enforce new signatures including both
> >>> WHIRLPOOL and SHA256, that would be great.
> >> How about if we put something like this in
> >> gentoo-x86/metadata/layout.conf now:
> > Did you mean profiles/layout.conf? I just want to make sure no scripts
> > that pull from CVS and expect that dir to not exist don't break.
> 
> No, it's metadata/layout.conf. I didn't choose the location. We actually
> inherited it from paludis about 1.5 years ago:
> 
> 
> http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=commit;h=f16aee82cefa95e9903fa46f448d30f6d4350f64
> 
> We're also using it to control thin-manifest support, among other things
> now:
> 
>   https://bugs.gentoo.org/show_bug.cgi?id=333691
> 
> >>    manifest2-sha1 = true
> >>    manifest2-whirlpool = false
> > Bikeshedding slightly, but can we figure something like a list or dict
> > instead? (Also gives us a chance to make the required hashes a list).
> > manifest2-hashes = ['SHA1', 'SHA256', 'RMD160']
> 
> Well, booleans are simpler. Also, note that I designed them to be
> removed from layout.conf eventually, which means that we will accumulate
> less bloat in layout.conf over time.

Should use a space delimited list instead named hashes instead; those 
being the hashes that should be generated, and that can be /used/.  
Not in the list, not an acceptable hash (even if a manifest2 carries 
that data).

If it's not set, then the pm defaults in a list; that default list 
should be tracked somewhere (rather than just whatever the PM author 
decides) also, although that's a seperate discussion.

Breaking it out into individual booleans isn't particularly great; we 
use lists for masters, a tristate for use-manifest, etc.  Having each 
CHF controlled by a seperate boolean adds more toggles than is worth 
it imo, and having the manifest2- prefix makes the parsing slightly 
more complex while also making the key name a bit daft if we ever 
switch to a manifest3. ;) 

~harring

Re: [gentoo-portage-dev] [GLEP59v2 5/5] GLEP59: Change live Manifest2 hashes to SHA256, SHA512, WHIRLPOOL

[Zac Medico]

On 10/02/2011 04:22 PM, Brian Harring wrote:
> On Sun, Oct 02, 2011 at 02:10:09PM -0700, Zac Medico wrote:
>> On 10/02/2011 01:54 PM, Robin H. Johnson wrote:
>>> On Sun, Oct 02, 2011 at 01:39:41PM -0700, Zac Medico wrote:
>>>> On 10/02/2011 05:46 AM, Robin H. Johnson wrote:
>>>>> On Sat, Oct 01, 2011 at 09:40:13PM -0700, Zac Medico wrote:
>>>>>> If we control these hashes via metadata/layout.conf, then we can toggle
>>>>>> it atomically for all commiters. Otherwise, we'll have an annoying
>>>>>> period of time where different committers are committing different sets
>>>>>> of hashes, depending on their portage version.
>>>>> How do you suggest doing it via layout.conf? I've kept SHA256 in both
>>>>> sets for now, but if you could enforce new signatures including both
>>>>> WHIRLPOOL and SHA256, that would be great.
>>>> How about if we put something like this in
>>>> gentoo-x86/metadata/layout.conf now:
>>> Did you mean profiles/layout.conf? I just want to make sure no scripts
>>> that pull from CVS and expect that dir to not exist don't break.
>>
>> No, it's metadata/layout.conf. I didn't choose the location. We actually
>> inherited it from paludis about 1.5 years ago:
>>
>>
>> http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=commit;h=f16aee82cefa95e9903fa46f448d30f6d4350f64
>>
>> We're also using it to control thin-manifest support, among other things
>> now:
>>
>>   https://bugs.gentoo.org/show_bug.cgi?id=333691
>>
>>>>    manifest2-sha1 = true
>>>>    manifest2-whirlpool = false
>>> Bikeshedding slightly, but can we figure something like a list or dict
>>> instead? (Also gives us a chance to make the required hashes a list).
>>> manifest2-hashes = ['SHA1', 'SHA256', 'RMD160']
>>
>> Well, booleans are simpler. Also, note that I designed them to be
>> removed from layout.conf eventually, which means that we will accumulate
>> less bloat in layout.conf over time.

I've implemented it with booleans in this commit:

http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=commit;h=c8cd3a985cc529299411d7343a11004b7d1330ef

> Should use a space delimited list instead named hashes instead; those 
> being the hashes that should be generated, and that can be /used/.  
> Not in the list, not an acceptable hash (even if a manifest2 carries 
> that data).

Why? Boolean flags are simpler and they work.

> If it's not set, then the pm defaults in a list; that default list 
> should be tracked somewhere (rather than just whatever the PM author 
> decides) also, although that's a seperate discussion.

Sure, it could be added to PMS or something.

> Breaking it out into individual booleans isn't particularly great; we 
> use lists for masters, a tristate for use-manifest, etc.  Having each 
> CHF controlled by a seperate boolean adds more toggles than is worth 

You can group them into a dictionary, like I did.

> it imo, and having the manifest2- prefix makes the parsing slightly 
> more complex while also making the key name a bit daft if we ever 
> switch to a manifest3. ;) 

I made it manifest- instead.
-- 
Thanks,
Zac

Re: [gentoo-portage-dev] [GLEP59v2 5/5] GLEP59: Change live Manifest2 hashes to SHA256, SHA512, WHIRLPOOL

[Zac Medico]

On 10/02/2011 05:21 PM, Zac Medico wrote:
> On 10/02/2011 04:22 PM, Brian Harring wrote:
>> On Sun, Oct 02, 2011 at 02:10:09PM -0700, Zac Medico wrote:
> I've implemented it with booleans in this commit:
> 
> http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=commit;h=c8cd3a985cc529299411d7343a11004b7d1330ef
> 
>> Should use a space delimited list instead named hashes instead; those 
>> being the hashes that should be generated, and that can be /used/.  
>> Not in the list, not an acceptable hash (even if a manifest2 carries 
>> that data).
> 
> Why? Boolean flags are simpler and they work.

After some thought, I like the space delimited approach better. Here's
the patch, which retains the ability to remove the manifest hash
settings from layout.conf after they become redundant:

http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=commit;h=d9d0606fe01618cc81fb0b862ada91149dad3746
-- 
Thanks,
Zac

Re: [gentoo-portage-dev] [GLEP59v2 5/5] GLEP59: Change live Manifest2 hashes to SHA256, SHA512, WHIRLPOOL

[Brian Harring]

On Mon, Oct 03, 2011 at 02:48:55AM -0700, Zac Medico wrote:
> On 10/02/2011 05:21 PM, Zac Medico wrote:
> > On 10/02/2011 04:22 PM, Brian Harring wrote:
> >> On Sun, Oct 02, 2011 at 02:10:09PM -0700, Zac Medico wrote:
> > I've implemented it with booleans in this commit:
> > 
> > http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=commit;h=c8cd3a985cc529299411d7343a11004b7d1330ef
> > 
> >> Should use a space delimited list instead named hashes instead; those 
> >> being the hashes that should be generated, and that can be /used/.  
> >> Not in the list, not an acceptable hash (even if a manifest2 carries 
> >> that data).
> > 
> > Why? Boolean flags are simpler and they work.
> 
> After some thought, I like the space delimited approach better. Here's
> the patch, which retains the ability to remove the manifest hash
> settings from layout.conf after they become redundant:
> 
> http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=commit;h=d9d0606fe01618cc81fb0b862ada91149dad3746

Suggest you go through the implementation a bit closer; quick look, if 
the repo no longer uses what portage considers a required hash (atm, 
sha1), it still will force it in; while that's rather annoying for 
manifest creation, the validation logic there strikes me as probably 
being buggy for that case.

~harring

Re: [gentoo-portage-dev] [GLEP59v2 5/5] GLEP59: Change live Manifest2 hashes to SHA256, SHA512, WHIRLPOOL

[Zac Medico]

On 10/03/2011 04:43 AM, Brian Harring wrote:
> On Mon, Oct 03, 2011 at 02:48:55AM -0700, Zac Medico wrote:
>> After some thought, I like the space delimited approach better. Here's
>> the patch, which retains the ability to remove the manifest hash
>> settings from layout.conf after they become redundant:
>>
>> http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=commit;h=d9d0606fe01618cc81fb0b862ada91149dad3746
> 
> Suggest you go through the implementation a bit closer; quick look, if 
> the repo no longer uses what portage considers a required hash (atm, 
> sha1), it still will force it in; while that's rather annoying for 
> manifest creation, the validation logic there strikes me as probably 
> being buggy for that case.

This case is only supposed to come up if the user is generating
manifests with a version of portage that has become obsolete for this
type of operation on the repository. I suppose that we could simply make
Manifest.write() raise an exception in this case, and make digestgen()
return unsuccessfully with a suitable error message.

We could also add a layout.conf setting to override
MANIFEST2_REQUIRED_HASH, but that seems unnecessary as long as we stick
to the plan:

  After WHIRLPOOL is supported in stable portage:
  - Add SHA256 and WHIRLPOOL to MANIFEST2_HASH_DEFAULTS.
  - Remove SHA1 and RMD160 from MANIFEST2_HASH_*.
  - Set manifest-hashes in gentoo-x86/metadata/layout.conf as follows:
      manifest-hashes = SHA256 SHA512 WHIRLPOOL

  After WHIRLPOOL is supported in stable portage for at least 1 year:
  - Change MANIFEST2_REQUIRED_HASH to WHIRLPOOL.
  - Remove SHA256 from MANIFEST2_HASH_*.
  - Set manifest-hashes in gentoo-x86/metadata/layout.conf as follows:
      manifest-hashes = SHA512 WHIRLPOOL

  After SHA-3 is approved:
  - Add new hashes to MANIFEST2_HASH_*.

  After SHA-3 is supported in stable portage:
  - Set manifest-hashes in gentoo-x86/metadata/layout.conf as follows:
      manifest-hashes = SHA3 SHA512 WHIRLPOOL

  After layout.conf settings correspond to defaults in stable portage:
  - Remove redundant settings from gentoo-x86/metadata/layout.conf.
-- 
Thanks,
Zac

Re: [gentoo-portage-dev] [GLEP59v2 0/5] GLEP59: Manifest2 hash types

[Zac Medico]

On 10/01/2011 12:40 AM, Robin H. Johnson wrote:
> Respun now with the help of ferringb. Cleans up the implementation and catches
> a few bug and improvements:
> - mhash priority moved lower than pycrypto/hashlib because mhash holds GIL
>   while the other implementations don't.
> - hashlib does offer whirlpool if it was built against openssl 1.0.
> 
> 1/5: Refactor RMD160 hashlib code for less-hardcoding
> 2/5: Manifest2 hash: Whirlpool
> 3/5: Manifest2 hash: SHA512
> 4/5: Manifest2 hash backend provider: mhash
> 5/5: GLEP59: Change live Manifest2 hashes to SHA256,

This is released in portage-2.1.10.21 and 2.2.0_alpha61, so please go
ahead and test the "manifest-hashes = SHA256 SHA512 WHIRLPOOL" setting
in metadata/layout.conf, which we can deploy after
>=sys-apps/portage-2.1.10.21 is stable.
-- 
Thanks,
Zac

Re: [gentoo-portage-dev] [GLEP59v2 0/5] GLEP59: Manifest2 hash types

[Robin H. Johnson]

On Wed, Oct 05, 2011 at 11:07:03AM -0700, Zac Medico wrote:
> On 10/01/2011 12:40 AM, Robin H. Johnson wrote:
> > Respun now with the help of ferringb. Cleans up the implementation and catches
> > a few bug and improvements:
> > - mhash priority moved lower than pycrypto/hashlib because mhash holds GIL
> >   while the other implementations don't.
> > - hashlib does offer whirlpool if it was built against openssl 1.0.
> > 
> > 1/5: Refactor RMD160 hashlib code for less-hardcoding
> > 2/5: Manifest2 hash: Whirlpool
> > 3/5: Manifest2 hash: SHA512
> > 4/5: Manifest2 hash backend provider: mhash
> > 5/5: GLEP59: Change live Manifest2 hashes to SHA256,
> 
> This is released in portage-2.1.10.21 and 2.2.0_alpha61, so please go
> ahead and test the "manifest-hashes = SHA256 SHA512 WHIRLPOOL" setting
> in metadata/layout.conf, which we can deploy after
> >=sys-apps/portage-2.1.10.21 is stable.
Works here in a local copy of metadata/layout.conf.

-- 
Robin Hugh Johnson
Gentoo Linux: Developer, Trustee & Infrastructure Lead
E-Mail     : robbat2@gentoo.org
GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85