[gentoo-portage-dev] [PATCH 1/3] portage.const: Remove obsolete manifest-hashes comment

[gentoo-portage-dev] [PATCH 1/3] portage.const: Remove obsolete manifest-hashes comment

[Michał Górny]

The comment is based upon the wrong assumption that the world is
Portage-centric. We are keeping the manifest-hashes in layout.conf
indefinitely, and we will most likely take BLAKE2 instead of SHA3.
---
 pym/portage/const.py | 23 -----------------------
 1 file changed, 23 deletions(-)

diff --git a/pym/portage/const.py b/pym/portage/const.py
index cbd2b6042..11e94b0a2 100644
--- a/pym/portage/const.py
+++ b/pym/portage/const.py
@@ -207,29 +207,6 @@ EAPI                     = 6
 HASHING_BLOCKSIZE        = 32768
 MANIFEST1_HASH_FUNCTIONS = ("MD5", "SHA256", "RMD160")
 
-# Past events:
-#
-# 20120704 - After WHIRLPOOL is supported in stable portage:
-# - Set manifest-hashes in gentoo-x86/metadata/layout.conf as follows:
-#     manifest-hashes = SHA256 SHA512 WHIRLPOOL
-# - Add SHA512 and WHIRLPOOL to MANIFEST2_HASH_DEFAULTS.
-# - Remove SHA1 and RMD160 from MANIFEST2_HASH_*.
-#
-# Future events:
-#
-# After WHIRLPOOL is supported in stable portage for at least 1 year:
-# - Change MANIFEST2_REQUIRED_HASH to WHIRLPOOL.
-# - Remove SHA256 from MANIFEST2_HASH_*.
-# - Set manifest-hashes in gentoo-x86/metadata/layout.conf as follows:
-#     manifest-hashes = SHA512 WHIRLPOOL
-#
-# After SHA-3 is supported in stable portage:
-# - Set manifest-hashes in gentoo-x86/metadata/layout.conf as follows:
-#     manifest-hashes = SHA3 SHA512 WHIRLPOOL
-#
-# After layout.conf settings correspond to defaults in stable portage:
-# - Remove redundant settings from gentoo-x86/metadata/layout.conf.
-
 MANIFEST2_HASH_FUNCTIONS = ("SHA256", "SHA512", "WHIRLPOOL",
 		"BLAKE2B", "BLAKE2S", "SHA3_256", "SHA3_512",
 		"STREEBOG256", "STREEBOG512")
-- 
2.15.0

[gentoo-portage-dev] [PATCH 2/3] Replace static MANIFEST2_HASH_FUNCTIONS with dynamic list

[Michał Górny]

Remove the MANIFEST2_HASH_FUNCTIONS const and replace it with
(deduplicated) calls to get_valid_checksum_keys(). We want Portage
to always complain whenever one of the hashes is not available even
if it is technically supported by the specific Portage version.

Closes: https://bugs.gentoo.org/634812
---
 man/portage.5                    | 4 ++--
 pym/portage/const.py             | 3 ---
 pym/portage/manifest.py          | 6 +++---
 pym/portage/repository/config.py | 5 +++--
 repoman/pym/repoman/repos.py     | 3 ++-
 5 files changed, 10 insertions(+), 11 deletions(-)

diff --git a/man/portage.5 b/man/portage.5
index 89dc8ce44..7605d7cfa 100644
--- a/man/portage.5
+++ b/man/portage.5
@@ -1261,8 +1261,8 @@ for every file), "true" (if an entry exists for a file, enforce it), or "false"
 .TP
 .BR manifest\-hashes
 List of hashes to generate/check in Manifest files.  Valid hashes depend on the
-current version of portage; see the portage.const.MANIFEST2_HASH_FUNCTIONS
-constant for the current list.
+current version of portage; see the portage.checksum module for the current
+list.
 .TP
 .BR update\-changelog " = [true|" false "]"
 The default setting for repoman's --echangelog option.
diff --git a/pym/portage/const.py b/pym/portage/const.py
index 11e94b0a2..98b7c88c9 100644
--- a/pym/portage/const.py
+++ b/pym/portage/const.py
@@ -207,9 +207,6 @@ EAPI                     = 6
 HASHING_BLOCKSIZE        = 32768
 MANIFEST1_HASH_FUNCTIONS = ("MD5", "SHA256", "RMD160")
 
-MANIFEST2_HASH_FUNCTIONS = ("SHA256", "SHA512", "WHIRLPOOL",
-		"BLAKE2B", "BLAKE2S", "SHA3_256", "SHA3_512",
-		"STREEBOG256", "STREEBOG512")
 MANIFEST2_HASH_DEFAULTS = frozenset(["SHA256", "SHA512", "WHIRLPOOL"])
 MANIFEST2_REQUIRED_HASH  = "SHA512"
 
diff --git a/pym/portage/manifest.py b/pym/portage/manifest.py
index 28a77ba87..0a68aa653 100644
--- a/pym/portage/manifest.py
+++ b/pym/portage/manifest.py
@@ -27,7 +27,7 @@ from portage.exception import DigestException, FileNotFound, \
 	InvalidDataType, MissingParameter, PermissionDenied, \
 	PortageException, PortagePackageException
 from portage.const import (MANIFEST1_HASH_FUNCTIONS, MANIFEST2_HASH_DEFAULTS,
-	MANIFEST2_HASH_FUNCTIONS, MANIFEST2_IDENTIFIERS, MANIFEST2_REQUIRED_HASH)
+	MANIFEST2_IDENTIFIERS, MANIFEST2_REQUIRED_HASH)
 from portage.localization import _
 
 _manifest_re = re.compile(
@@ -152,7 +152,7 @@ class Manifest(object):
 		if hashes is None:
 			hashes = MANIFEST2_HASH_DEFAULTS
 
-		self.hashes.update(hashes.intersection(MANIFEST2_HASH_FUNCTIONS))
+		self.hashes.update(hashes)
 		self.hashes.difference_update(hashname for hashname in \
 			list(self.hashes) if hashname not in get_valid_checksum_keys())
 		self.hashes.add("size")
@@ -251,7 +251,7 @@ class Manifest(object):
 		return myhashdict
 
 	def _createManifestEntries(self):
-		valid_hashes = set(MANIFEST2_HASH_FUNCTIONS)
+		valid_hashes = set(get_valid_checksum_keys())
 		valid_hashes.add('size')
 		mytypes = list(self.fhashdict)
 		mytypes.sort()
diff --git a/pym/portage/repository/config.py b/pym/portage/repository/config.py
index 902213014..3be0e8bda 100644
--- a/pym/portage/repository/config.py
+++ b/pym/portage/repository/config.py
@@ -11,7 +11,8 @@ import re
 
 import portage
 from portage import eclass_cache, os
-from portage.const import (MANIFEST2_HASH_FUNCTIONS, MANIFEST2_REQUIRED_HASH,
+from portage.checksum import get_valid_checksum_keys
+from portage.const import (MANIFEST2_REQUIRED_HASH,
 	PORTAGE_BASE_PATH, REPO_NAME_LOC, USER_CONFIG_PATH)
 from portage.eapi import eapi_allows_directories_on_profile_level_and_repository_level
 from portage.env.loaders import KeyValuePairFileLoader
@@ -1060,7 +1061,7 @@ def parse_layout_conf(repo_location, repo_name=None):
 				"layout_filename":layout_filename}),
 				DeprecationWarning)
 		unsupported_hashes = manifest_hashes.difference(
-			MANIFEST2_HASH_FUNCTIONS)
+			get_valid_checksum_keys())
 		if unsupported_hashes:
 			repo_name = _get_repo_name(repo_location, cached=repo_name)
 			warnings.warn((_("Repository named '%(repo_name)s' has a "
diff --git a/repoman/pym/repoman/repos.py b/repoman/pym/repoman/repos.py
index 39f53c180..11a6231de 100644
--- a/repoman/pym/repoman/repos.py
+++ b/repoman/pym/repoman/repos.py
@@ -13,6 +13,7 @@ from repoman._portage import portage
 from portage import os
 from portage import _encodings
 from portage import _unicode_encode
+from portage.checksum import get_valid_checksum_keys
 
 from repoman.errors import err
 from repoman.profile import ProfileDesc, valid_profile_types
@@ -116,7 +117,7 @@ class RepoSettings(object):
 				sys.exit(1)
 
 			unsupported_hashes = manifest_hashes.difference(
-				portage.const.MANIFEST2_HASH_FUNCTIONS)
+				get_valid_checksum_keys())
 			if unsupported_hashes:
 				msg = (
 					"The 'manifest-hashes' setting in the '%s' repository's "
-- 
2.15.0

[gentoo-portage-dev] [PATCH 3/3] Remove last traces of Manifest1 code

[Michał Górny]

---
 pym/portage/const.py    | 1 -
 pym/portage/manifest.py | 7 ++-----
 2 files changed, 2 insertions(+), 6 deletions(-)

diff --git a/pym/portage/const.py b/pym/portage/const.py
index 98b7c88c9..0af57d0e2 100644
--- a/pym/portage/const.py
+++ b/pym/portage/const.py
@@ -205,7 +205,6 @@ SUPPORTED_FEATURES       = frozenset([
 EAPI                     = 6
 
 HASHING_BLOCKSIZE        = 32768
-MANIFEST1_HASH_FUNCTIONS = ("MD5", "SHA256", "RMD160")
 
 MANIFEST2_HASH_DEFAULTS = frozenset(["SHA256", "SHA512", "WHIRLPOOL"])
 MANIFEST2_REQUIRED_HASH  = "SHA512"
diff --git a/pym/portage/manifest.py b/pym/portage/manifest.py
index 0a68aa653..36c82690c 100644
--- a/pym/portage/manifest.py
+++ b/pym/portage/manifest.py
@@ -26,7 +26,7 @@ from portage import _unicode_encode
 from portage.exception import DigestException, FileNotFound, \
 	InvalidDataType, MissingParameter, PermissionDenied, \
 	PortageException, PortagePackageException
-from portage.const import (MANIFEST1_HASH_FUNCTIONS, MANIFEST2_HASH_DEFAULTS,
+from portage.const import (MANIFEST2_HASH_DEFAULTS,
 	MANIFEST2_IDENTIFIERS, MANIFEST2_REQUIRED_HASH)
 from portage.localization import _
 
@@ -710,10 +710,7 @@ class Manifest(object):
 		myfile.close()
 		for l in lines:
 			mysplit = l.split()
-			if len(mysplit) == 4 and mysplit[0] in MANIFEST1_HASH_FUNCTIONS \
-				and 1 not in rVal:
-				rVal.append(1)
-			elif len(mysplit) > 4 and mysplit[0] in MANIFEST2_IDENTIFIERS \
+			if len(mysplit) > 4 and mysplit[0] in MANIFEST2_IDENTIFIERS \
 				and ((len(mysplit) - 3) % 2) == 0 and not 2 in rVal:
 				rVal.append(2)
 		return rVal
-- 
2.15.0

Re: [gentoo-portage-dev] [PATCH 3/3] Remove last traces of Manifest1 code

[Zac Medico]

[-- Attachment #1.1: Type: text/plain, Size: 249 bytes --]

On 11/06/2017 12:14 AM, Michał Górny wrote:
> ---
>  pym/portage/const.py    | 1 -
>  pym/portage/manifest.py | 7 ++-----
>  2 files changed, 2 insertions(+), 6 deletions(-)

This patch series looks good. Please merge.
-- 
Thanks,
Zac


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 224 bytes --]

Re: [gentoo-portage-dev] [PATCH 1/3] portage.const: Remove obsolete manifest-hashes comment

[Robin H. Johnson]

[-- Attachment #1: Type: text/plain, Size: 795 bytes --]

On Mon, Nov 06, 2017 at 09:14:56AM +0100, Michał Górny wrote:
> -# Future events:
> -#
> -# After WHIRLPOOL is supported in stable portage for at least 1 year:
> -# - Change MANIFEST2_REQUIRED_HASH to WHIRLPOOL.
> -# - Remove SHA256 from MANIFEST2_HASH_*.
> -# - Set manifest-hashes in gentoo-x86/metadata/layout.conf as follows:
> -#     manifest-hashes = SHA512 WHIRLPOOL
Can we please drop SHA256 as a required hash already? It was scheduled
for future removal a long time ago. I don't want that to get lost with
the removal of this comment block.

-- 
Robin Hugh Johnson
Gentoo Linux: Dev, Infra Lead, Foundation Asst. Treasurer
E-Mail   : robbat2@gentoo.org
GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 1113 bytes --]

Re: [gentoo-portage-dev] [PATCH 1/3] portage.const: Remove obsolete manifest-hashes comment

[Michał Górny]

Dnia 7 listopada 2017 00:01:18 CET, "Robin H. Johnson" <robbat2@gentoo.org> napisał(a):
>On Mon, Nov 06, 2017 at 09:14:56AM +0100, Michał Górny wrote:
>> -# Future events:
>> -#
>> -# After WHIRLPOOL is supported in stable portage for at least 1
>year:
>> -# - Change MANIFEST2_REQUIRED_HASH to WHIRLPOOL.
>> -# - Remove SHA256 from MANIFEST2_HASH_*.
>> -# - Set manifest-hashes in gentoo-x86/metadata/layout.conf as
>follows:
>> -#     manifest-hashes = SHA512 WHIRLPOOL
>Can we please drop SHA256 as a required hash already? It was scheduled
>for future removal a long time ago. I don't want that to get lost with
>the removal of this comment block.

I want to remove it when updating manifest-hashes, to avoid unnecessary interim Manifest rewrites.

Also SHA512 was the required hash for some time already.

-- 
Best regards,
Michał Górny (by phone)