[gentoo-dev] [PATCH 0/3] glep-0074: Explicitly specify hashes and compressed Manifest formats

[gentoo-dev] [PATCH 0/3] glep-0074: Explicitly specify hashes and compressed Manifest formats

[Michał Górny]

Hi,

Here's a batch of three followup patches to GLEP 74 (Manifest),
to be reviewed separately from the previous set:

1. Add a changelog to the GLEP.
2. Explicitly list supported hash algorithms.
3. Explicitly list supported compressed Manifest formats.


Originally I didn't want to include these lists in GLEP 74 because
I didn't want to update the spec whenever we were to add a new hash.
However, from retrospective this seems wrong.  In the end, new hashes
are not something that happens every month (we didn't add any since
the GLEP was published) and updating the GLEP is relatively cheap.
At the same time, having a formal, well-defined list of algorithms
is good for interoperability.

I have also considered the option of listing the algorithms
in a separate GLEP but I think this option would only cause unnecessary
confusion.

-- 
Best regards,
Michał Górny

Michał Górny (3):
  glep-0074: Add a changelog
  glep-0074: Specify supported hash algorithms
  glep-0074: Specify compressed file formats

 glep-0074.rst | 220 ++++++++++++++++++++++++++++++++++++--------------
 1 file changed, 161 insertions(+), 59 deletions(-)

-- 
2.37.3

[gentoo-dev] [PATCH 1/3] glep-0074: Add a changelog

[Michał Górny]

Signed-off-by: Michał Górny <mgorny@gentoo.org>
---
 glep-0074.rst | 15 +++++++++++++--
 1 file changed, 13 insertions(+), 2 deletions(-)

diff --git a/glep-0074.rst b/glep-0074.rst
index 182a871..8cec618 100644
--- a/glep-0074.rst
+++ b/glep-0074.rst
@@ -8,8 +8,8 @@ Type: Standards Track
 Status: Final
 Version: 1.2
 Created: 2017-10-21
-Last-Modified: 2022-09-08
-Post-History: 2017-10-26, 2017-11-16, 2018-02-08, 2022-09-08
+Last-Modified: 2022-09-11
+Post-History: 2017-10-26, 2017-11-16, 2018-02-08, 2022-09-08, 2022-09-11
 Content-Type: text/x-rst
 Requires: 59, 61
 Replaces: 44, 58, 60
@@ -23,6 +23,17 @@ integrity and authenticity checks. The format aims to be future-proof,
 efficient and provide means of backwards compatibility.
 
 
+Changes
+=======
+
+v1.2
+  Specified the newline convention used for Manifests.
+
+v1.1
+  Removed the restriction that all files covered by a Manifest tree
+  must reside on the same filesystem.
+
+
 Motivation
 ==========
 
-- 
2.37.3

[gentoo-dev] [PATCH 2/3] glep-0074: Specify supported hash algorithms

[Michał Górny]

Replace the informational hash name section with a formal specification
of allowed hash algorithms.  The original reasoning for leaving them
implementation-defined was poor.  After all, not a single new hash
was added since the initial version of the GLEP.  At the same time,
ensuring consistent support for at least a minimal set of hash
algorithms is crucial to interoperability.  Given that the effort needed
to update the GLEP is relatively small, it is better to require all
algorithms to be formally listed than to have to track all
implementations for new hashes and hope for consistency.

Signed-off-by: Michał Górny <mgorny@gentoo.org>
---
 glep-0074.rst | 125 +++++++++++++++++++++++++++++++-------------------
 1 file changed, 77 insertions(+), 48 deletions(-)

diff --git a/glep-0074.rst b/glep-0074.rst
index 8cec618..33f6f80 100644
--- a/glep-0074.rst
+++ b/glep-0074.rst
@@ -6,7 +6,7 @@ Author: Michał Górny <mgorny@gentoo.org>,
         Ulrich Müller <ulm@gentoo.org>
 Type: Standards Track
 Status: Final
-Version: 1.2
+Version: 1.3
 Created: 2017-10-21
 Last-Modified: 2022-09-11
 Post-History: 2017-10-26, 2017-11-16, 2018-02-08, 2022-09-08, 2022-09-11
@@ -26,6 +26,9 @@ efficient and provide means of backwards compatibility.
 Changes
 =======
 
+v1.3
+  Formally specified the current set of hash algorithms supported.
+
 v1.2
   Specified the newline convention used for Manifests.
 
@@ -364,27 +367,50 @@ up to and including the *original* directory. Note that those
 sub-Manifests can use different filenames than ``Manifest``.
 
 
-Checksum algorithms (informational)
------------------------------------
-
-This section is informational only. Specifying the exact set
-of supported algorithms is outside the scope of this specification.
-
-The algorithm names reserved at the time of writing are:
-
-- ``MD5`` [#MD5]_,
-- ``RMD160`` -- RIPEMD-160 [#RIPEMD160]_,
-- ``SHA1`` [#SHS]_,
-- ``SHA256`` and ``SHA512`` -- SHA-2 family of hashes [#SHS]_,
-- ``WHIRLPOOL`` [#WHIRLPOOL]_,
-- ``BLAKE2B`` and ``BLAKE2S`` -- BLAKE2 family of hashes [#BLAKE2]_,
-- ``SHA3_256`` and ``SHA3_512`` -- SHA-3 family of hashes [#SHA3]_,
-- ``STREEBOG256`` and ``STREEBOG512`` -- Streebog family of hashes
-  [#STREEBOG]_.
-
-The method of introducing new hashes is defined by GLEP 59 [#GLEP59]_.
-It is recommended that any new hashes are named after the Python
-``hashlib`` module algorithm names, transformed into uppercase.
+Checksum algorithms
+-------------------
+
+.. table:: Table 1. Defined hash algorithms
+   :widths: auto
+   :class: table table-bordered table-striped
+
+   =============== ======================= ==== ==== ===========
+   Name            Specification           Bits Enc. Notes
+   =============== ======================= ==== ==== ===========
+   ``BLAKE2B``     RFC 7693 [#RFC7693]_    512  Hex  Recommended
+   ``BLAKE2S``     ditto                   256  Hex
+   ``MD5``         RFC 1321 [#RFC1321]_    128  Hex  Deprecated
+   ``RMD160``      RIPEMD-160 [#RMD160]_   160  Hex  Deprecated
+   ``SHA1``        FIPS 180-4 [#SHS]_      160  Hex  Deprecated
+   ``SHA256``      ditto                   256  Hex
+   ``SHA512``      ditto                   512  Hex  Recommended
+   ``SHA3_256``    FIPS 202 [#SHA3]_       256  Hex
+   ``SHA3_512``    ditto                   512  Hex
+   ``STREEBOG256`` RFC 6986 [#RFC6986]_    256  Hex
+   ``STREEBOG512`` ditto                   512  Hex
+   ``WHIRLPOOL``   Whirlpool [#WHIRLPOOL]_ 512  Hex
+   =============== ======================= ==== ==== ===========
+
+Any new hashes must be added to this specification prior to being used
+in Manifest files. Adding a new hash is considered
+a backwards-compatible change to the GLEP. It is recommended that new
+hashes are named after the Python ``hashlib`` module algorithm names,
+transformed into uppercase, with dashes replaced by underscores.
+
+An implementation can implement an arbitrary subset of the listed
+hashes. For best interoperability, it should implement at least
+recommended hashes. If deprecated hashes are implemented, it is
+preferable to disallow their use by default.
+
+If an entry specifies multiple hashes using different algorithms,
+an implementation may choose to verify an arbitrary subset of them.
+However, should any tested hash yield a mismatch, the verification must
+fail.
+
+If a particular hash is either unsupported or unknown,
+the implementation can either ignore it or report a failure. However,
+at least one algorithm specified for a particular entry must be
+supported for the verification to succeed.
 
 
 Manifest compression
@@ -913,23 +939,25 @@ tool working with this Manifest format.
 Hash algorithms
 ---------------
 
-While maintaining a consistent supported hash set is important
-for interoperability, it is not a good fit for the generic layout
-of this GLEP. Furthermore, it would require updating the GLEP
-in the future every time the used algorithms change.
+Originally, this GLEP did not formally specify the complete set of hash
+algorithms. Instead, it only listed (informationally) the names already
+used at the time of writing. Since enforcing consistent use of algorithm
+names is important for interoperability, this was changed in version
+1.3.
 
-Instead, the specification focuses on listing the currently used
-algorithm names for interoperability, and sets a recommendation
-for consistent naming of algorithms in the future. The Python
-``hashlib`` module is used as a reference since it is used
-as the provider of hash functions for most of the Python software,
-including Portage and PkgCore.
+Since the effort needed to update the GLEP is small compared to the time
+needed for a new hash algorithm to be well-deployed, the GLEP needs
+to be updated prior to adding a new hash method.
 
-The basic rules for changing hash algorithms are defined in GLEP 59
-[#GLEP59]_. The implementations can focus only on those algorithms
-that are actually used or planned on being used. It may be feasible
-to devise a new GLEP that specifies the currently used hashes (or update
-GLEP 59 accordingly).
+The recommended naming is based off Python ``hashlib`` module,
+as most of the Gentoo tooling is written in Python. The names
+are transformed to match the historical naming used for hash functions
+in Manifests.
+
+Implementations are allowed to support and use only a subset of hashes
+listed in Manifest files. This could be used both to avoid the overhead
+of computing multiple hashes on non-performant systems, and to handle
+transition to new hashes gracefully.
 
 
 Manifest compression
@@ -1072,27 +1100,28 @@ References
 .. [#FILE-NAMING-RULES] Ebuild File Format -- Gentoo Development Guide
    (https://devmanual.gentoo.org/ebuild-writing/file-format/#file-naming-rules)
 
-.. [#MD5] RFC1321: The MD5 Message-Digest Algorithm
-   (https://www.ietf.org/rfc/rfc1321.txt)
+.. [#RFC7693] RFC 7693: The BLAKE2 Cryptographic Hash and Message Authentication
+   Code (MAC)
+   (https://www.rfc-editor.org/rfc/rfc7693)
+
+.. [#RFC1321] RFC 1321: The MD5 Message-Digest Algorithm
+   (https://www.rfc-editor.org/rfc/rfc1321)
 
-.. [#RIPEMD160] The hash function RIPEMD-160
+.. [#RMD160] The hash function RIPEMD-160
    (https://homes.esat.kuleuven.be/~bosselae/ripemd160.html)
 
 .. [#SHS] FIPS PUB 180-4: Secure Hash Standard (SHS)
    (http://nvlpubs.nist.gov/nistpubs/FIPS/NIST.FIPS.180-4.pdf)
 
-.. [#WHIRLPOOL] The WHIRLPOOL Hash Function
-   (http://www.larc.usp.br/~pbarreto/WhirlpoolPage.html)
-
-.. [#BLAKE2] BLAKE2 -- fast secure hashing
-   (https://blake2.net/)
-
 .. [#SHA3] FIPS PUB 202: SHA-3 Standard: Permutation-Based Hash
    and Extendable-Output Functions
    (http://nvlpubs.nist.gov/nistpubs/FIPS/NIST.FIPS.202.pdf)
 
-.. [#STREEBOG] GOST R 34.11-2012: Streebog Hash Function
-   (https://www.streebog.net/)
+.. [#RFC6986] RFC 6986: GOST R 34.11-2012: Hash Function
+   (https://www.rfc-editor.org/rfc/rfc6986)
+
+.. [#WHIRLPOOL] The WHIRLPOOL Hash Function (archived at 2017-11-29)
+   (https://web.archive.org/web/20171129084214/http://www.larc.usp.br/~pbarreto/WhirlpoolPage.html)
 
 .. [#C08] Cappos, J et al. (2008). "Attacks on Package Managers"
    (https://www2.cs.arizona.edu/stork/packagemanagersecurity/attacks-on-package-managers.html)
-- 
2.37.3

[gentoo-dev] [PATCH 3/3] glep-0074: Specify compressed file formats

[Michał Górny]

Signed-off-by: Michał Górny <mgorny@gentoo.org>
---
 glep-0074.rst | 82 ++++++++++++++++++++++++++++++++++++++++++++-------
 1 file changed, 72 insertions(+), 10 deletions(-)

diff --git a/glep-0074.rst b/glep-0074.rst
index 33f6f80..6deeaf8 100644
--- a/glep-0074.rst
+++ b/glep-0074.rst
@@ -27,7 +27,8 @@ Changes
 =======
 
 v1.3
-  Formally specified the current set of hash algorithms supported.
+  Formally specified the current set of hash algorithms and compressed
+  Manifest formats supported.
 
 v1.2
   Specified the newline convention used for Manifests.
@@ -422,9 +423,8 @@ compression and this specification.
 
 The compressed Manifest files are required to be suffixed for their
 compression algorithm. This suffix should be used to recognize
-the compression and decompress Manifests transparently. The exact list
-of algorithms and their corresponding suffixes are outside the scope
-of this specification.
+the compression and decompress Manifests transparently. The supported
+formats are specified in `compressed file formats`_ section.
 
 The top-level Manifest file must not be compressed. Since the OpenPGP
 signature covers the uncompressed text and is compressed itself,
@@ -445,6 +445,47 @@ uncompressed content and the specification is free to choose either
 of the files using the same base name.
 
 
+Compressed file formats
+-----------------------
+
+.. table:: Table 2. Defined compressed file formats
+   :widths: auto
+   :class: table table-bordered table-striped
+
+   =========== ====== ==================== ===========
+   Tool name   Suffix Specification        Notes
+   =========== ====== ==================== ===========
+   bzip2       .bz2   (none known)
+   gzip        .gz    RFC 1952 [#RFC1952]_ Recommended
+   lz4         .lz4   (none known)
+   lzip        .lz    RFC draft [#LZIP]_
+   lzma        .lzma  (none known)         Deprecated
+   lzop        .lzo   (none known)
+   xz          .xz    xz [#XZ]_
+   zstd        .zst   RFC 8878 [#RFC8878]_
+   =========== ====== ==================== ===========
+
+Any new formats must be added to this specification prior to being used
+for Manifest files. Adding a new compressed file format is considered
+a backwards-compatible change to the GLEP. It is recommended that new
+formats use their reference (most common) file suffixes.
+
+An implementation can implement an arbitrary subset of the listed
+formats. For best interoperability, it should implement at least
+the recommended formats. Using deprecated formats should be avoided.
+
+If multiple Manifest variants coexist using different compressed file
+formats, the implementation may choose to use an arbitrary subset
+of them. However, all of them must be verified against the hashes stored
+in the containing Manifest. Should they be decompressed, the resulting
+contents must be identical.
+
+If the compressed file format is unsupported and a variant using
+a supported format coexists, the other variant should be used. However,
+at least one supported variant must exist for the verification
+to succeed.
+
+
 Combining multiple Manifest trees (informational)
 -------------------------------------------------
 
@@ -986,12 +1027,19 @@ into a compressed sub-Manifest in the top directory (e.g.
 ``Manifest.sub.gz``), and including a ``MANIFEST`` entry for this file
 in a signed, uncompressed top-level Manifest.
 
-The existence of additional entries for uncompressed Manifest checksums
-was debated. However, plain entries for the uncompressed file would
-be confusing if only the compressed file existed, and conflicting
-if both uncompressed and compressed variants existed. Furthermore,
-it has been pointed out that ``DIST`` entries do not have
-an uncompressed variant either.
+The existence of additional entries for checksums of Manifest contents
+after uncompressing was debated. However, plain entries for
+the uncompressed file would be confusing if only the compressed file
+existed. Furthermore, it has been pointed out that ``DIST`` entries
+do not have an uncompressed variant either.
+
+The specification permits coexistence of multiple variants of the same
+Manifest file using different compression for historical compatibility.
+However, there does not seem to be any real benefit from including
+a compressed Manifest file if the uncompressed variant needs to exist
+anyway. Providing different compressed variants could technically
+improve interoperability, though the same result could probably
+be achieved by using a more commonly supported format (e.g. gzip).
 
 
 Performance considerations
@@ -1123,6 +1171,20 @@ References
 .. [#WHIRLPOOL] The WHIRLPOOL Hash Function (archived at 2017-11-29)
    (https://web.archive.org/web/20171129084214/http://www.larc.usp.br/~pbarreto/WhirlpoolPage.html)
 
+.. [#RFC1952] RFC 1952: GZIP file format specification version 4.3
+   (https://www.rfc-editor.org/rfc/rfc1952)
+
+.. [#LZIP] RFC draft: Lzip Compressed Format and the 'application/lzip'
+   Media Type
+   (https://datatracker.ietf.org/doc/html/draft-diaz-lzip)
+
+.. [#XZ] The .xz File Format
+   (https://tukaani.org/xz/xz-file-format.txt)
+
+.. [#RFC8878] RFC 8878: Zstandard Compression and the 'application/zstd'
+   Media Type
+   (https://www.rfc-editor.org/rfc/rfc8878)
+
 .. [#C08] Cappos, J et al. (2008). "Attacks on Package Managers"
    (https://www2.cs.arizona.edu/stork/packagemanagersecurity/attacks-on-package-managers.html)
 
-- 
2.37.3

Re: [gentoo-dev] [PATCH 2/3] glep-0074: Specify supported hash algorithms

[Florian Schmaus]

[-- Attachment #1.1.1: Type: text/plain, Size: 1285 bytes --]

On 11/09/2022 15.14, Michał Górny wrote:
> +.. table:: Table 1. Defined hash algorithms
> +   :widths: auto
> +   :class: table table-bordered table-striped
> +
> +   =============== ======================= ==== ==== ===========
> +   Name            Specification           Bits Enc. Notes
> +   =============== ======================= ==== ==== ===========
> +   ``BLAKE2B``     RFC 7693 [#RFC7693]_    512  Hex  Recommended
> +   ``BLAKE2S``     ditto                   256  Hex
> +   ``MD5``         RFC 1321 [#RFC1321]_    128  Hex  Deprecated
> +   ``RMD160``      RIPEMD-160 [#RMD160]_   160  Hex  Deprecated
> +   ``SHA1``        FIPS 180-4 [#SHS]_      160  Hex  Deprecated
> +   ``SHA256``      ditto                   256  Hex
> +   ``SHA512``      ditto                   512  Hex  Recommended
> +   ``SHA3_256``    FIPS 202 [#SHA3]_       256  Hex
> +   ``SHA3_512``    ditto                   512  Hex
> +   ``STREEBOG256`` RFC 6986 [#RFC6986]_    256  Hex
> +   ``STREEBOG512`` ditto                   512  Hex
> +   ``WHIRLPOOL``   Whirlpool [#WHIRLPOOL]_ 512  Hex
> +   =============== ======================= ==== ==== ===========

Personally, I would slightly prefer repeating the source specification, 
instead of using "ditto".

- Flow

[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 25757 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 495 bytes --]

Re: [gentoo-dev] [PATCH 2/3] glep-0074: Specify supported hash algorithms

[Michał Górny]

On Mon, 2022-09-12 at 10:28 +0200, Florian Schmaus wrote:
> On 11/09/2022 15.14, Michał Górny wrote:
> > +.. table:: Table 1. Defined hash algorithms
> > +   :widths: auto
> > +   :class: table table-bordered table-striped
> > +
> > +   =============== ======================= ==== ==== ===========
> > +   Name            Specification           Bits Enc. Notes
> > +   =============== ======================= ==== ==== ===========
> > +   ``BLAKE2B``     RFC 7693 [#RFC7693]_    512  Hex  Recommended
> > +   ``BLAKE2S``     ditto                   256  Hex
> > +   ``MD5``         RFC 1321 [#RFC1321]_    128  Hex  Deprecated
> > +   ``RMD160``      RIPEMD-160 [#RMD160]_   160  Hex  Deprecated
> > +   ``SHA1``        FIPS 180-4 [#SHS]_      160  Hex  Deprecated
> > +   ``SHA256``      ditto                   256  Hex
> > +   ``SHA512``      ditto                   512  Hex  Recommended
> > +   ``SHA3_256``    FIPS 202 [#SHA3]_       256  Hex
> > +   ``SHA3_512``    ditto                   512  Hex
> > +   ``STREEBOG256`` RFC 6986 [#RFC6986]_    256  Hex
> > +   ``STREEBOG512`` ditto                   512  Hex
> > +   ``WHIRLPOOL``   Whirlpool [#WHIRLPOOL]_ 512  Hex
> > +   =============== ======================= ==== ==== ===========
> 
> Personally, I would slightly prefer repeating the source specification, 
> instead of using "ditto".
> 

Unfortunately, that would mean each of them would have 2-3
backreferences in the 'References' section.

-- 
Best regards,
Michał Górny