[gentoo-dev] [PATCH v2 0/3] glep-0074: Explicitly specify hashes and compressed Manifest formats

[gentoo-dev] [PATCH v2 0/3] glep-0074: Explicitly specify hashes and compressed Manifest formats

[Michał Górny]

Hi,

Here's v2 of the patches.  Change from v1: removed explicit Bootstrap
table classes (they don't really belong in GLEPs) in favor of fixed
table rendering in docutils-glep 1.5 / matching www.g.o update.
Use rowspan="" instead of "ditto"s for hashes using the same spec.

Michał Górny (3):
  glep-0074: Add a changelog
  glep-0074: Specify supported hash algorithms
  glep-0074: Specify compressed file formats

 glep-0074.rst | 230 +++++++++++++++++++++++++++++++++++++-------------
 1 file changed, 171 insertions(+), 59 deletions(-)

-- 
2.37.3

[gentoo-dev] [PATCH v2 1/3] glep-0074: Add a changelog

[Michał Górny]

Signed-off-by: Michał Górny <mgorny@gentoo.org>
---
 glep-0074.rst | 15 +++++++++++++--
 1 file changed, 13 insertions(+), 2 deletions(-)

diff --git a/glep-0074.rst b/glep-0074.rst
index 182a871..8cec618 100644
--- a/glep-0074.rst
+++ b/glep-0074.rst
@@ -8,8 +8,8 @@ Type: Standards Track
 Status: Final
 Version: 1.2
 Created: 2017-10-21
-Last-Modified: 2022-09-08
-Post-History: 2017-10-26, 2017-11-16, 2018-02-08, 2022-09-08
+Last-Modified: 2022-09-11
+Post-History: 2017-10-26, 2017-11-16, 2018-02-08, 2022-09-08, 2022-09-11
 Content-Type: text/x-rst
 Requires: 59, 61
 Replaces: 44, 58, 60
@@ -23,6 +23,17 @@ integrity and authenticity checks. The format aims to be future-proof,
 efficient and provide means of backwards compatibility.
 
 
+Changes
+=======
+
+v1.2
+  Specified the newline convention used for Manifests.
+
+v1.1
+  Removed the restriction that all files covered by a Manifest tree
+  must reside on the same filesystem.
+
+
 Motivation
 ==========
 
-- 
2.37.3

[gentoo-dev] [PATCH v2 2/3] glep-0074: Specify supported hash algorithms

[Michał Górny]

Replace the informational hash name section with a formal specification
of allowed hash algorithms.  The original reasoning for leaving them
implementation-defined was poor.  After all, not a single new hash
was added since the initial version of the GLEP.  At the same time,
ensuring consistent support for at least a minimal set of hash
algorithms is crucial to interoperability.  Given that the effort needed
to update the GLEP is relatively small, it is better to require all
algorithms to be formally listed than to have to track all
implementations for new hashes and hope for consistency.

Signed-off-by: Michał Górny <mgorny@gentoo.org>
---
 glep-0074.rst | 136 ++++++++++++++++++++++++++++++++------------------
 1 file changed, 88 insertions(+), 48 deletions(-)

diff --git a/glep-0074.rst b/glep-0074.rst
index 8cec618..9a0e92b 100644
--- a/glep-0074.rst
+++ b/glep-0074.rst
@@ -6,7 +6,7 @@ Author: Michał Górny <mgorny@gentoo.org>,
         Ulrich Müller <ulm@gentoo.org>
 Type: Standards Track
 Status: Final
-Version: 1.2
+Version: 1.3
 Created: 2017-10-21
 Last-Modified: 2022-09-11
 Post-History: 2017-10-26, 2017-11-16, 2018-02-08, 2022-09-08, 2022-09-11
@@ -26,6 +26,9 @@ efficient and provide means of backwards compatibility.
 Changes
 =======
 
+v1.3
+  Formally specified the current set of hash algorithms supported.
+
 v1.2
   Specified the newline convention used for Manifests.
 
@@ -364,27 +367,60 @@ up to and including the *original* directory. Note that those
 sub-Manifests can use different filenames than ``Manifest``.
 
 
-Checksum algorithms (informational)
------------------------------------
-
-This section is informational only. Specifying the exact set
-of supported algorithms is outside the scope of this specification.
-
-The algorithm names reserved at the time of writing are:
-
-- ``MD5`` [#MD5]_,
-- ``RMD160`` -- RIPEMD-160 [#RIPEMD160]_,
-- ``SHA1`` [#SHS]_,
-- ``SHA256`` and ``SHA512`` -- SHA-2 family of hashes [#SHS]_,
-- ``WHIRLPOOL`` [#WHIRLPOOL]_,
-- ``BLAKE2B`` and ``BLAKE2S`` -- BLAKE2 family of hashes [#BLAKE2]_,
-- ``SHA3_256`` and ``SHA3_512`` -- SHA-3 family of hashes [#SHA3]_,
-- ``STREEBOG256`` and ``STREEBOG512`` -- Streebog family of hashes
-  [#STREEBOG]_.
-
-The method of introducing new hashes is defined by GLEP 59 [#GLEP59]_.
-It is recommended that any new hashes are named after the Python
-``hashlib`` module algorithm names, transformed into uppercase.
+Checksum algorithms
+-------------------
+
+.. table:: Table 1. Defined hash algorithms
+   :widths: auto
+
+   +-----------------+-----------------------+------+------+-------------+
+   | Name            | Specification         | Bits | Enc. | Notes       |
+   +=================+=======================+======+======+=============+
+   | ``BLAKE2B``     |                       | 512  | Hex  | Recommended |
+   +-----------------+ RFC 7693 [#RFC7693]_  +------+------+-------------+
+   | ``BLAKE2S``     |                       | 256  | Hex  |             |
+   +-----------------+-----------------------+------+------+-------------+
+   | ``MD5``         | RFC 1321 [#RFC1321]_  | 128  | Hex  | Deprecated  |
+   +-----------------+-----------------------+------+------+-------------+
+   | ``RMD160``      | RIPEMD-160 [#RMD160]_ | 160  | Hex  | Deprecated  |
+   +-----------------+-----------------------+------+------+-------------+
+   | ``SHA1``        |                       | 160  | Hex  | Deprecated  |
+   +-----------------+                       +------+------+-------------+
+   | ``SHA256``      | FIPS 180-4 [#SHS]_    | 256  | Hex  |             |
+   +-----------------+                       +------+------+-------------+
+   | ``SHA512``      |                       | 512  | Hex  | Recommended |
+   +-----------------+-----------------------+------+------+-------------+
+   | ``SHA3_256``    |                       | 256  | Hex  |             |
+   +-----------------+ FIPS 202 [#SHA3]_     +------+------+-------------+
+   | ``SHA3_512``    |                       | 512  | Hex  |             |
+   +-----------------+-----------------------+------+------+-------------+
+   | ``STREEBOG256`` |                       | 256  | Hex  |             |
+   +-----------------+ RFC 6986 [#RFC6986]_  +------+------+-------------+
+   | ``STREEBOG512`` |                       | 512  | Hex  |             |
+   +-----------------+-----------------------+------+------+-------------+
+   | ``WHIRLPOOL``   | Whirlpool [#BARRETO]_ | 512  | Hex  |             |
+   +-----------------+-----------------------+------+------+-------------+
+
+Any new hashes must be added to this specification prior to being used
+in Manifest files. Adding a new hash is considered
+a backwards-compatible change to the GLEP. It is recommended that new
+hashes are named after the Python ``hashlib`` module algorithm names,
+transformed into uppercase, with dashes replaced by underscores.
+
+An implementation can implement an arbitrary subset of the listed
+hashes. For best interoperability, it should implement at least
+recommended hashes. If deprecated hashes are implemented, it is
+preferable to disallow their use by default.
+
+If an entry specifies multiple hashes using different algorithms,
+an implementation may choose to verify an arbitrary subset of them.
+However, should any tested hash yield a mismatch, the verification must
+fail.
+
+If a particular hash is either unsupported or unknown,
+the implementation can either ignore it or report a failure. However,
+at least one algorithm specified for a particular entry must be
+supported for the verification to succeed.
 
 
 Manifest compression
@@ -913,23 +949,25 @@ tool working with this Manifest format.
 Hash algorithms
 ---------------
 
-While maintaining a consistent supported hash set is important
-for interoperability, it is not a good fit for the generic layout
-of this GLEP. Furthermore, it would require updating the GLEP
-in the future every time the used algorithms change.
+Originally, this GLEP did not formally specify the complete set of hash
+algorithms. Instead, it only listed (informationally) the names already
+used at the time of writing. Since enforcing consistent use of algorithm
+names is important for interoperability, this was changed in version
+1.3.
 
-Instead, the specification focuses on listing the currently used
-algorithm names for interoperability, and sets a recommendation
-for consistent naming of algorithms in the future. The Python
-``hashlib`` module is used as a reference since it is used
-as the provider of hash functions for most of the Python software,
-including Portage and PkgCore.
+Since the effort needed to update the GLEP is small compared to the time
+needed for a new hash algorithm to be well-deployed, the GLEP needs
+to be updated prior to adding a new hash method.
 
-The basic rules for changing hash algorithms are defined in GLEP 59
-[#GLEP59]_. The implementations can focus only on those algorithms
-that are actually used or planned on being used. It may be feasible
-to devise a new GLEP that specifies the currently used hashes (or update
-GLEP 59 accordingly).
+The recommended naming is based off Python ``hashlib`` module,
+as most of the Gentoo tooling is written in Python. The names
+are transformed to match the historical naming used for hash functions
+in Manifests.
+
+Implementations are allowed to support and use only a subset of hashes
+listed in Manifest files. This could be used both to avoid the overhead
+of computing multiple hashes on non-performant systems, and to handle
+transition to new hashes gracefully.
 
 
 Manifest compression
@@ -1072,27 +1110,29 @@ References
 .. [#FILE-NAMING-RULES] Ebuild File Format -- Gentoo Development Guide
    (https://devmanual.gentoo.org/ebuild-writing/file-format/#file-naming-rules)
 
-.. [#MD5] RFC1321: The MD5 Message-Digest Algorithm
-   (https://www.ietf.org/rfc/rfc1321.txt)
+.. [#RFC7693] RFC 7693: The BLAKE2 Cryptographic Hash and Message Authentication
+   Code (MAC)
+   (https://www.rfc-editor.org/rfc/rfc7693)
+
+.. [#RFC1321] RFC 1321: The MD5 Message-Digest Algorithm
+   (https://www.rfc-editor.org/rfc/rfc1321)
 
-.. [#RIPEMD160] The hash function RIPEMD-160
+.. [#RMD160] The hash function RIPEMD-160
    (https://homes.esat.kuleuven.be/~bosselae/ripemd160.html)
 
 .. [#SHS] FIPS PUB 180-4: Secure Hash Standard (SHS)
    (http://nvlpubs.nist.gov/nistpubs/FIPS/NIST.FIPS.180-4.pdf)
 
-.. [#WHIRLPOOL] The WHIRLPOOL Hash Function
-   (http://www.larc.usp.br/~pbarreto/WhirlpoolPage.html)
-
-.. [#BLAKE2] BLAKE2 -- fast secure hashing
-   (https://blake2.net/)
-
 .. [#SHA3] FIPS PUB 202: SHA-3 Standard: Permutation-Based Hash
    and Extendable-Output Functions
    (http://nvlpubs.nist.gov/nistpubs/FIPS/NIST.FIPS.202.pdf)
 
-.. [#STREEBOG] GOST R 34.11-2012: Streebog Hash Function
-   (https://www.streebog.net/)
+.. [#RFC6986] RFC 6986: GOST R 34.11-2012: Hash Function
+   (https://www.rfc-editor.org/rfc/rfc6986)
+
+.. [#BARRETO] Paulo S. L. M. Barreto, The WHIRLPOOL Hash Function
+   (archived at 2017-11-29)
+   (https://web.archive.org/web/20171129084214/http://www.larc.usp.br/~pbarreto/WhirlpoolPage.html)
 
 .. [#C08] Cappos, J et al. (2008). "Attacks on Package Managers"
    (https://www2.cs.arizona.edu/stork/packagemanagersecurity/attacks-on-package-managers.html)
-- 
2.37.3

[gentoo-dev] [PATCH v2 3/3] glep-0074: Specify compressed file formats

[Michał Górny]

Signed-off-by: Michał Górny <mgorny@gentoo.org>
---
 glep-0074.rst | 81 ++++++++++++++++++++++++++++++++++++++++++++-------
 1 file changed, 71 insertions(+), 10 deletions(-)

diff --git a/glep-0074.rst b/glep-0074.rst
index 9a0e92b..6523272 100644
--- a/glep-0074.rst
+++ b/glep-0074.rst
@@ -27,7 +27,8 @@ Changes
 =======
 
 v1.3
-  Formally specified the current set of hash algorithms supported.
+  Formally specified the current set of hash algorithms and compressed
+  Manifest formats supported.
 
 v1.2
   Specified the newline convention used for Manifests.
@@ -432,9 +433,8 @@ compression and this specification.
 
 The compressed Manifest files are required to be suffixed for their
 compression algorithm. This suffix should be used to recognize
-the compression and decompress Manifests transparently. The exact list
-of algorithms and their corresponding suffixes are outside the scope
-of this specification.
+the compression and decompress Manifests transparently. The supported
+formats are specified in `compressed file formats`_ section.
 
 The top-level Manifest file must not be compressed. Since the OpenPGP
 signature covers the uncompressed text and is compressed itself,
@@ -455,6 +455,46 @@ uncompressed content and the specification is free to choose either
 of the files using the same base name.
 
 
+Compressed file formats
+-----------------------
+
+.. table:: Table 2. Defined compressed file formats
+   :widths: auto
+
+   ===========  ======  ====================  ===========
+   Tool name    Suffix  Specification         Notes
+   ===========  ======  ====================  ===========
+   bzip2        .bz2    (none known)
+   gzip         .gz     RFC 1952 [#RFC1952]_  Recommended
+   lz4          .lz4    (none known)
+   lzip         .lz     RFC draft [#LZIP]_
+   lzma         .lzma   (none known)          Deprecated
+   lzop         .lzo    (none known)
+   xz           .xz     xz [#XZ]_
+   zstd         .zst    RFC 8878 [#RFC8878]_
+   ===========  ======  ====================  ===========
+
+Any new formats must be added to this specification prior to being used
+for Manifest files. Adding a new compressed file format is considered
+a backwards-compatible change to the GLEP. It is recommended that new
+formats use their reference (most common) file suffixes.
+
+An implementation can implement an arbitrary subset of the listed
+formats. For best interoperability, it should implement at least
+the recommended formats. Using deprecated formats should be avoided.
+
+If multiple Manifest variants coexist using different compressed file
+formats, the implementation may choose to use an arbitrary subset
+of them. However, all of them must be verified against the hashes stored
+in the containing Manifest. Should they be decompressed, the resulting
+contents must be identical.
+
+If the compressed file format is unsupported and a variant using
+a supported format coexists, the other variant should be used. However,
+at least one supported variant must exist for the verification
+to succeed.
+
+
 Combining multiple Manifest trees (informational)
 -------------------------------------------------
 
@@ -996,12 +1036,19 @@ into a compressed sub-Manifest in the top directory (e.g.
 ``Manifest.sub.gz``), and including a ``MANIFEST`` entry for this file
 in a signed, uncompressed top-level Manifest.
 
-The existence of additional entries for uncompressed Manifest checksums
-was debated. However, plain entries for the uncompressed file would
-be confusing if only the compressed file existed, and conflicting
-if both uncompressed and compressed variants existed. Furthermore,
-it has been pointed out that ``DIST`` entries do not have
-an uncompressed variant either.
+The existence of additional entries for checksums of Manifest contents
+after uncompressing was debated. However, plain entries for
+the uncompressed file would be confusing if only the compressed file
+existed. Furthermore, it has been pointed out that ``DIST`` entries
+do not have an uncompressed variant either.
+
+The specification permits coexistence of multiple variants of the same
+Manifest file using different compression for historical compatibility.
+However, there does not seem to be any real benefit from including
+a compressed Manifest file if the uncompressed variant needs to exist
+anyway. Providing different compressed variants could technically
+improve interoperability, though the same result could probably
+be achieved by using a more commonly supported format (e.g. gzip).
 
 
 Performance considerations
@@ -1134,6 +1181,20 @@ References
    (archived at 2017-11-29)
    (https://web.archive.org/web/20171129084214/http://www.larc.usp.br/~pbarreto/WhirlpoolPage.html)
 
+.. [#RFC1952] RFC 1952: GZIP file format specification version 4.3
+   (https://www.rfc-editor.org/rfc/rfc1952)
+
+.. [#LZIP] RFC draft: Lzip Compressed Format and the 'application/lzip'
+   Media Type
+   (https://datatracker.ietf.org/doc/html/draft-diaz-lzip)
+
+.. [#XZ] The .xz File Format
+   (https://tukaani.org/xz/xz-file-format.txt)
+
+.. [#RFC8878] RFC 8878: Zstandard Compression and the 'application/zstd'
+   Media Type
+   (https://www.rfc-editor.org/rfc/rfc8878)
+
 .. [#C08] Cappos, J et al. (2008). "Attacks on Package Managers"
    (https://www2.cs.arizona.edu/stork/packagemanagersecurity/attacks-on-package-managers.html)
 
-- 
2.37.3

Re: [gentoo-dev] [PATCH v2 2/3] glep-0074: Specify supported hash algorithms

[Michał Górny]

On Mon, 2022-09-12 at 15:38 +0200, Michał Górny wrote:
> +   +-----------------+-----------------------+------+------+-------------+
> +   | Name            | Specification         | Bits | Enc. | Notes       |
> +   +=================+=======================+======+======+=============+
> +   | ``BLAKE2B``     |                       | 512  | Hex  | Recommended |
> +   +-----------------+ RFC 7693 [#RFC7693]_  +------+------+-------------+
> +   | ``BLAKE2S``     |                       | 256  | Hex  |             |
> +   +-----------------+-----------------------+------+------+-------------+
> +   | ``MD5``         | RFC 1321 [#RFC1321]_  | 128  | Hex  | Deprecated  |
> +   +-----------------+-----------------------+------+------+-------------+
> +   | ``RMD160``      | RIPEMD-160 [#RMD160]_ | 160  | Hex  | Deprecated  |

Small change here after some internal and external consideration: I've
removed the "deprecated" tag from RIPEMD-160 since (unlike MD5/SHA1)
it's not officially considered weakened.

> +   +-----------------+-----------------------+------+------+-------------+
> +   | ``SHA1``        |                       | 160  | Hex  | Deprecated  |
> +   +-----------------+                       +------+------+-------------+
> +   | ``SHA256``      | FIPS 180-4 [#SHS]_    | 256  | Hex  |             |
> +   +-----------------+                       +------+------+-------------+
> +   | ``SHA512``      |                       | 512  | Hex  | Recommended |
> +   +-----------------+-----------------------+------+------+-------------+
> +   | ``SHA3_256``    |                       | 256  | Hex  |             |
> +   +-----------------+ FIPS 202 [#SHA3]_     +------+------+-------------+
> +   | ``SHA3_512``    |                       | 512  | Hex  |             |
> +   +-----------------+-----------------------+------+------+-------------+
> +   | ``STREEBOG256`` |                       | 256  | Hex  |             |
> +   +-----------------+ RFC 6986 [#RFC6986]_  +------+------+-------------+
> +   | ``STREEBOG512`` |                       | 512  | Hex  |             |
> +   +-----------------+-----------------------+------+------+-------------+
> +   | ``WHIRLPOOL``   | Whirlpool [#BARRETO]_ | 512  | Hex  |             |
> +   +-----------------+-----------------------+------+------+-------------+
> +
> +Any new hashes must be added to this specification prior to being used
> +in Manifest files. Adding a new hash is considered
> +a backwards-compatible change to the GLEP. It is recommended that new
> +hashes are named after the Python ``hashlib`` module algorithm names,
> +transformed into uppercase, with dashes replaced by underscores.
> +
> +An implementation can implement an arbitrary subset of the listed
> +hashes. For best interoperability, it should implement at least
> +recommended hashes. If deprecated hashes are implemented, it is
> +preferable to disallow their use by default.
> +
> +If an entry specifies multiple hashes using different algorithms,
> +an implementation may choose to verify an arbitrary subset of them.
> +However, should any tested hash yield a mismatch, the verification must
> +fail.
> +
> +If a particular hash is either unsupported or unknown,
> +the implementation can either ignore it or report a failure. However,
> +at least one algorithm specified for a particular entry must be
> +supported for the verification to succeed.
>  
>  
>  Manifest compression
> @@ -913,23 +949,25 @@ tool working with this Manifest format.
>  Hash algorithms
>  ---------------
>  
> -While maintaining a consistent supported hash set is important
> -for interoperability, it is not a good fit for the generic layout
> -of this GLEP. Furthermore, it would require updating the GLEP
> -in the future every time the used algorithms change.
> +Originally, this GLEP did not formally specify the complete set of hash
> +algorithms. Instead, it only listed (informationally) the names already
> +used at the time of writing. Since enforcing consistent use of algorithm
> +names is important for interoperability, this was changed in version
> +1.3.
>  
> -Instead, the specification focuses on listing the currently used
> -algorithm names for interoperability, and sets a recommendation
> -for consistent naming of algorithms in the future. The Python
> -``hashlib`` module is used as a reference since it is used
> -as the provider of hash functions for most of the Python software,
> -including Portage and PkgCore.
> +Since the effort needed to update the GLEP is small compared to the time
> +needed for a new hash algorithm to be well-deployed, the GLEP needs
> +to be updated prior to adding a new hash method.
>  
> -The basic rules for changing hash algorithms are defined in GLEP 59
> -[#GLEP59]_. The implementations can focus only on those algorithms
> -that are actually used or planned on being used. It may be feasible
> -to devise a new GLEP that specifies the currently used hashes (or update
> -GLEP 59 accordingly).
> +The recommended naming is based off Python ``hashlib`` module,
> +as most of the Gentoo tooling is written in Python. The names
> +are transformed to match the historical naming used for hash functions
> +in Manifests.
> +
> +Implementations are allowed to support and use only a subset of hashes
> +listed in Manifest files. This could be used both to avoid the overhead
> +of computing multiple hashes on non-performant systems, and to handle
> +transition to new hashes gracefully.
>  
>  
>  Manifest compression
> @@ -1072,27 +1110,29 @@ References
>  .. [#FILE-NAMING-RULES] Ebuild File Format -- Gentoo Development Guide
>     (https://devmanual.gentoo.org/ebuild-writing/file-format/#file-naming-rules)
>  
> -.. [#MD5] RFC1321: The MD5 Message-Digest Algorithm
> -   (https://www.ietf.org/rfc/rfc1321.txt)
> +.. [#RFC7693] RFC 7693: The BLAKE2 Cryptographic Hash and Message Authentication
> +   Code (MAC)
> +   (https://www.rfc-editor.org/rfc/rfc7693)
> +
> +.. [#RFC1321] RFC 1321: The MD5 Message-Digest Algorithm
> +   (https://www.rfc-editor.org/rfc/rfc1321)
>  
> -.. [#RIPEMD160] The hash function RIPEMD-160
> +.. [#RMD160] The hash function RIPEMD-160
>     (https://homes.esat.kuleuven.be/~bosselae/ripemd160.html)
>  
>  .. [#SHS] FIPS PUB 180-4: Secure Hash Standard (SHS)
>     (http://nvlpubs.nist.gov/nistpubs/FIPS/NIST.FIPS.180-4.pdf)
>  
> -.. [#WHIRLPOOL] The WHIRLPOOL Hash Function
> -   (http://www.larc.usp.br/~pbarreto/WhirlpoolPage.html)
> -
> -.. [#BLAKE2] BLAKE2 -- fast secure hashing
> -   (https://blake2.net/)
> -
>  .. [#SHA3] FIPS PUB 202: SHA-3 Standard: Permutation-Based Hash
>     and Extendable-Output Functions
>     (http://nvlpubs.nist.gov/nistpubs/FIPS/NIST.FIPS.202.pdf)
>  
> -.. [#STREEBOG] GOST R 34.11-2012: Streebog Hash Function
> -   (https://www.streebog.net/)
> +.. [#RFC6986] RFC 6986: GOST R 34.11-2012: Hash Function
> +   (https://www.rfc-editor.org/rfc/rfc6986)
> +
> +.. [#BARRETO] Paulo S. L. M. Barreto, The WHIRLPOOL Hash Function
> +   (archived at 2017-11-29)
> +   (https://web.archive.org/web/20171129084214/http://www.larc.usp.br/~pbarreto/WhirlpoolPage.html)
>  
>  .. [#C08] Cappos, J et al. (2008). "Attacks on Package Managers"
>     (https://www2.cs.arizona.edu/stork/packagemanagersecurity/attacks-on-package-managers.html)

-- 
Best regards,
Michał Górny

[gentoo-dev] [PATCH v3 0/3] glep-0074: Explicitly specify hashes and compressed Manifest formats

[Michał Górny]

Hi,

Here's v3 of the patches.  The changes since v2 are:

- RIPEMD-160 is no longer marked as deprecated (as noted in [1]) since
  there's no proof of it broken at this moment.

- The links to SHA specifications were changed from http:// to https://.

The patches are being resent to have a permanent trail for the version
that the Council will be voting on.


[1] https://archives.gentoo.org/gentoo-dev/message/6ea964b8276ff9530883976a92d16dda

Michał Górny (3):
  glep-0074: Add a changelog
  glep-0074: Specify supported hash algorithms
  glep-0074: Specify compressed file formats

 glep-0074.rst | 271 ++++++++++++++++++++++++++++++++++++++------------
 1 file changed, 210 insertions(+), 61 deletions(-)

-- 
2.37.3

[gentoo-dev] [PATCH v3 1/3] glep-0074: Add a changelog

[Michał Górny]

Signed-off-by: Michał Górny <mgorny@gentoo.org>
---
 glep-0074.rst | 15 +++++++++++++--
 1 file changed, 13 insertions(+), 2 deletions(-)

diff --git a/glep-0074.rst b/glep-0074.rst
index 182a871..8cec618 100644
--- a/glep-0074.rst
+++ b/glep-0074.rst
@@ -8,8 +8,8 @@ Type: Standards Track
 Status: Final
 Version: 1.2
 Created: 2017-10-21
-Last-Modified: 2022-09-08
-Post-History: 2017-10-26, 2017-11-16, 2018-02-08, 2022-09-08
+Last-Modified: 2022-09-11
+Post-History: 2017-10-26, 2017-11-16, 2018-02-08, 2022-09-08, 2022-09-11
 Content-Type: text/x-rst
 Requires: 59, 61
 Replaces: 44, 58, 60
@@ -23,6 +23,17 @@ integrity and authenticity checks. The format aims to be future-proof,
 efficient and provide means of backwards compatibility.
 
 
+Changes
+=======
+
+v1.2
+  Specified the newline convention used for Manifests.
+
+v1.1
+  Removed the restriction that all files covered by a Manifest tree
+  must reside on the same filesystem.
+
+
 Motivation
 ==========
 
-- 
2.37.3

[gentoo-dev] [PATCH v3 2/3] glep-0074: Specify supported hash algorithms

[Michał Górny]

Replace the informational hash name section with a formal specification
of allowed hash algorithms.  The original reasoning for leaving them
implementation-defined was poor.  After all, not a single new hash
was added since the initial version of the GLEP.  At the same time,
ensuring consistent support for at least a minimal set of hash
algorithms is crucial to interoperability.  Given that the effort needed
to update the GLEP is relatively small, it is better to require all
algorithms to be formally listed than to have to track all
implementations for new hashes and hope for consistency.

Signed-off-by: Michał Górny <mgorny@gentoo.org>
---
 glep-0074.rst | 177 ++++++++++++++++++++++++++++++++++++--------------
 1 file changed, 127 insertions(+), 50 deletions(-)

diff --git a/glep-0074.rst b/glep-0074.rst
index 8cec618..5a63f70 100644
--- a/glep-0074.rst
+++ b/glep-0074.rst
@@ -6,7 +6,7 @@ Author: Michał Górny <mgorny@gentoo.org>,
         Ulrich Müller <ulm@gentoo.org>
 Type: Standards Track
 Status: Final
-Version: 1.2
+Version: 1.3
 Created: 2017-10-21
 Last-Modified: 2022-09-11
 Post-History: 2017-10-26, 2017-11-16, 2018-02-08, 2022-09-08, 2022-09-11
@@ -26,6 +26,9 @@ efficient and provide means of backwards compatibility.
 Changes
 =======
 
+v1.3
+  Formally specified the current set of hash algorithms supported.
+
 v1.2
   Specified the newline convention used for Manifests.
 
@@ -364,27 +367,60 @@ up to and including the *original* directory. Note that those
 sub-Manifests can use different filenames than ``Manifest``.
 
 
-Checksum algorithms (informational)
------------------------------------
-
-This section is informational only. Specifying the exact set
-of supported algorithms is outside the scope of this specification.
-
-The algorithm names reserved at the time of writing are:
-
-- ``MD5`` [#MD5]_,
-- ``RMD160`` -- RIPEMD-160 [#RIPEMD160]_,
-- ``SHA1`` [#SHS]_,
-- ``SHA256`` and ``SHA512`` -- SHA-2 family of hashes [#SHS]_,
-- ``WHIRLPOOL`` [#WHIRLPOOL]_,
-- ``BLAKE2B`` and ``BLAKE2S`` -- BLAKE2 family of hashes [#BLAKE2]_,
-- ``SHA3_256`` and ``SHA3_512`` -- SHA-3 family of hashes [#SHA3]_,
-- ``STREEBOG256`` and ``STREEBOG512`` -- Streebog family of hashes
-  [#STREEBOG]_.
-
-The method of introducing new hashes is defined by GLEP 59 [#GLEP59]_.
-It is recommended that any new hashes are named after the Python
-``hashlib`` module algorithm names, transformed into uppercase.
+Checksum algorithms
+-------------------
+
+.. table:: Table 1. Defined hash algorithms
+   :widths: auto
+
+   +-----------------+-----------------------+------+------+-------------+
+   | Name            | Specification         | Bits | Enc. | Notes       |
+   +=================+=======================+======+======+=============+
+   | ``BLAKE2B``     |                       | 512  | Hex  | Recommended |
+   +-----------------+ RFC 7693 [#RFC7693]_  +------+------+-------------+
+   | ``BLAKE2S``     |                       | 256  | Hex  |             |
+   +-----------------+-----------------------+------+------+-------------+
+   | ``MD5``         | RFC 1321 [#RFC1321]_  | 128  | Hex  | Deprecated  |
+   +-----------------+-----------------------+------+------+-------------+
+   | ``RMD160``      | RIPEMD-160 [#RMD160]_ | 160  | Hex  |             |
+   +-----------------+-----------------------+------+------+-------------+
+   | ``SHA1``        |                       | 160  | Hex  | Deprecated  |
+   +-----------------+                       +------+------+-------------+
+   | ``SHA256``      | FIPS 180-4 [#SHS]_    | 256  | Hex  |             |
+   +-----------------+                       +------+------+-------------+
+   | ``SHA512``      |                       | 512  | Hex  | Recommended |
+   +-----------------+-----------------------+------+------+-------------+
+   | ``SHA3_256``    |                       | 256  | Hex  |             |
+   +-----------------+ FIPS 202 [#SHA3]_     +------+------+-------------+
+   | ``SHA3_512``    |                       | 512  | Hex  |             |
+   +-----------------+-----------------------+------+------+-------------+
+   | ``STREEBOG256`` |                       | 256  | Hex  |             |
+   +-----------------+ RFC 6986 [#RFC6986]_  +------+------+-------------+
+   | ``STREEBOG512`` |                       | 512  | Hex  |             |
+   +-----------------+-----------------------+------+------+-------------+
+   | ``WHIRLPOOL``   | Whirlpool [#BARRETO]_ | 512  | Hex  |             |
+   +-----------------+-----------------------+------+------+-------------+
+
+Any new hashes must be added to this specification prior to being used
+in Manifest files. Adding a new hash is considered
+a backwards-compatible change to the GLEP. It is recommended that new
+hashes are named after the Python ``hashlib`` module algorithm names,
+transformed into uppercase, with dashes replaced by underscores.
+
+An implementation can implement an arbitrary subset of the listed
+hashes. For best interoperability, it should implement at least
+recommended hashes. If deprecated hashes are implemented, it is
+preferable to disallow their use by default.
+
+If an entry specifies multiple hashes using different algorithms,
+an implementation may choose to verify an arbitrary subset of them.
+However, should any tested hash yield a mismatch, the verification must
+fail.
+
+If a particular hash is either unsupported or unknown,
+the implementation can either ignore it or report a failure. However,
+at least one algorithm specified for a particular entry must be
+supported for the verification to succeed.
 
 
 Manifest compression
@@ -498,6 +534,43 @@ for a package directory would have the following content::
     DIST sphinxtrain-1.0.8.tar.gz 8925803 SHA256 548e.. SHA512 465d..
 
 
+Security considerations (informational)
+---------------------------------------
+
+The Manifest files are text files that are transmitted as part of larger
+file sets in order to provide integrity and authenticity verification
+for other files. They are primarily intended to be processed locally
+to verify transferred files. They are commonly used along with the rsync
+protocol and inside tar archives.
+
+The format does not provide support for executable content,
+nor the ability to issue network requests. Its security is primarily
+considered in context of opening and reading local files for the purpose
+of computing hashes.
+
+Depending on the delivery method, it may be possible to include special
+files and symbolic links in the verified file set. Attempting to read
+special files (e.g. named pipes or devices like ``/dev/urandom``) could
+cause the tools to hang or enter an infinite loop. The specification
+explicitly requires implementations to verify the file type and reject
+processing non-regular files.
+
+The use of symbolic links permits computing checksums for arbitrary
+paths, including files with potentially sensitive content and files
+on special filesystems such as the ``/proc`` filesystem. Reading these
+files should not comprise an immediate risk, nor displaying checksum
+mismatches to the local risk. However, there is a risk of exposing
+sensitive information if the user reports checksum failures.
+Implementations can take steps to reduce the risk, e.g. by minimalizing
+the amount of information reported on checksum mismatches and warning
+about symbolic links.
+
+
+
+Portability considerations (informational)
+------------------------------------------
+
+
 Rationale
 =========
 
@@ -913,23 +986,25 @@ tool working with this Manifest format.
 Hash algorithms
 ---------------
 
-While maintaining a consistent supported hash set is important
-for interoperability, it is not a good fit for the generic layout
-of this GLEP. Furthermore, it would require updating the GLEP
-in the future every time the used algorithms change.
+Originally, this GLEP did not formally specify the complete set of hash
+algorithms. Instead, it only listed (informationally) the names already
+used at the time of writing. Since enforcing consistent use of algorithm
+names is important for interoperability, this was changed in version
+1.3.
 
-Instead, the specification focuses on listing the currently used
-algorithm names for interoperability, and sets a recommendation
-for consistent naming of algorithms in the future. The Python
-``hashlib`` module is used as a reference since it is used
-as the provider of hash functions for most of the Python software,
-including Portage and PkgCore.
+Since the effort needed to update the GLEP is small compared to the time
+needed for a new hash algorithm to be well-deployed, the GLEP needs
+to be updated prior to adding a new hash method.
 
-The basic rules for changing hash algorithms are defined in GLEP 59
-[#GLEP59]_. The implementations can focus only on those algorithms
-that are actually used or planned on being used. It may be feasible
-to devise a new GLEP that specifies the currently used hashes (or update
-GLEP 59 accordingly).
+The recommended naming is based off Python ``hashlib`` module,
+as most of the Gentoo tooling is written in Python. The names
+are transformed to match the historical naming used for hash functions
+in Manifests.
+
+Implementations are allowed to support and use only a subset of hashes
+listed in Manifest files. This could be used both to avoid the overhead
+of computing multiple hashes on non-performant systems, and to handle
+transition to new hashes gracefully.
 
 
 Manifest compression
@@ -1072,27 +1147,29 @@ References
 .. [#FILE-NAMING-RULES] Ebuild File Format -- Gentoo Development Guide
    (https://devmanual.gentoo.org/ebuild-writing/file-format/#file-naming-rules)
 
-.. [#MD5] RFC1321: The MD5 Message-Digest Algorithm
-   (https://www.ietf.org/rfc/rfc1321.txt)
+.. [#RFC7693] RFC 7693: The BLAKE2 Cryptographic Hash and Message Authentication
+   Code (MAC)
+   (https://www.rfc-editor.org/rfc/rfc7693)
+
+.. [#RFC1321] RFC 1321: The MD5 Message-Digest Algorithm
+   (https://www.rfc-editor.org/rfc/rfc1321)
 
-.. [#RIPEMD160] The hash function RIPEMD-160
+.. [#RMD160] The hash function RIPEMD-160
    (https://homes.esat.kuleuven.be/~bosselae/ripemd160.html)
 
 .. [#SHS] FIPS PUB 180-4: Secure Hash Standard (SHS)
-   (http://nvlpubs.nist.gov/nistpubs/FIPS/NIST.FIPS.180-4.pdf)
-
-.. [#WHIRLPOOL] The WHIRLPOOL Hash Function
-   (http://www.larc.usp.br/~pbarreto/WhirlpoolPage.html)
-
-.. [#BLAKE2] BLAKE2 -- fast secure hashing
-   (https://blake2.net/)
+   (https://nvlpubs.nist.gov/nistpubs/FIPS/NIST.FIPS.180-4.pdf)
 
 .. [#SHA3] FIPS PUB 202: SHA-3 Standard: Permutation-Based Hash
    and Extendable-Output Functions
-   (http://nvlpubs.nist.gov/nistpubs/FIPS/NIST.FIPS.202.pdf)
+   (https:://nvlpubs.nist.gov/nistpubs/FIPS/NIST.FIPS.202.pdf)
+
+.. [#RFC6986] RFC 6986: GOST R 34.11-2012: Hash Function
+   (https://www.rfc-editor.org/rfc/rfc6986)
 
-.. [#STREEBOG] GOST R 34.11-2012: Streebog Hash Function
-   (https://www.streebog.net/)
+.. [#BARRETO] Paulo S. L. M. Barreto, The WHIRLPOOL Hash Function
+   (archived at 2017-11-29)
+   (https://web.archive.org/web/20171129084214/http://www.larc.usp.br/~pbarreto/WhirlpoolPage.html)
 
 .. [#C08] Cappos, J et al. (2008). "Attacks on Package Managers"
    (https://www2.cs.arizona.edu/stork/packagemanagersecurity/attacks-on-package-managers.html)
-- 
2.37.3

[gentoo-dev] [PATCH v3 3/3] glep-0074: Specify compressed file formats

[Michał Górny]

Signed-off-by: Michał Górny <mgorny@gentoo.org>
---
 glep-0074.rst | 81 ++++++++++++++++++++++++++++++++++++++++++++-------
 1 file changed, 71 insertions(+), 10 deletions(-)

diff --git a/glep-0074.rst b/glep-0074.rst
index 5a63f70..d96a3dd 100644
--- a/glep-0074.rst
+++ b/glep-0074.rst
@@ -27,7 +27,8 @@ Changes
 =======
 
 v1.3
-  Formally specified the current set of hash algorithms supported.
+  Formally specified the current set of hash algorithms and compressed
+  Manifest formats supported.
 
 v1.2
   Specified the newline convention used for Manifests.
@@ -432,9 +433,8 @@ compression and this specification.
 
 The compressed Manifest files are required to be suffixed for their
 compression algorithm. This suffix should be used to recognize
-the compression and decompress Manifests transparently. The exact list
-of algorithms and their corresponding suffixes are outside the scope
-of this specification.
+the compression and decompress Manifests transparently. The supported
+formats are specified in `compressed file formats`_ section.
 
 The top-level Manifest file must not be compressed. Since the OpenPGP
 signature covers the uncompressed text and is compressed itself,
@@ -455,6 +455,46 @@ uncompressed content and the specification is free to choose either
 of the files using the same base name.
 
 
+Compressed file formats
+-----------------------
+
+.. table:: Table 2. Defined compressed file formats
+   :widths: auto
+
+   ===========  ======  ====================  ===========
+   Tool name    Suffix  Specification         Notes
+   ===========  ======  ====================  ===========
+   bzip2        .bz2    (none known)
+   gzip         .gz     RFC 1952 [#RFC1952]_  Recommended
+   lz4          .lz4    (none known)
+   lzip         .lz     RFC draft [#LZIP]_
+   lzma         .lzma   (none known)          Deprecated
+   lzop         .lzo    (none known)
+   xz           .xz     xz [#XZ]_
+   zstd         .zst    RFC 8878 [#RFC8878]_
+   ===========  ======  ====================  ===========
+
+Any new formats must be added to this specification prior to being used
+for Manifest files. Adding a new compressed file format is considered
+a backwards-compatible change to the GLEP. It is recommended that new
+formats use their reference (most common) file suffixes.
+
+An implementation can implement an arbitrary subset of the listed
+formats. For best interoperability, it should implement at least
+the recommended formats. Using deprecated formats should be avoided.
+
+If multiple Manifest variants coexist using different compressed file
+formats, the implementation may choose to use an arbitrary subset
+of them. However, all of them must be verified against the hashes stored
+in the containing Manifest. Should they be decompressed, the resulting
+contents must be identical.
+
+If the compressed file format is unsupported and a variant using
+a supported format coexists, the other variant should be used. However,
+at least one supported variant must exist for the verification
+to succeed.
+
+
 Combining multiple Manifest trees (informational)
 -------------------------------------------------
 
@@ -1033,12 +1073,19 @@ into a compressed sub-Manifest in the top directory (e.g.
 ``Manifest.sub.gz``), and including a ``MANIFEST`` entry for this file
 in a signed, uncompressed top-level Manifest.
 
-The existence of additional entries for uncompressed Manifest checksums
-was debated. However, plain entries for the uncompressed file would
-be confusing if only the compressed file existed, and conflicting
-if both uncompressed and compressed variants existed. Furthermore,
-it has been pointed out that ``DIST`` entries do not have
-an uncompressed variant either.
+The existence of additional entries for checksums of Manifest contents
+after uncompressing was debated. However, plain entries for
+the uncompressed file would be confusing if only the compressed file
+existed. Furthermore, it has been pointed out that ``DIST`` entries
+do not have an uncompressed variant either.
+
+The specification permits coexistence of multiple variants of the same
+Manifest file using different compression for historical compatibility.
+However, there does not seem to be any real benefit from including
+a compressed Manifest file if the uncompressed variant needs to exist
+anyway. Providing different compressed variants could technically
+improve interoperability, though the same result could probably
+be achieved by using a more commonly supported format (e.g. gzip).
 
 
 Performance considerations
@@ -1171,6 +1218,20 @@ References
    (archived at 2017-11-29)
    (https://web.archive.org/web/20171129084214/http://www.larc.usp.br/~pbarreto/WhirlpoolPage.html)
 
+.. [#RFC1952] RFC 1952: GZIP file format specification version 4.3
+   (https://www.rfc-editor.org/rfc/rfc1952)
+
+.. [#LZIP] RFC draft: Lzip Compressed Format and the 'application/lzip'
+   Media Type
+   (https://datatracker.ietf.org/doc/html/draft-diaz-lzip)
+
+.. [#XZ] The .xz File Format
+   (https://tukaani.org/xz/xz-file-format.txt)
+
+.. [#RFC8878] RFC 8878: Zstandard Compression and the 'application/zstd'
+   Media Type
+   (https://www.rfc-editor.org/rfc/rfc8878)
+
 .. [#C08] Cappos, J et al. (2008). "Attacks on Package Managers"
    (https://www2.cs.arizona.edu/stork/packagemanagersecurity/attacks-on-package-managers.html)
 
-- 
2.37.3