[gentoo-commits] data/glep:master commit in: /

["Michał Górny" <mgorny@gentoo.org>]

commit:     b00ade7b6467a3ae066d66f6e4ce71fb10309710
Author:     Michał Górny <mgorny <AT> gentoo <DOT> org>
AuthorDate: Wed Nov 22 11:40:34 2017 +0000
Commit:     Michał Górny <mgorny <AT> gentoo <DOT> org>
CommitDate: Sat Nov 25 20:49:17 2017 +0000
URL:        https://gitweb.gentoo.org/data/glep.git/commit/?id=b00ade7b

glep-0074: Provide encoding for disallowed characters

 glep-0074.rst | 75 ++++++++++++++++++++++++++++++++++++++++++++---------------
 1 file changed, 56 insertions(+), 19 deletions(-)

diff --git a/glep-0074.rst b/glep-0074.rst
index b0daa05..3dc6730 100644
--- a/glep-0074.rst
+++ b/glep-0074.rst
@@ -70,7 +70,8 @@ other space-separated values.
 
 Unless specified otherwise, the paths used in the Manifest files
 are relative to the directory containing the Manifest file. The paths
-must not reference the parent directory (``..``).
+must not reference the parent directory (``..``). Forward slash (``/``)
+is used as path component separator.
 
 The Manifest files use UTF-8 encoding.
 
@@ -132,13 +133,35 @@ are not otherwise ignored reside on a different filesystem, or symbolic
 links point to targets on a different filesystem, they must
 be explicitly excluded via ``IGNORE``.
 
-All paths specified in the Manifest file must consist of characters
+
+Path and filename encoding
+--------------------------
+
+The path fields in the Manifest file must consist of characters
 corresponding to valid UTF-8 code points excluding the NULL character
 (``U+0000``), the backwards slash (``\``) and characters classified
 as whitespace in the current version of the Unicode standard
-[#UNICODE]_. It is an error to use Manifest files in directories
-containing files whose names contain the disallowed characters.
-The forward slash (``/``) must be used as path separator.
+[#UNICODE]_.
+
+Any of the excluded characters that are present in path must be encoded
+using one of the following escape sequences:
+
+- characters in the ``U+0000`` to ``U+007F`` range can be encoded
+  as ``\xHH`` where ``HH`` specifies the zero-padded, hexadecimal
+  character code,
+
+- characters in the ``U+0000`` to ``U+FFFF`` range can be encoded
+  as ``\uHHHH`` where ``HHHH`` specifies the zero-padded, hexadecimal
+  character code,
+
+- characters in the UCS-4 range can be encoded as ``\UHHHHHHHH``
+  where ``HHHHHHHH`` specifies the zero-padded, hexadecimal character
+  code.
+
+It is invalid for backwards slash to be used in any other context,
+and a backwards slash present in filename must be encoded. Backwards
+slash used as path component separator should be replaced by forward
+slash instead.
 
 
 File verification
@@ -563,7 +586,7 @@ specification syntax [#PMS-FETCH]_ implicitly makes it impossible to use
 filenames containing whitespace.
 
 This specification aims to avoid arbitrary restrictions. For this
-reason, filename characters are only restricted by excluding two
+reason, filename characters are only restricted by excluding three
 technically problematic groups:
 
 1. The NULL character (``U+0000``) is normally used to indicate the end
@@ -571,12 +594,10 @@ technically problematic groups:
    written using C. Furthermore, it is not allowed in any known
    filesystem.
 
-2. The backwards slash character (``\``) is frequently used as an escape
-   character, in particular in the languages derived from C and in shell
-   script. Furthermore, it is used as path separator on Windows systems.
-   It is forbidden to avoid implementation mistakes (in particular,
-   attempting to use it to escape whitespace or as path separator
-   on Windows) but also reserved for possible future extension.
+2. The backwards slash character (``\``) is used as path separator
+   on Windows systems, so it's extremely unlikely to be used in real
+   filenames. For this reason it is used to implement character
+   encoding with minimal risk of breaking backwards compatibility.
 
 3. Whitespace characters are used to separate Manifest fields
    and entries. While technically it would be enough to restrict space
@@ -585,18 +606,34 @@ technically problematic groups:
    all whitespace characters are forbidden to avoid confusion
    and implementation errors.
 
-While the specification could be extended to allow such filenames
-by using some form of escaping, there is currently no apparent need
-for such a feature.
-
 Historically, Portage attempted to overcome the whitespace limitation
 by attempting to locate the size field and take everything before it
 as filename. This was terribly fragile and even if it worked, it would
 solve the problem only partially.
 
-Since the same restrictions apply to ``IGNORE`` rules, it is currently
-not possible to either list or ignore the file using whitespace
-characters. Therefore, the presence of such files is forbidden entirely.
+The character encoding method provides means to overcome the character
+restrictions to extend the tool usability beyond immediate Gentoo uses.
+The backslash escape form based on Python unicode strings is used
+since it can encode all characters within the Unicode range, the syntax
+is familiar to many programmers and the backwards slash character
+is extremely unlikely to appear in real filenames.
+
+Syntax is limited to the minimum necessary to implement the encoding.
+Shorthand forms (e.g. ``\t`` or ``\\``) are omitted to avoid unnecessary
+complexity, and to reduce the risk of shell users using backslash
+to escape space directly. The ``\x`` form is limited to ``\x00..\x7F``
+range to avoid ambiguity of higher values which might be interpreted
+either as UCS-2 code points or part of a UTF-8 encoded character.
+
+Encoding stores UCS-2/UCS-4 characters directly rather than hex-encoded
+UTF-8 string to simplify the implementation. In particular, it makes it
+possible to process the Manifest file as UTF-8 encoded text without
+having to perform additional UTF-8 decoding (and verification)
+of the escaped data.
+
+URL-encoding was considered as an alternative. However, it could collide
+with ``DIST`` entries that are implicitly named after the URL filename
+part where URL-encoding is pretty common.
 
 
 File verification model