[gentoo-dev] [PATCH] glep-0074: Remove single filesystem limitation

[gentoo-dev] [PATCH] glep-0074: Remove single filesystem limitation

[Michał Górny]

Remove the limitation that all files covered by the Manifest must reside
on a single filesystem. This breaks valid uses of overlayfs without
providing any real advantage.

The removal is justified further in the updated rationale section.
---
 glep-0074.rst | 66 +++++++++++++++++++++++++++++++++++------------------------
 1 file changed, 39 insertions(+), 27 deletions(-)

RST:  https://dev.gentoo.org/~mgorny/tmp/glep-0074.rst
HTML: https://dev.gentoo.org/~mgorny/tmp/glep-0074.html

diff --git a/glep-0074.rst b/glep-0074.rst
index 3835247..2f8deb2 100644
--- a/glep-0074.rst
+++ b/glep-0074.rst
@@ -6,10 +6,10 @@ Author: Michał Górny <mgorny@gentoo.org>,
         Ulrich Müller <ulm@gentoo.org>
 Type: Standards Track
 Status: Final
-Version: 1
+Version: 1.1
 Created: 2017-10-21
-Last-Modified: 2017-12-16
-Post-History: 2017-10-26, 2017-11-16
+Last-Modified: 2018-02-08
+Post-History: 2017-10-26, 2017-11-16, 2018-02-08
 Content-Type: text/x-rst
 Requires: 59, 61
 Replaces: 44, 58, 60
@@ -126,13 +126,6 @@ a different file type. If the tree contain files of other types
 that are not otherwise ignored, they need to be covered by an explicit
 ``IGNORE``.
 
-All the local (non-``DIST``) files covered by a Manifest tree must
-reside on the same filesystem. It is an error to specify entries
-applying to files on another filesystem. If files or directories that
-are not otherwise ignored reside on a different filesystem, or symbolic
-links point to targets on a different filesystem, they must
-be explicitly excluded via ``IGNORE``.
-
 
 Path and filename encoding
 --------------------------
@@ -325,22 +318,18 @@ Algorithm for finding parent Manifests
 In order to find the top-level Manifest from the current directory
 the following algorithm can be used:
 
-1. Store the current directory as *original* and the device ID
-   of the containing filesystem (``st_dev``) as *startdev*,
-
-2. If the device ID of the containing filesystem (``st_dev``)
-   of the current directory is different than *startdev*, stop.
+1. Store the current directory as *original*,
 
-3. If the current directory contains a ``Manifest`` file:
+2. If the current directory contains a ``Manifest`` file:
 
    a. If an ``IGNORE`` entry in the ``Manifest`` file covers
       the *original* directory (or one of the parent directories), stop.
 
    b. Otherwise, store the current directory as *last_found*.
 
-4. If the current directory is the root system directory (``/``), stop.
+3. If the current directory is the root system directory (``/``), stop.
 
-5. Otherwise, enter the parent directory and jump to step 2.
+4. Otherwise, enter the parent directory and jump to step 2.
 
 Once the algorithm stops, *last_found* will contain the relevant
 top-level Manifest. If *last_found* is null, then the directory tree
@@ -594,16 +583,39 @@ additional files in it, skipping some of the categories from syncing.
 This configuration can extend beyond the limits of this GLEP,
 e.g. by allowing wildcards or regular expressions.
 
-The algorithm is restricted to work on a single filesystem. This is
-mostly relevant when scanning for top-level Manifest -- we do not want
-to cross filesystem boundaries then. However, to ensure consistent
-bidirectional behavior we need to also ban them when operating downwards
-the tree.
 
-The directories and files on different filesystems need to be ignored
-explicitly as implicitly skipping them would cause confusion.
-In particular, tools might then claim that a file does not exist when
-it clearly does because it was skipped due to filesystem boundaries.
+Cross-filesystem Manifests
+--------------------------
+
+The first version of this specification had an additional requirement
+that all files covered by the Manifest tree must reside on a single
+filesystem. This requirement has been removed in version 1.1 for
+the reasons outlined in this section.
+
+The original rationale stated that this restriction aims to prevent
+crossing filesystem boundaries in the top-level Manifest lookup
+algorithm. While that seemed a good idea at the time, there is no real
+reason to prevent that and this particular method worked correctly only
+if the files were placed in a dedicated filesystem.
+
+Worse than that, the original rationale did not anticipate the use
+of overlayfs which combines multiple filesystems while preserving their
+original metadata, including device and inode numbers. As a result,
+if the repository was checked out to an overlayfs, it was quite possible
+that different files had different device numbers, and the Manifest
+checks failed due to crossing filesystem boundaries.
+
+Given no clear solution to that and no good reason to reject use
+of overlayfs, the restriction was lifted.
+
+The only potential drawback of this is that the implementation may now
+follow maliciously placed symbolic links pointing outside the tree.
+If a regular file was replaced by such a symlink, the user could
+be tricked into reporting the verification failure with the report
+containing the checksums of the target file. However, for this to happen
+the client would have to use rsync with ``--links`` option but without
+``--safe-links`` which is neither the default behavior of rsync nor
+the default configuration used by Portage.
 
 
 Filename character set restriction
-- 
2.16.1

Re: [gentoo-dev] [PATCH] glep-0074: Remove single filesystem limitation

[M. J. Everitt]

[-- Attachment #1.1: Type: text/plain, Size: 3286 bytes --]

On 08/02/18 17:09, Michał Górny wrote:
> Remove the limitation that all files covered by the Manifest must reside
> on a single filesystem. This breaks valid uses of overlayfs without
> providing any real advantage.
>
> The removal is justified further in the updated rationale section.
> ---
>  glep-0074.rst | 66 +++++++++++++++++++++++++++++++++++------------------------
>  1 file changed, 39 insertions(+), 27 deletions(-)
>
> RST:  https://dev.gentoo.org/~mgorny/tmp/glep-0074.rst
> HTML: https://dev.gentoo.org/~mgorny/tmp/glep-0074.html
>
> diff --git a/glep-0074.rst b/glep-0074.rst
> index 3835247..2f8deb2 100644
> --- a/glep-0074.rst
> +++ b/glep-0074.rst
> @@ -6,10 +6,10 @@ Author: Michał Górny <mgorny@gentoo.org>,
>          Ulrich Müller <ulm@gentoo.org>
>  Type: Standards Track
>  Status: Final
> -Version: 1
> +Version: 1.1
>  Created: 2017-10-21
> -Last-Modified: 2017-12-16
> -Post-History: 2017-10-26, 2017-11-16
> +Last-Modified: 2018-02-08
> +Post-History: 2017-10-26, 2017-11-16, 2018-02-08
>  Content-Type: text/x-rst
>  Requires: 59, 61
>  Replaces: 44, 58, 60
> @@ -126,13 +126,6 @@ a different file type. If the tree contain files of other types
>  that are not otherwise ignored, they need to be covered by an explicit
>  ``IGNORE``.
>  
> -All the local (non-``DIST``) files covered by a Manifest tree must
> -reside on the same filesystem. It is an error to specify entries
> -applying to files on another filesystem. If files or directories that
> -are not otherwise ignored reside on a different filesystem, or symbolic
> -links point to targets on a different filesystem, they must
> -be explicitly excluded via ``IGNORE``.
> -
>  
>  Path and filename encoding
>  --------------------------
> @@ -325,22 +318,18 @@ Algorithm for finding parent Manifests
>  In order to find the top-level Manifest from the current directory
>  the following algorithm can be used:
>  
> -1. Store the current directory as *original* and the device ID
> -   of the containing filesystem (``st_dev``) as *startdev*,
> -
> -2. If the device ID of the containing filesystem (``st_dev``)
> -   of the current directory is different than *startdev*, stop.
> +1. Store the current directory as *original*,
>  
> -3. If the current directory contains a ``Manifest`` file:
> +2. If the current directory contains a ``Manifest`` file:
>  
>     a. If an ``IGNORE`` entry in the ``Manifest`` file covers
>        the *original* directory (or one of the parent directories), stop.
>  
>     b. Otherwise, store the current directory as *last_found*.
>  
> -4. If the current directory is the root system directory (``/``), stop.
> +3. If the current directory is the root system directory (``/``), stop.
>  
> -5. Otherwise, enter the parent directory and jump to step 2.
> +4. Otherwise, enter the parent directory and jump to step 2.
>  
>  Once the algorithm stops, *last_found* will contain the relevant
>  top-level Manifest. If *last_found* is null, then the directory tree
>
<snip>

Observation: does RST not support auto-numbering like other formats? It
would make renumbering lists like this much easier (and from programming
experience, less error-prone). If not, #FEATUREREQ ... :]


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [gentoo-dev] [PATCH] glep-0074: Remove single filesystem limitation

[Michał Górny]

W dniu czw, 08.02.2018 o godzinie 17∶25 +0000, użytkownik M. J. Everitt
napisał:
> On 08/02/18 17:09, Michał Górny wrote:
> > Remove the limitation that all files covered by the Manifest must reside
> > on a single filesystem. This breaks valid uses of overlayfs without
> > providing any real advantage.
> > 
> > The removal is justified further in the updated rationale section.
> > ---
> >  glep-0074.rst | 66 +++++++++++++++++++++++++++++++++++------------------------
> >  1 file changed, 39 insertions(+), 27 deletions(-)
> > 
> > RST:  https://dev.gentoo.org/~mgorny/tmp/glep-0074.rst
> > HTML: https://dev.gentoo.org/~mgorny/tmp/glep-0074.html
> > 
> > diff --git a/glep-0074.rst b/glep-0074.rst
> > index 3835247..2f8deb2 100644
> > --- a/glep-0074.rst
> > +++ b/glep-0074.rst
> > @@ -6,10 +6,10 @@ Author: Michał Górny <mgorny@gentoo.org>,
> >          Ulrich Müller <ulm@gentoo.org>
> >  Type: Standards Track
> >  Status: Final
> > -Version: 1
> > +Version: 1.1
> >  Created: 2017-10-21
> > -Last-Modified: 2017-12-16
> > -Post-History: 2017-10-26, 2017-11-16
> > +Last-Modified: 2018-02-08
> > +Post-History: 2017-10-26, 2017-11-16, 2018-02-08
> >  Content-Type: text/x-rst
> >  Requires: 59, 61
> >  Replaces: 44, 58, 60
> > @@ -126,13 +126,6 @@ a different file type. If the tree contain files of other types
> >  that are not otherwise ignored, they need to be covered by an explicit
> >  ``IGNORE``.
> >  
> > -All the local (non-``DIST``) files covered by a Manifest tree must
> > -reside on the same filesystem. It is an error to specify entries
> > -applying to files on another filesystem. If files or directories that
> > -are not otherwise ignored reside on a different filesystem, or symbolic
> > -links point to targets on a different filesystem, they must
> > -be explicitly excluded via ``IGNORE``.
> > -
> >  
> >  Path and filename encoding
> >  --------------------------
> > @@ -325,22 +318,18 @@ Algorithm for finding parent Manifests
> >  In order to find the top-level Manifest from the current directory
> >  the following algorithm can be used:
> >  
> > -1. Store the current directory as *original* and the device ID
> > -   of the containing filesystem (``st_dev``) as *startdev*,
> > -
> > -2. If the device ID of the containing filesystem (``st_dev``)
> > -   of the current directory is different than *startdev*, stop.
> > +1. Store the current directory as *original*,
> >  
> > -3. If the current directory contains a ``Manifest`` file:
> > +2. If the current directory contains a ``Manifest`` file:
> >  
> >     a. If an ``IGNORE`` entry in the ``Manifest`` file covers
> >        the *original* directory (or one of the parent directories), stop.
> >  
> >     b. Otherwise, store the current directory as *last_found*.
> >  
> > -4. If the current directory is the root system directory (``/``), stop.
> > +3. If the current directory is the root system directory (``/``), stop.
> >  
> > -5. Otherwise, enter the parent directory and jump to step 2.
> > +4. Otherwise, enter the parent directory and jump to step 2.
> >  
> >  Once the algorithm stops, *last_found* will contain the relevant
> >  top-level Manifest. If *last_found* is null, then the directory tree
> > 
> 
> <snip>
> 
> Observation: does RST not support auto-numbering like other formats? It
> would make renumbering lists like this much easier (and from programming
> experience, less error-prone). If not, #FEATUREREQ ... :]

The whole purpose of WYWIWYG is to write *proper* text and not some
magical syntax that magically gets transformed to auto-numbered list.
Except when the format doesn't get it right.

Also, this comment is entirely off-topic and not useful to the thread
at all. Please cease those pointless off-topics.

-- 
Best regards,
Michał Górny

Re: [gentoo-dev] [PATCH] glep-0074: Remove single filesystem limitation

[Robin H. Johnson]

On Thu, Feb 08, 2018 at 06:09:22PM +0100, Michał Górny wrote:
> Remove the limitation that all files covered by the Manifest must reside
> on a single filesystem. This breaks valid uses of overlayfs without
> providing any real advantage.
Replying for the record:

I approve this change, because I didn't realize how overlayfs could
cause files & directories to show up in a single place with different
device IDs. Keeping overlayfs working is more important than the slight
security concerns of traversing device boundaries.

-- 
Robin Hugh Johnson
Gentoo Linux: Dev, Infra Lead, Foundation Treasurer
E-Mail   : robbat2@gentoo.org
GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136