From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by finch.gentoo.org (Postfix) with ESMTPS id C092F1396D9 for ; Thu, 2 Nov 2017 23:43:18 +0000 (UTC) Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id 83D1BE0CB8; Thu, 2 Nov 2017 23:43:12 +0000 (UTC) Received: from smtp.gentoo.org (smtp.gentoo.org [140.211.166.183]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by pigeon.gentoo.org (Postfix) with ESMTPS id 2BA20E0C7A for ; Thu, 2 Nov 2017 23:43:12 +0000 (UTC) Received: from grubbs.orbis-terrarum.net (localhost [127.0.0.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-SHA (128/128 bits)) (No client certificate requested) by smtp.gentoo.org (Postfix) with ESMTPS id 3D446341747 for ; Thu, 2 Nov 2017 23:43:09 +0000 (UTC) Received: (qmail 29454 invoked by uid 10000); 2 Nov 2017 23:43:07 -0000 Date: Thu, 2 Nov 2017 23:43:07 +0000 From: "Robin H. Johnson" To: gentoo-dev@lists.gentoo.org Subject: Re: [gentoo-dev] [v1.0.3] GLEP 74: Full-tree verification using Manifest files Message-ID: References: <1509048745.18656.6.camel@gentoo.org> <1509649919.21210.12.camel@gentoo.org> Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-dev@lists.gentoo.org Reply-to: gentoo-dev@lists.gentoo.org MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="3LDjyuwMyYgFmzqN" Content-Disposition: inline In-Reply-To: <1509649919.21210.12.camel@gentoo.org> User-Agent: Mutt/1.8.2 (2017-04-18) X-Archives-Salt: 853a4e0c-c6f6-4776-a09f-7fe0436e9b9d X-Archives-Hash: 71f0e50e5b8ea597c142943cdef4a261 --3LDjyuwMyYgFmzqN Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, Nov 02, 2017 at 08:11:59PM +0100, Micha=C5=82 G=C3=B3rny wrote: > Next version. Now without MISC/OPTIONAL, and with many clarifications. Huge improvements in this version, I found it much easier to understand. Nits:=20 - please stick to ASCII ellipsis. The unicode ellipsis is unreadable in some monospace fonts. Further items inline: > Directory tree coverage > ----------------------- =2E.. > The file entries (except for ``IGNORE``) can be specified for regular > files only. Symbolic links are followed when opening files > and traversing directories. It is an error to specify an entry for > a different file type. If the tree contain files of other types > that are not otherwise ignored, they need to be covered by an explicit > ``IGNORE``. >=20 > All the local (non-``DIST``) files covered by a Manifest tree must > reside on the same filesystem. It is an error to specify entries > applying to files on another filesystem. If subdirectories > that are not otherwise ignored reside on a different filesystem, they > must be explicitly excluded via ``IGNORE``. I would prefer this to say: 'If files that are not otherwise ignored reside on a different filesystem', as expanded from sub-directories. =20 This implicitly forbids following a symlink that crosses a filesystem boundary, and then matches the similar part of 'Tree layout restrictions'. > Rationale > =3D=3D=3D=3D=3D=3D=3D=3D=3D =2E.. > Tree layout restrictions > ------------------------ >=20 > The algorithm is meant to work primarily with ebuild repositories which > normally contain only files and directories. Directories provide > no useful metadata for verification, and specifying special entries > for additional file types is purposeless. Therefore, the specification > is restricted to dealing with regular files. >=20 > The Gentoo repository does not use symbolic links. Some Gentoo > repositories do, however. To provide a simple solution for dealing with > symlinks without having to take care to implement special handling for > them, the common behavior of implicitly resolving them is used. > Therefore, symbolic links to files are stored as if they were regular > files, and symbolic links to directories are followed as if they were > regular directories. >=20 > Dotfiles are implicitly ignored as that is a common notion used > in software written for POSIX systems. All other common filenames > require explicit ``IGNORE`` lines. 'common' in the second sentence seems odd. What about uncommon filenames? Maybe just s/other common filenames/other filenames/. > An ability to inject additional ignore entries is provided to account > for site configuration affecting the repository tree =E2=80=94 placing > additional files in it, skipping some of the categories from syncing. Mention that the package manager may provide wildcards or regex in the additional entries. Eg: 'IGNORE **/metadata.xml'=20 > Non-strict Manifest verification > -------------------------------- =2E.. > The cases for stripping unnecessary files mostly focused around space > savings. For this purpose, stripping ``metadata.xml`` and similar files > has little value. It is much more common for users to strip whole > categories which can not be handled via the ``MISC`` type, and needs > a dedicated package manager mechanism. The same mechanism can also > handle files that used the ``MISC`` type. Exclusion by package does happen as well. A list of categories or packages can be used for both the rsync exclusion and the IGNORE. > Splitting distfile checksums from file checksums > ------------------------------------------------ >=20 > Another problem with the current Manifest format is that the checksums > for fetched files are combined with checksums for local files > in a single file inside the package directory. It has been specifically > pointed out that: >=20 > - since distfiles are sometimes reused across different packages, > the repeating checksums are redundant, Comment: 8.4% of all DIST entries are duplicate, representing a 2MiB saving in tree size (25MiB of DIST entries altogether). > - mirror admins were interested in the possibility of verifying all > the distfiles with a single tool. >=20 > This specification does not provide a clean solution to this problem. > It technically permits moving ``DIST`` entries to higher-level Manifests > but the usefulness of such a solution is doubtful. This solution would require the packager manager to consider higher-level Manifests or all Manifests in the tree when searching for the DIST entry. The most useful implementation of this would be for the git->rsync process to move all DIST entries elsewhere (metadata/ maybe). Either way, this would have many downsides, and make manual work on the Manifest DIST entries painful. --=20 Robin Hugh Johnson Gentoo Linux: Dev, Infra Lead, Foundation Asst. Treasurer E-Mail : robbat2@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136 --3LDjyuwMyYgFmzqN Content-Type: application/pgp-signature; name="signature.asc" Content-Description: Digital signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 Comment: Robbat2 @ Orbis-Terrarum Networks - The text below is a digital signature. If it doesn't make any sense to you, ignore it. iQKTBAEBCgB9FiEEveu2pS8Vb98xaNkRGTlfI8WIJsQFAln7rYpfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldEJE RUJCNkE1MkYxNTZGREYzMTY4RDkxMTE5Mzk1RjIzQzU4ODI2QzQACgkQGTlfI8WI JsQ9VQ/+I3VwnREI6jomnsiWJY9EigRnFpaASY5KZ3LJ4wUTpb6QrFWK/l700QLx It9bi3JToCa1H6R2EZPPYaCpd9+kSbG+9Kvn93dT2YiibdLR1Lr7LLx5QnvtIVO0 aXahKDrz3qjDDQRSc/OIE/XwitBtgdTwR4aAPCZRyFkHEVEysNpYV9NyJ161XIwG 6CCmwI2Ju7fHF/lsUcL6evq5U/ced1MoITPeZGpgR3rF5DqzWlK8HtsWKNeV4Y75 HUXgUfZYOeiiubySONsq3CfrK4Af5P12EgNKyQS5Nwv3GWV7mmpccNjw7DLVu+le UMmkbiBxlZq25iilyQFuhxo1+5C8r5I0/Ot6r+cQhgCAN5PCC4AvHsY4U4us6DVv fHPG0iFBB4c06e4yL13uSDdv62QSozaqi/ACFr+P3GpOsu2bzeviyNK2uVLRB8fj 9eZI5PIAFdrtrLxU3dQ/ej6NQI8iZAhlNw67JEUhTxfwHh52awd+oa6DLbpL0cFO PWjxpRVl/ikhAE4vn0BtzATQQiR+YVXAYzGAq9CUzJk+H+mvVLrMGkBhIPx88qDV vgcTWR1o/RspNVaMhwpzgr5K6B620QA5+og8Q9RYBGGunq0SbTUgHC4y5TIz8zWW DI8p4PHgpanF+vGnaTy4Zd5GM57qpZPYViFgtkifL0Vbd5E1QbQ= =b/k/ -----END PGP SIGNATURE----- --3LDjyuwMyYgFmzqN--