From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from pigeon.gentoo.org ([208.92.234.80] helo=lists.gentoo.org) by finch.gentoo.org with esmtp (Exim 4.60) (envelope-from ) id 1NbWdD-0008Ka-Sg for garchives@archives.gentoo.org; Sun, 31 Jan 2010 10:02:16 +0000 Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id 0AB69E0983; Sun, 31 Jan 2010 10:02:05 +0000 (UTC) Received: from smtp.gentoo.org (smtp.gentoo.org [140.211.166.183]) by pigeon.gentoo.org (Postfix) with ESMTP id B574EE0983 for ; Sun, 31 Jan 2010 10:02:04 +0000 (UTC) Received: from mail.isohunt.com (b01.ext.isohunt.com [208.71.112.51]) by smtp.gentoo.org (Postfix) with ESMTP id 3434C1B4003 for ; Sun, 31 Jan 2010 10:02:04 +0000 (UTC) Received: (qmail 8787 invoked from network); 31 Jan 2010 10:02:02 -0000 Received: from tsi-static.orbis-terrarum.net (HELO grubbs.orbis-terrarum.net) (76.10.188.108) by mail.isohunt.com (qpsmtpd/0.33-dev on beta01) with (CAMELLIA256-SHA encrypted) ESMTPS; Sun, 31 Jan 2010 10:02:02 +0000 Received: (qmail 7813 invoked by uid 10000); 31 Jan 2010 10:01:58 -0000 Date: Sun, 31 Jan 2010 10:01:58 +0000 From: "Robin H. Johnson" To: gentoo-dev@lists.gentoo.org Subject: [gentoo-dev] GLEP60 - Manifest2 filetypes Message-ID: References: Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-dev@lists.gentoo.org Reply-to: gentoo-dev@lists.gentoo.org MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="NPukt5Otb9an/u20" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.20 (2009-06-14) X-Archives-Salt: 9f7db7c0-abdf-4a90-9657-01606c891241 X-Archives-Hash: 87bf429bbf673808f3a8aa4b28ea2034 --NPukt5Otb9an/u20 Content-Type: multipart/mixed; boundary="NGIwU0kFl1Z1A3An" Content-Disposition: inline --NGIwU0kFl1Z1A3An Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Changes: - This GLEP is able to stand independently of GLEP58. - 'UNKNOWN' type is now called 'OTHER', per ulm's suggestion. - Patches should be considered as type 'EXEC' - Type 'EXEC' may be used outside the files/ directory. (eg scripts, parts of profiles) - Logic for dealing with future unknown types. - Patterns are examples only, not exhaustive lists. --=20 Robin Hugh Johnson Gentoo Linux: Developer, Trustee & Infrastructure Lead E-Mail : robbat2@gentoo.org GnuPG FP : 11AC BA4F 4778 E3F6 E4ED F38E B27B 944E 3488 4E85 --NGIwU0kFl1Z1A3An Content-Type: text/plain; charset=us-ascii Content-Disposition: inline; filename="glep-0060.txt" Content-Transfer-Encoding: quoted-printable GLEP: 60 Title: Manifest2 filetypes Version: $Revision: 1.9 $ Last-Modified: $Date: 2010/01/31 09:55:43 $ Author: Robin Hugh Johnson =20 Status: Draft Type: Standards Track Content-Type: text/x-rst Requires: 44 Created: November 2007 Updated: June 2008, July 2008, October 2008, January 2010 Updates: 44 Post-History: December 2009, January 2010 Abstract =3D=3D=3D=3D=3D=3D=3D=3D Clarification of the Manifest2 [#GLEP44] specification, including new types= to help in the tree-signing specification. Motivation =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D [#GLEP44] was not entirely clear on the usage of filetype specifiers. This document serves to provide some of the internal logic used by Portage at the point of writing, as well as adding new types to cover the rest of the tree, for the purposes of tree-signing coverage. This GLEP is not mandatory for the tree-signing specification, but instead aims to clarify the usage of the Manifest2 filetype specifiers, and note which types signify files that are allowed to be missing from the tree (e.g. a user excluding a package or category). As such, it is also able to stand on it's own. Specification =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D General ------- For any given directory with a Manifest file, every file located in that directory, or a sub-directory must be listed in that Manifest file, unless stated otherwise in the following sections. The Manifest file must not contain an entry for itself. Excluded files -------------- When generating or validating a Manifest, or committing to a version control system, the package manager should endeavour to ignore files created by a version control system, backup files from text editors. A non-exhaustive list is suggested here: ``CVS/``, ``.svn/``, ``.bzr/``, ``.git/``, ``.hg/``, ``.#*``, ``*.rej``, ``*.orig``, ``*.bak``, ``*~``. Additionally, for a transitional Manifest1->Manifest2 system, old-style digest files located in a 'files/' directory, may be excluded from Manifest2 generation, or included with a type of MISC. Under strict security conditions, the exclusion list may be ignored during validation if the existence of a file would be considered a security risk. Existing filetypes: ------------------- AUX ~~~ - The AUX type is used for all items under the 'files' subdirectory.=20 - They should be verified relative to $FILESDIR.=20 - The string 'files/' is left out of the Manifest line.=20 - The absence of a file mentioned by AUX must be treated as an error.=20 - The AUX type is intended to denote potentially executable content (either directly or indirectly), that must be treated an error if modified or absent. EBUILD ~~~~~~ - The EBUILD type is used solely for files ending in .ebuild, or other suffixes as defined by the EAPI. - The files are located in the same directory as the Manifest file.=20 - The modification or absence of a file mentioned by EBUILD must be treated as an error. DIST ~~~~ - The DIST type is used for distfiles - They may be found directly via the $DISTDIR setting of the package manager.=20 - During simple verification of a Manifest, a missing DIST file should not be consider as a validation error (it is however a failure to fetch or unpack). MISC ~~~~ - The MISC type covers all remaining files in a directory.=20 - MISC is intended to mark all content that was not used in some way that directly affected execution of the package manager.=20 - This includes metadata.xml and ChangeLog entries, and any other purely informational content. - MISC entries where the file is missing may optionally be ignored as by non-strict package managers. - It should be possible to install a package while all MISC entries have been deleted from the tree. New filetypes: -------------- _INFO (new, abstract) ~~~~~~~~~~~~~~~~~~~~~ - This is the functionality of the old AUX, but does not include the implicit 'files/' prefix in the path, and is verified relative to the working directory instead of $FILESDIR. - The modification or absence of a file listed as a _INFO-derived type=20 is not an error unless the package manager is attempting to be strict. _CRIT (new, abstract) ~~~~~~~~~~~~~~~~~~~~~ - _CRIT is based off the _INFO type. - The modification or absence of a file listed as a _CRIT-derived type=20 MUST be treated as an error. EBUILD ~~~~~~ - Now derived from _CRIT. - Otherwise unchanged. DIST ~~~~ - Now derived from _CRIT. - Otherwise unchanged. MISC ~~~~ - Now derived from _INFO. - Otherwise unchanged. MANIFEST (new) ~~~~~~~~~~~~~~ - The MANIFEST type is explicitly to cover all nested Manifest files. - During validation, this serves as an indicator that the package manager may need to check subtree Manifest file.=20 - A missing MANIFEST file may be treated as a minor (e.g. excluding an entire category) or critical validation failure. - The failure should be considered as critical only if files that would be directly covered by this Manifest are missing. Deletion of a category-level Manifest while preserving the packages is forbidden. Deletion of an entire category is not. ECLASS (new) ~~~~~~~~~~~~ - uses _CRIT. - This type shall be used for all eclasses only. DATA (new) ~~~~~~~~~~ - uses _CRIT. - The DATA type shall be used for all files that directly affect the package manager, such as metadata/cache/* and profiles/. EXEC (new) ~~~~~~~~~~ - uses _CRIT. - If the file gets sourced, executed, or causes a change (patches) in how something is sourced or executed, it belongs in the EXEC filetype. - This filetype should be used for the scripts directories of a repository for important files. - This filetype is not limited to being used in the files/ subdirectory. OTHER (new) ~~~~~~~~~~~ - uses _CRIT. - All other files that are not covered by another type should be considered as 'OTHER'. - Any further new filetypes should be introduced to subtract files from the 'OTHER' set. - If a package manager runs into a unknown Manifest2 type, it should be treated as 'OTHER'. On Bloat -------- If repeated use of a common path prefix is considered a bloat problem, a Manifest file should be added inside the common directory, however this should not be done blindly, as bloat by inodes is more significant for the majority of use cases. See also [#GLEP58] on size reductions of Manifests. Chosing a filetype ------------------ 1. matches ``Manifest`` =3D> MANIFEST, stop. 2. matches ``*.ebuild`` =3D> EBUILD, stop. 3. matches ``*.eclass`` =3D> ECLASS, stop. 4. listed in SRC_URI=20 =3D> DIST, stop. 5. matches ``files/*`` =3D> AUX, continue [see note]. 6. matches any of ``*.sh``, ``*.bashrc``, ``*.patch``, ... =3D> EXEC, stop. 7. matches any of ``metadata/cache/*``, ``profiles/``, ``package.*``, ``use= =2Emask*``, ... =3D> DATA, stop. 8. matches any of ``ChangeLog``, ``metadata.xml``, ``*.desc``, ... =3D> MISC, stop. 9. not matched by any other rule=20 =3D> OTHER, stop. The logic behind 5, 6, 7 is ensuring that every item that by it's presence or absence may be dangerous should always be treated strictly. (Consider epatch given a directory of patches ``${FILESDIR}/${PV}/``, where it blindly includes them, or alternatively, the package.mask file or a profile being altered/missing). The above lists of file patterns are not intended to be exhaustive, but merely demonstrative. Note: The AUX entries should only be generated if we are generating a compatible Manifest that supports older versions of Portage. They should be generated along with the new type. Backwards Compatibility =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D For generation of existing package Manifests, the AUX entries must continue to be present for the standard Portage deprecation cycle. The new entries may be included already in all Manifest files, as they will be ignored by older Portage versions. Over time, ECLASS, DATA, EXEC, OTHER may replace the existing AUX type. The adoption of this proposal does also affect [#GLEP58] as part of this GLEP series, however this GLEP was an offset of the research in that GLEP. Thanks to =3D=3D=3D=3D=3D=3D=3D=3D=3D I'd like to thank the following people for input on this GLEP. - Marius Mauch (genone) & Zac Medico (zmedico): Portage Manifest2 References =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =2E. [#GLEP44] Mauch, M. (2005) GLEP44 - Manifest2 format. http://www.gentoo.org/proj/en/glep/glep-0044.html=09 Copyright =3D=3D=3D=3D=3D=3D=3D=3D=3D Copyright (c) 2007-2010 by Robin Hugh Johnson. This material may be distributed only subject to the terms and conditions set forth in the Open Publication License, v1.0. vim: tw=3D72 ts=3D2 expandtab: --NGIwU0kFl1Z1A3An-- --NPukt5Otb9an/u20 Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.14 (GNU/Linux) Comment: Robbat2 @ Orbis-Terrarum Networks - The text below is a digital signature. If it doesn't make any sense to you, ignore it. iEYEARECAAYFAktlVRYACgkQPpIsIjIzwiz4pACgz7u8inVFlmYxVyT8FDl7GoN7 MQQAoLiEuMp1IDvOWUiRzfWSprYh0AMk =V5ML -----END PGP SIGNATURE----- --NPukt5Otb9an/u20--