From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80]) by finch.gentoo.org (Postfix) with ESMTP id 243AF138200 for ; Fri, 9 Aug 2013 15:28:06 +0000 (UTC) Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id 660A1E0A69; Fri, 9 Aug 2013 15:28:04 +0000 (UTC) Received: from smtp.gentoo.org (smtp.gentoo.org [140.211.166.183]) (using TLSv1 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by pigeon.gentoo.org (Postfix) with ESMTPS id CE196E0A69 for ; Fri, 9 Aug 2013 15:28:03 +0000 (UTC) Received: from hornbill.gentoo.org (hornbill.gentoo.org [94.100.119.163]) (using TLSv1 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by smtp.gentoo.org (Postfix) with ESMTPS id A758F33EBF2 for ; Fri, 9 Aug 2013 15:28:02 +0000 (UTC) Received: from localhost.localdomain (localhost [127.0.0.1]) by hornbill.gentoo.org (Postfix) with ESMTP id EFAC4E5469 for ; Fri, 9 Aug 2013 15:27:59 +0000 (UTC) From: "André Erdmann" To: gentoo-commits@lists.gentoo.org Content-Transfer-Encoding: 8bit Content-type: text/plain; charset=UTF-8 Reply-To: gentoo-dev@lists.gentoo.org, "André Erdmann" Message-ID: <1376061750.2f9215125b0044a799137f3fc2478a06620f5b5b.dywi@gentoo> Subject: [gentoo-commits] proj/R_overlay:master commit in: doc/rst/ X-VCS-Repository: proj/R_overlay X-VCS-Files: doc/rst/usage.rst X-VCS-Directories: doc/rst/ X-VCS-Committer: dywi X-VCS-Committer-Name: André Erdmann X-VCS-Revision: 2f9215125b0044a799137f3fc2478a06620f5b5b X-VCS-Branch: master Date: Fri, 9 Aug 2013 15:27:59 +0000 (UTC) Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-commits@lists.gentoo.org X-Archives-Salt: b1a1ed07-52ed-4b39-af96-8cabddce979a X-Archives-Hash: 88f6f4cb70c3163a1075d7d49d2b0950 commit: 2f9215125b0044a799137f3fc2478a06620f5b5b Author: André Erdmann mailerd de> AuthorDate: Fri Aug 9 15:22:30 2013 +0000 Commit: André Erdmann mailerd de> CommitDate: Fri Aug 9 15:22:30 2013 +0000 URL: http://git.overlays.gentoo.org/gitweb/?p=proj/R_overlay.git;a=commit;h=2f921512 doc/rst: implementation overview, stats config --- doc/rst/usage.rst | 145 +++++++++++++++++++++++++++++++++++++----------------- 1 file changed, 99 insertions(+), 46 deletions(-) diff --git a/doc/rst/usage.rst b/doc/rst/usage.rst index 88761e5..787c9aa 100644 --- a/doc/rst/usage.rst +++ b/doc/rst/usage.rst @@ -89,37 +89,43 @@ Expected prior knowlegde: Prerequisites --------------- -* python >= 2.7 (tested with python 2.7 and 3.2) +* python >= 2.7 built with ssl support (tested with python 2.7 and 3.2) -* argparse (http://code.google.com/p/argparse) + extra dependencies for python 2.7 (built-in since python 3.2) + + * argparse (http://code.google.com/p/argparse) + * concurrent.futures (http://pypi.python.org/pypi/futures) (optional) * setuptools (http://pypi.python.org/pypi/setuptools) * rsync (for using rsync repositories) -* for Manifest creation: +* portage (*ebuild*) for Manifest creation and downloading source files for + imported ebuilds - * portage (*ebuild*) - * *true* or *echo* from coreutils or busybox for preventing - package downloads during Manifest creation (optional) +* *true* or *echo* from coreutils or busybox for preventing + package downloads during Manifest creation (optional) * for generating documentation files: python docutils >= 0.9 * hardware requirements (when using the default configuration): disk - * 50-55GB disk space for the R packages - * a filesystem that supports symbolic or hard links - * there will be many small-sized files (ebuilds), - so a filesystem with lots of inodes and a small block size - may be advantageous + + * 50-55GB disk space for the R packages + * a filesystem that supports symbolic or hard links + * there will be many small-sized files (ebuilds), + so a filesystem with lots of inodes and a small block size + may be advantageous memory + up to 600MB which depends on the amount of processed packages and the write mechanism in use. The amount can be halved (approximately) when using a slower one. time + Expect 1h execution time for the first run, depending on computing and I/O speed. *roverlay* is able to work in incremental mode, thus making subsequent runs need a lot less time. @@ -402,10 +408,19 @@ Main Config See `Configuration Reference`_ for all main config options like log file rotation and assistance for writing new *dependency rules*. +Patching generated ebuilds / Importing ebuilds + See `Additions Directory`_. + +R_SUGGESTS + See `USE_EXPAND flag rename file`_, which describes how R_SUGGESTS + USE_EXPAND flags can be renamed. + Field Definition Refer to `Field Definition`_ in case you want to change *how* R packages are being read, e.g. if you want the 'Depents' information field (obviously - a typo) to be understood as 'Depends'. + a typo) to be understood as 'Depends'. Also see `License Map file`_ for + translating *license strings* into valid portage licenses. + ------------ Running it @@ -543,6 +558,10 @@ apply_rules This command implies the **sync** command unless the *--nosync* option is specified. +setupdirs + Creates all configured directories with proper permissions. + + ---------------------------- Providing a package mirror ---------------------------- @@ -582,15 +601,21 @@ An installation of roverlay includes some helper programs: true + .. Note:: + + *roverlay-sh* requires a valid config file. + + *roverlay-mkconfig* a script that creates a config file for roverlay -*name_is_todo--roverlay_creation_helper* - <> - Safely runs overlay creation <>. +*run-roverlay.sh* + A script that safely runs overlay creation and repoman afterwards. + Uses filesystem locks to ensure that it is run at most once at a time. Suitable for being run as cron job. - + This file is not part of the roverlay installation. It can be found + at ``files/scripts/run-roverlay.sh`` in the `roverlay git repo`_. =============================== @@ -603,71 +628,84 @@ An installation of roverlay includes some helper programs: These are the steps that *roverlay* performs: -1. **sync** - get R packages using various methods +#. **sync** - get R packages using various methods (rsync, http, local directory) -2. scan the R Overlay directory (if it exists) for valid ebuilds +#. scan the R Overlay directory (if it exists) for valid ebuilds -3. import ebuilds from the additions dir +#. import ebuilds from the additions dir -4. **add** - queue all R packages for ebuild creation +#. **add** - queue all R packages for ebuild creation * all repositories are asked to list their packages which are then added to a queue - * packages may be declined by the overlay creator if they already have - an ebuild - * packages may be declined or manipulated by package rules See also: `Package Rules`_ -5. **create** - process each package *p* from the package queue + * packages may be declined by the overlay creator if they already have + an ebuild + + The overlay is able to detect changes to the package file, e.g. when + upstream modified a package without renaming it. To realize this, + the package file's checksum (sha256) is compared with the ebuild's + entry in the `distmap`_. If they do match, then the package is declined + as it already exists. Otherwise, a revision bump is triggered. + +#. **create** - process each package *p* from the package queue (thread-able on a per package basis) - * read *p*'s DESCRIPTION file that contains information fields - like 'Depends', 'Description' and 'Suggests' + * read *p*'s DESCRIPTION file that contains information fields + like 'Depends', 'Description' and 'Suggests' + + * resolve *p*'s dependencies - * resolve *p*'s dependencies + * differentiate between *required* and *optional* dependencies + (for example, dependencies from the 'Depends' field are required, + while those from 'Suggests' are optional) - * differentiate between *required* and *optional* dependencies - (for example, dependencies from the 'Depends' field are required, - while those from 'Suggests' are optional) + * **immediately stop** processing *p* if a *required* dependency + cannot be resolved in which case a valid ebuild cannot be created - * **immediately stop** processing *p* if a *required* dependency - cannot be resolved in which case a valid ebuild cannot be created + * verify dependencies on packages found in the overlay, whether their + ebuilds already exist or not (*selfdep validation*) - * verify dependencies on packages found in the overlay, whether their - ebuilds already exist or not (*selfdep validation*) + * drop unsatisfiable *selfdeps* - See also: `Dependency Resolution`_ + * **stop** processing *p* if a *required selfdep* is missing - * create an ebuild for *p* by using the dependency resolution results - and a few information fields like 'Description' + See also: `Dependency Resolution`_ - * **done** with *p* - the overlay creator takes control over *p* - and may decide to write *p*'s ebuild now (or later) + * create an ebuild for *p* by using the dependency resolution results + and a few information fields like 'Description' -6. write the overlay + * **done** with *p* - the overlay creator takes control over *p* + and may decide to write *p*'s ebuild now (or later) - * write/update the profiles dir +#. write the overlay + + * write/update the profiles dir and metadata/layout.conf * *roverlay* respects manual changes to USE_EXPAND description files * write all ebuilds and apply patches to them - (supports threads on a per package basis) * write the *metadata.xml* files - (supports threads on a per package basis) * this uses the latest created ebuild available for a package * write the *Manifest* files - (does not support threads by default / supports threads on a per package - basis when using *portage* directly) * this uses all ebuilds availabe for a package + Most write operations support threading. + +#. call the *overlay_success* hook + +#. write the stats database file (optional) + + * call the *db_written* hook -------------------------------------------------------------- Expected Overlay Result / Structure of the generated overlay @@ -1085,7 +1123,7 @@ A local directory will never be modified. websync_pkglist_. --------- - distmap + Distmap --------- *roverlay* uses a text file to store information about files in the package @@ -2452,6 +2490,21 @@ CACHEDIR This option is **required**. +.. _STATS_DB_FILE: + +STATS_DB + Path to the stats database file. This has to be a round-robin database + file (rrdtool). *roverlay* creates it if necessary. + + Defaults to , which disable database writing. + +STATS_INTERVAL + Database update interval, which should be set to the expected timespan + between running overlay creation, in seconds. Only meaningful when + creating a new database file. + + Defaults to 7200 (2 hours). + .. _DISTFILES: DISTFILES