* [gentoo-commits] proj/R_overlay:master commit in: doc/rst/
@ 2014-07-18 16:20 André Erdmann
2014-07-18 14:55 ` [gentoo-commits] proj/R_overlay:wip/addition_control " André Erdmann
0 siblings, 1 reply; 41+ messages in thread
From: André Erdmann @ 2014-07-18 16:20 UTC (permalink / raw
To: gentoo-commits
commit: ddc44b19a1310813ccbf1a533a1884685bd27ce3
Author: André Erdmann <dywi <AT> mailerd <DOT> de>
AuthorDate: Fri Jul 18 14:53:53 2014 +0000
Commit: André Erdmann <dywi <AT> mailerd <DOT> de>
CommitDate: Fri Jul 18 14:53:53 2014 +0000
URL: http://git.overlays.gentoo.org/gitweb/?p=proj/R_overlay.git;a=commit;h=ddc44b19
doc/rst/usage.rst: document add-policy args
---
doc/rst/usage.rst | 45 +++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 45 insertions(+)
diff --git a/doc/rst/usage.rst b/doc/rst/usage.rst
index f841f36..c8ffff9 100644
--- a/doc/rst/usage.rst
+++ b/doc/rst/usage.rst
@@ -514,6 +514,51 @@ to know in detail what *roverlay* does before running it.
--no-incremental
Force recreation of existing ebuilds
+--package-revbump <package>
+ Enforces a revbump of ``<package>``.
+
+ ``<package>`` can be a package name (``${PN}``), a package name with
+ version (``${P}`` or ``${PF}``), optionally prefixed with its category
+ (``${CATEGORY}/${PN}``, ``${CATEGORY}/${PF}`` etc.).
+ Wildcard characters (``*``, ``?``) are supported, too.
+
+ Incremental overlay creation only.
+ A revbump only occurs if the package exists in the overlay and is queued
+ for overlay creation.
+
+--package-force-replace <package>
+ Enforces a recreation of ``<package>``.
+
+ See ``--package-revbump``.
+
+--package-replace-only <package>
+ Forbids ebuild creation for ``<package>`` if it does not exist in the
+ overlay already.
+
+ Useful in conjunction with ``--package-revbump``, ``--package-replace``,
+ ``--package-list`` or ``--ebuild-list``:
+
+ .. code:: text
+
+ roverlay --package-revbump "sci-R/*" --package-replace-only "*"
+
+
+--package-list <file>
+ A file that lists packages that should be revbumped.
+
+ Same as specifying ``--package-replace-only <pkg> --package-revbump <pkg>``
+ for each line in the file.
+
+ The file can also be in extended format. (TODO: NOT DOCUMENTED)
+
+
+--ebuild-list <file>
+ Same as ``--package-list``, but expects a file that lists ebuild files.
+
+ Useful in combination with ``grep -rl <expr> <overlay>`` scripts.
+
+ The file can also be in extended format. (TODO: NOT DOCUMENTED)
+
--no-revbump
Disable revbump checks in incremental overlay creation mode
^ permalink raw reply related [flat|nested] 41+ messages in thread* [gentoo-commits] proj/R_overlay:wip/addition_control commit in: doc/rst/
2014-07-18 16:20 [gentoo-commits] proj/R_overlay:master commit in: doc/rst/ André Erdmann
@ 2014-07-18 14:55 ` André Erdmann
0 siblings, 0 replies; 41+ messages in thread
From: André Erdmann @ 2014-07-18 14:55 UTC (permalink / raw
To: gentoo-commits
commit: ddc44b19a1310813ccbf1a533a1884685bd27ce3
Author: André Erdmann <dywi <AT> mailerd <DOT> de>
AuthorDate: Fri Jul 18 14:53:53 2014 +0000
Commit: André Erdmann <dywi <AT> mailerd <DOT> de>
CommitDate: Fri Jul 18 14:53:53 2014 +0000
URL: http://git.overlays.gentoo.org/gitweb/?p=proj/R_overlay.git;a=commit;h=ddc44b19
doc/rst/usage.rst: document add-policy args
---
doc/rst/usage.rst | 45 +++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 45 insertions(+)
diff --git a/doc/rst/usage.rst b/doc/rst/usage.rst
index f841f36..c8ffff9 100644
--- a/doc/rst/usage.rst
+++ b/doc/rst/usage.rst
@@ -514,6 +514,51 @@ to know in detail what *roverlay* does before running it.
--no-incremental
Force recreation of existing ebuilds
+--package-revbump <package>
+ Enforces a revbump of ``<package>``.
+
+ ``<package>`` can be a package name (``${PN}``), a package name with
+ version (``${P}`` or ``${PF}``), optionally prefixed with its category
+ (``${CATEGORY}/${PN}``, ``${CATEGORY}/${PF}`` etc.).
+ Wildcard characters (``*``, ``?``) are supported, too.
+
+ Incremental overlay creation only.
+ A revbump only occurs if the package exists in the overlay and is queued
+ for overlay creation.
+
+--package-force-replace <package>
+ Enforces a recreation of ``<package>``.
+
+ See ``--package-revbump``.
+
+--package-replace-only <package>
+ Forbids ebuild creation for ``<package>`` if it does not exist in the
+ overlay already.
+
+ Useful in conjunction with ``--package-revbump``, ``--package-replace``,
+ ``--package-list`` or ``--ebuild-list``:
+
+ .. code:: text
+
+ roverlay --package-revbump "sci-R/*" --package-replace-only "*"
+
+
+--package-list <file>
+ A file that lists packages that should be revbumped.
+
+ Same as specifying ``--package-replace-only <pkg> --package-revbump <pkg>``
+ for each line in the file.
+
+ The file can also be in extended format. (TODO: NOT DOCUMENTED)
+
+
+--ebuild-list <file>
+ Same as ``--package-list``, but expects a file that lists ebuild files.
+
+ Useful in combination with ``grep -rl <expr> <overlay>`` scripts.
+
+ The file can also be in extended format. (TODO: NOT DOCUMENTED)
+
--no-revbump
Disable revbump checks in incremental overlay creation mode
^ permalink raw reply related [flat|nested] 41+ messages in thread
* [gentoo-commits] proj/R_overlay:master commit in: doc/rst/
@ 2014-08-23 19:03 André Erdmann
0 siblings, 0 replies; 41+ messages in thread
From: André Erdmann @ 2014-08-23 19:03 UTC (permalink / raw
To: gentoo-commits
commit: cf093636b7458403591634c74e043d95bbdeec03
Author: André Erdmann <dywi <AT> mailerd <DOT> de>
AuthorDate: Sat Aug 23 19:00:13 2014 +0000
Commit: André Erdmann <dywi <AT> mailerd <DOT> de>
CommitDate: Sat Aug 23 19:00:13 2014 +0000
URL: http://sources.gentoo.org/gitweb/?p=proj/R_overlay.git;a=commit;h=cf093636
doc/rst: fix slot rule examples
---
doc/rst/usage.rst | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/doc/rst/usage.rst b/doc/rst/usage.rst
index 1f6dbb2..c6046c0 100644
--- a/doc/rst/usage.rst
+++ b/doc/rst/usage.rst
@@ -1535,7 +1535,7 @@ Example 5 - fuzzy slot rule
.. code-block:: text
- ~sci-libs/fftw:s=0..1 :: fftw
+ ~sci-libs/fftw:wide_match:s=0..1 :: fftw
Example 6 - slot-restricted fuzzy slot rule
Similar to example 5, but this rule does not resolve anything unless the
@@ -1543,7 +1543,7 @@ Example 6 - slot-restricted fuzzy slot rule
.. code-block:: text
- ~sci-libs/fftw:s=0..1:restrict=2.1,3.0: :: fftw
+ ~sci-libs/fftw:wide_match:s=0..1:restrict=2.1,3.0: :: fftw
Example 7 - slot-restricted fuzzy slot rule with *immediate* value
Example 6 is not quite correct, as sci-libs/fftw currently uses slot 3.0
@@ -1552,7 +1552,7 @@ Example 7 - slot-restricted fuzzy slot rule with *immediate* value
.. code-block:: text
- ~sci-libs/fftw:s=i3.0:restrict=3.0,3.1,3.2,3.3 :: fftw
+ ~sci-libs/fftw:wide_match:s=i3.0:restrict=3.0,3.1,3.2,3.3 :: fftw
Please see the default rule files for more extensive examples that cover
^ permalink raw reply related [flat|nested] 41+ messages in thread* [gentoo-commits] proj/R_overlay:master commit in: doc/rst/
@ 2014-07-29 18:29 André Erdmann
2014-08-23 19:03 ` André Erdmann
0 siblings, 1 reply; 41+ messages in thread
From: André Erdmann @ 2014-07-29 18:29 UTC (permalink / raw
To: gentoo-commits
commit: 49009d462b003ccfd5c534680db4ab1e07c50e4f
Author: André Erdmann <dywi <AT> mailerd <DOT> de>
AuthorDate: Tue Jul 29 17:49:49 2014 +0000
Commit: André Erdmann <dywi <AT> mailerd <DOT> de>
CommitDate: Tue Jul 29 17:49:49 2014 +0000
URL: http://git.overlays.gentoo.org/gitweb/?p=proj/R_overlay.git;a=commit;h=49009d46
doc/rst: document slot deprule changes
---
doc/rst/usage.rst | 6 ++++++
1 file changed, 6 insertions(+)
diff --git a/doc/rst/usage.rst b/doc/rst/usage.rst
index c8ffff9..1f6dbb2 100644
--- a/doc/rst/usage.rst
+++ b/doc/rst/usage.rst
@@ -1680,6 +1680,12 @@ Fuzzy Slot Rules
* ``with_version`` or ``+v``: include version, too (``=<cat>/<pkg>-<pkgver>:<SLOT>``)
* ``open``: non-versioned slot (``<cat>/<pkg>:*`` or ``<cat>/<pkg>:=``)
+ * match behavior
+
+ By default, slot rules match ``<= <version>`` and ``= <version>``
+ statements only.
+ The ``wide_match`` option allows to match ``>= <version>``, too.
+
* accepted *calculated* slot values can be restricted with
``restrict=<list of accepted values`` or ``r=<list>``
* relevant slot parts can be set with ``slotparts=<selection>`` or
^ permalink raw reply related [flat|nested] 41+ messages in thread* [gentoo-commits] proj/R_overlay:master commit in: doc/rst/
2014-07-29 18:29 André Erdmann
@ 2014-08-23 19:03 ` André Erdmann
0 siblings, 0 replies; 41+ messages in thread
From: André Erdmann @ 2014-08-23 19:03 UTC (permalink / raw
To: gentoo-commits
commit: 49009d462b003ccfd5c534680db4ab1e07c50e4f
Author: André Erdmann <dywi <AT> mailerd <DOT> de>
AuthorDate: Tue Jul 29 17:49:49 2014 +0000
Commit: André Erdmann <dywi <AT> mailerd <DOT> de>
CommitDate: Tue Jul 29 17:49:49 2014 +0000
URL: http://sources.gentoo.org/gitweb/?p=proj/R_overlay.git;a=commit;h=49009d46
doc/rst: document slot deprule changes
---
doc/rst/usage.rst | 6 ++++++
1 file changed, 6 insertions(+)
diff --git a/doc/rst/usage.rst b/doc/rst/usage.rst
index c8ffff9..1f6dbb2 100644
--- a/doc/rst/usage.rst
+++ b/doc/rst/usage.rst
@@ -1680,6 +1680,12 @@ Fuzzy Slot Rules
* ``with_version`` or ``+v``: include version, too (``=<cat>/<pkg>-<pkgver>:<SLOT>``)
* ``open``: non-versioned slot (``<cat>/<pkg>:*`` or ``<cat>/<pkg>:=``)
+ * match behavior
+
+ By default, slot rules match ``<= <version>`` and ``= <version>``
+ statements only.
+ The ``wide_match`` option allows to match ``>= <version>``, too.
+
* accepted *calculated* slot values can be restricted with
``restrict=<list of accepted values`` or ``r=<list>``
* relevant slot parts can be set with ``slotparts=<selection>`` or
^ permalink raw reply related [flat|nested] 41+ messages in thread
* [gentoo-commits] proj/R_overlay:master commit in: doc/rst/
@ 2014-06-05 22:09 André Erdmann
0 siblings, 0 replies; 41+ messages in thread
From: André Erdmann @ 2014-06-05 22:09 UTC (permalink / raw
To: gentoo-commits
commit: 1c50c6deb1f14af0b48c9f7a9a3b4c856e9dd404
Author: André Erdmann <dywi <AT> mailerd <DOT> de>
AuthorDate: Wed May 7 01:29:52 2014 +0000
Commit: André Erdmann <dywi <AT> mailerd <DOT> de>
CommitDate: Wed May 7 01:29:52 2014 +0000
URL: http://git.overlays.gentoo.org/gitweb/?p=proj/R_overlay.git;a=commit;h=1c50c6de
doc/rst/usage: typo
---
doc/rst/usage.rst | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/doc/rst/usage.rst b/doc/rst/usage.rst
index f27e933..6d2879e 100644
--- a/doc/rst/usage.rst
+++ b/doc/rst/usage.rst
@@ -4240,7 +4240,7 @@ that tries to create an ebuild for it.
An *EbuildCreationJob* does the following steps:
-#. Read the DESCRIPTION file of *p* R package tarball and stores the
+#. Read the DESCRIPTION file of *p*'s R package tarball and store the
data in an associative array ('DESCRIPTION field' -> 'data')
#. Call `dependency resolution`_
^ permalink raw reply related [flat|nested] 41+ messages in thread* [gentoo-commits] proj/R_overlay:master commit in: doc/rst/
@ 2013-11-14 18:24 André Erdmann
0 siblings, 0 replies; 41+ messages in thread
From: André Erdmann @ 2013-11-14 18:24 UTC (permalink / raw
To: gentoo-commits
commit: 22601ebdc08ac1692779bcb3432e0d08234df004
Author: André Erdmann <dywi <AT> mailerd <DOT> de>
AuthorDate: Thu Nov 14 17:57:12 2013 +0000
Commit: André Erdmann <dywi <AT> mailerd <DOT> de>
CommitDate: Thu Nov 14 17:57:12 2013 +0000
URL: http://git.overlays.gentoo.org/gitweb/?p=proj/R_overlay.git;a=commit;h=22601ebd
doc/rst: hook env, EVENT_HOOK_RC config entry
---
doc/rst/usage.rst | 20 ++++++++++++++++++++
1 file changed, 20 insertions(+)
diff --git a/doc/rst/usage.rst b/doc/rst/usage.rst
index c341d42..15aa3c1 100644
--- a/doc/rst/usage.rst
+++ b/doc/rst/usage.rst
@@ -2337,6 +2337,11 @@ the config file. An empty string sets the policy to *deny all*.
+----------------+-------------------+-----------------------------------------+
| HOME | os.environ | |
+----------------+-------------------+-----------------------------------------+
+ | LANG | os.environ | |
+ +----------------+-------------------+-----------------------------------------+
+ | *LC_\** | os.environ | all environment variables starting |
+ | | | with LC\_ |
+ +----------------+-------------------+-----------------------------------------+
| ROVERLAY_PHASE | event | event that caused the script to run |
+----------------+-------------------+-----------------------------------------+
| HAS_CHANGES | *internal* | a shbool (``y`` or ``n``) that |
@@ -2390,6 +2395,9 @@ the config file. An empty string sets the policy to *deny all*.
| | *$ROVERLAY\ | executable (which creates the overlay) |
| | _HELPER_EXE* | |
+----------------+-------------------+-----------------------------------------+
+ | ROVERLAY\ | config | hook config file (`EVENT_HOOK_RC`_, |
+ | _HOOKRC | | optional) |
+ +----------------+-------------------+-----------------------------------------+
| STATS_DB | config | stats database file |
| | | (optional, only set if configured) |
+----------------+-------------------+-----------------------------------------+
@@ -3101,6 +3109,13 @@ EVENT_HOOK
Defaults to <libexec dir>/hooks/mux.sh if roverlay has been installed
and ADDITIONS_DIR_/hooks/mux.sh otherwise.
+.. _EVENT_HOOK_RC:
+
+EVENT_HOOK_RC
+ Config file for hook scripts (in shell script syntax).
+
+ Defaults to <not set>.
+
.. _EVENT_HOOK_RESTRICT:
EVENT_HOOK_RESTRICT
@@ -3128,6 +3143,11 @@ FILTER_SHELL_ENV
HOOK
Alias to EVENT_HOOK_.
+.. _HOOK_RC:
+
+HOOK_RC
+ Alias to EVENT_HOOK_RC_.
+
.. _HOOK_RESTRICT:
HOOK_RESTRICT
^ permalink raw reply related [flat|nested] 41+ messages in thread* [gentoo-commits] proj/R_overlay:master commit in: doc/rst/
@ 2013-09-23 15:30 André Erdmann
0 siblings, 0 replies; 41+ messages in thread
From: André Erdmann @ 2013-09-23 15:30 UTC (permalink / raw
To: gentoo-commits
commit: 487b3f0a42ff6681cf23fb8d8abe121dc7996cc1
Author: André Erdmann <dywi <AT> mailerd <DOT> de>
AuthorDate: Mon Sep 23 13:26:51 2013 +0000
Commit: André Erdmann <dywi <AT> mailerd <DOT> de>
CommitDate: Mon Sep 23 13:26:51 2013 +0000
URL: http://git.overlays.gentoo.org/gitweb/?p=proj/R_overlay.git;a=commit;h=487b3f0a
doc/rst: roverlay-status script
---
doc/rst/usage.rst | 59 ++++++++++++++++++++++++++++++++++++++++++++++---------
1 file changed, 50 insertions(+), 9 deletions(-)
diff --git a/doc/rst/usage.rst b/doc/rst/usage.rst
index 56c91bb..803be6d 100644
--- a/doc/rst/usage.rst
+++ b/doc/rst/usage.rst
@@ -6,6 +6,9 @@
.. _git repository: `roverlay git repo`_
+.. _mako templates:
+ http://www.makotemplates.org/
+
.. _omegahat's PACKAGES file:
http://www.omegahat.org/R/src/contrib/PACKAGES
@@ -623,7 +626,8 @@ OVERLAY_DISTDIR_FLAT_.
roverlay helpers
------------------
-An installation of roverlay includes some helper programs:
+An installation of roverlay includes some helper programs, most of them
+can also be found in ``<R overlay src directory>/bin/``:
*roverlay-sh*
a wrapper around /bin/sh that runs a shell or shell script in roverlay's
@@ -666,6 +670,31 @@ An installation of roverlay includes some helper programs:
* ``roverlay-setup hooks del <hook> <event|"all"> [<event|"all">...]``
removes a hook (referenced by name) from one or more (or all) *events*
+
+*roverlay-status*
+ A script that generates status reports based on `mako templates`_
+ and roverlay's stats (`STATS_DB_FILE`_). Supported output formats include
+ html, cgi (html with cgi header) and plain text.
+
+ Notable options accepted by *roverlay-status*:
+
+ --output <file|"-">, -O <file|"-">
+ output file/stream, defaults to stdout ("-")
+
+ --template <file|name>, -t <file|name>
+ name of or path to the mako template file. A ``--mode`` specific
+ file extension is automatically appended when searching for the template
+ file.
+
+ --mode <"cli"|"cgi"|"html">
+ Sets the output format. Defaults to *cli* (plain text).
+
+ --config <file>, -c <file>
+ Path to roverlay's main config file.
+
+ See ``roverlay-status --help`` for a full listing.
+
+
*run-roverlay.sh*
A script that safely runs overlay creation and repoman afterwards.
Uses filesystem locks to ensure that it is run at most once at a time.
@@ -2654,17 +2683,29 @@ CACHEDIR
.. _STATS_DB_FILE:
STATS_DB
- Path to the stats database file. This has to be a round-robin database
- file (rrdtool). *roverlay* creates it if necessary.
+ Path to the stats database file. *roverlay* creates it if necessary.
+
+ Defaults to <not set>, which disables database writing.
+
+.. _TEMPLATE_ROOT:
+
+TEMPLATE_ROOT
+ List of directories with templates for generating status reports.
+
+ */usr/share/roverlay/mako_templates* is automatically added to this list
+ if roverlay has been installed.
+
+ This option is **required** for *roverlay-status*, but doesn't need to be
+ set if roverlay has been installed.
+
+ Defaults to <not set>.
- Defaults to <not set>, which disable database writing.
+.. _TEMPLATE_MODULE_DIR:
-STATS_INTERVAL
- Database update interval, which should be set to the expected timespan
- between running overlay creation, in seconds. Only meaningful when
- creating a new database file.
+TEMPLATE_MODULE_DIR
+ Directory for caching mako templates.
- Defaults to 7200 (2 hours).
+ Defaults to CACHEDIR_/mako_templates.
.. _DISTFILES:
^ permalink raw reply related [flat|nested] 41+ messages in thread* [gentoo-commits] proj/R_overlay:master commit in: doc/rst/
@ 2013-09-23 12:34 André Erdmann
0 siblings, 0 replies; 41+ messages in thread
From: André Erdmann @ 2013-09-23 12:34 UTC (permalink / raw
To: gentoo-commits
commit: bbba27be6af13ac4fc0b37c5452e310258b1722f
Author: André Erdmann <dywi <AT> mailerd <DOT> de>
AuthorDate: Mon Sep 23 12:34:12 2013 +0000
Commit: André Erdmann <dywi <AT> mailerd <DOT> de>
CommitDate: Mon Sep 23 12:34:12 2013 +0000
URL: http://git.overlays.gentoo.org/gitweb/?p=proj/R_overlay.git;a=commit;h=bbba27be
doc/rst/usage.rst: roverlay-setup[-interactive]
---
doc/rst/usage.rst | 38 ++++++++++++++++++++++----------------
1 file changed, 22 insertions(+), 16 deletions(-)
diff --git a/doc/rst/usage.rst b/doc/rst/usage.rst
index b42ac47..56c91bb 100644
--- a/doc/rst/usage.rst
+++ b/doc/rst/usage.rst
@@ -263,23 +263,14 @@ It accepts a number of options, in particular:
See ``roverlay-setup --help`` for a full listing.
-
-
-\*\*\*\*\*\*\*\*\*\*\*\* NOT VALID BEGIN \*\*\*\*\*\*\*\*\*\*\*\*
-
-[interactive setup -- will be re-added]
-
-If roverlay has been installed, then ``emerge --config roverlay`` can be
-used to set up the config file as well as to create essential directories.
-It can be run multiple times in order to configure roverlay for more than
-one user.
+``roverlay-setup-interactive`` (``bin/roverlay-setup-interactive``) is a
+script that asks a few questions and then calls ``roverlay-setup`` based on
+the answers.
+It's also accessible via ``emerge --config roverlay`` if roverlay
+has been installed.
.. Important::
-
- ``emerge --config roverlay`` will overwrite the user's config file (or
- /etc/roverlay/R-overlay.conf when configuring for root).
-
-\*\*\*\*\*\*\*\*\*\*\*\* NOT VALID END \*\*\*\*\*\*\*\*\*\*\*\*
+ ``roverlay-setup[-interactive]`` overwrites the user's config.
++++++++++++++++++++++
@@ -658,7 +649,22 @@ An installation of roverlay includes some helper programs:
*roverlay-setup*
- <<DOC TODO>>
+ A script with various subcommands for setting up roverlay:
+
+ * ``init`` for initial setup as described in
+ `Automated Configuration and Setup`_
+
+ * ``mkconfig`` for generating the main config file
+
+ * ``hooks`` for managings hooks
+
+ * ``roverlay-setup hooks [show]`` prints information
+
+ * ``roverlay-setup hooks add <hook> <event> [<event>...]`` adds a hook
+ (referenced by name) to one or more *events*
+
+ * ``roverlay-setup hooks del <hook> <event|"all"> [<event|"all">...]``
+ removes a hook (referenced by name) from one or more (or all) *events*
*run-roverlay.sh*
A script that safely runs overlay creation and repoman afterwards.
^ permalink raw reply related [flat|nested] 41+ messages in thread* [gentoo-commits] proj/R_overlay:master commit in: doc/rst/
@ 2013-09-16 8:56 André Erdmann
0 siblings, 0 replies; 41+ messages in thread
From: André Erdmann @ 2013-09-16 8:56 UTC (permalink / raw
To: gentoo-commits
commit: 479d4653729d38421a6eeb133ddc60fef5b3651b
Author: André Erdmann <dywi <AT> mailerd <DOT> de>
AuthorDate: Mon Sep 16 08:55:58 2013 +0000
Commit: André Erdmann <dywi <AT> mailerd <DOT> de>
CommitDate: Mon Sep 16 08:55:58 2013 +0000
URL: http://git.overlays.gentoo.org/gitweb/?p=proj/R_overlay.git;a=commit;h=479d4653
doc/rst: automated configuration and setup
---
doc/rst/usage.rst | 93 ++++++++++++++++++++++++++++++++++++++++++-------------
1 file changed, 72 insertions(+), 21 deletions(-)
diff --git a/doc/rst/usage.rst b/doc/rst/usage.rst
index 6a48e99..b42ac47 100644
--- a/doc/rst/usage.rst
+++ b/doc/rst/usage.rst
@@ -188,18 +188,9 @@ Make sure to meet the dependencies as listed in Prerequisites_.
Then, simply clone the git repository to a local directory that is referenced
as the *R Overlay src directory* from now on.
-.. Note::
- You have to *cd* into the *R Overlay src directory* before running
- *roverlay* to ensure that the python modules can be imported correctly.
-
- You can work around this by setting up a wrapper script:
-
- .. code-block:: sh
-
- #!/bin/sh
- # /usr/local/bin/roverlay.sh
- # example wrapper script for roverlay
- cd ${ROVERLAY_SRC:-~/roverlay/src} && ./roverlay.py "$@"
+The *R Overlay src directory* provides a wrapper script (``roverlay.py`` or
+``bin/roverlay``) which changes the working directory and adjusts the
+``PYTHONPATH`` variable so that *roverlay* can be run from anywhere.
==================
@@ -218,12 +209,65 @@ it will look for the config file in that order:
#. */etc/roverlay/R-overlay.conf*,
which is part of the installation but has to be modified.
-Otherwise, *roverlay* uses *R-overlay.conf* in the current directory.
+Otherwise, *roverlay* uses *R-overlay.conf.local* or *R-overlay.conf*
+in the current directory.
An example config file is available in the *R Overlay src directory*.
-The config file is a text file with '<option> = <value>' syntax. Some options
-accept multiple values (e.g. <option> = file1, file2), in which case the
-values have to be enclosed with quotes (-> ``<option> = "file1 file2"``).
+
+Overall, the following steps have to be taken before running *roverlay*:
+
+* create/adjust the main config file
+* provide additional files (dependency rule files, hook event script etc.)
+ at the locations pointed out in the main config file
+* optional: create directories in advance with proper permissions, e.g.
+ a shared package files directory
+
+These steps can be automated by using *roverlay-setup*.
+See `Automated configuration and setup` for details.
+`Manual configuration` covers how to configure *roverlay* by hand.
+
+
++++++++++++++++++++++++++++++++++++
+ Automated configuration and setup
++++++++++++++++++++++++++++++++++++
+
+``roverlay-setup`` (``bin/roverlay-setup`` in the *R overlay src directory*)
+is a script that is able to generate *roverlay's* main config file, import
+other config files, create essential directories and activate/manage
+*event hooks*.
+
+It accepts a number of options, in particular:
+
+--pretend, -p
+ Show what would be done. **Recommended** (prior to running the command
+ without --pretend).
+
+--ask, -a
+ Ask for confirmation before each action. This allows to skip certain
+ actions by answering with *n*, *no* or *skip*.
+
+--work-root <dir>, -W <dir>
+ The user's root directory for work data (distfiles, overlay, ...).
+ Defaults to ``${HOME}/roverlay``.
+
+--conf-dir <dir>, --my-conf-root <dir>, -C <dir>
+ Directory for the user's private config.
+ Defaults to ``${HOME}/roverlay/config``.
+
+--variable <name=value>, -v <name=value>
+ This can be used to set config options.
+
+ Example: ``--variable DISTFILES=/nfs/roverlay/distfiles`` for setting
+ up a shared package files directory.
+
+
+See ``roverlay-setup --help`` for a full listing.
+
+
+
+\*\*\*\*\*\*\*\*\*\*\*\* NOT VALID BEGIN \*\*\*\*\*\*\*\*\*\*\*\*
+
+[interactive setup -- will be re-added]
If roverlay has been installed, then ``emerge --config roverlay`` can be
used to set up the config file as well as to create essential directories.
@@ -235,6 +279,16 @@ one user.
``emerge --config roverlay`` will overwrite the user's config file (or
/etc/roverlay/R-overlay.conf when configuring for root).
+\*\*\*\*\*\*\*\*\*\*\*\* NOT VALID END \*\*\*\*\*\*\*\*\*\*\*\*
+
+
+++++++++++++++++++++++
+ Manual configuration
+++++++++++++++++++++++
+
+The config file is a text file with '<option> = <value>' syntax. Some options
+accept multiple values (e.g. <option> = file1, file2), in which case the
+values have to be enclosed with quotes (-> ``<option> = "file1 file2"``).
The following options should be set before running *roverlay*:
@@ -558,9 +612,6 @@ apply_rules
This command implies the **sync** command unless the *--no-sync* option
is specified.
-setupdirs
- Creates all configured directories with proper permissions.
-
----------------------------
Providing a package mirror
@@ -606,8 +657,8 @@ An installation of roverlay includes some helper programs:
*roverlay-sh* requires a valid config file.
-*roverlay-mkconfig*
- a script that creates a config file for roverlay
+*roverlay-setup*
+ <<DOC TODO>>
*run-roverlay.sh*
A script that safely runs overlay creation and repoman afterwards.
^ permalink raw reply related [flat|nested] 41+ messages in thread* [gentoo-commits] proj/R_overlay:master commit in: doc/rst/
@ 2013-09-03 14:32 André Erdmann
0 siblings, 0 replies; 41+ messages in thread
From: André Erdmann @ 2013-09-03 14:32 UTC (permalink / raw
To: gentoo-commits
commit: 8a6277f219f7150f92dbd001ede2858ceaeda8a1
Author: André Erdmann <dywi <AT> mailerd <DOT> de>
AuthorDate: Tue Sep 3 14:12:58 2013 +0000
Commit: André Erdmann <dywi <AT> mailerd <DOT> de>
CommitDate: Tue Sep 3 14:12:58 2013 +0000
URL: http://git.overlays.gentoo.org/gitweb/?p=proj/R_overlay.git;a=commit;h=8a6277f2
doc/rst: document dynamic selfdep pool changes
---
doc/rst/usage.rst | 8 ++++++--
1 file changed, 6 insertions(+), 2 deletions(-)
diff --git a/doc/rst/usage.rst b/doc/rst/usage.rst
index 9db0ff3..36ad090 100644
--- a/doc/rst/usage.rst
+++ b/doc/rst/usage.rst
@@ -4388,8 +4388,12 @@ R packages" to create rules.
*roverlay* uses one dynamic rule pool, the **Dynamic Selfdep Rule Pool**.
This pool contains *selfdep* rules for all known R packages and is able
-to resolve R package dependencies.
-By convention, it will never resolve any system dependencies.
+to resolve R package dependencies. Moreover, it knows to which repository
+a rule belongs, and tries to satisfy dependencies on a package within its
+own repository first, before trying the other repositories in the order as
+they appear in the repo config file(s).
+
+By convention, this rule pool never resolves any system dependencies.
+++++++++++++++++++++++++++++
Dependency Resolver Modules
^ permalink raw reply related [flat|nested] 41+ messages in thread
* [gentoo-commits] proj/R_overlay:master commit in: doc/rst/
@ 2013-08-27 15:39 André Erdmann
0 siblings, 0 replies; 41+ messages in thread
From: André Erdmann @ 2013-08-27 15:39 UTC (permalink / raw
To: gentoo-commits
commit: 7cf05a080255ad13a857ff2a8a4ab68a30c823f3
Author: André Erdmann <dywi <AT> mailerd <DOT> de>
AuthorDate: Tue Aug 27 15:33:49 2013 +0000
Commit: André Erdmann <dywi <AT> mailerd <DOT> de>
CommitDate: Tue Aug 27 15:33:49 2013 +0000
URL: http://git.overlays.gentoo.org/gitweb/?p=proj/R_overlay.git;a=commit;h=7cf05a08
doc/rst: #! ERROR, deptype logging
---
doc/rst/usage.rst | 12 ++++++++++--
1 file changed, 10 insertions(+), 2 deletions(-)
diff --git a/doc/rst/usage.rst b/doc/rst/usage.rst
index 1379aad..10da45a 100644
--- a/doc/rst/usage.rst
+++ b/doc/rst/usage.rst
@@ -1402,7 +1402,7 @@ Keychars
simple rule.
Keywords
- There are two keywords that control how a rule file is read.
+ There are three keywords that control how a rule file is read.
The important one is the *#deptype <dependency type>* directive that
defines that all rules until the next *deptype* directory or end of file,
@@ -1424,7 +1424,9 @@ Keywords
Check the *dependency type* if a newly added rule has no effect.
- The other keyword is *#! NOPARSE* which stops parsing of a rule file.
+ The other keywords are *#! NOPARSE*, which stops parsing of a rule file
+ (ignore remaining file content), and *#! ERROR*, which raises a python
+ exception (program exits).
Dependencies
are strings that are recognized by portage as **Dynamic DEPENDs**
@@ -3190,6 +3192,12 @@ DESCRIPTION_DIR
LOG_FILE_UNRESOLVABLE
A file where all unresolved dependency strings will be written into
on *roverlay* exit. Primarily useful for creating new rules.
+ The file's format is ``<dependency type mask in hex>, <dependency string>``,
+ where the *dependency type mask* is usually
+ ``0x7`` (mandatory system dependency),
+ ``0x8`` (optional R package dependency)
+ or ``0xb`` (mandatory R package dependency).
+
Defaults to <not set>, which disables this feature.
^ permalink raw reply related [flat|nested] 41+ messages in thread* [gentoo-commits] proj/R_overlay:master commit in: doc/rst/
@ 2013-08-23 13:52 André Erdmann
0 siblings, 0 replies; 41+ messages in thread
From: André Erdmann @ 2013-08-23 13:52 UTC (permalink / raw
To: gentoo-commits
commit: 56d8c5f526e4139ac4f1207bbf23ac8b58101ac6
Author: André Erdmann <dywi <AT> mailerd <DOT> de>
AuthorDate: Fri Aug 23 13:32:41 2013 +0000
Commit: André Erdmann <dywi <AT> mailerd <DOT> de>
CommitDate: Fri Aug 23 13:32:41 2013 +0000
URL: http://git.overlays.gentoo.org/gitweb/?p=proj/R_overlay.git;a=commit;h=56d8c5f5
doc/rst, package rules: 'add' action
---
doc/rst/usage.rst | 120 ++++++++++++++++++++++++++++++------------------------
1 file changed, 67 insertions(+), 53 deletions(-)
diff --git a/doc/rst/usage.rst b/doc/rst/usage.rst
index 991a4b9..1379aad 100644
--- a/doc/rst/usage.rst
+++ b/doc/rst/usage.rst
@@ -1833,60 +1833,74 @@ control *where*) and the number of values they accept:
.. table:: action keywords
- +----------------+------------------+-------------+------------------------+
- | action keyword | affects | # of values | description |
- +================+==================+=============+========================+
- | ignore | | | ignore package, |
- +----------------+ overlay creation | none | do not try to create |
- | do-not-process | | | an ebuild for it |
- +----------------+------------------+-------------+------------------------+
- | keywords | ebuild variables | >= 1 | set per-package |
- | | | | ``KEYWORDS`` |
- +----------------+------------------+-------------+------------------------+
- | trace | package rules | none | mark a package as |
- | | | | modified |
- | | +-------------+------------------------+
- | | | 1 | add the stored string |
- | | | | to a package's |
- | | | | *modified* variable |
- | | | | whenever this action |
- | | | | is applied |
- +----------------+------------------+-------------+------------------------+
- | set | package | 2 | set package |
- +----------------+ metadata, +-------------+ information |
- | set_<key> | overlay creaton | 1 | |
- +----------------+------------------+-------------+------------------------+
- | rename | package | 2 | modify package |
- +----------------+ metadata, +-------------+ information with |
- | rename_<key> | overlay creation | 1 | sed-like |
- | | | | *s/expr/repl/* |
- | | | | statements |
- +----------------+------------------+-------------+------------------------+
- | null | *n/a* | none | does nothing |
- +----------------+ | | (can be used for |
- | pass | | | readability) |
- +----------------+------------------+-------------+------------------------+
-
-The two-arg form of the set/rename keywords expect a <key> as first and
+ +----------------+-------------------+-------------+------------------------+
+ | action keyword | affects | # of values | description |
+ +================+===================+=============+========================+
+ | ignore | | | ignore package, |
+ +----------------+ overlay creation | none | do not try to create |
+ | do-not-process | | | an ebuild for it |
+ +----------------+-------------------+-------------+------------------------+
+ | keywords | ebuild variables | >= 1 | set per-package |
+ | | | | ``KEYWORDS`` |
+ +----------------+-------------------+-------------+------------------------+
+ | trace | package rules | none | mark a package as |
+ | | | | modified |
+ | | +-------------+------------------------+
+ | | | 1 | add the stored string |
+ | | | | to a package's |
+ | | | | *modified* variable |
+ | | | | whenever this action |
+ | | | | is applied |
+ +----------------+-------------------+-------------+------------------------+
+ | set | package metadata, | 2 | set package |
+ +----------------+ overlay creation, +-------------+ information |
+ | set_<key> | ebuild creation | 1 | |
+ +----------------+ +-------------+------------------------+
+ | rename | | 2 | modify package |
+ +----------------+ +-------------+ information with |
+ | rename_<key> | | 1 | sed-like |
+ | | | | *s/expr/repl/* |
+ | | | | statements |
+ +----------------+ +-------------+------------------------+
+ | add | | 2 | similar to *set*, but |
+ +----------------+ +-------------+ can be applied |
+ | add_<key> | | 1 | multiple times without |
+ | | | | overwriting existing |
+ | | | | information |
+ +----------------+-------------------+-------------+------------------------+
+ | null | *n/a* | none | does nothing |
+ +----------------+ | | (can be used for |
+ | pass | | | improving readability) |
+ +----------------+-------------------+-------------+------------------------+
+
+The two-arg form of the set/rename/add keywords expect a <key> as first and
a value / sed expression as second arg. The one-arg form expects the latter
-one only. The "/" delimitier in the sed expression can be any character.
+one only. keys are case-insensitive. The "/" delimitier in the sed expression
+can be any other character.
The following *info keys* can be set and/or modified:
-.. table:: info keys for set/rename
-
- +--------------+---------------------+-------------------------------------+
- | info key | supports set/rename | description |
- +==============+=====================+=====================================+
- | name | yes / yes | rename the ebuild |
- +--------------+---------------------+-------------------------------------+
- | category | yes / yes | set or rename (based on the |
- | | (*repo_name*) | repository's name) the package's |
- | | | category |
- +--------------+---------------------+-------------------------------------+
- | destfile | yes / yes | rename ebuild destfile by using the |
- | | | '->' operator in ``${SRC_URI}`` |
- +--------------+---------------------+-------------------------------------+
+.. table:: info keys for set/rename/add
+
+ +--------------+-------------------------+-------------------------------------+
+ | info key | supports set/rename/add | description |
+ +==============+=========================+=====================================+
+ | name | yes / yes / no | rename the ebuild |
+ +--------------+-------------------------+-------------------------------------+
+ | category | yes / yes (*repo name*) | set or rename (based on the |
+ | | / no | repository's name) the package's |
+ | | | category |
+ +--------------+-------------------------+-------------------------------------+
+ | destfile | yes / yes / no | rename ebuild destfile by using the |
+ | | | '->' operator in ``${SRC_URI}`` |
+ +--------------+-------------------------+-------------------------------------+
+ | DEPEND | no / no / **yes** | add additional dependencies to |
+ +--------------+ | *DEPEND/RDEPEND/RSUGGESTS* |
+ | RDEPEND | | |
+ +--------------+ | The value has to be a string |
+ | RSUGGESTS | | accepted by the package manager, |
+ | | | e.g. a *DEPEND Atom*. |
+ +--------------+-------------------------+-------------------------------------+
The *set* action also supports variable substitution by means of python
@@ -1930,9 +1944,9 @@ The following *info keys* can be accessed:
.. Note::
- Applying the same (non-incremental) ebuild variable, set or rename action
- more than once is possible, but only the last one will have an effect
- on ebuild creation.
+ Applying the same ebuild variable, set or rename action more than once
+ is possible, but only the last one will have an effect on ebuild creation.
+ add actions are cumulative by definition.
Extended Action Block Syntax
^ permalink raw reply related [flat|nested] 41+ messages in thread* [gentoo-commits] proj/R_overlay:master commit in: doc/rst/
@ 2013-08-19 15:42 André Erdmann
0 siblings, 0 replies; 41+ messages in thread
From: André Erdmann @ 2013-08-19 15:42 UTC (permalink / raw
To: gentoo-commits
commit: 7c4fd191b0070020e8560cb64284b8113742703a
Author: André Erdmann <dywi <AT> mailerd <DOT> de>
AuthorDate: Mon Aug 19 15:40:22 2013 +0000
Commit: André Erdmann <dywi <AT> mailerd <DOT> de>
CommitDate: Mon Aug 19 15:40:22 2013 +0000
URL: http://git.overlays.gentoo.org/gitweb/?p=proj/R_overlay.git;a=commit;h=7c4fd191
doc/rst: document package rule changes
This commit also removes any occurence of --nosync.
---
doc/rst/usage.rst | 80 ++++++++++++++++++++++++++++++++++++++++++++++++++-----
1 file changed, 73 insertions(+), 7 deletions(-)
diff --git a/doc/rst/usage.rst b/doc/rst/usage.rst
index 46164e4..e493420 100644
--- a/doc/rst/usage.rst
+++ b/doc/rst/usage.rst
@@ -534,7 +534,7 @@ For reference, these **commands** are accepted by *roverlay*:
create
As described above, this will run ebuild, metadata creation, including
overlay and Manifest file writing.
- This command implies the **sync** command unless the *--nosync* option
+ This command implies the **sync** command unless the *--no-sync* option
is specified.
sync
@@ -555,7 +555,7 @@ apply_rules
Meant for testing.
- This command implies the **sync** command unless the *--nosync* option
+ This command implies the **sync** command unless the *--no-sync* option
is specified.
setupdirs
@@ -1746,7 +1746,7 @@ their *match keywords* and their meaning:
| boolean function | match | description |
| | keyword(s) | |
+==================+=============+========================================+
- | AND | and all && | all listed conditions must match |
+ | AND | all && | all listed conditions must match |
+------------------+-------------+----------------------------------------+
| OR | or \|\| | any |
| | | of the listed conditions must match |
@@ -1757,6 +1757,10 @@ their *match keywords* and their meaning:
| NOR | nor none | none |
| | | of the listed conditions must match |
+------------------+-------------+----------------------------------------+
+ | VERUM | any true | always true |
+ +------------------+-------------+----------------------------------------+
+ | FALSUM | none false | always false |
+ +------------------+-------------+----------------------------------------+
In other words, a (boolean) match keyword starts a *nested match block*
@@ -1779,6 +1783,8 @@ The nested block is terminated by indenting out, i.e. decreasing the
Correspondingly, the logic for the top-level match block is *AND* by
convention.
+ *VERUM* and *FALSUM* do accept any nested condition.
+
Using this syntax, match blocks can be nested indefinitely (minus technical
limitations):
@@ -1874,12 +1880,44 @@ The following *info keys* can be set and/or modified:
+==============+=====================+=====================================+
| name | yes / yes | rename the ebuild |
+--------------+---------------------+-------------------------------------+
- | category | yes / **no** | set package category |
+ | category | yes / yes | set or rename (based on the |
+ | | (*repo_name*) | repository's name) the package's |
+ | | | category |
+--------------+---------------------+-------------------------------------+
| destfile | yes / yes | rename ebuild destfile by using the |
| | | '->' operator in ``${SRC_URI}`` |
+--------------+---------------------+-------------------------------------+
+
+The *set* action also supports variable substitution by means of python
+string formatting (``{info_key}``), for example:
+
+.. code::
+
+ set category sci-{repo_name}
+
+
+The following *info keys* can be accessed:
+
+.. table:: info keys available for variable substitution
+
+ +---------------------+----------------------------------------------------+
+ | info key | description |
+ +=====================+====================================================+
+ | repo_name | name of the repo, e.g. *CRAN* |
+ +---------------------+----------------------------------------------------+
+ | version | package version ``${PV}`` |
+ +---------------------+----------------------------------------------------+
+ | package_name | package file name without version |
+ | | and file extension, e.g. *seewave* |
+ +---------------------+----------------------------------------------------+
+ | name | ebuild name ``${PN}`` |
+ +---------------------+----------------------------------------------------+
+ | package_filename | package file name with file extension, |
+ | | e.g. *seewave_1.7.0.tar.gz* |
+ +---------------------+----------------------------------------------------+
+
+
.. Caution::
Category moves are not handled automatically. In incremental mode, overlay
@@ -2002,6 +2040,20 @@ A rule that assigns all packages from BIOC-2.10/bioc to sci-bioc:
END;
+A more generic rule that sets per-repo categories:
+
+.. code::
+
+ MATCH:
+ any
+ ACTION:
+ # set category
+ # CRAN->CRAN, CRAN-Archive->CRAN, BIOC-2.10/experimental->BIOC, ...
+ #
+ rename category s=^(?P<repo>[^-/]+)([-/].*)?$=sci-\g<repo>=
+ END;
+
+
The following example prefixes all *yaml* packages with *Rpkg_*:
.. code::
@@ -2009,24 +2061,38 @@ The following example prefixes all *yaml* packages with *Rpkg_*:
MATCH:
ebuild_name ,= yaml
ACTION:
+ set destfile Rpkg_{package_filename}
+ END;
+
+ # alternatively:
+ MATCH:
+ ebuild_name ,= yaml
+ ACTION:
rename destfile s/^/Rpkg_/
END;
-Moving such packages to a "R-package" sub directory would be possible, too:
+Moving such packages to a "R-packages" sub directory would be possible, too:
.. code::
MATCH:
name ,= yaml
ACTION:
- rename_destfile s=^=R-package/=
+ set_destfile R-packages/{package_filename}
+ END;
+
+ # alternatively:
+ MATCH:
+ name ,= yaml
+ ACTION:
+ rename_destfile s=^=R-packages/=
END;
.. Hint::
- ``roverlay [--nosync] [--dump-file <file>] apply_rules`` can be used to
+ ``roverlay [--no-sync] [--dump-file <file>] apply_rules`` can be used to
test rules. It applies the rules to all packages without running overlay
creation. Furthermore, ``roverlay --ppr`` parses the package rules,
prints them and exits afterwards.
^ permalink raw reply related [flat|nested] 41+ messages in thread* [gentoo-commits] proj/R_overlay:master commit in: doc/rst/
@ 2013-08-09 15:27 André Erdmann
0 siblings, 0 replies; 41+ messages in thread
From: André Erdmann @ 2013-08-09 15:27 UTC (permalink / raw
To: gentoo-commits
commit: 2f9215125b0044a799137f3fc2478a06620f5b5b
Author: André Erdmann <dywi <AT> mailerd <DOT> de>
AuthorDate: Fri Aug 9 15:22:30 2013 +0000
Commit: André Erdmann <dywi <AT> mailerd <DOT> de>
CommitDate: Fri Aug 9 15:22:30 2013 +0000
URL: http://git.overlays.gentoo.org/gitweb/?p=proj/R_overlay.git;a=commit;h=2f921512
doc/rst: implementation overview, stats config
---
doc/rst/usage.rst | 145 +++++++++++++++++++++++++++++++++++++-----------------
1 file changed, 99 insertions(+), 46 deletions(-)
diff --git a/doc/rst/usage.rst b/doc/rst/usage.rst
index 88761e5..787c9aa 100644
--- a/doc/rst/usage.rst
+++ b/doc/rst/usage.rst
@@ -89,37 +89,43 @@ Expected prior knowlegde:
Prerequisites
---------------
-* python >= 2.7 (tested with python 2.7 and 3.2)
+* python >= 2.7 built with ssl support (tested with python 2.7 and 3.2)
-* argparse (http://code.google.com/p/argparse)
+ extra dependencies for python 2.7 (built-in since python 3.2)
+
+ * argparse (http://code.google.com/p/argparse)
+ * concurrent.futures (http://pypi.python.org/pypi/futures) (optional)
* setuptools (http://pypi.python.org/pypi/setuptools)
* rsync (for using rsync repositories)
-* for Manifest creation:
+* portage (*ebuild*) for Manifest creation and downloading source files for
+ imported ebuilds
- * portage (*ebuild*)
- * *true* or *echo* from coreutils or busybox for preventing
- package downloads during Manifest creation (optional)
+* *true* or *echo* from coreutils or busybox for preventing
+ package downloads during Manifest creation (optional)
* for generating documentation files: python docutils >= 0.9
* hardware requirements (when using the default configuration):
disk
- * 50-55GB disk space for the R packages
- * a filesystem that supports symbolic or hard links
- * there will be many small-sized files (ebuilds),
- so a filesystem with lots of inodes and a small block size
- may be advantageous
+
+ * 50-55GB disk space for the R packages
+ * a filesystem that supports symbolic or hard links
+ * there will be many small-sized files (ebuilds),
+ so a filesystem with lots of inodes and a small block size
+ may be advantageous
memory
+
up to 600MB which depends on the amount of processed packages and the
write mechanism in use. The amount can be halved (approximately) when
using a slower one.
time
+
Expect 1h execution time for the first run, depending on computing
and I/O speed. *roverlay* is able to work in incremental mode,
thus making subsequent runs need a lot less time.
@@ -402,10 +408,19 @@ Main Config
See `Configuration Reference`_ for all main config options like log file
rotation and assistance for writing new *dependency rules*.
+Patching generated ebuilds / Importing ebuilds
+ See `Additions Directory`_.
+
+R_SUGGESTS
+ See `USE_EXPAND flag rename file`_, which describes how R_SUGGESTS
+ USE_EXPAND flags can be renamed.
+
Field Definition
Refer to `Field Definition`_ in case you want to change *how* R packages
are being read, e.g. if you want the 'Depents' information field (obviously
- a typo) to be understood as 'Depends'.
+ a typo) to be understood as 'Depends'. Also see `License Map file`_ for
+ translating *license strings* into valid portage licenses.
+
------------
Running it
@@ -543,6 +558,10 @@ apply_rules
This command implies the **sync** command unless the *--nosync* option
is specified.
+setupdirs
+ Creates all configured directories with proper permissions.
+
+
----------------------------
Providing a package mirror
----------------------------
@@ -582,15 +601,21 @@ An installation of roverlay includes some helper programs:
true
+ .. Note::
+
+ *roverlay-sh* requires a valid config file.
+
+
*roverlay-mkconfig*
a script that creates a config file for roverlay
-*name_is_todo--roverlay_creation_helper*
- <<TODO>>
- Safely runs overlay creation <<and $$afterwards>>.
+*run-roverlay.sh*
+ A script that safely runs overlay creation and repoman afterwards.
+ Uses filesystem locks to ensure that it is run at most once at a time.
Suitable for being run as cron job.
-
+ This file is not part of the roverlay installation. It can be found
+ at ``files/scripts/run-roverlay.sh`` in the `roverlay git repo`_.
===============================
@@ -603,71 +628,84 @@ An installation of roverlay includes some helper programs:
These are the steps that *roverlay* performs:
-1. **sync** - get R packages using various methods
+#. **sync** - get R packages using various methods
(rsync, http, local directory)
-2. scan the R Overlay directory (if it exists) for valid ebuilds
+#. scan the R Overlay directory (if it exists) for valid ebuilds
-3. import ebuilds from the additions dir
+#. import ebuilds from the additions dir
-4. **add** - queue all R packages for ebuild creation
+#. **add** - queue all R packages for ebuild creation
* all repositories are asked to list their packages which are then added
to a queue
- * packages may be declined by the overlay creator if they already have
- an ebuild
-
* packages may be declined or manipulated by package rules
See also: `Package Rules`_
-5. **create** - process each package *p* from the package queue
+ * packages may be declined by the overlay creator if they already have
+ an ebuild
+
+ The overlay is able to detect changes to the package file, e.g. when
+ upstream modified a package without renaming it. To realize this,
+ the package file's checksum (sha256) is compared with the ebuild's
+ entry in the `distmap`_. If they do match, then the package is declined
+ as it already exists. Otherwise, a revision bump is triggered.
+
+#. **create** - process each package *p* from the package queue
(thread-able on a per package basis)
- * read *p*'s DESCRIPTION file that contains information fields
- like 'Depends', 'Description' and 'Suggests'
+ * read *p*'s DESCRIPTION file that contains information fields
+ like 'Depends', 'Description' and 'Suggests'
+
+ * resolve *p*'s dependencies
- * resolve *p*'s dependencies
+ * differentiate between *required* and *optional* dependencies
+ (for example, dependencies from the 'Depends' field are required,
+ while those from 'Suggests' are optional)
- * differentiate between *required* and *optional* dependencies
- (for example, dependencies from the 'Depends' field are required,
- while those from 'Suggests' are optional)
+ * **immediately stop** processing *p* if a *required* dependency
+ cannot be resolved in which case a valid ebuild cannot be created
- * **immediately stop** processing *p* if a *required* dependency
- cannot be resolved in which case a valid ebuild cannot be created
+ * verify dependencies on packages found in the overlay, whether their
+ ebuilds already exist or not (*selfdep validation*)
- * verify dependencies on packages found in the overlay, whether their
- ebuilds already exist or not (*selfdep validation*)
+ * drop unsatisfiable *selfdeps*
- See also: `Dependency Resolution`_
+ * **stop** processing *p* if a *required selfdep* is missing
- * create an ebuild for *p* by using the dependency resolution results
- and a few information fields like 'Description'
+ See also: `Dependency Resolution`_
- * **done** with *p* - the overlay creator takes control over *p*
- and may decide to write *p*'s ebuild now (or later)
+ * create an ebuild for *p* by using the dependency resolution results
+ and a few information fields like 'Description'
-6. write the overlay
+ * **done** with *p* - the overlay creator takes control over *p*
+ and may decide to write *p*'s ebuild now (or later)
- * write/update the profiles dir
+#. write the overlay
+
+ * write/update the profiles dir and metadata/layout.conf
* *roverlay* respects manual changes to USE_EXPAND description files
* write all ebuilds and apply patches to them
- (supports threads on a per package basis)
* write the *metadata.xml* files
- (supports threads on a per package basis)
* this uses the latest created ebuild available for a package
* write the *Manifest* files
- (does not support threads by default / supports threads on a per package
- basis when using *portage* directly)
* this uses all ebuilds availabe for a package
+ Most write operations support threading.
+
+#. call the *overlay_success* hook
+
+#. write the stats database file (optional)
+
+ * call the *db_written* hook
--------------------------------------------------------------
Expected Overlay Result / Structure of the generated overlay
@@ -1085,7 +1123,7 @@ A local directory will never be modified.
websync_pkglist_.
---------
- distmap
+ Distmap
---------
*roverlay* uses a text file to store information about files in the package
@@ -2452,6 +2490,21 @@ CACHEDIR
This option is **required**.
+.. _STATS_DB_FILE:
+
+STATS_DB
+ Path to the stats database file. This has to be a round-robin database
+ file (rrdtool). *roverlay* creates it if necessary.
+
+ Defaults to <not set>, which disable database writing.
+
+STATS_INTERVAL
+ Database update interval, which should be set to the expected timespan
+ between running overlay creation, in seconds. Only meaningful when
+ creating a new database file.
+
+ Defaults to 7200 (2 hours).
+
.. _DISTFILES:
DISTFILES
^ permalink raw reply related [flat|nested] 41+ messages in thread* [gentoo-commits] proj/R_overlay:master commit in: doc/rst/
@ 2013-08-07 16:10 André Erdmann
0 siblings, 0 replies; 41+ messages in thread
From: André Erdmann @ 2013-08-07 16:10 UTC (permalink / raw
To: gentoo-commits
commit: a7bdfc6904b64fae37e9b8a18134f07a9c55f0c6
Author: André Erdmann <dywi <AT> mailerd <DOT> de>
AuthorDate: Wed Aug 7 15:54:19 2013 +0000
Commit: André Erdmann <dywi <AT> mailerd <DOT> de>
CommitDate: Wed Aug 7 15:54:19 2013 +0000
URL: http://git.overlays.gentoo.org/gitweb/?p=proj/R_overlay.git;a=commit;h=a7bdfc69
doc/rst: license map syntax, using roverlay's API
+ documentation about ebuild variable protection features
---
doc/rst/usage.rst | 148 ++++++++++++++++++++++++++++++++++++++++++++++++------
1 file changed, 133 insertions(+), 15 deletions(-)
diff --git a/doc/rst/usage.rst b/doc/rst/usage.rst
index 2da8dd0..88761e5 100644
--- a/doc/rst/usage.rst
+++ b/doc/rst/usage.rst
@@ -52,20 +52,28 @@ This document is targeted at
* *roverlay* maintainers who **control and test overlay creation**,
e.g. configure which R packages will be part of the generated overlay
- Depending on what you want to configure, chapters 5-10 are relevant,
+ Depending on what you want to configure, chapters 5-11 are relevant,
namely `Repositories / Getting Packages`_, `Additions Directory`,
`Dependency Rules`_, `Package Rules`_, `Configuration Reference`_
and `Field Definition Config`_.
There is another chapter that is only interesting for testing, the
- `Dependency Resolution Console`_ (11), which can be used to interactively
- test dependency rules.
+ `Dependency Resolution Console`_ (12.1), which can be used to
+ interactively test dependency rules.
+
* *roverlay* code maintainers who want to know **how roverlay works** for
code improvements etc.
- The most important chapter is `Implementation Overview`_ (12) which has
- references to other chapters (4-10) where required.
+ The most important chapter is `Implementation Overview`_ (14) which has
+ references to other chapters (4-13) where required.
+
+ * developers who intend to **use roverlay's functionality outside of roverlay**
+
+ The most important chapter is `Roverlay Interface`_, which gives an
+ overview of *roverlay's* API.
+ Reading the other chapters (4-14) is recommended.
+
Expected prior knowlegde:
@@ -197,17 +205,15 @@ as the *R Overlay src directory* from now on.
------------------------------
*roverlay* needs a configuration file to run.
-If roverlay has been installed with *emerge*, it will look for the config file in
-that order:
+If roverlay has been installed with *emerge*,
+it will look for the config file in that order:
-1. *<current directory>/R-overlay.conf*
#. *~/roverlay/R-overlay.conf*
#. */etc/roverlay/R-overlay.conf*,
which is part of the installation but has to be modified.
-Otherwise, *roverlay* will only look for *R-overlay.conf* in the current
-directory. An example config file is available in the
-*R Overlay src directory*.
+Otherwise, *roverlay* uses *R-overlay.conf* in the current directory.
+An example config file is available in the *R Overlay src directory*.
The config file is a text file with '<option> = <value>' syntax. Some options
accept multiple values (e.g. <option> = file1, file2), in which case the
@@ -432,6 +438,10 @@ to know in detail what *roverlay* does before running it.
--nosync, --no-sync
Disable downloading of R packages.
+--strict
+ Enable strict behavior.
+ For example, this causes *roverlay* to exit if any repo cannot be synced.
+
--distmap-verify
Enforce verification of R packages in the package mirror directory.
This also tries to recreate the distmap.
@@ -671,6 +681,7 @@ eclass file are used, the result should look like:
<overlay root>/
<overlay root>/eclass
<overlay root>/eclass/R-packages.eclass
+ <overlay root>/metadata/layout.conf
<overlay root>/profiles
<overlay root>/profiles/categories
<overlay root>/profiles/repo_name
@@ -3085,7 +3096,7 @@ Example:
Each flag is renamed at most once, so the following example renames 'sound'
-to media, but 'audio' to 'sound':;
+to media, but 'audio' to 'sound':
.. code-block:: text
@@ -3099,6 +3110,63 @@ to media, but 'audio' to 'sound':;
unpredictable results.
+------------------
+ License Map File
+------------------
+
+The license map file is a file with dictionary-like entries that is used
+to translate *license strings* (read from the package's DESCRIPTION) into
+licenses accepted by portage, e.g. ``GPL-3`` or ``|| ( GPL-3+ BSD )``.
+Its syntax is similar to the dependency rule file syntax:
+
+.. code-block:: text
+
+ # this is a comment line
+
+ # 1,1 mapping
+ portage_license :: license_str
+
+ # 1,n mapping (n>=0)
+ portage_license {
+ license_str_0
+ license_str_1
+ ...
+ license_str_n
+ }
+
+ # 0,1 mapping (ignore license_str)
+ ! :: license_str
+
+ # 0,n mapping (ignore several license_str)
+ ! {
+ license_str_0
+ license_str_1
+ ...
+ license_str_n
+ }
+
+
+
+Example (excerpt from *roverlay's* license map file):
+
+.. code-block:: text
+
+ # freestyle text
+ ! {
+ foo
+ freeforresearchpurpose.
+ whatlicenseisitunder?
+ }
+
+ BSD :: freebsd
+
+ || ( GPL-2+ BSD ) {
+ gpl>2|bsd
+ gpl>2|freebsd
+ }
+
+
+
.. _Field Definition:
-------------------------
@@ -3267,6 +3335,7 @@ This is the default field definition file (without any ignored fields):
roverlay provides an interactive console for accessing certain subsystems,
e.g. dependency resolution. Its features include tab completion for filesystem
paths and a command history that supports navigation with the arrow keys.
+Line continuation with a backslash character ``\`` is supported, too.
The console also implements a subset of typical tools like ``cat``, which
behave similar but not identical to their counterparts.
@@ -3347,7 +3416,7 @@ It is an interactive console with the following features:
* save rules to a file
Rules are managed in a set. These so-called *rule pools* are organized in
-a *first-in-first-out* data structure that allows to create or remove
+a *last-in-first-out* data structure that allows to create or remove
them easily at runtime.
Running ``roverlay depres_console`` prints a short welcome message and starts
@@ -3500,6 +3569,46 @@ For extending the API, roverlay provides the abstract *RoverlayInterface* and
*RoverlaySubInterface* classes.
+The following code snippet gives an idea on how to include roverlay's API in
+your code:
+
+.. code-block:: python
+
+ #!/usr/bin/python
+ #
+ # Initializes logging and roverlay's interfaces
+ #
+
+ import logging
+
+ import roverlay.core
+ import roverlay.interface.main
+
+ def main():
+ # log everything to console
+ roverlay.core.force_console_logging ( log_level=logging.INFO )
+
+ # load roverlay's config
+ config = roverlay.core.load_locate_config_file (
+ ROVERLAY_INSTALLED=False, setup_logger=False
+ )
+
+ # create the main interface
+ main_interface = roverlay.interface.main.MainInterface ( config=config )
+
+ # create subinterfaces, as needed
+ depres_interface = main_interface.spawn_interface ( "depres" )
+ remote_interface = main_interface.spawn_interface ( "remote" )
+
+ # use them
+ pass
+ # --- end of main (...) ---
+
+ if __name__ == '__main__':
+ main()
+
+
+
------------------
DepRes Interface
------------------
@@ -3713,8 +3822,8 @@ package directories) and implements all overlay-related functionality:
* Write the overlay to its filesystem location
- * initialize the overlay (write the *profiles/* directory,
- import eclass files)
+ * initialize the overlay (write the *profiles/* directory and
+ *metadata*/layout.conf, import eclass files)
* Write ebuilds; all *PackageInfo* instances with an ebuild will be written
* Generate and write metadata
* Write Manifest files
@@ -3797,6 +3906,15 @@ class. An *EbuildVar* defines its position in the ebuild and how its text
output should look like. Ebuild text is created by adding ebuild variables
to an *Ebuilder* that automatically sorts them and creates the ebuild.
+Most ebuild variables, e.g. all variables that contain text from a package's
+DESCRIPTION data, have basic protection against code injection:
+
+* only ASCII characters are allowed
+* the following chars are always removed: ``"``, ``'``, ````` and ``;``
+* backslash chars ``\`` at the end of the variable's value are removed
+* any char sequence starting with ``$(`` is completely dropped
+
+
------------------
Overlay Creation
------------------
^ permalink raw reply related [flat|nested] 41+ messages in thread* [gentoo-commits] proj/R_overlay:master commit in: doc/rst/
@ 2013-08-06 16:02 André Erdmann
0 siblings, 0 replies; 41+ messages in thread
From: André Erdmann @ 2013-08-06 16:02 UTC (permalink / raw
To: gentoo-commits
commit: df165f4d6cd457b40d85c2594981505b3454467d
Author: André Erdmann <dywi <AT> mailerd <DOT> de>
AuthorDate: Tue Aug 6 15:56:19 2013 +0000
Commit: André Erdmann <dywi <AT> mailerd <DOT> de>
CommitDate: Tue Aug 6 15:56:19 2013 +0000
URL: http://git.overlays.gentoo.org/gitweb/?p=proj/R_overlay.git;a=commit;h=df165f4d
doc/rst: new console, selfdep validation, misc
* rewritten console section (generic + depres console)
* implementation overview: more detailed overlay creation doc,
selfdep validation
* config referene: license map options
* api reference: a few notes about the RemoteInterface
---
doc/rst/usage.rst | 481 +++++++++++++++++++++++++++++++++++++++---------------
1 file changed, 348 insertions(+), 133 deletions(-)
diff --git a/doc/rst/usage.rst b/doc/rst/usage.rst
index dab9b8c..2da8dd0 100644
--- a/doc/rst/usage.rst
+++ b/doc/rst/usage.rst
@@ -559,23 +559,21 @@ An installation of roverlay includes some helper programs:
`hook environment`_.
roverlay-related scripts can use it to automatically inherit roverlay's
- config and ``$FUNCTIONS`` file:
+ config and get access to its ``$FUNCTIONS`` file:
.. code-block:: sh
#!/usr/bin/roverlay-sh
- # reset DEBUG, VERBOSE, QUIET
- DEBUG=n; QUIET=n; VERBOSE=y
-
# load the functions file (optional)
. "${FUNCTIONS?}" || exit
# script body
true
- <<TODO: maybe there's a better place for the details>>
+*roverlay-mkconfig*
+ a script that creates a config file for roverlay
*name_is_todo--roverlay_creation_helper*
<<TODO>>
@@ -953,7 +951,9 @@ Examples:
This is your best bet if the remote is a repository but does not offer rsync
access. Basic digest verification is supported (MD5). The remote has to have
a package list file, typically named *PACKAGES*,
-with a special syntax (debian control file syntax).
+with a special syntax (debian control file syntax). Syncing is retried up to
+3 times, for example if a connection timeout occurs or a remote file
+disappears after reading the package list file.
A package list example,
excerpt from `omegahat's PACKAGES file`_ (as of Aug 2012):
@@ -1098,9 +1098,6 @@ The *additions directory* is a directory with overlay-like structure that
contains extra files for overlay creation. Currently, ebuild patches and
ebuild files are supported.
-To give an idea of how this directory could
-
-
------------------
Patching ebuilds
@@ -1158,17 +1155,16 @@ Foreign ebuilds can be imported into the overlay by simple putting them into
the additions directory.
The naming convention is similar to ebuild patches and identical to the
-portage tree:
-
-.. code::
+portage tree, ``${CATEGORY}/${PN}/${PF}.ebuild``.
- ${CATEGORY}/${PN}/${PF}.ebuild
-
-
-Ebuilds imported that way can not be overwritten by generated ebuilds and
+Ebuilds imported that way cannot be overwritten by generated ebuilds and
benefit from most overlay creation features, e.g. Manifest file creation.
However, they cannot be used for metadata creation.
+The ``${CATEGORY}/${PN}/metadata.xml`` file will be imported if it exists and
+if ``${PN}`` in the overlay (a) has no metadata.xml file or (b) has no
+generated ebuilds (only imports).
+
==================
Dependency Rules
@@ -1469,6 +1465,10 @@ Comments
*#! NOPARSE* keywords. Comments inside rule blocks are not allowed and
will be read as normal *dependency strings*.
+
+.. _SELFDEP:
+.. _SELFDEPS:
+
Selfdep
This is a classification for dependencies on packages which are also part
of the overlay being created. Typically, selfdeps refer to other R
@@ -2486,6 +2486,13 @@ RSYNC_BWLIMIT
Defaults to <not set>, which disables bandwidth limitation.
+.. _PORTDIR:
+
+PORTDIR
+ Path to the portage tree. This option is **recommended**, but not always
+ required.
+
+ Defaults to "/usr/portage".
-----------------
@@ -2979,6 +2986,47 @@ LOG_FILE_ROTATE_COUNT
Defaults to *3* and has no effect if LOG_FILE_ROTATE_ is disabled.
+
+---------------------
+ license map options
+---------------------
+
+.. _CREATE_LICENSE_FILE:
+
+CREATE_LICENSE_FILE
+ Alias to CREATE_LICENSES_FILE_.
+
+.. _CREATE_LICENSES_FILE:
+
+CREATE_LICENSES_FILE
+ Create the CACHEDIR_/license file after scanning PORTDIR_/licenses for
+ licenses. This file can serve as fallback if PORTDIR_ is not available
+ or if USE_PORTAGE_LICENSES_ is set to *no*.
+
+ Defaults to *yes*.
+
+.. _LICENSE_MAP:
+
+LICENSE_MAP
+ Path to the license map file, which is used to translate license strings
+ into valid licenses (accepted by portage). Its syntax is similar to
+ dependency rules.
+
+ This option is **not required**, but recommended if the
+ `field definition config`_ file contains a field with the *isLicense* flag.
+
+ Defaults to <not set>.
+
+.. _USE_PORTAGE_LICENSES:
+
+USE_PORTAGE_LICENSES
+ A *bool* that controls whether PORTDIR_/licenses should be scanned for
+ licenses. As fallback, or if this option is set to *no*, the
+ CACHEDIR_/license file is read.
+
+ Defaults to *yes*.
+
+
--------------------------------------------------------------------
options for debugging, manual dependency rule creation and testing
--------------------------------------------------------------------
@@ -3216,13 +3264,72 @@ This is the default field definition file (without any ignored fields):
Roverlay Console
==================
-<<section is TODO>>
+roverlay provides an interactive console for accessing certain subsystems,
+e.g. dependency resolution. Its features include tab completion for filesystem
+paths and a command history that supports navigation with the arrow keys.
+
+The console also implements a subset of typical tools like ``cat``, which
+behave similar but not identical to their counterparts.
+
+See the following sections for how to run specific consoles.
+
+
+The following table lists all basic commands, which are available in all
+roverlay consoles. Some commands have more detailed help messages, which are
+printed when running ``<cmd> --help``.
+
+
+.. table:: basic console commands (subsystem-independent)
+
+ +---------------------+----------+----------------------------------------------------+
+ | command | extended | description |
+ | | --help | |
+ +=====================+==========+====================================================+
+ | exit, quit, q, qq | no | exit |
+ +---------------------+----------+----------------------------------------------------+
+ | help, ? | no | print help message (list known commands) |
+ +---------------------+----------+----------------------------------------------------+
+ | help *cmd* | no | show command-specific help message |
+ +---------------------+----------+----------------------------------------------------+
+ | alias | no | show command aliases |
+ +---------------------+----------+----------------------------------------------------+
+ | unalias | no | unset command aliases (*not implemented*) |
+ +---------------------+----------+----------------------------------------------------+
+ | cat | **yes** | read files and print them (supports compressed |
+ | | | files) |
+ +---------------------+----------+----------------------------------------------------+
+ | cd | no | change working directory |
+ | | | |
+ | | | Actually, this does not change the working |
+ | | | directory. It simply sets the prefix which is used |
+ | | | when dealing with relative filesystem paths. |
+ +---------------------+----------+----------------------------------------------------+
+ | chroot | no | enter/leave/show command chroot |
+ | | | |
+ | | | A command chroot prefixes all input with a |
+ | | | specific command. |
+ +---------------------+----------+----------------------------------------------------+
+ | declare | no | show variables |
+ +---------------------+----------+----------------------------------------------------+
+ | echo | no | print text (supports python string formatting) |
+ +---------------------+----------+----------------------------------------------------+
+ | exec | no | switch to another subsystem (*not implemented*) |
+ +---------------------+----------+----------------------------------------------------+
+ | history | no | show command history |
+ +---------------------+----------+----------------------------------------------------+
+ | ls | no | print directory content |
+ +---------------------+----------+----------------------------------------------------+
+ | pwd | no | print current working directory |
+ +---------------------+----------+----------------------------------------------------+
+ | set | no | set variables |
+ +---------------------+----------+----------------------------------------------------+
+ | unset | no | unset variables |
+ +---------------------+----------+----------------------------------------------------+
-<<links to depres console chapter may need to be fixed>>
-<<basic commands, table>>
+.. Note::
-<<note regarding python -OO and missing help texts>>
+ Running the console with ``python -OO`` removes most of the help messages.
.. _DepRes Console:
@@ -3231,144 +3338,133 @@ This is the default field definition file (without any ignored fields):
Dependency Resolution Console
-------------------------------
-.. Warning::
-
- This section is out-of-date.
-
-
As previously stated, the *DepRes Console* is only meant for **testing**.
It is an interactive console with the following features:
* resolve dependencies
* create new dependency rules
-* create dependency rules for each R package found in a directory
* load rules from files
* save rules to a file
Rules are managed in a set. These so-called *rule pools* are organized in
-a *first-in-first-out* data structure that allows
-to create or remove them easily at runtime.
-
-Running ``roverlay depres_console`` will print a welcome message that
-lists all available commands and a few usage hints.
-
-For reference, these commands are currently available:
-
-<<TODO: rewrite/update command table, it's out-of-date>>
-
-+---------------------+----------------------------------------------------+
-| command | description |
-+=====================+====================================================+
-| help, | lists all commands |
-| h, ? | |
-+---------------------+----------------------------------------------------+
-| help *<cmd>*, | prints a command-specific help message |
-| h *<cmd>* | |
-+---------------------+----------------------------------------------------+
-| load *<file|dir>*, | loads a rule file or a directory with rule files |
-| l *<file|dir>* | into a new *rule pool* |
-+---------------------+----------------------------------------------------+
-| load_conf, | loads the rule files listed in the config file |
-| lc | into a new *rule pool* |
-+---------------------+----------------------------------------------------+
-| add_rule *<rule>* | creates a new rule and adds it to the topmost, |
-| + *<rule>* | i.e. latest *rule pool*. This command uses |
-| | `Rule File Syntax`_. Multi line rules are |
-| | supported. |
-+---------------------+----------------------------------------------------+
-| add_pool, | creates a new *rule pool* |
-| << | |
-+---------------------+----------------------------------------------------+
-| unwind_pool, | removes the topmost *rule pool* and all of its |
-| >> | rules |
-+---------------------+----------------------------------------------------+
-| resolve *<dep>*, | tries to resolve the given dependency string and |
-| ?? *<dep>* | prints the result |
-+---------------------+----------------------------------------------------+
-| ! | enter the resolve *command chroot* |
-| | all input will be prefixed with "resolve" |
-+---------------------+----------------------------------------------------+
-| !! | leave any *command chroot* |
-+---------------------+----------------------------------------------------+
-| print, p | prints the rules of the topmost *rule pool* |
-+---------------------+----------------------------------------------------+
-| print --all|-a | prints the rules of all *rule pools* |
-+---------------------+----------------------------------------------------+
-| print <id> [<id>..] | prints the rules of a specific *rule pool* |
-+---------------------+----------------------------------------------------+
-| write *<file>*, | writes the rules of the topmost *rule pool* into |
-| w *<file>* | *<file>*. See write --help for advanced usage. |
-+---------------------+----------------------------------------------------+
-| cd *<dir>* | changes the working directory |
-| | This is a virtual command. <<TODO:EXPLAIN>> |
-+---------------------+----------------------------------------------------+
-| set VAR=VALUE | set variables |
-+---------------------+----------------------------------------------------+
-| unset VAR | unset variables |
-+---------------------+----------------------------------------------------+
-| declare | show declared variables |
-+---------------------+----------------------------------------------------+
-| alias | show command aliases |
-+---------------------+----------------------------------------------------+
-| unalias | reserved for future usage |
-+---------------------+----------------------------------------------------+
-| exit, qq, q | exits the *DepRes Console* |
-+---------------------+----------------------------------------------------+
+a *first-in-first-out* data structure that allows to create or remove
+them easily at runtime.
+
+Running ``roverlay depres_console`` prints a short welcome message and starts
+the console.
+
+For reference, these commands are available (in addition to the basic ones):
+
+.. table:: depres commands
+
+ +---------------------+----------+----------------------------------------------------+
+ | command | extended | description |
+ | | --help | |
+ +=====================+==========+====================================================+
+ | add_pool, << | no | creates a new, empty *rule pool* |
+ +---------------------+----------+----------------------------------------------------+
+ | add_rule, + | no | creates a new rule and adds it to the topmost, |
+ | | | i.e. latest *rule pool*. This command uses |
+ | | | `Rule File Syntax`_. Multi line rules are |
+ | | | supported. |
+ +---------------------+----------+----------------------------------------------------+
+ | load, l | yes | load rule files |
+ +---------------------+----------+----------------------------------------------------+
+ | load_conf, lc | no | load configured rule files |
+ +---------------------+----------+----------------------------------------------------+
+ | print_pool, print, | yes | print one or more *rule pools* |
+ | p | | |
+ +---------------------+----------+----------------------------------------------------+
+ | resolve, ?? | no | tries to resolve the given dependency string and |
+ | | | prints the result |
+ +---------------------+----------+----------------------------------------------------+
+ | unwind_pool, >> | yes | removes the topmost *rule pool* and all of its |
+ | | | rules |
+ +---------------------+----------+----------------------------------------------------+
+ | write, w | yes | writes rules to a file |
+ +---------------------+----------+----------------------------------------------------+
+ | ! | no | enter the *resolve command chroot*, which prefixes |
+ | | | all input with *resolve* |
+ +---------------------+----------+----------------------------------------------------+
+ | !! | no | leave any *command chroot* |
+ +---------------------+----------+----------------------------------------------------+
-.. Note::
-
- Some commands also offer more detailed help via ``<command> --help``.
+Example Session:
+ .. code-block::
-<<TODO>>
+ [roverlay depres_console]
-<<Example Session is out-of-date>>
+ == dependency resolution console (r2) ==
+ Run 'help' to list all known commands.
+ More specifically, 'help <cmd>' prints a help message for
+ the given command, and 'help --list' lists all help topics.
+ Use 'load_conf' or 'lc' to load the configured rule files.
-Example Session:
- .. code-block::
+ cmd % help
- == depres console ==
- Run 'help' to list all known commands
- More specifically, 'help <cmd>' prints a help message for the given
- command, and 'help --list' lists all help topics available
- Use 'load_conf' or 'lc' to load the configured rule files
+ Documented commands (type help <topic>):
+ ========================================
+ EOF cat echo history print qq unalias
+ add_pool cd exec load print_pool quit unset
+ add_rule chroot exit load_conf pwd resolve unwind_pool
+ alias declare help ls q set write
- commands: load, unwind, set, help, >>, load_conf, <<, cd, mkhelp,
- resolve, lc, add_pool, addrule, h, +, l, li, write, p, r, ?, w, print,
- sd, unset, scandir
- exit-commands: q, qq, exit
+ cmd % <tab><tab>
+ EOF chroot history pwd unalias
+ add_pool declare load q unset
+ add_rule echo load_conf qq unwind_pool
+ alias exec ls quit write
+ cat exit print resolve
+ cd help print_pool set
cmd % + ~dev-lang/R :: R language
- new rules:
+
+ cmd % print --help
+ usage: print_pool [-h] [--all] [<id> [<id> ...]]
+
+ positional arguments:
+ <id> print specific pools (by id)
+
+ optional arguments:
+ -h, --help show this help message and exit
+ --all, -a print all pools
+
+ cmd % print -a
~dev-lang/R :: R language
- --- ---
- command succeeded.
- cmd % ? R language
- Trying to resolve ('R language',).
- Resolved as: ('dev-lang/R',)
+ cmd % !
+
+ (resolve) % R language
+ 'R language' has been resolved as dev-lang/R.
+
+ (resolve) % R language [ 2.15 ]
+ 'R language [ 2.15 ]' has been resolved as >=dev-lang/R-2.15.
- cmd % ? R language [ 2.15 ]
- Trying to resolve ('R language [ 2.15 ]',).
- Resolved as: ('>=dev-lang/R-2.15',)
+ (resolve) % R
+ 'R' could not be resolved.
- cmd % ? R
- Trying to resolve ('R',).
- Channel returned None. At least one dep could not be resolved.
+ (resolve) % !!
cmd % p
~dev-lang/R :: R language
cmd % >>
- Pool removed from resolver.
+ pool has been removed.
+
+ cmd % >>
+ resolver has no pools.
cmd % p
- cmd % ? R language
- Trying to resolve ('R language',).
- Channel returned None. At least one dep could not be resolved.
- cmd % exit
+ cmd % ?? R language
+ 'R language' could not be resolved.
+
+ cmd % set PS1=#!
+
+ #! exit
+
====================
@@ -3376,7 +3472,7 @@ Example Session:
====================
Roverlay provides an API for accessing its functionality independently of
-R overlay creation. Only dependency resolution is available, currently.
+R overlay creation.
Note, however, that a minimal config file may still be required for accessing
*roverlay interfaces*.
@@ -3439,7 +3535,16 @@ for usage examples.
Remote Interface
------------------
-<<TODO; this interface isn't mature enough yet (it will likely change in future)>>
+The *RemoteInterface* is experimental/incomplete and currently offers the
+following functionality:
+
+* set sync mode to online/offline
+* sync/nosync
+* load repo config
+* list repos
+* list repo packages
+
+Refer to in-code documentation for details.
=========================
@@ -3654,14 +3759,14 @@ would require variable interpolation, e.g. for ``${PN}``).
Ebuild creation is the process centered around one *PackageInfo* instance *p*
that tries to create an ebuild for it.
-It does the following steps:
+An *EbuildCreationJob* does the following steps:
#. Read the DESCRIPTION file of *p* R package tarball and stores the
data in an associative array ('DESCRIPTION field' -> 'data')
#. Call `dependency resolution`_
-#. If dependency resolution was successful and any *selfdeps* found
+#. If dependency resolution was successful and any selfdeps_ found
(dependencies on other packages): *pause* ebuild creation for *p* until
it has been verified whether these dependencies are satisfiable.
This is necessary because dependency resolution does not know whether a
@@ -3709,11 +3814,116 @@ afterwards. Overlay creation keeps going if an ebuild cannot be created,
instead the event is logged. Running more than one *OverlayWorker* in parallel
is possible.
+The following pseudo-code illustrates how overlay creation basically works:
+
+.. code-block:: text
+
+ ACCEPT_PACKAGES:
+
+ for each received PackageInfo <p>
+
+ create an EbuildCreationJob for p and add it to the work queue
+
+ end for
+
+
+
+ CREATE_OVERLAY:
+
+ while work_queue is not empty
+
+ work_queue_next <= empty
+
+
+ in parallel with N OverlayWorkers (>=0 threads):
+
+ for each EbuildCreationJob ebuild_job from the work_queue
+
+ run/resume ebuild_job (as described in Ebuild Creation)
+
+ if ebuild_job is paused
+
+ add ebuild_job to work_queue_next
+
+ end if
+
+ end for
+
+ end in parallel
+
+
+ if work_queue_next is not empty
+
+ <run selfdep validation>
+
+ work_queue <= work_queue_next
+
+ end if
+
+ end while
+
+
++++++++++++++++++++
Selfdep Validation
++++++++++++++++++++
-<<TODO: specify algo here>>
+EbuildCreationJobs are processed in no specific order and possibly
+concurrently. This leads to the problem that dependency resolution cannot
+know whether a successfully resolved selfdep_ is actually satisfiable.
+
+For example, if a package *A* depends on another package *B* and *B* is
+uncreatable (ebuild creation *will* fail for *B*), then dependency resolution
+still resolves *B* (e.g. as sci-R/B). *A* has a **dangling selfdep** now.
+
+*Selfdep validation* is the process of **identifying** *dangling selfdeps* and
+**removing** packages with such dependencies or simply **dropping**
+the dependencies, depending on their `dependency type`_.
+
+It is an algorithm with 3 phases:
+
+#. **prepare**: create a graph containing all selfdeps
+
+ * **collect**: find ebuild creation jobs with selfdeps (*direct* selfdeps)
+
+ * **link**: expand the selfdep graph, find selfdep-selfdep dependencies
+ (*indirect* selfdeps)
+
+#. **reduce**: find *dangling selfdeps* and mark them as invalid
+
+ * a selfdep is *dangling* iff the overlay contains no suitable
+ *PackageInfo* with valid selfdeps
+
+ * **repeat this step** until no more *dangling selfdeps* found
+
+#. **balance**: find ebuild creation jobs with invalid selfdeps (*inversed collect*)
+
+ * drop optional dependencies
+ * let ebuild creation fail if a mandatory selfdep is not valid
+
+
+The actual implementation in *roverlay* is spread across several modules:
+
+.. table:: modules/packages involved in selfdep validation
+
+ +-------------------------------+------------------------------------------+
+ | module/package | phase(s) / description |
+ +===============================+==========================================+
+ | roverlay.overlay.creator | **all** (controls selfdep validation), |
+ | | especially *collect* and |
+ | | *reduction loop* |
+ +-------------------------------+------------------------------------------+
+ | roverlay.ebuild.creation | **prepare** (*collect*), |
+ | roverlay.ebuild.depres | **balance** (*drop dependencies*, |
+ | | *ebuild creation failure*) |
+ +-------------------------------+------------------------------------------+
+ | roverlay.overlay.root | **prepare** (*link*), |
+ | roverlay.overlay.category | **balance** (*remove packages*) |
+ | roverlay.overlay.pkgdir | |
+ +-------------------------------+------------------------------------------+
+ | roverlay.depres.depresult | **all** (selfdep data object) |
+ +-------------------------------+------------------------------------------+
+ | roverlay.packageinfo | **all** (package data object), |
+ +-------------------------------+------------------------------------------+
-----------------------
@@ -3747,6 +3957,9 @@ From the ebuild creation perspective, dependency resolution works like this:
Details about dependency resolution like how *channels* work can be found
in the following subsections.
+
+.. _DEPENDENCY TYPE:
+
++++++++++++++++++
Dependency types
++++++++++++++++++
@@ -3969,13 +4182,15 @@ Its mode of operation of operation is best described in pseudo-code:
else
if <a rule pool accepts depenv's type and resolves depenv>
+
resolved <= True
else if <depenv's type contains TRY_OTHER>
-
if <a rule pool supports TRY_OTHER and does not accept depenv's type and resolves depenv>
resolved <= True
+
+ end if
end if
end if
^ permalink raw reply related [flat|nested] 41+ messages in thread* [gentoo-commits] proj/R_overlay:master commit in: doc/rst/
@ 2013-08-02 14:30 André Erdmann
0 siblings, 0 replies; 41+ messages in thread
From: André Erdmann @ 2013-08-02 14:30 UTC (permalink / raw
To: gentoo-commits
commit: da72a7ccfbf7b54d11ed50e554cd92dd8430f787
Author: André Erdmann <dywi <AT> mailerd <DOT> de>
AuthorDate: Fri Aug 2 14:27:53 2013 +0000
Commit: André Erdmann <dywi <AT> mailerd <DOT> de>
CommitDate: Fri Aug 2 14:27:53 2013 +0000
URL: http://git.overlays.gentoo.org/gitweb/?p=proj/R_overlay.git;a=commit;h=da72a7cc
doc/rst: hook env, OVERLAY_MASTERS config option
---
doc/rst/usage.rst | 38 +++++++++++++++++++++++++++-----------
1 file changed, 27 insertions(+), 11 deletions(-)
diff --git a/doc/rst/usage.rst b/doc/rst/usage.rst
index 68b8e40..dab9b8c 100644
--- a/doc/rst/usage.rst
+++ b/doc/rst/usage.rst
@@ -2113,13 +2113,20 @@ the config file. An empty string sets the policy to *deny all*.
+----------------+-------------------+-----------------------------------------+
| ROVERLAY_PHASE | event | event that caused the script to run |
+----------------+-------------------+-----------------------------------------+
+ | HAS_CHANGES | *internal* | a shbool (``y`` or ``n``) that |
+ | | | indicates whether the overlay has |
+ | | | any changes |
+ +----------------+-------------------+-----------------------------------------+
| OVERLAY_NAME | config | name of the overlay |
+----------------+-------------------+-----------------------------------------+
| OVERLAY | config | overlay directory (`OVERLAY_DIR`_), |
- +----------------+-------------------+ initial working directory |
- | S | *$OVERLAY* | |
+----------------+-------------------+ |
- | PWD | *$OVERLAY* | |
+ | S | *$OVERLAY* | |
+ +----------------+-------------------+-----------------------------------------+
+ | PWD | *$OVERLAY* | initial working directory |
+ | | *$ROVERLAY_PHASE* | |
+ | | | depends on $ROVERLAY_PHASE (usually set |
+ | | | to $OVERLAY or left unchanged) |
+----------------+-------------------+-----------------------------------------+
| DISTROOT | config | package mirror directory |
| | | (`OVERLAY_DISTDIR_ROOT`_) |
@@ -2337,17 +2344,19 @@ Shell function files should be put into ``<ADDITIONS_DIR>/shlib``.
Hook event table
------------------
-The following table lists all known events:
+The following table lists all known events (``ROVERLAY_PHASE``):
.. table::
- +-------------------+-------------+----------------------------------------+
- | name | conditional | description |
- +===================+=============+========================================+
- | overlay_success | yes | overlay creation succeeded |
- +-------------------+-------------+----------------------------------------+
- | db_written | yes | stats database file written |
- +-------------------+-------------+----------------------------------------+
+ +-------------------+---------------------------+------------------------------+
+ | name | initial working directory | description |
+ +===================+===========================+==============================+
+ | overlay_success | *$OVERLAY* | overlay creation succeeded |
+ +-------------------+---------------------------+------------------------------+
+ | db_written | *$OVERLAY* | stats database file written |
+ +-------------------+---------------------------+------------------------------+
+ | user | unchanged | user-triggered event |
+ +-------------------+---------------------------+------------------------------+
---------------------
@@ -2688,6 +2697,13 @@ OVERLAY_MANIFEST_IMPLEMENTATION
Use the '--no-manifest' command line option to disable manifest
writing.
+.. _OVERLAY_MASTERS:
+
+OVERLAY_MASTERS
+ A list of repo names that are used as 'masters' attribute when generating
+ ``<overlay>/metadata/layout.conf``.
+
+ Defaults to "gentoo".
.. _OVERLAY_NAME:
^ permalink raw reply related [flat|nested] 41+ messages in thread* [gentoo-commits] proj/R_overlay:master commit in: doc/rst/
@ 2013-08-01 17:14 André Erdmann
0 siblings, 0 replies; 41+ messages in thread
From: André Erdmann @ 2013-08-01 17:14 UTC (permalink / raw
To: gentoo-commits
commit: fbf04e6b3afb7909cded6b5700513fd80ef0e5ed
Author: André Erdmann <dywi <AT> mailerd <DOT> de>
AuthorDate: Thu Aug 1 17:11:05 2013 +0000
Commit: André Erdmann <dywi <AT> mailerd <DOT> de>
CommitDate: Thu Aug 1 17:11:05 2013 +0000
URL: http://git.overlays.gentoo.org/gitweb/?p=proj/R_overlay.git;a=commit;h=fbf04e6b
doc/rst, package rules: else-action block
Other changes included in this commit:
* --no-revbump, --dump-stats cmdline options
* hook environment: $STATS_DB
* hook event table
* field definition file: update joinValues description + [License] field
* Dependency Resolution: is a subsection of Roverlay Console now,
TODO notes added
* added doc stub for RemoteInterface
---
doc/rst/usage.rst | 194 ++++++++++++++++++++++++++++++++++++++++++------------
1 file changed, 153 insertions(+), 41 deletions(-)
diff --git a/doc/rst/usage.rst b/doc/rst/usage.rst
index 204de09..68b8e40 100644
--- a/doc/rst/usage.rst
+++ b/doc/rst/usage.rst
@@ -439,6 +439,9 @@ to know in detail what *roverlay* does before running it.
--no-incremental
Force recreation of existing ebuilds
+--no-revbump
+ Disable revbump checks in incremental overlay creation mode
+
--immediate-ebuild-writes
Immediately write ebuilds when they are ready.
@@ -481,6 +484,9 @@ For **testing** *roverlay*, these **options** may be convenient:
--no-write
Disable overlay writing
+--dump-stats
+ Print all stats
+
--show
Print all ebuilds and metadata to console
@@ -1520,7 +1526,7 @@ Package Rules can be used to control both overlay and ebuild creation.
Each package rule consists of conditions, e.g. *package name contains amd64*,
and actions, e.g. *set KEYWORDS="-x86 amd64"*.
The actions of a rule will only be applied if a package meets all conditions,
-otherwise the rule does nothing.
+otherwise the rule applies alternative actions (if defined) or does nothing.
Moreover, rules can contain further rules which will only take effect if all
enclosing rules match a given package.
@@ -1531,11 +1537,14 @@ enclosing rules match a given package.
As stated above, each rule has two parts, a *match block* that lists the
rule's conditions and an *action block* that defines which actions and
nested rules are applied to a package if the rule matches it, i.e. if all
-conditions are met.
+conditions are met. A rule can also contain an *alternative action block*
+whose actions are applied to a package if the rule does not match it.
A rule file contains zero or more package rules.
-Each rule has to declare one *match* and one *action statement* at least.
-The basic syntax for a rule with 1..m *match* and 1..n *action statements* is
+Each rule has to declare one *match* and one
+*[alternative] action statement* at least.
+The basic syntax for a rule with 1..m *match*, 1..n *action statements* and
+1..k *alternative action statements* is
.. code::
@@ -1549,6 +1558,11 @@ The basic syntax for a rule with 1..m *match* and 1..n *action statements* is
<action statement 2>
...
<action statement n>
+ ELSE:
+ <alternative action statement 1>
+ <alternative action statement 2>
+ ...
+ <alternative action statement k>
END;
@@ -1556,7 +1570,9 @@ A rule is introduced by the ``MATCH:`` keyword, which starts the
*match block* and is followed by one or more *match statements*, one per line.
The *match block* ends with the ``ACTION:`` keyword, which also starts the
*action block* and is followed by one or more *action statements*
-(again, one per line). Finally, the rule is terminated by the ``END;`` keyword.
+(again, one per line). The *alternative action block* introduced by the
+``ELSE:`` keyword is optional and lists *action statements*.
+Finally, the rule is terminated by the ``END;`` keyword.
Indention is purely optional, leading and ending whitespace will be discarded.
Lines starting with ``#`` or ``;`` are considered to be comments and will be
@@ -1774,7 +1790,7 @@ control *where*) and the number of values they accept:
+----------------+------------------+-------------+------------------------+
| trace | package rules | none | mark a package as |
| | | | modified |
- + + +-------------+------------------------+
+ | | +-------------+------------------------+
| | | 1 | add the stored string |
| | | | to a package's |
| | | | *modified* variable |
@@ -1791,6 +1807,10 @@ control *where*) and the number of values they accept:
| | | | *s/expr/repl/* |
| | | | statements |
+----------------+------------------+-------------+------------------------+
+ | null | *n/a* | none | does nothing |
+ +----------------+ | | (can be used for |
+ | pass | | | readability) |
+ +----------------+------------------+-------------+------------------------+
The two-arg form of the set/rename keywords expect a <key> as first and
a value / sed expression as second arg. The one-arg form expects the latter
@@ -1851,6 +1871,19 @@ is exactly the same as for the normal package rules:
END;
# top-level rule, action block continues
...
+ ELSE:
+ # top-level rule, alternative action block
+ ...
+ MATCH:
+ # (alternative) nested rule, match block
+ ...
+ ACTION:
+ # (alternative) nested rule, action block
+ ...
+ ELSE:
+ # (alternative) nested rule, alternative action block
+ ...
+ END;
END;
Rules can be nested indefinitely, whitespace indention is optional.
@@ -1863,15 +1896,21 @@ checks necessary for a given package.
+++++++++++++++++++++++
A rule that ignores the 'yaqcaffy' package from CRAN, which is also available
-from BIOC:
+from BIOC. Additionally, it sets the category for all non-ignored packages
+from CRAN to sci-CRAN:
.. code::
MATCH:
- repo == CRAN
- package_name == yaqcaffy
+ repo CRAN
ACTION:
- do-not-process
+ MATCH:
+ package_name == yaqcaffy
+ ACTION:
+ do-not-process
+ ELSE:
+ set category sci-CRAN
+ END;
END;
@@ -1886,11 +1925,12 @@ if the package is from BIOC/experiment, and otherwise to ``-x86 amd64``:
* package_name ~ x86_64
* package_name ~ amd64
ACTION:
- keywords "-x86 amd64"
MATCH:
repo == BIOC/experiment
ACTION:
keywords "-x86 ~amd64"
+ ELSE:
+ keywords "-x86 amd64"
END;
END;
@@ -1935,6 +1975,15 @@ Moving such packages to a "R-package" sub directory would be possible, too:
END;
+.. Hint::
+
+ ``roverlay [--nosync] [--dump-file <file>] apply_rules`` can be used to
+ test rules. It applies the rules to all packages without running overlay
+ creation. Furthermore, ``roverlay --ppr`` parses the package rules,
+ prints them and exits afterwards.
+
+
+
=============
Event Hooks
=============
@@ -2108,6 +2157,9 @@ the config file. An empty string sets the policy to *deny all*.
| | *$ROVERLAY\ | executable (which creates the overlay) |
| | _HELPER_EXE* | |
+----------------+-------------------+-----------------------------------------+
+ | STATS_DB | config | stats database file |
+ | | | (optional, only set if configured) |
+ +----------------+-------------------+-----------------------------------------+
| DEBUG | log level | *shbool* (``y`` or ``n``) that |
| | | indicates whether debug messages should |
| | | be printed |
@@ -2281,6 +2333,23 @@ The template below illustrates how to write function files:
Shell function files should be put into ``<ADDITIONS_DIR>/shlib``.
+------------------
+ Hook event table
+------------------
+
+The following table lists all known events:
+
+.. table::
+
+ +-------------------+-------------+----------------------------------------+
+ | name | conditional | description |
+ +===================+=============+========================================+
+ | overlay_success | yes | overlay creation succeeded |
+ +-------------------+-------------+----------------------------------------+
+ | db_written | yes | stats database file written |
+ +-------------------+-------------+----------------------------------------+
+
+
---------------------
Adding a hook event
---------------------
@@ -3033,9 +3102,9 @@ Known field flags:
joinValues
Declares that a field's value is one string even if it spans over
- multiple lines.
- The lines will be joined with a single space character ' '.
- The default behavior is to merge lines.
+ multiple lines. The lines will be joined with a single space
+ character ' '. The default behavior is to merge lines.
+ This flag can be used in conjunction with any "is list" flag.
.. _field flag\: isList:
@@ -3120,19 +3189,42 @@ This is the default field definition file (without any ignored fields):
alias_nocase = OS_TYPE
allowed_values = unix
+ [License]
+ alias_nocase = License, Licence, Lisence
+ isLicense
+
+
+.. _Roverlay Console:
+
+==================
+ Roverlay Console
+==================
+
+<<section is TODO>>
+
+<<links to depres console chapter may need to be fixed>>
+
+<<basic commands, table>>
+
+<<note regarding python -OO and missing help texts>>
.. _DepRes Console:
-===============================
+-------------------------------
Dependency Resolution Console
-===============================
+-------------------------------
+
+.. Warning::
+
+ This section is out-of-date.
+
As previously stated, the *DepRes Console* is only meant for **testing**.
It is an interactive console with the following features:
* resolve dependencies
-* create new dependency rules (**only single line rules**)
+* create new dependency rules
* create dependency rules for each R package found in a directory
* load rules from files
* save rules to a file
@@ -3146,14 +3238,13 @@ lists all available commands and a few usage hints.
For reference, these commands are currently available:
+<<TODO: rewrite/update command table, it's out-of-date>>
+
+---------------------+----------------------------------------------------+
| command | description |
+=====================+====================================================+
| help, | lists all commands |
-| h | |
-+---------------------+----------------------------------------------------+
-| help --list, | lists all help topics for which help is available |
-| h --list | |
+| h, ? | |
+---------------------+----------------------------------------------------+
| help *<cmd>*, | prints a command-specific help message |
| h *<cmd>* | |
@@ -3164,46 +3255,58 @@ For reference, these commands are currently available:
| load_conf, | loads the rule files listed in the config file |
| lc | into a new *rule pool* |
+---------------------+----------------------------------------------------+
-| addrule *<rule>* | creates a new rule and adds it to the topmost, |
+| add_rule *<rule>* | creates a new rule and adds it to the topmost, |
| + *<rule>* | i.e. latest *rule pool*. This command uses |
-| | `Rule File Syntax`_, but multi line rules are |
-| | not supported. |
+| | `Rule File Syntax`_. Multi line rules are |
+| | supported. |
+---------------------+----------------------------------------------------+
| add_pool, | creates a new *rule pool* |
| << | |
+---------------------+----------------------------------------------------+
-| unwind, | removes the topmost *rule pool* and all of its |
+| unwind_pool, | removes the topmost *rule pool* and all of its |
| >> | rules |
+---------------------+----------------------------------------------------+
| resolve *<dep>*, | tries to resolve the given dependency string and |
-| ? *<dep>* | prints the result |
+| ?? *<dep>* | prints the result |
++---------------------+----------------------------------------------------+
+| ! | enter the resolve *command chroot* |
+| | all input will be prefixed with "resolve" |
++---------------------+----------------------------------------------------+
+| !! | leave any *command chroot* |
+---------------------+----------------------------------------------------+
| print, p | prints the rules of the topmost *rule pool* |
+---------------------+----------------------------------------------------+
-| print all, p all | prints the rules of all *rule pools* |
+| print --all|-a | prints the rules of all *rule pools* |
++---------------------+----------------------------------------------------+
+| print <id> [<id>..] | prints the rules of a specific *rule pool* |
+---------------------+----------------------------------------------------+
| write *<file>*, | writes the rules of the topmost *rule pool* into |
-| w *<file>* | *<file>* |
+| w *<file>* | *<file>*. See write --help for advanced usage. |
++---------------------+----------------------------------------------------+
+| cd *<dir>* | changes the working directory |
+| | This is a virtual command. <<TODO:EXPLAIN>> |
+---------------------+----------------------------------------------------+
-| cd *<dir>* | changes the working directory, also creates it if |
-| | necessary |
+| set VAR=VALUE | set variables |
+---------------------+----------------------------------------------------+
-| scandir *<dir>*, | creates dependency rules for each R package found |
-| sd *<dir>* | in *<dir>* |
+| unset VAR | unset variables |
+---------------------+----------------------------------------------------+
-| set, unset | prints the status of currently (in)active modes |
+| declare | show declared variables |
+---------------------+----------------------------------------------------+
-| set *<mode>*, | sets or unsets *<mode>*. There is only one mode to |
-| unset *<mode>* | control, the *shlex mode* which controls how |
-| | command arguments are parsed |
+| alias | show command aliases |
+---------------------+----------------------------------------------------+
-| mkhelp | verifies that each accessible command has a help |
-| | message |
+| unalias | reserved for future usage |
+---------------------+----------------------------------------------------+
| exit, qq, q | exits the *DepRes Console* |
+---------------------+----------------------------------------------------+
+.. Note::
+
+ Some commands also offer more detailed help via ``<command> --help``.
+
+<<TODO>>
+
+<<Example Session is out-of-date>>
Example Session:
.. code-block::
@@ -3277,6 +3380,8 @@ The table below lists all interfaces and where to find them:
+-----------------+---------------------------+-----------------------------+
| DepresInterface | roverlay.interface.depres | dependency resolution |
+-----------------+---------------------------+-----------------------------+
+ | RemoteInterface | roverlay.interface.remote | remote interaction (sync) |
+ +-----------------+---------------------------+-----------------------------+
For extending the API, roverlay provides the abstract *RoverlayInterface* and
@@ -3310,9 +3415,16 @@ The *DepResInterface* offers the following functionality:
Refer to in-code documentation on how to use this interface.
-See the dependency resolution test suite for a usage example
-(``tests.depres``, not part of the roverlay installation).
-(The depres console is also a candidate for using this interface in future.)
+See the dependency resolution test suite (``tests.depres``, not part of the
+roverlay installation) and the dependency resolution console
+for usage examples.
+
+------------------
+ Remote Interface
+------------------
+
+<<TODO; this interface isn't mature enough yet (it will likely change in future)>>
+
=========================
Implementation Overview
^ permalink raw reply related [flat|nested] 41+ messages in thread* [gentoo-commits] proj/R_overlay:gsoc13/next commit in: doc/rst/
@ 2013-07-23 18:34 André Erdmann
2013-07-23 18:34 ` [gentoo-commits] proj/R_overlay:master " André Erdmann
0 siblings, 1 reply; 41+ messages in thread
From: André Erdmann @ 2013-07-23 18:34 UTC (permalink / raw
To: gentoo-commits
commit: 1dc64039a45eb41f785f9f64a4ddbf9464e018ba
Author: André Erdmann <dywi <AT> mailerd <DOT> de>
AuthorDate: Tue Jul 23 18:33:54 2013 +0000
Commit: André Erdmann <dywi <AT> mailerd <DOT> de>
CommitDate: Tue Jul 23 18:33:54 2013 +0000
URL: http://git.overlays.gentoo.org/gitweb/?p=proj/R_overlay.git;a=commit;h=1dc64039
doc/rst: add hook environment changes
---
doc/rst/usage.rst | 55 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 55 insertions(+)
diff --git a/doc/rst/usage.rst b/doc/rst/usage.rst
index 01c3411..204de09 100644
--- a/doc/rst/usage.rst
+++ b/doc/rst/usage.rst
@@ -542,6 +542,43 @@ to symbolic links if hard links are not supported. This should be fine in
most cases, but fine-tuning can be done via OVERLAY_DISTDIR_STRATEGY_ and
OVERLAY_DISTDIR_FLAT_.
+------------------
+ roverlay helpers
+------------------
+
+An installation of roverlay includes some helper programs:
+
+*roverlay-sh*
+ a wrapper around /bin/sh that runs a shell or shell script in roverlay's
+ `hook environment`_.
+
+ roverlay-related scripts can use it to automatically inherit roverlay's
+ config and ``$FUNCTIONS`` file:
+
+ .. code-block:: sh
+
+ #!/usr/bin/roverlay-sh
+
+ # reset DEBUG, VERBOSE, QUIET
+ DEBUG=n; QUIET=n; VERBOSE=y
+
+ # load the functions file (optional)
+ . "${FUNCTIONS?}" || exit
+
+ # script body
+ true
+
+ <<TODO: maybe there's a better place for the details>>
+
+
+*name_is_todo--roverlay_creation_helper*
+ <<TODO>>
+ Safely runs overlay creation <<and $$afterwards>>.
+ Suitable for being run as cron job.
+
+
+
+
===============================
Basic Implementation Overview
===============================
@@ -2027,6 +2064,8 @@ the config file. An empty string sets the policy to *deny all*.
+----------------+-------------------+-----------------------------------------+
| ROVERLAY_PHASE | event | event that caused the script to run |
+----------------+-------------------+-----------------------------------------+
+ | OVERLAY_NAME | config | name of the overlay |
+ +----------------+-------------------+-----------------------------------------+
| OVERLAY | config | overlay directory (`OVERLAY_DIR`_), |
+----------------+-------------------+ initial working directory |
| S | *$OVERLAY* | |
@@ -2046,6 +2085,9 @@ the config file. An empty string sets the policy to *deny all*.
+----------------+-------------------+ |
| FILESDIR | *$ADDITIONS_DIR* | |
+----------------+-------------------+-----------------------------------------+
+ | WORKDIR | config | directory for work data |
+ | | | (`CACHEDIR`_) |
+ +----------------+-------------------+-----------------------------------------+
| SHLIB | *$ADDITIONS_DIR*, | A list of directories with shell |
| | *installed?* | function files |
| | | (optional, only set if any dir exists) |
@@ -2053,6 +2095,19 @@ the config file. An empty string sets the policy to *deny all*.
| FUNCTIONS | *$ADDITIONS_DIR*, | file with essential shell functions |
| | *installed?* | (optional, only set if it exists) |
+----------------+-------------------+-----------------------------------------+
+ | DATADIR | *installed?* | location of installed data files |
+ | | | (directory, usually |
+ | | | */usr/share/roverlay*) |
+ +----------------+-------------------+-----------------------------------------+
+ | ROVERLAY\ | sys.argv | path to the roverlay executable that |
+ | _HELPER_EXE | | created the hook environment |
+ | | | (usually */usr/bin/roverlay* or |
+ | | | */usr/bin/roverlay-sh*) |
+ +----------------+-------------------+-----------------------------------------+
+ | ROVERLAY_EXE | guessed, | guessed path to the roverlay "main" |
+ | | *$ROVERLAY\ | executable (which creates the overlay) |
+ | | _HELPER_EXE* | |
+ +----------------+-------------------+-----------------------------------------+
| DEBUG | log level | *shbool* (``y`` or ``n``) that |
| | | indicates whether debug messages should |
| | | be printed |
^ permalink raw reply related [flat|nested] 41+ messages in thread* [gentoo-commits] proj/R_overlay:master commit in: doc/rst/
2013-07-23 18:34 [gentoo-commits] proj/R_overlay:gsoc13/next " André Erdmann
@ 2013-07-23 18:34 ` André Erdmann
0 siblings, 0 replies; 41+ messages in thread
From: André Erdmann @ 2013-07-23 18:34 UTC (permalink / raw
To: gentoo-commits
commit: 1dc64039a45eb41f785f9f64a4ddbf9464e018ba
Author: André Erdmann <dywi <AT> mailerd <DOT> de>
AuthorDate: Tue Jul 23 18:33:54 2013 +0000
Commit: André Erdmann <dywi <AT> mailerd <DOT> de>
CommitDate: Tue Jul 23 18:33:54 2013 +0000
URL: http://git.overlays.gentoo.org/gitweb/?p=proj/R_overlay.git;a=commit;h=1dc64039
doc/rst: add hook environment changes
---
doc/rst/usage.rst | 55 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 55 insertions(+)
diff --git a/doc/rst/usage.rst b/doc/rst/usage.rst
index 01c3411..204de09 100644
--- a/doc/rst/usage.rst
+++ b/doc/rst/usage.rst
@@ -542,6 +542,43 @@ to symbolic links if hard links are not supported. This should be fine in
most cases, but fine-tuning can be done via OVERLAY_DISTDIR_STRATEGY_ and
OVERLAY_DISTDIR_FLAT_.
+------------------
+ roverlay helpers
+------------------
+
+An installation of roverlay includes some helper programs:
+
+*roverlay-sh*
+ a wrapper around /bin/sh that runs a shell or shell script in roverlay's
+ `hook environment`_.
+
+ roverlay-related scripts can use it to automatically inherit roverlay's
+ config and ``$FUNCTIONS`` file:
+
+ .. code-block:: sh
+
+ #!/usr/bin/roverlay-sh
+
+ # reset DEBUG, VERBOSE, QUIET
+ DEBUG=n; QUIET=n; VERBOSE=y
+
+ # load the functions file (optional)
+ . "${FUNCTIONS?}" || exit
+
+ # script body
+ true
+
+ <<TODO: maybe there's a better place for the details>>
+
+
+*name_is_todo--roverlay_creation_helper*
+ <<TODO>>
+ Safely runs overlay creation <<and $$afterwards>>.
+ Suitable for being run as cron job.
+
+
+
+
===============================
Basic Implementation Overview
===============================
@@ -2027,6 +2064,8 @@ the config file. An empty string sets the policy to *deny all*.
+----------------+-------------------+-----------------------------------------+
| ROVERLAY_PHASE | event | event that caused the script to run |
+----------------+-------------------+-----------------------------------------+
+ | OVERLAY_NAME | config | name of the overlay |
+ +----------------+-------------------+-----------------------------------------+
| OVERLAY | config | overlay directory (`OVERLAY_DIR`_), |
+----------------+-------------------+ initial working directory |
| S | *$OVERLAY* | |
@@ -2046,6 +2085,9 @@ the config file. An empty string sets the policy to *deny all*.
+----------------+-------------------+ |
| FILESDIR | *$ADDITIONS_DIR* | |
+----------------+-------------------+-----------------------------------------+
+ | WORKDIR | config | directory for work data |
+ | | | (`CACHEDIR`_) |
+ +----------------+-------------------+-----------------------------------------+
| SHLIB | *$ADDITIONS_DIR*, | A list of directories with shell |
| | *installed?* | function files |
| | | (optional, only set if any dir exists) |
@@ -2053,6 +2095,19 @@ the config file. An empty string sets the policy to *deny all*.
| FUNCTIONS | *$ADDITIONS_DIR*, | file with essential shell functions |
| | *installed?* | (optional, only set if it exists) |
+----------------+-------------------+-----------------------------------------+
+ | DATADIR | *installed?* | location of installed data files |
+ | | | (directory, usually |
+ | | | */usr/share/roverlay*) |
+ +----------------+-------------------+-----------------------------------------+
+ | ROVERLAY\ | sys.argv | path to the roverlay executable that |
+ | _HELPER_EXE | | created the hook environment |
+ | | | (usually */usr/bin/roverlay* or |
+ | | | */usr/bin/roverlay-sh*) |
+ +----------------+-------------------+-----------------------------------------+
+ | ROVERLAY_EXE | guessed, | guessed path to the roverlay "main" |
+ | | *$ROVERLAY\ | executable (which creates the overlay) |
+ | | _HELPER_EXE* | |
+ +----------------+-------------------+-----------------------------------------+
| DEBUG | log level | *shbool* (``y`` or ``n``) that |
| | | indicates whether debug messages should |
| | | be printed |
^ permalink raw reply related [flat|nested] 41+ messages in thread
* [gentoo-commits] proj/R_overlay:gsoc13/next commit in: doc/rst/
@ 2013-07-23 14:57 André Erdmann
2013-07-23 14:57 ` [gentoo-commits] proj/R_overlay:master " André Erdmann
0 siblings, 1 reply; 41+ messages in thread
From: André Erdmann @ 2013-07-23 14:57 UTC (permalink / raw
To: gentoo-commits
commit: c9055b8a4cc87a7518710bd7a2ff7a1f57ad97ee
Author: André Erdmann <dywi <AT> mailerd <DOT> de>
AuthorDate: Tue Jul 23 14:30:05 2013 +0000
Commit: André Erdmann <dywi <AT> mailerd <DOT> de>
CommitDate: Tue Jul 23 14:30:42 2013 +0000
URL: http://git.overlays.gentoo.org/gitweb/?p=proj/R_overlay.git;a=commit;h=c9055b8a
doc/rst: "fix" example 3 in dependency rules
Due to changes in the depresolver, the old example does no longer work.
---
doc/rst/usage.rst | 11 ++++-------
1 file changed, 4 insertions(+), 7 deletions(-)
diff --git a/doc/rst/usage.rst b/doc/rst/usage.rst
index df18824..01c3411 100644
--- a/doc/rst/usage.rst
+++ b/doc/rst/usage.rst
@@ -1230,16 +1230,13 @@ Example 2 - *default* simple rule stub
be resolved. See *Selfdep* in `Rule File Syntax`_ for details.
Example 3 - *default* simple rule
- A rule that matches several *dependency strings* and resolves them
- as "sci-libs/gdal and sci-libs/proj":
+ A rule that matches a *dependency string* and resolves it
+ as "virtual/blas and virtual/lapack":
.. code-block:: text
- ( sci-libs/gdal sci-libs/proj ) {
- for building from source: GDAL >= 1.3.1 && GDAL < 1.6.0 (until tested) library and PROJ.4 (proj >= 4.4.9)
- for building from source: GDAL >= 1.3.1 library and PROJ.4 (proj >= 4.4.9)
- for building from source: GDAL >= 1.3.1 library and PROJ.4(proj >= 4.4.9)
- for building from source: GDAL >= 1.6.0 library and PROJ.4(proj >= 4.4.9)
+ ( virtual/blas virtual/lapack ) {
+ BLAS/LAPACK libraries
}
Example 4 - *ignore* simple rule
^ permalink raw reply related [flat|nested] 41+ messages in thread* [gentoo-commits] proj/R_overlay:master commit in: doc/rst/
2013-07-23 14:57 [gentoo-commits] proj/R_overlay:gsoc13/next " André Erdmann
@ 2013-07-23 14:57 ` André Erdmann
0 siblings, 0 replies; 41+ messages in thread
From: André Erdmann @ 2013-07-23 14:57 UTC (permalink / raw
To: gentoo-commits
commit: c9055b8a4cc87a7518710bd7a2ff7a1f57ad97ee
Author: André Erdmann <dywi <AT> mailerd <DOT> de>
AuthorDate: Tue Jul 23 14:30:05 2013 +0000
Commit: André Erdmann <dywi <AT> mailerd <DOT> de>
CommitDate: Tue Jul 23 14:30:42 2013 +0000
URL: http://git.overlays.gentoo.org/gitweb/?p=proj/R_overlay.git;a=commit;h=c9055b8a
doc/rst: "fix" example 3 in dependency rules
Due to changes in the depresolver, the old example does no longer work.
---
doc/rst/usage.rst | 11 ++++-------
1 file changed, 4 insertions(+), 7 deletions(-)
diff --git a/doc/rst/usage.rst b/doc/rst/usage.rst
index df18824..01c3411 100644
--- a/doc/rst/usage.rst
+++ b/doc/rst/usage.rst
@@ -1230,16 +1230,13 @@ Example 2 - *default* simple rule stub
be resolved. See *Selfdep* in `Rule File Syntax`_ for details.
Example 3 - *default* simple rule
- A rule that matches several *dependency strings* and resolves them
- as "sci-libs/gdal and sci-libs/proj":
+ A rule that matches a *dependency string* and resolves it
+ as "virtual/blas and virtual/lapack":
.. code-block:: text
- ( sci-libs/gdal sci-libs/proj ) {
- for building from source: GDAL >= 1.3.1 && GDAL < 1.6.0 (until tested) library and PROJ.4 (proj >= 4.4.9)
- for building from source: GDAL >= 1.3.1 library and PROJ.4 (proj >= 4.4.9)
- for building from source: GDAL >= 1.3.1 library and PROJ.4(proj >= 4.4.9)
- for building from source: GDAL >= 1.6.0 library and PROJ.4(proj >= 4.4.9)
+ ( virtual/blas virtual/lapack ) {
+ BLAS/LAPACK libraries
}
Example 4 - *ignore* simple rule
^ permalink raw reply related [flat|nested] 41+ messages in thread
* [gentoo-commits] proj/R_overlay:gsoc13/next commit in: doc/rst/
@ 2013-07-16 16:35 André Erdmann
2013-07-16 16:36 ` [gentoo-commits] proj/R_overlay:master " André Erdmann
0 siblings, 1 reply; 41+ messages in thread
From: André Erdmann @ 2013-07-16 16:35 UTC (permalink / raw
To: gentoo-commits
commit: d7db304d7a087f04613cc0d62cf0d39504311f14
Author: André Erdmann <dywi <AT> mailerd <DOT> de>
AuthorDate: Tue Jul 16 16:33:22 2013 +0000
Commit: André Erdmann <dywi <AT> mailerd <DOT> de>
CommitDate: Tue Jul 16 16:33:22 2013 +0000
URL: http://git.overlays.gentoo.org/gitweb/?p=proj/R_overlay.git;a=commit;h=d7db304d
doc/rst: api, manifest creation
---
doc/rst/usage.rst | 77 +++++++++++++++++++++++++++++++++++++++++++++++--------
1 file changed, 67 insertions(+), 10 deletions(-)
diff --git a/doc/rst/usage.rst b/doc/rst/usage.rst
index 36fa031..df18824 100644
--- a/doc/rst/usage.rst
+++ b/doc/rst/usage.rst
@@ -2311,8 +2311,6 @@ CACHEDIR
This option is **required**.
- <<TODO: default value!>>
-
.. _DISTFILES:
DISTFILES
@@ -3206,9 +3204,63 @@ Example Session:
Roverlay Interface
====================
-<<TODO: stub only>>
+Roverlay provides an API for accessing its functionality independently of
+R overlay creation. Only dependency resolution is available, currently.
+
+Note, however, that a minimal config file may still be required for accessing
+*roverlay interfaces*.
+
+The table below lists all interfaces and where to find them:
+
+.. table:: roverlay interfaces
+
+ +-----------------+---------------------------+-----------------------------+
+ | name | module | description |
+ +=================+===========================+=============================+
+ | RootInterface | roverlay.interface.root | meta interface for managing |
+ | | | other interfaces |
+ +-----------------+---------------------------+-----------------------------+
+ | MainInterface | roverlay.interface.main | RootInterface with delayed |
+ | | | initialization |
+ +-----------------+---------------------------+-----------------------------+
+ | DepresInterface | roverlay.interface.depres | dependency resolution |
+ +-----------------+---------------------------+-----------------------------+
+
+
+For extending the API, roverlay provides the abstract *RoverlayInterface* and
+*RoverlaySubInterface* classes.
+
+
+------------------
+ DepRes Interface
+------------------
+
+The *DepResInterface* offers the following functionality:
+
+* rule creation
+
+ * load rules from text input, files and/or configured files (SIMPLE_RULES_FILE_)
+
+ * *compile* rules after loading them
+
+* manage dependency rule pools (container for rules)
+
+ * rule pools are kept in a stack-like data structure, which makes it easy
+ to "forget" the most recent rules (*discard_pool()* et al.)
-API to roverlay functionality. Currently supports dependency resolution.
+ * *visualize* pools: convert rules into text (inverse of rule creation)
+
+* easy access to the resolver via *resolve()*, *can_resolve()*
+ and *cannot_resolve()*
+
+* "low-level" access to dependency resolution, e.g. *EbuildJobChannels*, is
+ also possible
+
+
+Refer to in-code documentation on how to use this interface.
+See the dependency resolution test suite for a usage example
+(``tests.depres``, not part of the roverlay installation).
+(The depres console is also a candidate for using this interface in future.)
=========================
Implementation Overview
@@ -3401,13 +3453,18 @@ for which an ebuild has been created.
Manifest Creation
+++++++++++++++++++
-<<TODO:this section is totally out of date>>
+Two implementations for writing manifest files are available, *ebuild* and
+*next*.
+
+The *ebuild* implementation calls the ``ebuild`` executable for each package
+directory in a filtered environment where where FETCHCOMMAND and
+RESUMECOMMAND are set to no-operation (/bin/true on systems that support it).
-Manifest files are created by calling the *ebuild* executable for each
-package directory in a filtered environment where FETCHCOMMAND and
-RESUMECOMMAND are set to no-operation. The directories that contain the R
-package files are passed in the PORTAGE_RO_DISTDIRS variable and DISTDIR
-is set to `DISTFILES_ROOT`_/__tmp__.
+*next* is an internal implementation that uses the
+``roverlay.overlay.pkgdir.manifest`` module for creating Manifest files.
+It falls back to *ebuild* when creating Manifest entries for imported ebuilds,
+since roverlay does not support reading *SRC_URI* from ebuilds reliably (that
+would require variable interpolation, e.g. for ``${PN}``).
-----------------
^ permalink raw reply related [flat|nested] 41+ messages in thread* [gentoo-commits] proj/R_overlay:master commit in: doc/rst/
2013-07-16 16:35 [gentoo-commits] proj/R_overlay:gsoc13/next " André Erdmann
@ 2013-07-16 16:36 ` André Erdmann
0 siblings, 0 replies; 41+ messages in thread
From: André Erdmann @ 2013-07-16 16:36 UTC (permalink / raw
To: gentoo-commits
commit: d7db304d7a087f04613cc0d62cf0d39504311f14
Author: André Erdmann <dywi <AT> mailerd <DOT> de>
AuthorDate: Tue Jul 16 16:33:22 2013 +0000
Commit: André Erdmann <dywi <AT> mailerd <DOT> de>
CommitDate: Tue Jul 16 16:33:22 2013 +0000
URL: http://git.overlays.gentoo.org/gitweb/?p=proj/R_overlay.git;a=commit;h=d7db304d
doc/rst: api, manifest creation
---
doc/rst/usage.rst | 77 +++++++++++++++++++++++++++++++++++++++++++++++--------
1 file changed, 67 insertions(+), 10 deletions(-)
diff --git a/doc/rst/usage.rst b/doc/rst/usage.rst
index 36fa031..df18824 100644
--- a/doc/rst/usage.rst
+++ b/doc/rst/usage.rst
@@ -2311,8 +2311,6 @@ CACHEDIR
This option is **required**.
- <<TODO: default value!>>
-
.. _DISTFILES:
DISTFILES
@@ -3206,9 +3204,63 @@ Example Session:
Roverlay Interface
====================
-<<TODO: stub only>>
+Roverlay provides an API for accessing its functionality independently of
+R overlay creation. Only dependency resolution is available, currently.
+
+Note, however, that a minimal config file may still be required for accessing
+*roverlay interfaces*.
+
+The table below lists all interfaces and where to find them:
+
+.. table:: roverlay interfaces
+
+ +-----------------+---------------------------+-----------------------------+
+ | name | module | description |
+ +=================+===========================+=============================+
+ | RootInterface | roverlay.interface.root | meta interface for managing |
+ | | | other interfaces |
+ +-----------------+---------------------------+-----------------------------+
+ | MainInterface | roverlay.interface.main | RootInterface with delayed |
+ | | | initialization |
+ +-----------------+---------------------------+-----------------------------+
+ | DepresInterface | roverlay.interface.depres | dependency resolution |
+ +-----------------+---------------------------+-----------------------------+
+
+
+For extending the API, roverlay provides the abstract *RoverlayInterface* and
+*RoverlaySubInterface* classes.
+
+
+------------------
+ DepRes Interface
+------------------
+
+The *DepResInterface* offers the following functionality:
+
+* rule creation
+
+ * load rules from text input, files and/or configured files (SIMPLE_RULES_FILE_)
+
+ * *compile* rules after loading them
+
+* manage dependency rule pools (container for rules)
+
+ * rule pools are kept in a stack-like data structure, which makes it easy
+ to "forget" the most recent rules (*discard_pool()* et al.)
-API to roverlay functionality. Currently supports dependency resolution.
+ * *visualize* pools: convert rules into text (inverse of rule creation)
+
+* easy access to the resolver via *resolve()*, *can_resolve()*
+ and *cannot_resolve()*
+
+* "low-level" access to dependency resolution, e.g. *EbuildJobChannels*, is
+ also possible
+
+
+Refer to in-code documentation on how to use this interface.
+See the dependency resolution test suite for a usage example
+(``tests.depres``, not part of the roverlay installation).
+(The depres console is also a candidate for using this interface in future.)
=========================
Implementation Overview
@@ -3401,13 +3453,18 @@ for which an ebuild has been created.
Manifest Creation
+++++++++++++++++++
-<<TODO:this section is totally out of date>>
+Two implementations for writing manifest files are available, *ebuild* and
+*next*.
+
+The *ebuild* implementation calls the ``ebuild`` executable for each package
+directory in a filtered environment where where FETCHCOMMAND and
+RESUMECOMMAND are set to no-operation (/bin/true on systems that support it).
-Manifest files are created by calling the *ebuild* executable for each
-package directory in a filtered environment where FETCHCOMMAND and
-RESUMECOMMAND are set to no-operation. The directories that contain the R
-package files are passed in the PORTAGE_RO_DISTDIRS variable and DISTDIR
-is set to `DISTFILES_ROOT`_/__tmp__.
+*next* is an internal implementation that uses the
+``roverlay.overlay.pkgdir.manifest`` module for creating Manifest files.
+It falls back to *ebuild* when creating Manifest entries for imported ebuilds,
+since roverlay does not support reading *SRC_URI* from ebuilds reliably (that
+would require variable interpolation, e.g. for ``${PN}``).
-----------------
^ permalink raw reply related [flat|nested] 41+ messages in thread
* [gentoo-commits] proj/R_overlay:gsoc13/next commit in: doc/rst/
@ 2013-07-15 22:31 André Erdmann
2013-07-16 16:36 ` [gentoo-commits] proj/R_overlay:master " André Erdmann
0 siblings, 1 reply; 41+ messages in thread
From: André Erdmann @ 2013-07-15 22:31 UTC (permalink / raw
To: gentoo-commits
commit: e09da4f72660a0e564b57c9439f9dddaaf7c9186
Author: André Erdmann <dywi <AT> mailerd <DOT> de>
AuthorDate: Mon Jul 15 22:31:14 2013 +0000
Commit: André Erdmann <dywi <AT> mailerd <DOT> de>
CommitDate: Mon Jul 15 22:31:14 2013 +0000
URL: http://git.overlays.gentoo.org/gitweb/?p=proj/R_overlay.git;a=commit;h=e09da4f7
doc/rst: selfdep validation, add TODO sections
---
doc/rst/usage.rst | 128 ++++++++++++++++++++++++++++++++++++++++++------------
1 file changed, 100 insertions(+), 28 deletions(-)
diff --git a/doc/rst/usage.rst b/doc/rst/usage.rst
index 54ed6d3..36fa031 100644
--- a/doc/rst/usage.rst
+++ b/doc/rst/usage.rst
@@ -85,11 +85,13 @@ Expected prior knowlegde:
* argparse (http://code.google.com/p/argparse)
+* setuptools (http://pypi.python.org/pypi/setuptools)
+
* rsync (for using rsync repositories)
* for Manifest creation:
- * portage (*ebuild* and/or the *portage libs* directly)
+ * portage (*ebuild*)
* *true* or *echo* from coreutils or busybox for preventing
package downloads during Manifest creation (optional)
@@ -110,7 +112,7 @@ Expected prior knowlegde:
using a slower one.
time
- Expect 3-6h execution time for the first run, depending on computing
+ Expect 1h execution time for the first run, depending on computing
and I/O speed. *roverlay* is able to work in incremental mode,
thus making subsequent runs need a lot less time.
@@ -211,6 +213,17 @@ The config file is a text file with '<option> = <value>' syntax. Some options
accept multiple values (e.g. <option> = file1, file2), in which case the
values have to be enclosed with quotes (-> ``<option> = "file1 file2"``).
+If roverlay has been installed, then ``emerge --config roverlay`` can be
+used to set up the config file as well as to create essential directories.
+It can be run multiple times in order to configure roverlay for more than
+one user.
+
+.. Important::
+
+ ``emerge --config roverlay`` will overwrite the user's config file (or
+ /etc/roverlay/R-overlay.conf when configuring for root).
+
+
The following options should be set before running *roverlay*:
OVERLAY_DIR
@@ -350,6 +363,10 @@ have reasonable defaults if *roverlay* has been installed using *emerge*:
Example: DISTDIR_FLAT = yes
+ PORTDIR
+ Portage directory, which is used to scan for valid licenses.
+
+ Example: PORTDIR = /usr/portage
There is another option that is useful for creating new dependency rules,
LOG_FILE_UNRESOLVABLE_, which will write all unresolvable dependencies
@@ -569,6 +586,9 @@ These are the steps that *roverlay* performs:
* **immediately stop** processing *p* if a *required* dependency
cannot be resolved in which case a valid ebuild cannot be created
+ * verify dependencies on packages found in the overlay, whether their
+ ebuilds already exist or not (*selfdep validation*)
+
See also: `Dependency Resolution`_
* create an ebuild for *p* by using the dependency resolution results
@@ -1106,11 +1126,6 @@ Ebuilds imported that way can not be overwritten by generated ebuilds and
benefit from most overlay creation features, e.g. Manifest file creation.
However, they cannot be used for metadata creation.
-.. Important::
-
- Importing ebuilds is only supported by the default Manifest implementation
- (*ebuildmanifest*).
-
==================
Dependency Rules
@@ -1314,6 +1329,15 @@ Keywords
* *all* (no type restrictions)
* *pkg* (resolve only R package dependencies)
* *sys* (resolve only system dependencies)
+ * *selfdep* (subtype: dependency is part of the overlay, see selfdep below)
+
+ The dependency type can also be a comma-separated list of the listed types.
+ Actually, *all* is an abbreviated version of ``pkg,sys``.
+ Specifying *selfdep* alone does not resolve anything.
+
+ .. Hint::
+
+ Check the *dependency type* if a newly added rule has no effect.
The other keyword is *#! NOPARSE* which stops parsing of a rule file.
@@ -1406,16 +1430,28 @@ Comments
will be read as normal *dependency strings*.
Selfdep
- This is another name for *dependency strings* that are resolved by an
- R package with the same name, which is also part of the overlay being
- created.
+ This is a classification for dependencies on packages which are also part
+ of the overlay being created. Typically, selfdeps refer to other R
+ packages, but there may be a few exceptions.
+
+ roverlay validates *selfdeps* during overlay creation, which is the most
+ significant difference to non-*selfdeps*.
+
+ Selfdep rules can be written by prefixing a single rule with ``@selfdep``
+ (in a separate text line) or by adding ``selfdep`` to the dependency rule
+ type.
+
+
+ There is a second variant of selfdeps, *true selfdeps*, which resolve
+ a *dependency string* as R package with the same name.
+
- Example: *zoo* is resolved as *sci-R/zoo*, assuming that `OVERLAY_CATEGORY`_
- is set to *sci-R*
+ Example: *zoo* is resolved as *sci-R/zoo*, assuming that
+ `OVERLAY_CATEGORY`_ is set to *sci-R*.
- Writing selfdep rules is not necessary since *roverlay* automatically
- creates rules for all known R packages (see `Dynamic Selfdep Rule Pool`_
- for details).
+ Writing such selfdep rules is not necessary since *roverlay* automatically
+ creates rules for all known R packages at runtime (see
+ `Dynamic Selfdep Rule Pool`_ for details).
There are a few exceptions to that in which case selfdep rules have to
be written:
@@ -1424,15 +1460,10 @@ Selfdep
R package). This is likely a "bug" in the DESCRIPTION file of the
R package being processed.
- * The R package name is not ebuild-name compliant (e.g. contains the '.'
- char, which is remapped to '_'.).
- Most *char remap* cases are handled properly, but there may be a few
- exceptions.
-
.. Caution::
- Writing unnecessary selfdep rules slows dependency resolution down!
- Each rule will exist twice, once as *dynamic* rule and once as
+ Writing unnecessary *true selfdep* rules slows dependency resolution
+ down! Each rule will exist twice, once as *dynamic* rule and once as
the written one.
@@ -2524,10 +2555,12 @@ OVERLAY_KEEP_NTH_LATEST
OVERLAY_MANIFEST_IMPLEMENTATION
Sets the implementation that will be used for Manifest file writing.
- Available choices are *ebuild*, *portage*, *default* and *none*.
- Defaults to *default* (-> *ebuild*).
- *portage* is highly experimental and therefore not recommended
- for production usage.
+ Available choices are *ebuild*, *next*, *default* and *none*.
+ Defaults to *default* (-> *next*).
+
+ *next* is an mostly internal implementation that is considerably faster
+ than *ebuild*. *ebuild* is still used when creating Manifest files for
+ imported ebuilds.
.. Note::
@@ -2966,6 +2999,13 @@ Known field flags:
Declares that a field's value is a list whose values are separated by
whitespace. Has no effect if `field flag: isList` is set.
+ .. _field flag\: isLicense:
+
+ isLicense
+ Declares that a field's value should be interpreted as license string.
+ This disables *allowed_value*/*allowed_values* and all other flags
+ except for *mandatory*/*optional*.
+
.. _field flag\: mandatory:
.. _'mandatory' field flag:
@@ -3162,6 +3202,14 @@ Example Session:
cmd % exit
+====================
+ Roverlay Interface
+====================
+
+<<TODO: stub only>>
+
+API to roverlay functionality. Currently supports dependency resolution.
+
=========================
Implementation Overview
=========================
@@ -3353,6 +3401,8 @@ for which an ebuild has been created.
Manifest Creation
+++++++++++++++++++
+<<TODO:this section is totally out of date>>
+
Manifest files are created by calling the *ebuild* executable for each
package directory in a filtered environment where FETCHCOMMAND and
RESUMECOMMAND are set to no-operation. The directories that contain the R
@@ -3374,6 +3424,14 @@ It does the following steps:
#. Call `dependency resolution`_
+#. If dependency resolution was successful and any *selfdeps* found
+ (dependencies on other packages): *pause* ebuild creation for *p* until
+ it has been verified whether these dependencies are satisfiable.
+ This is necessary because dependency resolution does not know whether a
+ resolved dependency is actually valid. To realize this, *roverlay* collects
+ paused ebuild creation jobs after processing all packages, performs
+ *selfdep validation* and then continues ebuild creation.
+
#. If dependency resolution was successful, dependency ebuild variables are
created (*DEPEND*, *RDEPEND* and *R_SUGGESTS*, also *IUSE*, *MISSINGDEPS*).
Otherwise **ebuild creation stops** and *p* is marked as
@@ -3414,6 +3472,13 @@ afterwards. Overlay creation keeps going if an ebuild cannot be created,
instead the event is logged. Running more than one *OverlayWorker* in parallel
is possible.
+++++++++++++++++++++
+ Selfdep Validation
+++++++++++++++++++++
+
+<<TODO: specify algo here>>
+
+
-----------------------
Dependency Resolution
-----------------------
@@ -3467,6 +3532,13 @@ System Dependency
This declares that the *dependency string* could be a system dependency,
e.g. a scientific library or a video encoder.
+Selfdep
+ This declares that the resolved dependency has to pass
+ *selfdep validation*. While *selfdep* usually implies *package*, these
+ types are not the same. The *package* type is applied to *dependency strings*
+ based on their origin (DESCRIPTION field), whereas *selfdep* is a property
+ of the resolving rule.
+
Try other dependency types
This declares that the *dependency string* can be resolved by ignoring its
dependency type partially. This property allows to resolve package
@@ -3602,8 +3674,8 @@ R packages" to create rules.
.. _Dynamic Selfdep Rule Pool:
*roverlay* uses one dynamic rule pool, the **Dynamic Selfdep Rule Pool**.
-This pool contains rules for all known R packages and is able to resolve
-R package dependencies.
+This pool contains *selfdep* rules for all known R packages and is able
+to resolve R package dependencies.
By convention, it will never resolve any system dependencies.
+++++++++++++++++++++++++++++
^ permalink raw reply related [flat|nested] 41+ messages in thread* [gentoo-commits] proj/R_overlay:master commit in: doc/rst/
2013-07-15 22:31 [gentoo-commits] proj/R_overlay:gsoc13/next " André Erdmann
@ 2013-07-16 16:36 ` André Erdmann
0 siblings, 0 replies; 41+ messages in thread
From: André Erdmann @ 2013-07-16 16:36 UTC (permalink / raw
To: gentoo-commits
commit: e09da4f72660a0e564b57c9439f9dddaaf7c9186
Author: André Erdmann <dywi <AT> mailerd <DOT> de>
AuthorDate: Mon Jul 15 22:31:14 2013 +0000
Commit: André Erdmann <dywi <AT> mailerd <DOT> de>
CommitDate: Mon Jul 15 22:31:14 2013 +0000
URL: http://git.overlays.gentoo.org/gitweb/?p=proj/R_overlay.git;a=commit;h=e09da4f7
doc/rst: selfdep validation, add TODO sections
---
doc/rst/usage.rst | 128 ++++++++++++++++++++++++++++++++++++++++++------------
1 file changed, 100 insertions(+), 28 deletions(-)
diff --git a/doc/rst/usage.rst b/doc/rst/usage.rst
index 54ed6d3..36fa031 100644
--- a/doc/rst/usage.rst
+++ b/doc/rst/usage.rst
@@ -85,11 +85,13 @@ Expected prior knowlegde:
* argparse (http://code.google.com/p/argparse)
+* setuptools (http://pypi.python.org/pypi/setuptools)
+
* rsync (for using rsync repositories)
* for Manifest creation:
- * portage (*ebuild* and/or the *portage libs* directly)
+ * portage (*ebuild*)
* *true* or *echo* from coreutils or busybox for preventing
package downloads during Manifest creation (optional)
@@ -110,7 +112,7 @@ Expected prior knowlegde:
using a slower one.
time
- Expect 3-6h execution time for the first run, depending on computing
+ Expect 1h execution time for the first run, depending on computing
and I/O speed. *roverlay* is able to work in incremental mode,
thus making subsequent runs need a lot less time.
@@ -211,6 +213,17 @@ The config file is a text file with '<option> = <value>' syntax. Some options
accept multiple values (e.g. <option> = file1, file2), in which case the
values have to be enclosed with quotes (-> ``<option> = "file1 file2"``).
+If roverlay has been installed, then ``emerge --config roverlay`` can be
+used to set up the config file as well as to create essential directories.
+It can be run multiple times in order to configure roverlay for more than
+one user.
+
+.. Important::
+
+ ``emerge --config roverlay`` will overwrite the user's config file (or
+ /etc/roverlay/R-overlay.conf when configuring for root).
+
+
The following options should be set before running *roverlay*:
OVERLAY_DIR
@@ -350,6 +363,10 @@ have reasonable defaults if *roverlay* has been installed using *emerge*:
Example: DISTDIR_FLAT = yes
+ PORTDIR
+ Portage directory, which is used to scan for valid licenses.
+
+ Example: PORTDIR = /usr/portage
There is another option that is useful for creating new dependency rules,
LOG_FILE_UNRESOLVABLE_, which will write all unresolvable dependencies
@@ -569,6 +586,9 @@ These are the steps that *roverlay* performs:
* **immediately stop** processing *p* if a *required* dependency
cannot be resolved in which case a valid ebuild cannot be created
+ * verify dependencies on packages found in the overlay, whether their
+ ebuilds already exist or not (*selfdep validation*)
+
See also: `Dependency Resolution`_
* create an ebuild for *p* by using the dependency resolution results
@@ -1106,11 +1126,6 @@ Ebuilds imported that way can not be overwritten by generated ebuilds and
benefit from most overlay creation features, e.g. Manifest file creation.
However, they cannot be used for metadata creation.
-.. Important::
-
- Importing ebuilds is only supported by the default Manifest implementation
- (*ebuildmanifest*).
-
==================
Dependency Rules
@@ -1314,6 +1329,15 @@ Keywords
* *all* (no type restrictions)
* *pkg* (resolve only R package dependencies)
* *sys* (resolve only system dependencies)
+ * *selfdep* (subtype: dependency is part of the overlay, see selfdep below)
+
+ The dependency type can also be a comma-separated list of the listed types.
+ Actually, *all* is an abbreviated version of ``pkg,sys``.
+ Specifying *selfdep* alone does not resolve anything.
+
+ .. Hint::
+
+ Check the *dependency type* if a newly added rule has no effect.
The other keyword is *#! NOPARSE* which stops parsing of a rule file.
@@ -1406,16 +1430,28 @@ Comments
will be read as normal *dependency strings*.
Selfdep
- This is another name for *dependency strings* that are resolved by an
- R package with the same name, which is also part of the overlay being
- created.
+ This is a classification for dependencies on packages which are also part
+ of the overlay being created. Typically, selfdeps refer to other R
+ packages, but there may be a few exceptions.
+
+ roverlay validates *selfdeps* during overlay creation, which is the most
+ significant difference to non-*selfdeps*.
+
+ Selfdep rules can be written by prefixing a single rule with ``@selfdep``
+ (in a separate text line) or by adding ``selfdep`` to the dependency rule
+ type.
+
+
+ There is a second variant of selfdeps, *true selfdeps*, which resolve
+ a *dependency string* as R package with the same name.
+
- Example: *zoo* is resolved as *sci-R/zoo*, assuming that `OVERLAY_CATEGORY`_
- is set to *sci-R*
+ Example: *zoo* is resolved as *sci-R/zoo*, assuming that
+ `OVERLAY_CATEGORY`_ is set to *sci-R*.
- Writing selfdep rules is not necessary since *roverlay* automatically
- creates rules for all known R packages (see `Dynamic Selfdep Rule Pool`_
- for details).
+ Writing such selfdep rules is not necessary since *roverlay* automatically
+ creates rules for all known R packages at runtime (see
+ `Dynamic Selfdep Rule Pool`_ for details).
There are a few exceptions to that in which case selfdep rules have to
be written:
@@ -1424,15 +1460,10 @@ Selfdep
R package). This is likely a "bug" in the DESCRIPTION file of the
R package being processed.
- * The R package name is not ebuild-name compliant (e.g. contains the '.'
- char, which is remapped to '_'.).
- Most *char remap* cases are handled properly, but there may be a few
- exceptions.
-
.. Caution::
- Writing unnecessary selfdep rules slows dependency resolution down!
- Each rule will exist twice, once as *dynamic* rule and once as
+ Writing unnecessary *true selfdep* rules slows dependency resolution
+ down! Each rule will exist twice, once as *dynamic* rule and once as
the written one.
@@ -2524,10 +2555,12 @@ OVERLAY_KEEP_NTH_LATEST
OVERLAY_MANIFEST_IMPLEMENTATION
Sets the implementation that will be used for Manifest file writing.
- Available choices are *ebuild*, *portage*, *default* and *none*.
- Defaults to *default* (-> *ebuild*).
- *portage* is highly experimental and therefore not recommended
- for production usage.
+ Available choices are *ebuild*, *next*, *default* and *none*.
+ Defaults to *default* (-> *next*).
+
+ *next* is an mostly internal implementation that is considerably faster
+ than *ebuild*. *ebuild* is still used when creating Manifest files for
+ imported ebuilds.
.. Note::
@@ -2966,6 +2999,13 @@ Known field flags:
Declares that a field's value is a list whose values are separated by
whitespace. Has no effect if `field flag: isList` is set.
+ .. _field flag\: isLicense:
+
+ isLicense
+ Declares that a field's value should be interpreted as license string.
+ This disables *allowed_value*/*allowed_values* and all other flags
+ except for *mandatory*/*optional*.
+
.. _field flag\: mandatory:
.. _'mandatory' field flag:
@@ -3162,6 +3202,14 @@ Example Session:
cmd % exit
+====================
+ Roverlay Interface
+====================
+
+<<TODO: stub only>>
+
+API to roverlay functionality. Currently supports dependency resolution.
+
=========================
Implementation Overview
=========================
@@ -3353,6 +3401,8 @@ for which an ebuild has been created.
Manifest Creation
+++++++++++++++++++
+<<TODO:this section is totally out of date>>
+
Manifest files are created by calling the *ebuild* executable for each
package directory in a filtered environment where FETCHCOMMAND and
RESUMECOMMAND are set to no-operation. The directories that contain the R
@@ -3374,6 +3424,14 @@ It does the following steps:
#. Call `dependency resolution`_
+#. If dependency resolution was successful and any *selfdeps* found
+ (dependencies on other packages): *pause* ebuild creation for *p* until
+ it has been verified whether these dependencies are satisfiable.
+ This is necessary because dependency resolution does not know whether a
+ resolved dependency is actually valid. To realize this, *roverlay* collects
+ paused ebuild creation jobs after processing all packages, performs
+ *selfdep validation* and then continues ebuild creation.
+
#. If dependency resolution was successful, dependency ebuild variables are
created (*DEPEND*, *RDEPEND* and *R_SUGGESTS*, also *IUSE*, *MISSINGDEPS*).
Otherwise **ebuild creation stops** and *p* is marked as
@@ -3414,6 +3472,13 @@ afterwards. Overlay creation keeps going if an ebuild cannot be created,
instead the event is logged. Running more than one *OverlayWorker* in parallel
is possible.
+++++++++++++++++++++
+ Selfdep Validation
+++++++++++++++++++++
+
+<<TODO: specify algo here>>
+
+
-----------------------
Dependency Resolution
-----------------------
@@ -3467,6 +3532,13 @@ System Dependency
This declares that the *dependency string* could be a system dependency,
e.g. a scientific library or a video encoder.
+Selfdep
+ This declares that the resolved dependency has to pass
+ *selfdep validation*. While *selfdep* usually implies *package*, these
+ types are not the same. The *package* type is applied to *dependency strings*
+ based on their origin (DESCRIPTION field), whereas *selfdep* is a property
+ of the resolving rule.
+
Try other dependency types
This declares that the *dependency string* can be resolved by ignoring its
dependency type partially. This property allows to resolve package
@@ -3602,8 +3674,8 @@ R packages" to create rules.
.. _Dynamic Selfdep Rule Pool:
*roverlay* uses one dynamic rule pool, the **Dynamic Selfdep Rule Pool**.
-This pool contains rules for all known R packages and is able to resolve
-R package dependencies.
+This pool contains *selfdep* rules for all known R packages and is able
+to resolve R package dependencies.
By convention, it will never resolve any system dependencies.
+++++++++++++++++++++++++++++
^ permalink raw reply related [flat|nested] 41+ messages in thread
* [gentoo-commits] proj/R_overlay:master commit in: doc/rst/
@ 2013-07-03 10:05 André Erdmann
0 siblings, 0 replies; 41+ messages in thread
From: André Erdmann @ 2013-07-03 10:05 UTC (permalink / raw
To: gentoo-commits
commit: 22b1e80c37192d3049f95293cedd03b30ca2c7d2
Author: André Erdmann <dywi <AT> mailerd <DOT> de>
AuthorDate: Fri Jun 28 17:16:32 2013 +0000
Commit: André Erdmann <dywi <AT> mailerd <DOT> de>
CommitDate: Fri Jun 28 17:16:32 2013 +0000
URL: http://git.overlays.gentoo.org/gitweb/?p=proj/R_overlay.git;a=commit;h=22b1e80c
doc/rst: SLOT handling (incomplete)
Draft for documentation about fuzzy slot rules.
(No doc/html commit will be created since documentation is todo.)
---
doc/rst/usage.rst | 69 +++++++++++++++++++++++++++++++++++++++++++++++++------
1 file changed, 62 insertions(+), 7 deletions(-)
diff --git a/doc/rst/usage.rst b/doc/rst/usage.rst
index 2daedce..f3e2e18 100644
--- a/doc/rst/usage.rst
+++ b/doc/rst/usage.rst
@@ -1160,7 +1160,8 @@ Simple Rules
Fuzzy Rules
Fuzzy Rules are **extended Simple Rules**. If the basic lookup
as described above fails for a *dependency string*,
- they will *try* to resolve it as a **version-relative match**.
+ they will *try* to resolve it as a **version-relative**,
+ **slot-relative** or **version,slot-relative match**.
To do this, the *dependency string* will be split into components like
*dependency name*, *dependency version* and useless comments, which are
@@ -1177,7 +1178,7 @@ Fuzzy Rules
* "R 2.12" as ">=dev-lang/R-2.12"
* "The R PROGRAMMING LANGUAGE [<2.14] from http://www.r-project.org/"
as "<dev-lang/R-2.14"
- * "R ( !2.10 )" as "( !=dev-lang/R-2.10 dev-lang/R )"
+ * "R ( !=2.10 )" as "( !=dev-lang/R-2.10 dev-lang/R )"
++++++++++++++++++++
@@ -1190,7 +1191,7 @@ See `Rule File Syntax`_ for a formal description.
Example 1 - *default* fuzzy rule
A rule that matches many dependencies on dev-lang/R, for example
- "r 2.12", "R(>= 2.14)", "R [<2.10]", "r{ !2.12 }", and "R", and
+ "r 2.12", "R(>= 2.14)", "R [<2.10]", "r{ !=2.12 }", and "R", and
resolves them as '>=dev-lang/R-2.12', '>=dev-lang/R-2.14',
'<dev-lang/R-2.10', etc.:
@@ -1239,6 +1240,25 @@ Example 4 - *ignore* simple rule
Will use djmrgl or rgl packages for rendering if present
}
+Example 5 - fuzzy slot rule
+ A rule that matches many dependencies on sci-libs/fftw and resolves them
+ as slotted depencency:
+
+ .. code-block:: text
+
+ ~sci-libs/fftw: :: fftw
+
+Example 6 - slot-restricted fuzzy slot rule
+ Similar to example 5, but this rule does not resolve anything unless the
+ calculated slot is allowed.
+
+ .. code-block:: text
+
+ ~sci-libs/fftw:2.1,3.0: :: fftw
+
+ .. caution::
+
+ TODO! (only integers accepted)
Please see the default rule files for more extensive examples that cover
other aspects like limiting a rule to certain dependency types.
@@ -1307,14 +1327,13 @@ Dependencies
Dependency strings cannot start with *~* as it is a keychar.
Use braces *( ~... )* to work around that.
-
Single line rules
resolve exactly one *dependency string*. Their rule separator is ' :: '.
Syntax:
.. code:: text
- [<keychar>]<dependency> :: <dependency string>
+ [<keychar>]<dependency>[<rule options>] :: <dependency string>
Multi line rules
resolve several *dependency strings*.
@@ -1324,7 +1343,7 @@ Multi line rules
Syntax:
.. code-block:: text
- [<keychar>]<dependency> {
+ [<keychar>]<dependency>[<rule options>] {
<dependency string>
[<dependency string>]
...
@@ -1336,6 +1355,42 @@ Multi line rules
zero or more *dependency strings*. An empty rule makes little sense,
though.
+Rule Options
+ Certain rule types accept options that control the rule's behavior.
+ For example, *default* fuzzy rules can be set up to yield slotted
+ dependencies.
+
+Fuzzy Slot Rules
+ <<TODO>>
+
+ Fuzzy Slot rules are a subtype of *default* fuzzy rules. Appending a colon
+ character ``:`` to the *dependency string* of a fuzzy rules
+ (as *rule option*) turns it into a slot rule.
+
+ Fuzzy slot rules accept even more options:
+
+ * accepted slot values can be restricted (integer range or list)
+ * *version,slot-relative* matches can be enabled (example, <<TODO>>: ``>=${CATEGORY}/${PN}-${PV}:${SLOT}${SLOT_SUFFIX-}``)
+ * a *slot suffix* can be specified (e.g. for using *Atom Slot Operators*)
+
+ .. code-block:: text
+
+ fuzzy slot options := [:<slot restrict>]:{+<flag>}[<slot suffix>]
+ flag := v
+ slot restrict := <restrict range> | <restrict list>
+ restrict range := [<number>]..[<number>]
+ restrict list := <number>{,<number>}
+
+ # slot suffix can be any string that starts with '/', '=' or '*'
+ # number is a natural number (0..N)
+
+
+
+ .. Note::
+
+ Fuzzy Slot rules cannot resolve "not <version>" statements, e.g.
+ "R ( != 2.14 )".
+
Comments
start with **#**. There are a few exceptions to that, the *#deptype* and
*#! NOPARSE* keywords. Comments inside rule blocks are not allowed and
@@ -1380,7 +1435,7 @@ Rule Stubs
Syntax:
.. code:: text
- [<keychar>]<short dependency>
+ [<keychar>]<short dependency>[<rule options>]
===============
^ permalink raw reply related [flat|nested] 41+ messages in thread* [gentoo-commits] proj/R_overlay:gsoc13/next commit in: doc/rst/
@ 2013-07-03 10:04 André Erdmann
2013-07-03 10:05 ` [gentoo-commits] proj/R_overlay:master " André Erdmann
0 siblings, 1 reply; 41+ messages in thread
From: André Erdmann @ 2013-07-03 10:04 UTC (permalink / raw
To: gentoo-commits
commit: 1e797b7bcafc6b364190961ef74b63febe10b9e9
Author: André Erdmann <dywi <AT> mailerd <DOT> de>
AuthorDate: Wed Jul 3 09:57:49 2013 +0000
Commit: André Erdmann <dywi <AT> mailerd <DOT> de>
CommitDate: Wed Jul 3 09:57:49 2013 +0000
URL: http://git.overlays.gentoo.org/gitweb/?p=proj/R_overlay.git;a=commit;h=1e797b7b
doc/rst: SLOT handling
---
doc/rst/usage.rst | 53 +++++++++++++++++++++++++++++++----------------------
1 file changed, 31 insertions(+), 22 deletions(-)
diff --git a/doc/rst/usage.rst b/doc/rst/usage.rst
index f3e2e18..54ed6d3 100644
--- a/doc/rst/usage.rst
+++ b/doc/rst/usage.rst
@@ -1242,11 +1242,15 @@ Example 4 - *ignore* simple rule
Example 5 - fuzzy slot rule
A rule that matches many dependencies on sci-libs/fftw and resolves them
- as slotted depencency:
+ as slotted depencency. The ``s=<range>`` option controls which parts of the
+ version (from the dependency string) are relevant for calculating the
+ slot. The following example resolves "fftw 2.1", "fftw 2.1.2" and
+ "fftw 2.1.3" as "sci-libs/fftw:2.1", "fftw 3.0" as "sci-libs/fftw:3.0"
+ and so on:
.. code-block:: text
- ~sci-libs/fftw: :: fftw
+ ~sci-libs/fftw:s=0..1 :: fftw
Example 6 - slot-restricted fuzzy slot rule
Similar to example 5, but this rule does not resolve anything unless the
@@ -1254,11 +1258,17 @@ Example 6 - slot-restricted fuzzy slot rule
.. code-block:: text
- ~sci-libs/fftw:2.1,3.0: :: fftw
+ ~sci-libs/fftw:s=0..1:restrict=2.1,3.0: :: fftw
- .. caution::
+Example 7 - slot-restricted fuzzy slot rule with *immediate* value
+ Example 6 is not quite correct, as sci-libs/fftw currently uses slot 3.0
+ for various versions from the 3.x range. The following rule resolves
+ "fftw 3.0", ..., "fftw 3.3" as "sci-libs/fftw:3.0":
+
+ .. code-block:: text
+
+ ~sci-libs/fftw:s=i3.0:restrict=3.0,3.1,3.2,3.3 :: fftw
- TODO! (only integers accepted)
Please see the default rule files for more extensive examples that cover
other aspects like limiting a rule to certain dependency types.
@@ -1361,30 +1371,29 @@ Rule Options
dependencies.
Fuzzy Slot Rules
- <<TODO>>
-
Fuzzy Slot rules are a subtype of *default* fuzzy rules. Appending a colon
- character ``:`` to the *dependency string* of a fuzzy rules
+ character ``:`` to the *dependency string* of a fuzzy rule
(as *rule option*) turns it into a slot rule.
- Fuzzy slot rules accept even more options:
-
- * accepted slot values can be restricted (integer range or list)
- * *version,slot-relative* matches can be enabled (example, <<TODO>>: ``>=${CATEGORY}/${PN}-${PV}:${SLOT}${SLOT_SUFFIX-}``)
- * a *slot suffix* can be specified (e.g. for using *Atom Slot Operators*)
-
- .. code-block:: text
+ Fuzzy slot rules accept even more options, each of them separated by one
+ colong char ``:``:
- fuzzy slot options := [:<slot restrict>]:{+<flag>}[<slot suffix>]
- flag := v
- slot restrict := <restrict range> | <restrict list>
- restrict range := [<number>]..[<number>]
- restrict list := <number>{,<number>}
+ * slot mode:
- # slot suffix can be any string that starts with '/', '=' or '*'
- # number is a natural number (0..N)
+ * ``default``: calculate a slot value (``<cat>/<pkg>:<SLOT>``)
+ * ``with_version`` or ``+v``: include version, too (``=<cat>/<pkg>-<pkgver>:<SLOT>``)
+ * ``open``: non-versioned slot (``<cat>/<pkg>:*`` or ``<cat>/<pkg>:=``)
+ * accepted *calculated* slot values can be restricted with
+ ``restrict=<list of accepted values`` or ``r=<list>``
+ * relevant slot parts can be set with ``slotparts=<selection>`` or
+ ``s=<selection>``
+ * relevant subslot parts can be set with ``subslotparts=<selection>`` or
+ ``/<selection>``
+ * slot operator can be set to ``*`` or ``=``
+ ``<selection>`` can be an index (integer) range
+ ``[<low>:=0]..[<high>:=<low>]`` or a fixed value ``i<value>``.
.. Note::
^ permalink raw reply related [flat|nested] 41+ messages in thread* [gentoo-commits] proj/R_overlay:master commit in: doc/rst/
2013-07-03 10:04 [gentoo-commits] proj/R_overlay:gsoc13/next " André Erdmann
@ 2013-07-03 10:05 ` André Erdmann
0 siblings, 0 replies; 41+ messages in thread
From: André Erdmann @ 2013-07-03 10:05 UTC (permalink / raw
To: gentoo-commits
commit: 1e797b7bcafc6b364190961ef74b63febe10b9e9
Author: André Erdmann <dywi <AT> mailerd <DOT> de>
AuthorDate: Wed Jul 3 09:57:49 2013 +0000
Commit: André Erdmann <dywi <AT> mailerd <DOT> de>
CommitDate: Wed Jul 3 09:57:49 2013 +0000
URL: http://git.overlays.gentoo.org/gitweb/?p=proj/R_overlay.git;a=commit;h=1e797b7b
doc/rst: SLOT handling
---
doc/rst/usage.rst | 53 +++++++++++++++++++++++++++++++----------------------
1 file changed, 31 insertions(+), 22 deletions(-)
diff --git a/doc/rst/usage.rst b/doc/rst/usage.rst
index f3e2e18..54ed6d3 100644
--- a/doc/rst/usage.rst
+++ b/doc/rst/usage.rst
@@ -1242,11 +1242,15 @@ Example 4 - *ignore* simple rule
Example 5 - fuzzy slot rule
A rule that matches many dependencies on sci-libs/fftw and resolves them
- as slotted depencency:
+ as slotted depencency. The ``s=<range>`` option controls which parts of the
+ version (from the dependency string) are relevant for calculating the
+ slot. The following example resolves "fftw 2.1", "fftw 2.1.2" and
+ "fftw 2.1.3" as "sci-libs/fftw:2.1", "fftw 3.0" as "sci-libs/fftw:3.0"
+ and so on:
.. code-block:: text
- ~sci-libs/fftw: :: fftw
+ ~sci-libs/fftw:s=0..1 :: fftw
Example 6 - slot-restricted fuzzy slot rule
Similar to example 5, but this rule does not resolve anything unless the
@@ -1254,11 +1258,17 @@ Example 6 - slot-restricted fuzzy slot rule
.. code-block:: text
- ~sci-libs/fftw:2.1,3.0: :: fftw
+ ~sci-libs/fftw:s=0..1:restrict=2.1,3.0: :: fftw
- .. caution::
+Example 7 - slot-restricted fuzzy slot rule with *immediate* value
+ Example 6 is not quite correct, as sci-libs/fftw currently uses slot 3.0
+ for various versions from the 3.x range. The following rule resolves
+ "fftw 3.0", ..., "fftw 3.3" as "sci-libs/fftw:3.0":
+
+ .. code-block:: text
+
+ ~sci-libs/fftw:s=i3.0:restrict=3.0,3.1,3.2,3.3 :: fftw
- TODO! (only integers accepted)
Please see the default rule files for more extensive examples that cover
other aspects like limiting a rule to certain dependency types.
@@ -1361,30 +1371,29 @@ Rule Options
dependencies.
Fuzzy Slot Rules
- <<TODO>>
-
Fuzzy Slot rules are a subtype of *default* fuzzy rules. Appending a colon
- character ``:`` to the *dependency string* of a fuzzy rules
+ character ``:`` to the *dependency string* of a fuzzy rule
(as *rule option*) turns it into a slot rule.
- Fuzzy slot rules accept even more options:
-
- * accepted slot values can be restricted (integer range or list)
- * *version,slot-relative* matches can be enabled (example, <<TODO>>: ``>=${CATEGORY}/${PN}-${PV}:${SLOT}${SLOT_SUFFIX-}``)
- * a *slot suffix* can be specified (e.g. for using *Atom Slot Operators*)
-
- .. code-block:: text
+ Fuzzy slot rules accept even more options, each of them separated by one
+ colong char ``:``:
- fuzzy slot options := [:<slot restrict>]:{+<flag>}[<slot suffix>]
- flag := v
- slot restrict := <restrict range> | <restrict list>
- restrict range := [<number>]..[<number>]
- restrict list := <number>{,<number>}
+ * slot mode:
- # slot suffix can be any string that starts with '/', '=' or '*'
- # number is a natural number (0..N)
+ * ``default``: calculate a slot value (``<cat>/<pkg>:<SLOT>``)
+ * ``with_version`` or ``+v``: include version, too (``=<cat>/<pkg>-<pkgver>:<SLOT>``)
+ * ``open``: non-versioned slot (``<cat>/<pkg>:*`` or ``<cat>/<pkg>:=``)
+ * accepted *calculated* slot values can be restricted with
+ ``restrict=<list of accepted values`` or ``r=<list>``
+ * relevant slot parts can be set with ``slotparts=<selection>`` or
+ ``s=<selection>``
+ * relevant subslot parts can be set with ``subslotparts=<selection>`` or
+ ``/<selection>``
+ * slot operator can be set to ``*`` or ``=``
+ ``<selection>`` can be an index (integer) range
+ ``[<low>:=0]..[<high>:=<low>]`` or a fixed value ``i<value>``.
.. Note::
^ permalink raw reply related [flat|nested] 41+ messages in thread
* [gentoo-commits] proj/R_overlay:gsoc13/next commit in: doc/rst/
@ 2013-06-26 17:29 André Erdmann
2013-06-30 15:58 ` [gentoo-commits] proj/R_overlay:master " André Erdmann
0 siblings, 1 reply; 41+ messages in thread
From: André Erdmann @ 2013-06-26 17:29 UTC (permalink / raw
To: gentoo-commits
commit: 0678580564dcf6aa8e2587d3b04d62544265ca74
Author: André Erdmann <dywi <AT> mailerd <DOT> de>
AuthorDate: Wed Jun 26 17:28:39 2013 +0000
Commit: André Erdmann <dywi <AT> mailerd <DOT> de>
CommitDate: Wed Jun 26 17:28:39 2013 +0000
URL: http://git.overlays.gentoo.org/gitweb/?p=proj/R_overlay.git;a=commit;h=06785805
doc/rst: "run hooks", "distmap"
---
doc/rst/usage.rst | 643 ++++++++++++++++++++++++++++++++----------------------
1 file changed, 383 insertions(+), 260 deletions(-)
diff --git a/doc/rst/usage.rst b/doc/rst/usage.rst
index 078a7ac..2daedce 100644
--- a/doc/rst/usage.rst
+++ b/doc/rst/usage.rst
@@ -1010,6 +1010,22 @@ A local directory will never be modified.
consider using one of the **websync** repo types, websync_repo_ and
websync_pkglist_.
+---------
+ distmap
+---------
+
+*roverlay* uses a text file to store information about files in the package
+mirror directory (OVERLAY_DISTDIR_ROOT_). This is necessary for comparing
+package files from repos with files for which an ebuild has already been
+created (in previous runs).
+
+With the help of the *distmap file*, *roverlay* is able to determine whether
+upstream has changed a package file silently and creates a revision-bumped
+ebuild for the *new* package file.
+
+The *distmap file* can optionally be compressed (bzip2 or gzip), which
+reduces its size considerably.
+
=====================
Additions Directory
@@ -1790,6 +1806,354 @@ Moving such packages to a "R-package" sub directory would be possible, too:
END;
+=============
+ Event Hooks
+=============
+
+*roverlay* is able to call a script when certain events occur, e.g. after
+successful overlay creation, which can be used to perform additional actions
+without touching *roverlay's* source code.
+
+To realize this, *roverlay* determines whether a given event is permitted
+(`event policy`_) and, if so, creates a `hook environment`_ and runs the
+script. Additionally, shell scripts can load *roverlay's* *$FUNCTIONS* file,
+which provides extra functionality.
+
+.. Note::
+
+ *roverlay* waits until the script terminates and thus possibly waits
+ forever.
+
+
+----------------------
+ Default event script
+----------------------
+
+The default event script (``mux.sh``) loads *$FUNCTIONS* and then runs the
+following script files (by sourcing them), in order:
+
+#. all files from ADDITIONS_DIR_/hooks/<event> that end with *.sh*
+ (``<ADDITIONS_DIR>/hooks/<event>/*.sh``)
+#. all files ADDITIONS_DIR_/hooks that end with *.<event>*
+ (``<ADDITIONS_DIR>/hooks/*.<event>``)
+
+So, there are two naming schemes for *hook scripts*.
+Either one is acceptable, but it is advised to stay consistent.
+Having the same script at both locations results in executing it twice.
+
+.. Note::
+
+ The default event script enables *nounset* behavior, which causes the
+ shell command interpreter to exit abnormally if an unset variable is
+ accessed.
+
+
+++++++++++++++++++++++++++
+ Activating a hook script
+++++++++++++++++++++++++++
+
+Activating a hook script can be done by symlinking it:
+
+.. code-block:: text
+
+ ln -s <real script> ${ADDITIONS_DIR}/hooks/<event>/<name>.sh
+ # or
+ ln -s <real script> ${ADDITIONS_DIR}/hooks/<name>.<event>
+
+
+++++++++++++++++++++++++++
+ Adding a new hook script
+++++++++++++++++++++++++++
+
+As hinted before, *hook scripts* are simple shell scripts. The following
+template gives an idea of how to write them:
+
+.. code-block:: sh
+
+ #!/bin/sh
+ #set -u
+
+ # load essential functions
+ # (not necessary when using the default event script)
+ . "${FUNCTIONS?}" || exit
+
+ ## load additional function files, if any
+ #$lf <name(s)>
+
+ # script body
+ #
+ # when redirecting output to $DEVNULL, use ">>" instead of ">" as
+ # $DEVNULL could be a file
+ #ls >>${DEVNULL}
+ #
+ # ...
+
+
+ # the script must not exit if everything went well (return is ok)
+ return 0
+
+
+--------------
+ Event Policy
+--------------
+
+The *event policy* controls whether a certain event actually triggers a script
+call or not.
+It is constructed by parsing the EVENT_HOOK_RESTRICT_ config option:
+
+* a word prefixed by ``-`` means *deny specific event* (-> blacklist)
+* the asterisk char ``*`` (or ``+*``) sets the policy to
+ *allow unless denied* (blacklist) or *allow all*
+* a word prefixed by ``+`` or without a prefix char means
+ *allow specific event* (-> whitelist)
+* the asterisk char with ``-`` as prefix (``-*``) sets the policy to
+ *deny unless allowed* (whitelist) or *deny all*
+
+
+The policy defaults to *allow all* if ``EVENT_HOOK_RESTRICT`` is not set in
+the config file. An empty string sets the policy to *deny all*.
+
+
+------------------
+ Hook Environment
+------------------
+
+.. table:: environment variables provided by *roverlay*
+
+ +----------------+-------------------+-----------------------------------------+
+ | variable | source | notes / description |
+ +================+===================+=========================================+
+ | PATH | os.environ | |
+ +----------------+-------------------+-----------------------------------------+
+ | LOGNAME | os.environ | |
+ +----------------+-------------------+-----------------------------------------+
+ | SHLVL | os.environ | |
+ +----------------+-------------------+-----------------------------------------+
+ | TERM | os.environ | |
+ +----------------+-------------------+-----------------------------------------+
+ | HOME | os.environ | |
+ +----------------+-------------------+-----------------------------------------+
+ | ROVERLAY_PHASE | event | event that caused the script to run |
+ +----------------+-------------------+-----------------------------------------+
+ | OVERLAY | config | overlay directory (`OVERLAY_DIR`_), |
+ +----------------+-------------------+ initial working directory |
+ | S | *$OVERLAY* | |
+ +----------------+-------------------+ |
+ | PWD | *$OVERLAY* | |
+ +----------------+-------------------+-----------------------------------------+
+ | DISTROOT | config | package mirror directory |
+ | | | (`OVERLAY_DISTDIR_ROOT`_) |
+ +----------------+-------------------+-----------------------------------------+
+ | TMPDIR | os.environ, | directory for temporary files |
+ | | *fallback* | |
+ +----------------+-------------------+ |
+ | T | *$TMPDIR* | |
+ +----------------+-------------------+-----------------------------------------+
+ | ADDITIONS_DIR | config | directory with supplementary files |
+ | | | (`OVERLAY_ADDITIONS_DIR`_) |
+ +----------------+-------------------+ |
+ | FILESDIR | *$ADDITIONS_DIR* | |
+ +----------------+-------------------+-----------------------------------------+
+ | SHLIB | *$ADDITIONS_DIR*, | A list of directories with shell |
+ | | *installed?* | function files |
+ | | | (optional, only set if any dir exists) |
+ +----------------+-------------------+-----------------------------------------+
+ | FUNCTIONS | *$ADDITIONS_DIR*, | file with essential shell functions |
+ | | *installed?* | (optional, only set if it exists) |
+ +----------------+-------------------+-----------------------------------------+
+ | DEBUG | log level | *shbool* (``y`` or ``n``) that |
+ | | | indicates whether debug messages should |
+ | | | be printed |
+ +----------------+-------------------+-----------------------------------------+
+ | VERBOSE | log level | *shbool* |
+ +----------------+-------------------+-----------------------------------------+
+ | QUIET | log level | *shbool* that indicates whether scripts |
+ | | | should be quiet |
+ +----------------+-------------------+-----------------------------------------+
+ | NO_COLOR | *n/a* | *shbool*. Always set to *y* since |
+ | | | colored output should not be produced |
+ +----------------+-------------------+-----------------------------------------+
+ | NOSYNC | config | *shbool* that indicates whether data |
+ | | | transfers from/to remote machines is |
+ | | | allowed (NOSYNC_) |
+ +----------------+-------------------+-----------------------------------------+
+ | EBUILD | config | the *ebuild* executable |
+ +----------------+-------------------+-----------------------------------------+
+ | GIT_EDITOR | *n/a* | set to */bin/false* |
+ +----------------+-------------------+ |
+ | GIT_ASKPASS | *n/a* | |
+ +----------------+-------------------+-----------------------------------------+
+
+
+The default *essential shell functions* file (*$FUNCTIONS*) makes,
+when included in the hook script, most of the enviroment variables readonly.
+
+
+.. table:: variables provided by *$FUNCTIONS*
+
+ +-----------------+-------------------------------------------------------+
+ | variable | description |
+ +=================+=======================================================+
+ | IFS_DEFAULT | default *internal field separator* |
+ +-----------------+-------------------------------------------------------+
+ | IFS_NEWLINE | *IFS* for iterating over text lines |
+ +-----------------+-------------------------------------------------------+
+ | DEVNULL | */dev/null* target (could also be a file) |
+ +-----------------+-------------------------------------------------------+
+ | EX_ERR | default error exit code |
+ +-----------------+-------------------------------------------------------+
+ | EX_ARG_ERR | default exit code for arg errors |
+ +-----------------+-------------------------------------------------------+
+ | SCRIPT_FILENAME | file name of the hook script |
+ +-----------------+-------------------------------------------------------+
+ | SCRIPT_NAME | name of the hook script (without file extension) |
+ +-----------------+-------------------------------------------------------+
+ | lf | reference to a function that loads additional shell |
+ | | function files |
+ +-----------------+-------------------------------------------------------+
+
+
+*$FUNCTIONS* also provides a number of shell functions:
+
+.. code-block:: sh
+
+ # --- message ---
+ #
+ # void veinfo ( message )
+ # Prints a message to stdout if $DEBUG is set to 'y'.
+ #
+ # void einfo ( message )
+ # Prints a message to stdout if $VERBOSE is set to 'y'.
+ #
+ # void ewarn ( message )
+ # Prints a message to stderr unless $QUIET is set to 'y'.
+ #
+ # void eerror ( message )
+ # Prints a message to stderr.
+ #
+ #
+ # --- core ---
+ #
+ # @noreturn die ( [message], [exit_code] ), raises exit()
+ # Lets the script die with the given message/exit code.
+ #
+ # @noreturn OUT_OF_BOUNDS(), raises die()
+ # Lets the script die due to insufficient arg count.
+ #
+ # int run_command ( *cmdv )
+ # Logs a command and runs it afterwards.
+ #
+ # int run_command_logged ( *cmdv )
+ # Logs a command, runs it and logs the result.
+ #
+ # void autodie ( *cmdv ), raises die()
+ # Runs a command. Lets the script die if the command fails.
+ #
+ #
+ # void load_functions ( *filenames, **SHLIB ), raises die()
+ # Loads additional shell functions file(s) from $SHLIB.
+ # (Referenced by $lf.)
+ #
+ # void dont_run_as_root(), raises die()
+ # Lets the script die if it is run as root.
+ #
+ # int list_has ( word, *list_items )
+ # Returns 0 if $word is in the given list, else 1.
+ #
+ # int qwhich ( *command )
+ # Returns 0 if all listed commands are found by "which", else 1.
+ #
+ #
+ # --- fs ---
+ #
+ # int dodir ( *dir )
+ # Ensures that zero or more directories exist by creating them if
+ # necessary. Returns the number of directories that could not be created.
+ #
+ #
+ # --- str ---
+ #
+ # int yesno ( word, **YESNO_YES=0, **YESNO_NO=1, **YESNO_EMPTY=2 )
+ # Returns $YESNO_YES if $word means "yes", $YESNO_EMPTY if $word is empty
+ # and $YESNO_NO otherwise (if $word probably means "no").
+ #
+ # ~int str_trim ( *args )
+ # Removes whitespace at the beginning and end of a string and replaces
+ # any whitespace sequence within the string with a single space char.
+ # Passes the return value of the underlying sed command.
+ #
+ # ~int str_upper ( *args )
+ # Echoes the uppercase variant of stdin or *args.
+ # Passes tr's return value.
+ #
+ # ~int str_lower ( *args )
+ # Echoes the lowercase variant of stdin or *args.
+ # Passes tr's return value.
+ #
+ # ~int str_field ( fieldspec, *args, **FIELD_SEPARATOR=' ' )
+ # Echoes the requested fields of stdin or *args.
+ # Passes cut's return value.
+ #
+ #
+ # --- int ---
+ #
+ # int is_int ( word )
+ # Returns 0 if $word is an integer, else 1.
+ #
+ # int is_natural ( word )
+ # Returns 0 if $word is an integer >= 0, else 1.
+ #
+ # int is_positive ( word )
+ # Returns 0 if $word is an integer >= 1, else 1.
+ #
+ # int is_negative ( word )
+ # Returns 0 if $word is an integer < 0, else 1.
+ #
+
+
+------------------------
+ Adding a function file
+------------------------
+
+Function files are shell script files that provide functions and variables.
+They should, however, not execute any code directly.
+
+The template below illustrates how to write function files:
+
+.. code-block:: sh
+
+ # protect against repeated inclusion of this file
+ # (replace <name> with a unique identifier)
+ if [ -z "${__HAVE_<name>__-}" ]; then
+ readonly __HAVE_<name>__=y
+
+ # function file body
+ # ...
+
+ fi
+
+Shell function files should be put into ``<ADDITIONS_DIR>/shlib``.
+
+---------------------
+ Adding a hook event
+---------------------
+
+Adding a new event has to be done in *roverlay's* source code and is a rather
+trivial task. The ``roverlay.hook`` module implements a function for running
+the event script:
+
+.. code-block:: python
+
+ # import hook module
+ import roverlay.hook
+
+ # ...
+ # then, somewhere in the code
+ roverlay.hook.run ( "<event>" )
+ # ...
+ roverlay.hook.run ( "<another event>" )
+
+
=========================
Configuration Reference
=========================
@@ -1936,6 +2300,11 @@ DISTDIR_STRATEGY
DISTDIR_VERIFY
Alias to OVERLAY_DISTDIR_VERIFY_.
+.. _DISTMAP_COMPRESSION:
+
+DISTMAP_COMPRESSION
+ Alias to OVERLAY_DISTMAP_COMPRESSION_.
+
.. _DISTMAP_FILE:
DISTMAP_FILE
@@ -2051,6 +2420,13 @@ OVERLAY_DISTDIR_VERIFY
Defaults to *no* as the verification is normally not needed.
+.. _OVERLAY_DISTMAP_COMPRESSION:
+
+OVERLAY_DISTMAP_COMPRESSION
+ Compression format for the distmap file. Choices are none, gzip/gz and
+ bzip2/bz2.
+
+ Defaults to bzip2.
.. _OVERLAY_DISTMAP_FILE:
@@ -2212,9 +2588,10 @@ USE_EXPAND_RENAME
.. _EVENT_HOOK:
EVENT_HOOK
- A script that is called for handling *events* (see Hooks_).
+ A script that is called for handling *events* (see `Event Hooks`_).
- Defaults to ADDITIONS_DIR_/hooks/mux.sh.
+ Defaults to <libexec dir>/hooks/mux.sh if roverlay has been installed
+ and ADDITIONS_DIR_/hooks/mux.sh otherwise.
.. _EVENT_HOOK_RESTRICT:
@@ -2226,6 +2603,10 @@ EVENT_HOOK_RESTRICT
Defaults to <not set>, which is equivalent to *deny all*.
+ ``EVENT_HOOK_RESTRICT="overlay_success"`` would allow the ``overlay_success``
+ event only, whereas ``EVENT_HOOK_RESTRICT="* -overlay_success"`` would
+ allow any event except for ``overlay_success``. Also see `event policy`_.
+
.. _FILTER_SHELL_ENV:
FILTER_SHELL_ENV
@@ -3242,261 +3623,3 @@ Its mode of operation of operation is best described in pseudo-code:
The dependency resolver can be **run as thread**, in which case the while loop
becomes "loop until resolver closes".
-
-
-=======
- undef
-=======
-
-<<TODO; this section has to merged/moved>>
-
----------
- distmap
----------
-
-The distmap is a file that stores information about the files in the package
-mirror directory (`OVERLAY_DISTDIR_ROOT`_).
-<<TODO>>
-
-
--------
- hooks
--------
-
-*roverlay* calls a script when certain events occur, e.g. after successful
-overlay creation.
-
-<<TODO>>
-
-* real work is offloaded to the called script
-* roverlay provides some env vars
-* roverlay provides a *$FUNCTIONS* file and a way to load other function files
-* ``EVENT_HOOK_RESTRICT="this -not_that *"``
-
-<<^^TODO>>
-
-.. Note::
-
- *roverlay* waits until the script terminates.
-
-
-++++++++++++++++++
- Hook environment
-++++++++++++++++++
-
-.. table:: environment variables provided by *roverlay*
-
- +----------------+------------------+-----------------------------------------+
- | variable | source | notes / description |
- +================+==================+=========================================+
- | PATH | os.environ | |
- +----------------+------------------+-----------------------------------------+
- | LOGNAME | os.environ | |
- +----------------+------------------+-----------------------------------------+
- | SHLVL | os.environ | |
- +----------------+------------------+-----------------------------------------+
- | TERM | os.environ | |
- +----------------+------------------+-----------------------------------------+
- | HOME | os.environ | |
- +----------------+------------------+-----------------------------------------+
- | ROVERLAY_PHASE | event | event that caused the script to run |
- +----------------+------------------+-----------------------------------------+
- | OVERLAY | config | overlay directory (`OVERLAY_DIR`_), |
- +----------------+------------------+ initial working directory |
- | S | *$OVERLAY* | |
- +----------------+------------------+ |
- | PWD | *$OVERLAY* | |
- +----------------+------------------+-----------------------------------------+
- | DISTROOT | config | package mirror directory |
- | | | (`OVERLAY_DISTDIR_ROOT`_) |
- +----------------+------------------+-----------------------------------------+
- | TMPDIR | os.environ, | directory for temporary files |
- | | *fallback* | |
- +----------------+------------------+ |
- | T | *$TMPDIR* | |
- +----------------+------------------+-----------------------------------------+
- | ADDITIONS_DIR | config | directory with supplementary files |
- | | | (`OVERLAY_ADDITIONS_DIR`_) |
- +----------------+------------------+ |
- | FILESDIR | *$ADDITIONS_DIR* | |
- +----------------+------------------+-----------------------------------------+
- | SHLIB | *$ADDITIONS_DIR* | directory with shell function files |
- | | | (optional, only set if it exists) |
- +----------------+------------------+-----------------------------------------+
- | FUNCTIONS | *$ADDITIONS_DIR* | file with essential shell functions |
- | | | (optional, only set if it exists) |
- +----------------+------------------+-----------------------------------------+
- | DEBUG | log level | *shbool* (``y`` or ``n``) that |
- | | | indicates whether debug messages should |
- | | | be printed |
- +----------------+------------------+-----------------------------------------+
- | VERBOSE | log level | *shbool* |
- +----------------+------------------+-----------------------------------------+
- | QUIET | log level | *shbool* that indicates whether scripts |
- | | | should be quiet |
- +----------------+------------------+-----------------------------------------+
- | NO_COLOR | *n/a* | *shbool*. Always set to *y* since |
- | | | colored output should not be produced |
- +----------------+------------------+-----------------------------------------+
- | NOSYNC | config | *shbool* that indicates whether data |
- | | | transfers from/to remote machines is |
- | | | allowed (NOSYNC_) |
- +----------------+------------------+-----------------------------------------+
- | EBUILD | config | the *ebuild* executable |
- +----------------+------------------+-----------------------------------------+
- | GIT_EDITOR | *n/a* | set to */bin/false* |
- +----------------+------------------+ |
- | GIT_ASKPASS | *n/a* | |
- +----------------+------------------+-----------------------------------------+
-
-
-The default *essential shell functions* file (*$FUNCTIONS*) makes,
-when included in the hook script, most of the enviroment variables readonly.
-
-
-.. table:: variables provided by *$FUNCTIONS*
-
- +-----------------+-------------------------------------------------------+
- | variable | description |
- +=================+=======================================================+
- | IFS_DEFAULT | default *internal field separator* |
- +-----------------+-------------------------------------------------------+
- | IFS_NEWLINE | *IFS* for iterating over text lines |
- +-----------------+-------------------------------------------------------+
- | DEVNULL | */dev/null* target (could also be a file) |
- +-----------------+-------------------------------------------------------+
- | EX_ERR | default error exit code |
- +-----------------+-------------------------------------------------------+
- | EX_ARG_ERR | default exit code for arg errors |
- +-----------------+-------------------------------------------------------+
- | SCRIPT_FILENAME | file name of the hook script |
- +-----------------+-------------------------------------------------------+
- | SCRIPT_NAME | name of the hook script (without file extension) |
- +-----------------+-------------------------------------------------------+
- | lf | reference to a function that loads additional shell |
- | | function files |
- +-----------------+-------------------------------------------------------+
-
-
-*$FUNCTIONS* also provides a number of shell functions:
-
-.. code-block:: sh
-
- # --- message ---
- #
- # void veinfo ( message )
- # Prints a message to stdout if $DEBUG is set to 'y'.
- #
- # void einfo ( message )
- # Prints a message to stdout if $VERBOSE is set to 'y'.
- #
- # void ewarn ( message )
- # Prints a message to stderr unless $QUIET is set to 'y'.
- #
- # void eerror ( message )
- # Prints a message to stderr.
- #
- #
- # --- core ---
- #
- # @noreturn die ( [message], [exit_code] ), raises exit()
- # Lets the script die with the given message/exit code.
- #
- # @noreturn OUT_OF_BOUNDS(), raises die()
- # Lets the script die due to insufficient arg count.
- #
- # int run_command ( *cmdv )
- # Logs a command and runs it afterwards.
- #
- # int run_command_logged ( *cmdv )
- # Logs a command, runs it and logs the result.
- #
- # void autodie ( *cmdv ), raises die()
- # Runs a command. Lets the script die if the command fails.
- #
- #
- # void load_functions ( *filenames, **SHLIB ), raises die()
- # Loads additional shell functions file(s) from $SHLIB.
- # (Referenced by $lf.)
- #
- # void dont_run_as_root(), raises die()
- # Lets the script die if it is run as root.
- #
- # int list_has ( word, *list_items )
- # Returns 0 if $word is in the given list.
- #
- # int qwhich ( *command )
- # Returns 0 if all listed commands are found by "which".
- #
- #
- # --- fs ---
- #
- # int dodir ( *dir )
- # Ensures that zero or more directories exist by creating them if
- # necessary. Returns the number of directories that could not be created.
- #
- #
- # --- str ---
- #
- # int yesno ( word, **YESNO_YES=0, **YESNO_NO=1, **YESNO_EMPTY=2 )
- # Returns $YESNO_YES if $word means "yes", $YESNO_EMPTY if $word is empty
- # and $YESNO_NO otherwise (if $word probably means "no").
- #
- # ~int str_trim ( *args )
- # Removes whitespace at the beginning and end of a string and replaces
- # any whitespace sequence within the string with a single space char.
- # Passes the return value of the underlying sed command.
- #
- # ~int str_upper ( *args )
- # Echoes the uppercase variant of stdin or *args.
- # Passes tr's return value.
- #
- # ~int str_lower ( *args )
- # Echoes the lowercase variant of stdin or *args.
- # Passes tr's return value.
- #
- # ~int str_field ( fieldspec, *args, **FIELD_SEPARATOR=' ' )
- # Echoes the requested fields of stdin or *args.
- # Passes cut's return value.
- #
- #
- # --- int ---
- #
- # int is_int ( word )
- # Returns 0 if $word is an integer, else 1.
- #
- # int is_natural ( word )
- # Returns 0 if $word is an integer >= 0, else 1.
- #
- # int is_positive ( word )
- # Returns 0 if $word is an integer >= 1, else 1.
- #
- # int is_negative ( word )
- # Returns 0 if $word is an integer < 0, else 1.
- #
-
-
-+++++++++++++++++++++
- Adding a hook event
-+++++++++++++++++++++
-
-in roverlay
-
-.. code-block:: python
-
- # import hook module
- import roverlay.hook
-
- # ...
- # then:
- roverlay.hook.run ( "<event>" )
- # ...
- roverlay.hook.run ( "<another event>" )
-
-
-in additions dir (assuming that mux.sh is used)
-
-.. code-block:: text
-
- <somehow create> ${ADDITIONS_DIR}/<hook>.<event>
- <somehow create> ${ADDITIONS_DIR}/<hook>/<event>.sh
^ permalink raw reply related [flat|nested] 41+ messages in thread* [gentoo-commits] proj/R_overlay:master commit in: doc/rst/
2013-06-26 17:29 [gentoo-commits] proj/R_overlay:gsoc13/next " André Erdmann
@ 2013-06-30 15:58 ` André Erdmann
0 siblings, 0 replies; 41+ messages in thread
From: André Erdmann @ 2013-06-30 15:58 UTC (permalink / raw
To: gentoo-commits
commit: 0678580564dcf6aa8e2587d3b04d62544265ca74
Author: André Erdmann <dywi <AT> mailerd <DOT> de>
AuthorDate: Wed Jun 26 17:28:39 2013 +0000
Commit: André Erdmann <dywi <AT> mailerd <DOT> de>
CommitDate: Wed Jun 26 17:28:39 2013 +0000
URL: http://git.overlays.gentoo.org/gitweb/?p=proj/R_overlay.git;a=commit;h=06785805
doc/rst: "run hooks", "distmap"
---
doc/rst/usage.rst | 643 ++++++++++++++++++++++++++++++++----------------------
1 file changed, 383 insertions(+), 260 deletions(-)
diff --git a/doc/rst/usage.rst b/doc/rst/usage.rst
index 078a7ac..2daedce 100644
--- a/doc/rst/usage.rst
+++ b/doc/rst/usage.rst
@@ -1010,6 +1010,22 @@ A local directory will never be modified.
consider using one of the **websync** repo types, websync_repo_ and
websync_pkglist_.
+---------
+ distmap
+---------
+
+*roverlay* uses a text file to store information about files in the package
+mirror directory (OVERLAY_DISTDIR_ROOT_). This is necessary for comparing
+package files from repos with files for which an ebuild has already been
+created (in previous runs).
+
+With the help of the *distmap file*, *roverlay* is able to determine whether
+upstream has changed a package file silently and creates a revision-bumped
+ebuild for the *new* package file.
+
+The *distmap file* can optionally be compressed (bzip2 or gzip), which
+reduces its size considerably.
+
=====================
Additions Directory
@@ -1790,6 +1806,354 @@ Moving such packages to a "R-package" sub directory would be possible, too:
END;
+=============
+ Event Hooks
+=============
+
+*roverlay* is able to call a script when certain events occur, e.g. after
+successful overlay creation, which can be used to perform additional actions
+without touching *roverlay's* source code.
+
+To realize this, *roverlay* determines whether a given event is permitted
+(`event policy`_) and, if so, creates a `hook environment`_ and runs the
+script. Additionally, shell scripts can load *roverlay's* *$FUNCTIONS* file,
+which provides extra functionality.
+
+.. Note::
+
+ *roverlay* waits until the script terminates and thus possibly waits
+ forever.
+
+
+----------------------
+ Default event script
+----------------------
+
+The default event script (``mux.sh``) loads *$FUNCTIONS* and then runs the
+following script files (by sourcing them), in order:
+
+#. all files from ADDITIONS_DIR_/hooks/<event> that end with *.sh*
+ (``<ADDITIONS_DIR>/hooks/<event>/*.sh``)
+#. all files ADDITIONS_DIR_/hooks that end with *.<event>*
+ (``<ADDITIONS_DIR>/hooks/*.<event>``)
+
+So, there are two naming schemes for *hook scripts*.
+Either one is acceptable, but it is advised to stay consistent.
+Having the same script at both locations results in executing it twice.
+
+.. Note::
+
+ The default event script enables *nounset* behavior, which causes the
+ shell command interpreter to exit abnormally if an unset variable is
+ accessed.
+
+
+++++++++++++++++++++++++++
+ Activating a hook script
+++++++++++++++++++++++++++
+
+Activating a hook script can be done by symlinking it:
+
+.. code-block:: text
+
+ ln -s <real script> ${ADDITIONS_DIR}/hooks/<event>/<name>.sh
+ # or
+ ln -s <real script> ${ADDITIONS_DIR}/hooks/<name>.<event>
+
+
+++++++++++++++++++++++++++
+ Adding a new hook script
+++++++++++++++++++++++++++
+
+As hinted before, *hook scripts* are simple shell scripts. The following
+template gives an idea of how to write them:
+
+.. code-block:: sh
+
+ #!/bin/sh
+ #set -u
+
+ # load essential functions
+ # (not necessary when using the default event script)
+ . "${FUNCTIONS?}" || exit
+
+ ## load additional function files, if any
+ #$lf <name(s)>
+
+ # script body
+ #
+ # when redirecting output to $DEVNULL, use ">>" instead of ">" as
+ # $DEVNULL could be a file
+ #ls >>${DEVNULL}
+ #
+ # ...
+
+
+ # the script must not exit if everything went well (return is ok)
+ return 0
+
+
+--------------
+ Event Policy
+--------------
+
+The *event policy* controls whether a certain event actually triggers a script
+call or not.
+It is constructed by parsing the EVENT_HOOK_RESTRICT_ config option:
+
+* a word prefixed by ``-`` means *deny specific event* (-> blacklist)
+* the asterisk char ``*`` (or ``+*``) sets the policy to
+ *allow unless denied* (blacklist) or *allow all*
+* a word prefixed by ``+`` or without a prefix char means
+ *allow specific event* (-> whitelist)
+* the asterisk char with ``-`` as prefix (``-*``) sets the policy to
+ *deny unless allowed* (whitelist) or *deny all*
+
+
+The policy defaults to *allow all* if ``EVENT_HOOK_RESTRICT`` is not set in
+the config file. An empty string sets the policy to *deny all*.
+
+
+------------------
+ Hook Environment
+------------------
+
+.. table:: environment variables provided by *roverlay*
+
+ +----------------+-------------------+-----------------------------------------+
+ | variable | source | notes / description |
+ +================+===================+=========================================+
+ | PATH | os.environ | |
+ +----------------+-------------------+-----------------------------------------+
+ | LOGNAME | os.environ | |
+ +----------------+-------------------+-----------------------------------------+
+ | SHLVL | os.environ | |
+ +----------------+-------------------+-----------------------------------------+
+ | TERM | os.environ | |
+ +----------------+-------------------+-----------------------------------------+
+ | HOME | os.environ | |
+ +----------------+-------------------+-----------------------------------------+
+ | ROVERLAY_PHASE | event | event that caused the script to run |
+ +----------------+-------------------+-----------------------------------------+
+ | OVERLAY | config | overlay directory (`OVERLAY_DIR`_), |
+ +----------------+-------------------+ initial working directory |
+ | S | *$OVERLAY* | |
+ +----------------+-------------------+ |
+ | PWD | *$OVERLAY* | |
+ +----------------+-------------------+-----------------------------------------+
+ | DISTROOT | config | package mirror directory |
+ | | | (`OVERLAY_DISTDIR_ROOT`_) |
+ +----------------+-------------------+-----------------------------------------+
+ | TMPDIR | os.environ, | directory for temporary files |
+ | | *fallback* | |
+ +----------------+-------------------+ |
+ | T | *$TMPDIR* | |
+ +----------------+-------------------+-----------------------------------------+
+ | ADDITIONS_DIR | config | directory with supplementary files |
+ | | | (`OVERLAY_ADDITIONS_DIR`_) |
+ +----------------+-------------------+ |
+ | FILESDIR | *$ADDITIONS_DIR* | |
+ +----------------+-------------------+-----------------------------------------+
+ | SHLIB | *$ADDITIONS_DIR*, | A list of directories with shell |
+ | | *installed?* | function files |
+ | | | (optional, only set if any dir exists) |
+ +----------------+-------------------+-----------------------------------------+
+ | FUNCTIONS | *$ADDITIONS_DIR*, | file with essential shell functions |
+ | | *installed?* | (optional, only set if it exists) |
+ +----------------+-------------------+-----------------------------------------+
+ | DEBUG | log level | *shbool* (``y`` or ``n``) that |
+ | | | indicates whether debug messages should |
+ | | | be printed |
+ +----------------+-------------------+-----------------------------------------+
+ | VERBOSE | log level | *shbool* |
+ +----------------+-------------------+-----------------------------------------+
+ | QUIET | log level | *shbool* that indicates whether scripts |
+ | | | should be quiet |
+ +----------------+-------------------+-----------------------------------------+
+ | NO_COLOR | *n/a* | *shbool*. Always set to *y* since |
+ | | | colored output should not be produced |
+ +----------------+-------------------+-----------------------------------------+
+ | NOSYNC | config | *shbool* that indicates whether data |
+ | | | transfers from/to remote machines is |
+ | | | allowed (NOSYNC_) |
+ +----------------+-------------------+-----------------------------------------+
+ | EBUILD | config | the *ebuild* executable |
+ +----------------+-------------------+-----------------------------------------+
+ | GIT_EDITOR | *n/a* | set to */bin/false* |
+ +----------------+-------------------+ |
+ | GIT_ASKPASS | *n/a* | |
+ +----------------+-------------------+-----------------------------------------+
+
+
+The default *essential shell functions* file (*$FUNCTIONS*) makes,
+when included in the hook script, most of the enviroment variables readonly.
+
+
+.. table:: variables provided by *$FUNCTIONS*
+
+ +-----------------+-------------------------------------------------------+
+ | variable | description |
+ +=================+=======================================================+
+ | IFS_DEFAULT | default *internal field separator* |
+ +-----------------+-------------------------------------------------------+
+ | IFS_NEWLINE | *IFS* for iterating over text lines |
+ +-----------------+-------------------------------------------------------+
+ | DEVNULL | */dev/null* target (could also be a file) |
+ +-----------------+-------------------------------------------------------+
+ | EX_ERR | default error exit code |
+ +-----------------+-------------------------------------------------------+
+ | EX_ARG_ERR | default exit code for arg errors |
+ +-----------------+-------------------------------------------------------+
+ | SCRIPT_FILENAME | file name of the hook script |
+ +-----------------+-------------------------------------------------------+
+ | SCRIPT_NAME | name of the hook script (without file extension) |
+ +-----------------+-------------------------------------------------------+
+ | lf | reference to a function that loads additional shell |
+ | | function files |
+ +-----------------+-------------------------------------------------------+
+
+
+*$FUNCTIONS* also provides a number of shell functions:
+
+.. code-block:: sh
+
+ # --- message ---
+ #
+ # void veinfo ( message )
+ # Prints a message to stdout if $DEBUG is set to 'y'.
+ #
+ # void einfo ( message )
+ # Prints a message to stdout if $VERBOSE is set to 'y'.
+ #
+ # void ewarn ( message )
+ # Prints a message to stderr unless $QUIET is set to 'y'.
+ #
+ # void eerror ( message )
+ # Prints a message to stderr.
+ #
+ #
+ # --- core ---
+ #
+ # @noreturn die ( [message], [exit_code] ), raises exit()
+ # Lets the script die with the given message/exit code.
+ #
+ # @noreturn OUT_OF_BOUNDS(), raises die()
+ # Lets the script die due to insufficient arg count.
+ #
+ # int run_command ( *cmdv )
+ # Logs a command and runs it afterwards.
+ #
+ # int run_command_logged ( *cmdv )
+ # Logs a command, runs it and logs the result.
+ #
+ # void autodie ( *cmdv ), raises die()
+ # Runs a command. Lets the script die if the command fails.
+ #
+ #
+ # void load_functions ( *filenames, **SHLIB ), raises die()
+ # Loads additional shell functions file(s) from $SHLIB.
+ # (Referenced by $lf.)
+ #
+ # void dont_run_as_root(), raises die()
+ # Lets the script die if it is run as root.
+ #
+ # int list_has ( word, *list_items )
+ # Returns 0 if $word is in the given list, else 1.
+ #
+ # int qwhich ( *command )
+ # Returns 0 if all listed commands are found by "which", else 1.
+ #
+ #
+ # --- fs ---
+ #
+ # int dodir ( *dir )
+ # Ensures that zero or more directories exist by creating them if
+ # necessary. Returns the number of directories that could not be created.
+ #
+ #
+ # --- str ---
+ #
+ # int yesno ( word, **YESNO_YES=0, **YESNO_NO=1, **YESNO_EMPTY=2 )
+ # Returns $YESNO_YES if $word means "yes", $YESNO_EMPTY if $word is empty
+ # and $YESNO_NO otherwise (if $word probably means "no").
+ #
+ # ~int str_trim ( *args )
+ # Removes whitespace at the beginning and end of a string and replaces
+ # any whitespace sequence within the string with a single space char.
+ # Passes the return value of the underlying sed command.
+ #
+ # ~int str_upper ( *args )
+ # Echoes the uppercase variant of stdin or *args.
+ # Passes tr's return value.
+ #
+ # ~int str_lower ( *args )
+ # Echoes the lowercase variant of stdin or *args.
+ # Passes tr's return value.
+ #
+ # ~int str_field ( fieldspec, *args, **FIELD_SEPARATOR=' ' )
+ # Echoes the requested fields of stdin or *args.
+ # Passes cut's return value.
+ #
+ #
+ # --- int ---
+ #
+ # int is_int ( word )
+ # Returns 0 if $word is an integer, else 1.
+ #
+ # int is_natural ( word )
+ # Returns 0 if $word is an integer >= 0, else 1.
+ #
+ # int is_positive ( word )
+ # Returns 0 if $word is an integer >= 1, else 1.
+ #
+ # int is_negative ( word )
+ # Returns 0 if $word is an integer < 0, else 1.
+ #
+
+
+------------------------
+ Adding a function file
+------------------------
+
+Function files are shell script files that provide functions and variables.
+They should, however, not execute any code directly.
+
+The template below illustrates how to write function files:
+
+.. code-block:: sh
+
+ # protect against repeated inclusion of this file
+ # (replace <name> with a unique identifier)
+ if [ -z "${__HAVE_<name>__-}" ]; then
+ readonly __HAVE_<name>__=y
+
+ # function file body
+ # ...
+
+ fi
+
+Shell function files should be put into ``<ADDITIONS_DIR>/shlib``.
+
+---------------------
+ Adding a hook event
+---------------------
+
+Adding a new event has to be done in *roverlay's* source code and is a rather
+trivial task. The ``roverlay.hook`` module implements a function for running
+the event script:
+
+.. code-block:: python
+
+ # import hook module
+ import roverlay.hook
+
+ # ...
+ # then, somewhere in the code
+ roverlay.hook.run ( "<event>" )
+ # ...
+ roverlay.hook.run ( "<another event>" )
+
+
=========================
Configuration Reference
=========================
@@ -1936,6 +2300,11 @@ DISTDIR_STRATEGY
DISTDIR_VERIFY
Alias to OVERLAY_DISTDIR_VERIFY_.
+.. _DISTMAP_COMPRESSION:
+
+DISTMAP_COMPRESSION
+ Alias to OVERLAY_DISTMAP_COMPRESSION_.
+
.. _DISTMAP_FILE:
DISTMAP_FILE
@@ -2051,6 +2420,13 @@ OVERLAY_DISTDIR_VERIFY
Defaults to *no* as the verification is normally not needed.
+.. _OVERLAY_DISTMAP_COMPRESSION:
+
+OVERLAY_DISTMAP_COMPRESSION
+ Compression format for the distmap file. Choices are none, gzip/gz and
+ bzip2/bz2.
+
+ Defaults to bzip2.
.. _OVERLAY_DISTMAP_FILE:
@@ -2212,9 +2588,10 @@ USE_EXPAND_RENAME
.. _EVENT_HOOK:
EVENT_HOOK
- A script that is called for handling *events* (see Hooks_).
+ A script that is called for handling *events* (see `Event Hooks`_).
- Defaults to ADDITIONS_DIR_/hooks/mux.sh.
+ Defaults to <libexec dir>/hooks/mux.sh if roverlay has been installed
+ and ADDITIONS_DIR_/hooks/mux.sh otherwise.
.. _EVENT_HOOK_RESTRICT:
@@ -2226,6 +2603,10 @@ EVENT_HOOK_RESTRICT
Defaults to <not set>, which is equivalent to *deny all*.
+ ``EVENT_HOOK_RESTRICT="overlay_success"`` would allow the ``overlay_success``
+ event only, whereas ``EVENT_HOOK_RESTRICT="* -overlay_success"`` would
+ allow any event except for ``overlay_success``. Also see `event policy`_.
+
.. _FILTER_SHELL_ENV:
FILTER_SHELL_ENV
@@ -3242,261 +3623,3 @@ Its mode of operation of operation is best described in pseudo-code:
The dependency resolver can be **run as thread**, in which case the while loop
becomes "loop until resolver closes".
-
-
-=======
- undef
-=======
-
-<<TODO; this section has to merged/moved>>
-
----------
- distmap
----------
-
-The distmap is a file that stores information about the files in the package
-mirror directory (`OVERLAY_DISTDIR_ROOT`_).
-<<TODO>>
-
-
--------
- hooks
--------
-
-*roverlay* calls a script when certain events occur, e.g. after successful
-overlay creation.
-
-<<TODO>>
-
-* real work is offloaded to the called script
-* roverlay provides some env vars
-* roverlay provides a *$FUNCTIONS* file and a way to load other function files
-* ``EVENT_HOOK_RESTRICT="this -not_that *"``
-
-<<^^TODO>>
-
-.. Note::
-
- *roverlay* waits until the script terminates.
-
-
-++++++++++++++++++
- Hook environment
-++++++++++++++++++
-
-.. table:: environment variables provided by *roverlay*
-
- +----------------+------------------+-----------------------------------------+
- | variable | source | notes / description |
- +================+==================+=========================================+
- | PATH | os.environ | |
- +----------------+------------------+-----------------------------------------+
- | LOGNAME | os.environ | |
- +----------------+------------------+-----------------------------------------+
- | SHLVL | os.environ | |
- +----------------+------------------+-----------------------------------------+
- | TERM | os.environ | |
- +----------------+------------------+-----------------------------------------+
- | HOME | os.environ | |
- +----------------+------------------+-----------------------------------------+
- | ROVERLAY_PHASE | event | event that caused the script to run |
- +----------------+------------------+-----------------------------------------+
- | OVERLAY | config | overlay directory (`OVERLAY_DIR`_), |
- +----------------+------------------+ initial working directory |
- | S | *$OVERLAY* | |
- +----------------+------------------+ |
- | PWD | *$OVERLAY* | |
- +----------------+------------------+-----------------------------------------+
- | DISTROOT | config | package mirror directory |
- | | | (`OVERLAY_DISTDIR_ROOT`_) |
- +----------------+------------------+-----------------------------------------+
- | TMPDIR | os.environ, | directory for temporary files |
- | | *fallback* | |
- +----------------+------------------+ |
- | T | *$TMPDIR* | |
- +----------------+------------------+-----------------------------------------+
- | ADDITIONS_DIR | config | directory with supplementary files |
- | | | (`OVERLAY_ADDITIONS_DIR`_) |
- +----------------+------------------+ |
- | FILESDIR | *$ADDITIONS_DIR* | |
- +----------------+------------------+-----------------------------------------+
- | SHLIB | *$ADDITIONS_DIR* | directory with shell function files |
- | | | (optional, only set if it exists) |
- +----------------+------------------+-----------------------------------------+
- | FUNCTIONS | *$ADDITIONS_DIR* | file with essential shell functions |
- | | | (optional, only set if it exists) |
- +----------------+------------------+-----------------------------------------+
- | DEBUG | log level | *shbool* (``y`` or ``n``) that |
- | | | indicates whether debug messages should |
- | | | be printed |
- +----------------+------------------+-----------------------------------------+
- | VERBOSE | log level | *shbool* |
- +----------------+------------------+-----------------------------------------+
- | QUIET | log level | *shbool* that indicates whether scripts |
- | | | should be quiet |
- +----------------+------------------+-----------------------------------------+
- | NO_COLOR | *n/a* | *shbool*. Always set to *y* since |
- | | | colored output should not be produced |
- +----------------+------------------+-----------------------------------------+
- | NOSYNC | config | *shbool* that indicates whether data |
- | | | transfers from/to remote machines is |
- | | | allowed (NOSYNC_) |
- +----------------+------------------+-----------------------------------------+
- | EBUILD | config | the *ebuild* executable |
- +----------------+------------------+-----------------------------------------+
- | GIT_EDITOR | *n/a* | set to */bin/false* |
- +----------------+------------------+ |
- | GIT_ASKPASS | *n/a* | |
- +----------------+------------------+-----------------------------------------+
-
-
-The default *essential shell functions* file (*$FUNCTIONS*) makes,
-when included in the hook script, most of the enviroment variables readonly.
-
-
-.. table:: variables provided by *$FUNCTIONS*
-
- +-----------------+-------------------------------------------------------+
- | variable | description |
- +=================+=======================================================+
- | IFS_DEFAULT | default *internal field separator* |
- +-----------------+-------------------------------------------------------+
- | IFS_NEWLINE | *IFS* for iterating over text lines |
- +-----------------+-------------------------------------------------------+
- | DEVNULL | */dev/null* target (could also be a file) |
- +-----------------+-------------------------------------------------------+
- | EX_ERR | default error exit code |
- +-----------------+-------------------------------------------------------+
- | EX_ARG_ERR | default exit code for arg errors |
- +-----------------+-------------------------------------------------------+
- | SCRIPT_FILENAME | file name of the hook script |
- +-----------------+-------------------------------------------------------+
- | SCRIPT_NAME | name of the hook script (without file extension) |
- +-----------------+-------------------------------------------------------+
- | lf | reference to a function that loads additional shell |
- | | function files |
- +-----------------+-------------------------------------------------------+
-
-
-*$FUNCTIONS* also provides a number of shell functions:
-
-.. code-block:: sh
-
- # --- message ---
- #
- # void veinfo ( message )
- # Prints a message to stdout if $DEBUG is set to 'y'.
- #
- # void einfo ( message )
- # Prints a message to stdout if $VERBOSE is set to 'y'.
- #
- # void ewarn ( message )
- # Prints a message to stderr unless $QUIET is set to 'y'.
- #
- # void eerror ( message )
- # Prints a message to stderr.
- #
- #
- # --- core ---
- #
- # @noreturn die ( [message], [exit_code] ), raises exit()
- # Lets the script die with the given message/exit code.
- #
- # @noreturn OUT_OF_BOUNDS(), raises die()
- # Lets the script die due to insufficient arg count.
- #
- # int run_command ( *cmdv )
- # Logs a command and runs it afterwards.
- #
- # int run_command_logged ( *cmdv )
- # Logs a command, runs it and logs the result.
- #
- # void autodie ( *cmdv ), raises die()
- # Runs a command. Lets the script die if the command fails.
- #
- #
- # void load_functions ( *filenames, **SHLIB ), raises die()
- # Loads additional shell functions file(s) from $SHLIB.
- # (Referenced by $lf.)
- #
- # void dont_run_as_root(), raises die()
- # Lets the script die if it is run as root.
- #
- # int list_has ( word, *list_items )
- # Returns 0 if $word is in the given list.
- #
- # int qwhich ( *command )
- # Returns 0 if all listed commands are found by "which".
- #
- #
- # --- fs ---
- #
- # int dodir ( *dir )
- # Ensures that zero or more directories exist by creating them if
- # necessary. Returns the number of directories that could not be created.
- #
- #
- # --- str ---
- #
- # int yesno ( word, **YESNO_YES=0, **YESNO_NO=1, **YESNO_EMPTY=2 )
- # Returns $YESNO_YES if $word means "yes", $YESNO_EMPTY if $word is empty
- # and $YESNO_NO otherwise (if $word probably means "no").
- #
- # ~int str_trim ( *args )
- # Removes whitespace at the beginning and end of a string and replaces
- # any whitespace sequence within the string with a single space char.
- # Passes the return value of the underlying sed command.
- #
- # ~int str_upper ( *args )
- # Echoes the uppercase variant of stdin or *args.
- # Passes tr's return value.
- #
- # ~int str_lower ( *args )
- # Echoes the lowercase variant of stdin or *args.
- # Passes tr's return value.
- #
- # ~int str_field ( fieldspec, *args, **FIELD_SEPARATOR=' ' )
- # Echoes the requested fields of stdin or *args.
- # Passes cut's return value.
- #
- #
- # --- int ---
- #
- # int is_int ( word )
- # Returns 0 if $word is an integer, else 1.
- #
- # int is_natural ( word )
- # Returns 0 if $word is an integer >= 0, else 1.
- #
- # int is_positive ( word )
- # Returns 0 if $word is an integer >= 1, else 1.
- #
- # int is_negative ( word )
- # Returns 0 if $word is an integer < 0, else 1.
- #
-
-
-+++++++++++++++++++++
- Adding a hook event
-+++++++++++++++++++++
-
-in roverlay
-
-.. code-block:: python
-
- # import hook module
- import roverlay.hook
-
- # ...
- # then:
- roverlay.hook.run ( "<event>" )
- # ...
- roverlay.hook.run ( "<another event>" )
-
-
-in additions dir (assuming that mux.sh is used)
-
-.. code-block:: text
-
- <somehow create> ${ADDITIONS_DIR}/<hook>.<event>
- <somehow create> ${ADDITIONS_DIR}/<hook>/<event>.sh
^ permalink raw reply related [flat|nested] 41+ messages in thread
* [gentoo-commits] proj/R_overlay:gsoc13/next commit in: doc/rst/
@ 2013-06-25 21:10 André Erdmann
2013-06-30 15:58 ` [gentoo-commits] proj/R_overlay:master " André Erdmann
0 siblings, 1 reply; 41+ messages in thread
From: André Erdmann @ 2013-06-25 21:10 UTC (permalink / raw
To: gentoo-commits
commit: adbe0bb8d1edc7cbff598b4834fbaceb7335fbb3
Author: André Erdmann <dywi <AT> mailerd <DOT> de>
AuthorDate: Tue Jun 25 21:10:02 2013 +0000
Commit: André Erdmann <dywi <AT> mailerd <DOT> de>
CommitDate: Tue Jun 25 21:10:02 2013 +0000
URL: http://git.overlays.gentoo.org/gitweb/?p=proj/R_overlay.git;a=commit;h=adbe0bb8
doc/rst: doc for "run hooks" (incomplete)
---
doc/rst/usage.rst | 354 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 354 insertions(+)
diff --git a/doc/rst/usage.rst b/doc/rst/usage.rst
index 8d06bab..078a7ac 100644
--- a/doc/rst/usage.rst
+++ b/doc/rst/usage.rst
@@ -314,6 +314,12 @@ have reasonable defaults if *roverlay* has been installed using *emerge*:
Example: USE_EXPAND_RENAME = ~/roverlay/config/useflag_rename
+ CACHEDIR
+ Directory for generated files that do not belong to the overlay, e.g.
+ the *distmap* file. This option is **required**.
+
+ Example: CACHEDIR = ~/roverlay/workdir/cache
+
OVERLAY_ECLASS
This option lists eclass files that should be imported into the overlay
(into *OVERLAY_DIR*/eclass/) and inherited in all ebuilds.
@@ -409,6 +415,10 @@ to know in detail what *roverlay* does before running it.
--nosync, --no-sync
Disable downloading of R packages.
+--distmap-verify
+ Enforce verification of R packages in the package mirror directory.
+ This also tries to recreate the distmap.
+
--no-incremental
Force recreation of existing ebuilds
@@ -1834,6 +1844,16 @@ The following sections will list all config entries.
misc options
--------------
+.. _CACHEDIR:
+
+CACHEDIR
+ Directory for persistent files that don't belong to the overlay, e.g.
+ the distmap file.
+
+ This option is **required**.
+
+ <<TODO: default value!>>
+
.. _DISTFILES:
DISTFILES
@@ -1863,6 +1883,14 @@ EBUILD_PROG
Defaults to *ebuild*, which should be fine in most cases.
+.. _NOSYNC:
+
+NOSYNC
+ A *bool* that controls whether *syncing*, i.e. data transfers from/to
+ remote machines, is allowed or forbidden.
+
+ Defaults to *no*.
+
.. _RSYNC_BWLIMIT:
RSYNC_BWLIMIT
@@ -1872,6 +1900,7 @@ RSYNC_BWLIMIT
Defaults to <not set>, which disables bandwidth limitation.
+
-----------------
overlay options
-----------------
@@ -1901,6 +1930,17 @@ DISTDIR_FLAT
DISTDIR_STRATEGY
Alias to OVERLAY_DISTDIR_STRATEGY_.
+
+.. _DISTDIR_VERIFY:
+
+DISTDIR_VERIFY
+ Alias to OVERLAY_DISTDIR_VERIFY_.
+
+.. _DISTMAP_FILE:
+
+DISTMAP_FILE
+ Alias to OVERLAY_DISTMAP_FILE_.
+
.. _EBUILD_USE_EXPAND_NAME:
EBUILD_USE_EXPAND_NAME
@@ -2002,6 +2042,23 @@ OVERLAY_DISTDIR_STRATEGY
This option is **required**, but has a reasonable value in the default
config file, "hardlink symlink".
+.. _OVERLAY_DISTDIR_VERIFY:
+
+OVERLAY_DISTDIR_VERIFY
+ A *bool* that controls whether file integrity of *OVERLAY_DISTDIR_ROOT*
+ should be checked on startup. This is an expensive operation since each
+ file have to be read once.
+
+ Defaults to *no* as the verification is normally not needed.
+
+
+.. _OVERLAY_DISTMAP_FILE:
+
+OVERLAY_DISTMAP_FILE:
+ File path to the distmap file.
+
+ Defaults to <not set>, which results in CACHEDIR_/distmap.db.
+
.. _OVERLAY_ECLASS:
OVERLAY_ECLASS
@@ -2148,6 +2205,45 @@ USE_EXPAND_DESC
USE_EXPAND_RENAME
Alias to EBUILD_USE_EXPAND_RENAME_.
+---------------------------
+ shell environment / hooks
+---------------------------
+
+.. _EVENT_HOOK:
+
+EVENT_HOOK
+ A script that is called for handling *events* (see Hooks_).
+
+ Defaults to ADDITIONS_DIR_/hooks/mux.sh.
+
+.. _EVENT_HOOK_RESTRICT:
+
+EVENT_HOOK_RESTRICT
+ A list of *events* that are allowed (``<event>``, ``+<event>``) or
+ forbidden (``-<event>``). The asterisk wildcard character can be used to
+ set the default policy to *allow unless forbidden* (``*``) or
+ *deny unless allowed* (``-*``).
+
+ Defaults to <not set>, which is equivalent to *deny all*.
+
+.. _FILTER_SHELL_ENV:
+
+FILTER_SHELL_ENV
+ A *bool* that controls whether the hook environment should be filtered
+ or not.
+
+ Defaults to *true*.
+
+.. _HOOK:
+
+HOOK
+ Alias to EVENT_HOOK_.
+
+.. _HOOK_RESTRICT:
+
+HOOK_RESTRICT
+ Alias to EVENT_HOOK_RESTRICT_.
+
---------
logging
---------
@@ -3146,3 +3242,261 @@ Its mode of operation of operation is best described in pseudo-code:
The dependency resolver can be **run as thread**, in which case the while loop
becomes "loop until resolver closes".
+
+
+=======
+ undef
+=======
+
+<<TODO; this section has to merged/moved>>
+
+---------
+ distmap
+---------
+
+The distmap is a file that stores information about the files in the package
+mirror directory (`OVERLAY_DISTDIR_ROOT`_).
+<<TODO>>
+
+
+-------
+ hooks
+-------
+
+*roverlay* calls a script when certain events occur, e.g. after successful
+overlay creation.
+
+<<TODO>>
+
+* real work is offloaded to the called script
+* roverlay provides some env vars
+* roverlay provides a *$FUNCTIONS* file and a way to load other function files
+* ``EVENT_HOOK_RESTRICT="this -not_that *"``
+
+<<^^TODO>>
+
+.. Note::
+
+ *roverlay* waits until the script terminates.
+
+
+++++++++++++++++++
+ Hook environment
+++++++++++++++++++
+
+.. table:: environment variables provided by *roverlay*
+
+ +----------------+------------------+-----------------------------------------+
+ | variable | source | notes / description |
+ +================+==================+=========================================+
+ | PATH | os.environ | |
+ +----------------+------------------+-----------------------------------------+
+ | LOGNAME | os.environ | |
+ +----------------+------------------+-----------------------------------------+
+ | SHLVL | os.environ | |
+ +----------------+------------------+-----------------------------------------+
+ | TERM | os.environ | |
+ +----------------+------------------+-----------------------------------------+
+ | HOME | os.environ | |
+ +----------------+------------------+-----------------------------------------+
+ | ROVERLAY_PHASE | event | event that caused the script to run |
+ +----------------+------------------+-----------------------------------------+
+ | OVERLAY | config | overlay directory (`OVERLAY_DIR`_), |
+ +----------------+------------------+ initial working directory |
+ | S | *$OVERLAY* | |
+ +----------------+------------------+ |
+ | PWD | *$OVERLAY* | |
+ +----------------+------------------+-----------------------------------------+
+ | DISTROOT | config | package mirror directory |
+ | | | (`OVERLAY_DISTDIR_ROOT`_) |
+ +----------------+------------------+-----------------------------------------+
+ | TMPDIR | os.environ, | directory for temporary files |
+ | | *fallback* | |
+ +----------------+------------------+ |
+ | T | *$TMPDIR* | |
+ +----------------+------------------+-----------------------------------------+
+ | ADDITIONS_DIR | config | directory with supplementary files |
+ | | | (`OVERLAY_ADDITIONS_DIR`_) |
+ +----------------+------------------+ |
+ | FILESDIR | *$ADDITIONS_DIR* | |
+ +----------------+------------------+-----------------------------------------+
+ | SHLIB | *$ADDITIONS_DIR* | directory with shell function files |
+ | | | (optional, only set if it exists) |
+ +----------------+------------------+-----------------------------------------+
+ | FUNCTIONS | *$ADDITIONS_DIR* | file with essential shell functions |
+ | | | (optional, only set if it exists) |
+ +----------------+------------------+-----------------------------------------+
+ | DEBUG | log level | *shbool* (``y`` or ``n``) that |
+ | | | indicates whether debug messages should |
+ | | | be printed |
+ +----------------+------------------+-----------------------------------------+
+ | VERBOSE | log level | *shbool* |
+ +----------------+------------------+-----------------------------------------+
+ | QUIET | log level | *shbool* that indicates whether scripts |
+ | | | should be quiet |
+ +----------------+------------------+-----------------------------------------+
+ | NO_COLOR | *n/a* | *shbool*. Always set to *y* since |
+ | | | colored output should not be produced |
+ +----------------+------------------+-----------------------------------------+
+ | NOSYNC | config | *shbool* that indicates whether data |
+ | | | transfers from/to remote machines is |
+ | | | allowed (NOSYNC_) |
+ +----------------+------------------+-----------------------------------------+
+ | EBUILD | config | the *ebuild* executable |
+ +----------------+------------------+-----------------------------------------+
+ | GIT_EDITOR | *n/a* | set to */bin/false* |
+ +----------------+------------------+ |
+ | GIT_ASKPASS | *n/a* | |
+ +----------------+------------------+-----------------------------------------+
+
+
+The default *essential shell functions* file (*$FUNCTIONS*) makes,
+when included in the hook script, most of the enviroment variables readonly.
+
+
+.. table:: variables provided by *$FUNCTIONS*
+
+ +-----------------+-------------------------------------------------------+
+ | variable | description |
+ +=================+=======================================================+
+ | IFS_DEFAULT | default *internal field separator* |
+ +-----------------+-------------------------------------------------------+
+ | IFS_NEWLINE | *IFS* for iterating over text lines |
+ +-----------------+-------------------------------------------------------+
+ | DEVNULL | */dev/null* target (could also be a file) |
+ +-----------------+-------------------------------------------------------+
+ | EX_ERR | default error exit code |
+ +-----------------+-------------------------------------------------------+
+ | EX_ARG_ERR | default exit code for arg errors |
+ +-----------------+-------------------------------------------------------+
+ | SCRIPT_FILENAME | file name of the hook script |
+ +-----------------+-------------------------------------------------------+
+ | SCRIPT_NAME | name of the hook script (without file extension) |
+ +-----------------+-------------------------------------------------------+
+ | lf | reference to a function that loads additional shell |
+ | | function files |
+ +-----------------+-------------------------------------------------------+
+
+
+*$FUNCTIONS* also provides a number of shell functions:
+
+.. code-block:: sh
+
+ # --- message ---
+ #
+ # void veinfo ( message )
+ # Prints a message to stdout if $DEBUG is set to 'y'.
+ #
+ # void einfo ( message )
+ # Prints a message to stdout if $VERBOSE is set to 'y'.
+ #
+ # void ewarn ( message )
+ # Prints a message to stderr unless $QUIET is set to 'y'.
+ #
+ # void eerror ( message )
+ # Prints a message to stderr.
+ #
+ #
+ # --- core ---
+ #
+ # @noreturn die ( [message], [exit_code] ), raises exit()
+ # Lets the script die with the given message/exit code.
+ #
+ # @noreturn OUT_OF_BOUNDS(), raises die()
+ # Lets the script die due to insufficient arg count.
+ #
+ # int run_command ( *cmdv )
+ # Logs a command and runs it afterwards.
+ #
+ # int run_command_logged ( *cmdv )
+ # Logs a command, runs it and logs the result.
+ #
+ # void autodie ( *cmdv ), raises die()
+ # Runs a command. Lets the script die if the command fails.
+ #
+ #
+ # void load_functions ( *filenames, **SHLIB ), raises die()
+ # Loads additional shell functions file(s) from $SHLIB.
+ # (Referenced by $lf.)
+ #
+ # void dont_run_as_root(), raises die()
+ # Lets the script die if it is run as root.
+ #
+ # int list_has ( word, *list_items )
+ # Returns 0 if $word is in the given list.
+ #
+ # int qwhich ( *command )
+ # Returns 0 if all listed commands are found by "which".
+ #
+ #
+ # --- fs ---
+ #
+ # int dodir ( *dir )
+ # Ensures that zero or more directories exist by creating them if
+ # necessary. Returns the number of directories that could not be created.
+ #
+ #
+ # --- str ---
+ #
+ # int yesno ( word, **YESNO_YES=0, **YESNO_NO=1, **YESNO_EMPTY=2 )
+ # Returns $YESNO_YES if $word means "yes", $YESNO_EMPTY if $word is empty
+ # and $YESNO_NO otherwise (if $word probably means "no").
+ #
+ # ~int str_trim ( *args )
+ # Removes whitespace at the beginning and end of a string and replaces
+ # any whitespace sequence within the string with a single space char.
+ # Passes the return value of the underlying sed command.
+ #
+ # ~int str_upper ( *args )
+ # Echoes the uppercase variant of stdin or *args.
+ # Passes tr's return value.
+ #
+ # ~int str_lower ( *args )
+ # Echoes the lowercase variant of stdin or *args.
+ # Passes tr's return value.
+ #
+ # ~int str_field ( fieldspec, *args, **FIELD_SEPARATOR=' ' )
+ # Echoes the requested fields of stdin or *args.
+ # Passes cut's return value.
+ #
+ #
+ # --- int ---
+ #
+ # int is_int ( word )
+ # Returns 0 if $word is an integer, else 1.
+ #
+ # int is_natural ( word )
+ # Returns 0 if $word is an integer >= 0, else 1.
+ #
+ # int is_positive ( word )
+ # Returns 0 if $word is an integer >= 1, else 1.
+ #
+ # int is_negative ( word )
+ # Returns 0 if $word is an integer < 0, else 1.
+ #
+
+
++++++++++++++++++++++
+ Adding a hook event
++++++++++++++++++++++
+
+in roverlay
+
+.. code-block:: python
+
+ # import hook module
+ import roverlay.hook
+
+ # ...
+ # then:
+ roverlay.hook.run ( "<event>" )
+ # ...
+ roverlay.hook.run ( "<another event>" )
+
+
+in additions dir (assuming that mux.sh is used)
+
+.. code-block:: text
+
+ <somehow create> ${ADDITIONS_DIR}/<hook>.<event>
+ <somehow create> ${ADDITIONS_DIR}/<hook>/<event>.sh
^ permalink raw reply related [flat|nested] 41+ messages in thread* [gentoo-commits] proj/R_overlay:master commit in: doc/rst/
2013-06-25 21:10 [gentoo-commits] proj/R_overlay:gsoc13/next " André Erdmann
@ 2013-06-30 15:58 ` André Erdmann
0 siblings, 0 replies; 41+ messages in thread
From: André Erdmann @ 2013-06-30 15:58 UTC (permalink / raw
To: gentoo-commits
commit: adbe0bb8d1edc7cbff598b4834fbaceb7335fbb3
Author: André Erdmann <dywi <AT> mailerd <DOT> de>
AuthorDate: Tue Jun 25 21:10:02 2013 +0000
Commit: André Erdmann <dywi <AT> mailerd <DOT> de>
CommitDate: Tue Jun 25 21:10:02 2013 +0000
URL: http://git.overlays.gentoo.org/gitweb/?p=proj/R_overlay.git;a=commit;h=adbe0bb8
doc/rst: doc for "run hooks" (incomplete)
---
doc/rst/usage.rst | 354 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 354 insertions(+)
diff --git a/doc/rst/usage.rst b/doc/rst/usage.rst
index 8d06bab..078a7ac 100644
--- a/doc/rst/usage.rst
+++ b/doc/rst/usage.rst
@@ -314,6 +314,12 @@ have reasonable defaults if *roverlay* has been installed using *emerge*:
Example: USE_EXPAND_RENAME = ~/roverlay/config/useflag_rename
+ CACHEDIR
+ Directory for generated files that do not belong to the overlay, e.g.
+ the *distmap* file. This option is **required**.
+
+ Example: CACHEDIR = ~/roverlay/workdir/cache
+
OVERLAY_ECLASS
This option lists eclass files that should be imported into the overlay
(into *OVERLAY_DIR*/eclass/) and inherited in all ebuilds.
@@ -409,6 +415,10 @@ to know in detail what *roverlay* does before running it.
--nosync, --no-sync
Disable downloading of R packages.
+--distmap-verify
+ Enforce verification of R packages in the package mirror directory.
+ This also tries to recreate the distmap.
+
--no-incremental
Force recreation of existing ebuilds
@@ -1834,6 +1844,16 @@ The following sections will list all config entries.
misc options
--------------
+.. _CACHEDIR:
+
+CACHEDIR
+ Directory for persistent files that don't belong to the overlay, e.g.
+ the distmap file.
+
+ This option is **required**.
+
+ <<TODO: default value!>>
+
.. _DISTFILES:
DISTFILES
@@ -1863,6 +1883,14 @@ EBUILD_PROG
Defaults to *ebuild*, which should be fine in most cases.
+.. _NOSYNC:
+
+NOSYNC
+ A *bool* that controls whether *syncing*, i.e. data transfers from/to
+ remote machines, is allowed or forbidden.
+
+ Defaults to *no*.
+
.. _RSYNC_BWLIMIT:
RSYNC_BWLIMIT
@@ -1872,6 +1900,7 @@ RSYNC_BWLIMIT
Defaults to <not set>, which disables bandwidth limitation.
+
-----------------
overlay options
-----------------
@@ -1901,6 +1930,17 @@ DISTDIR_FLAT
DISTDIR_STRATEGY
Alias to OVERLAY_DISTDIR_STRATEGY_.
+
+.. _DISTDIR_VERIFY:
+
+DISTDIR_VERIFY
+ Alias to OVERLAY_DISTDIR_VERIFY_.
+
+.. _DISTMAP_FILE:
+
+DISTMAP_FILE
+ Alias to OVERLAY_DISTMAP_FILE_.
+
.. _EBUILD_USE_EXPAND_NAME:
EBUILD_USE_EXPAND_NAME
@@ -2002,6 +2042,23 @@ OVERLAY_DISTDIR_STRATEGY
This option is **required**, but has a reasonable value in the default
config file, "hardlink symlink".
+.. _OVERLAY_DISTDIR_VERIFY:
+
+OVERLAY_DISTDIR_VERIFY
+ A *bool* that controls whether file integrity of *OVERLAY_DISTDIR_ROOT*
+ should be checked on startup. This is an expensive operation since each
+ file have to be read once.
+
+ Defaults to *no* as the verification is normally not needed.
+
+
+.. _OVERLAY_DISTMAP_FILE:
+
+OVERLAY_DISTMAP_FILE:
+ File path to the distmap file.
+
+ Defaults to <not set>, which results in CACHEDIR_/distmap.db.
+
.. _OVERLAY_ECLASS:
OVERLAY_ECLASS
@@ -2148,6 +2205,45 @@ USE_EXPAND_DESC
USE_EXPAND_RENAME
Alias to EBUILD_USE_EXPAND_RENAME_.
+---------------------------
+ shell environment / hooks
+---------------------------
+
+.. _EVENT_HOOK:
+
+EVENT_HOOK
+ A script that is called for handling *events* (see Hooks_).
+
+ Defaults to ADDITIONS_DIR_/hooks/mux.sh.
+
+.. _EVENT_HOOK_RESTRICT:
+
+EVENT_HOOK_RESTRICT
+ A list of *events* that are allowed (``<event>``, ``+<event>``) or
+ forbidden (``-<event>``). The asterisk wildcard character can be used to
+ set the default policy to *allow unless forbidden* (``*``) or
+ *deny unless allowed* (``-*``).
+
+ Defaults to <not set>, which is equivalent to *deny all*.
+
+.. _FILTER_SHELL_ENV:
+
+FILTER_SHELL_ENV
+ A *bool* that controls whether the hook environment should be filtered
+ or not.
+
+ Defaults to *true*.
+
+.. _HOOK:
+
+HOOK
+ Alias to EVENT_HOOK_.
+
+.. _HOOK_RESTRICT:
+
+HOOK_RESTRICT
+ Alias to EVENT_HOOK_RESTRICT_.
+
---------
logging
---------
@@ -3146,3 +3242,261 @@ Its mode of operation of operation is best described in pseudo-code:
The dependency resolver can be **run as thread**, in which case the while loop
becomes "loop until resolver closes".
+
+
+=======
+ undef
+=======
+
+<<TODO; this section has to merged/moved>>
+
+---------
+ distmap
+---------
+
+The distmap is a file that stores information about the files in the package
+mirror directory (`OVERLAY_DISTDIR_ROOT`_).
+<<TODO>>
+
+
+-------
+ hooks
+-------
+
+*roverlay* calls a script when certain events occur, e.g. after successful
+overlay creation.
+
+<<TODO>>
+
+* real work is offloaded to the called script
+* roverlay provides some env vars
+* roverlay provides a *$FUNCTIONS* file and a way to load other function files
+* ``EVENT_HOOK_RESTRICT="this -not_that *"``
+
+<<^^TODO>>
+
+.. Note::
+
+ *roverlay* waits until the script terminates.
+
+
+++++++++++++++++++
+ Hook environment
+++++++++++++++++++
+
+.. table:: environment variables provided by *roverlay*
+
+ +----------------+------------------+-----------------------------------------+
+ | variable | source | notes / description |
+ +================+==================+=========================================+
+ | PATH | os.environ | |
+ +----------------+------------------+-----------------------------------------+
+ | LOGNAME | os.environ | |
+ +----------------+------------------+-----------------------------------------+
+ | SHLVL | os.environ | |
+ +----------------+------------------+-----------------------------------------+
+ | TERM | os.environ | |
+ +----------------+------------------+-----------------------------------------+
+ | HOME | os.environ | |
+ +----------------+------------------+-----------------------------------------+
+ | ROVERLAY_PHASE | event | event that caused the script to run |
+ +----------------+------------------+-----------------------------------------+
+ | OVERLAY | config | overlay directory (`OVERLAY_DIR`_), |
+ +----------------+------------------+ initial working directory |
+ | S | *$OVERLAY* | |
+ +----------------+------------------+ |
+ | PWD | *$OVERLAY* | |
+ +----------------+------------------+-----------------------------------------+
+ | DISTROOT | config | package mirror directory |
+ | | | (`OVERLAY_DISTDIR_ROOT`_) |
+ +----------------+------------------+-----------------------------------------+
+ | TMPDIR | os.environ, | directory for temporary files |
+ | | *fallback* | |
+ +----------------+------------------+ |
+ | T | *$TMPDIR* | |
+ +----------------+------------------+-----------------------------------------+
+ | ADDITIONS_DIR | config | directory with supplementary files |
+ | | | (`OVERLAY_ADDITIONS_DIR`_) |
+ +----------------+------------------+ |
+ | FILESDIR | *$ADDITIONS_DIR* | |
+ +----------------+------------------+-----------------------------------------+
+ | SHLIB | *$ADDITIONS_DIR* | directory with shell function files |
+ | | | (optional, only set if it exists) |
+ +----------------+------------------+-----------------------------------------+
+ | FUNCTIONS | *$ADDITIONS_DIR* | file with essential shell functions |
+ | | | (optional, only set if it exists) |
+ +----------------+------------------+-----------------------------------------+
+ | DEBUG | log level | *shbool* (``y`` or ``n``) that |
+ | | | indicates whether debug messages should |
+ | | | be printed |
+ +----------------+------------------+-----------------------------------------+
+ | VERBOSE | log level | *shbool* |
+ +----------------+------------------+-----------------------------------------+
+ | QUIET | log level | *shbool* that indicates whether scripts |
+ | | | should be quiet |
+ +----------------+------------------+-----------------------------------------+
+ | NO_COLOR | *n/a* | *shbool*. Always set to *y* since |
+ | | | colored output should not be produced |
+ +----------------+------------------+-----------------------------------------+
+ | NOSYNC | config | *shbool* that indicates whether data |
+ | | | transfers from/to remote machines is |
+ | | | allowed (NOSYNC_) |
+ +----------------+------------------+-----------------------------------------+
+ | EBUILD | config | the *ebuild* executable |
+ +----------------+------------------+-----------------------------------------+
+ | GIT_EDITOR | *n/a* | set to */bin/false* |
+ +----------------+------------------+ |
+ | GIT_ASKPASS | *n/a* | |
+ +----------------+------------------+-----------------------------------------+
+
+
+The default *essential shell functions* file (*$FUNCTIONS*) makes,
+when included in the hook script, most of the enviroment variables readonly.
+
+
+.. table:: variables provided by *$FUNCTIONS*
+
+ +-----------------+-------------------------------------------------------+
+ | variable | description |
+ +=================+=======================================================+
+ | IFS_DEFAULT | default *internal field separator* |
+ +-----------------+-------------------------------------------------------+
+ | IFS_NEWLINE | *IFS* for iterating over text lines |
+ +-----------------+-------------------------------------------------------+
+ | DEVNULL | */dev/null* target (could also be a file) |
+ +-----------------+-------------------------------------------------------+
+ | EX_ERR | default error exit code |
+ +-----------------+-------------------------------------------------------+
+ | EX_ARG_ERR | default exit code for arg errors |
+ +-----------------+-------------------------------------------------------+
+ | SCRIPT_FILENAME | file name of the hook script |
+ +-----------------+-------------------------------------------------------+
+ | SCRIPT_NAME | name of the hook script (without file extension) |
+ +-----------------+-------------------------------------------------------+
+ | lf | reference to a function that loads additional shell |
+ | | function files |
+ +-----------------+-------------------------------------------------------+
+
+
+*$FUNCTIONS* also provides a number of shell functions:
+
+.. code-block:: sh
+
+ # --- message ---
+ #
+ # void veinfo ( message )
+ # Prints a message to stdout if $DEBUG is set to 'y'.
+ #
+ # void einfo ( message )
+ # Prints a message to stdout if $VERBOSE is set to 'y'.
+ #
+ # void ewarn ( message )
+ # Prints a message to stderr unless $QUIET is set to 'y'.
+ #
+ # void eerror ( message )
+ # Prints a message to stderr.
+ #
+ #
+ # --- core ---
+ #
+ # @noreturn die ( [message], [exit_code] ), raises exit()
+ # Lets the script die with the given message/exit code.
+ #
+ # @noreturn OUT_OF_BOUNDS(), raises die()
+ # Lets the script die due to insufficient arg count.
+ #
+ # int run_command ( *cmdv )
+ # Logs a command and runs it afterwards.
+ #
+ # int run_command_logged ( *cmdv )
+ # Logs a command, runs it and logs the result.
+ #
+ # void autodie ( *cmdv ), raises die()
+ # Runs a command. Lets the script die if the command fails.
+ #
+ #
+ # void load_functions ( *filenames, **SHLIB ), raises die()
+ # Loads additional shell functions file(s) from $SHLIB.
+ # (Referenced by $lf.)
+ #
+ # void dont_run_as_root(), raises die()
+ # Lets the script die if it is run as root.
+ #
+ # int list_has ( word, *list_items )
+ # Returns 0 if $word is in the given list.
+ #
+ # int qwhich ( *command )
+ # Returns 0 if all listed commands are found by "which".
+ #
+ #
+ # --- fs ---
+ #
+ # int dodir ( *dir )
+ # Ensures that zero or more directories exist by creating them if
+ # necessary. Returns the number of directories that could not be created.
+ #
+ #
+ # --- str ---
+ #
+ # int yesno ( word, **YESNO_YES=0, **YESNO_NO=1, **YESNO_EMPTY=2 )
+ # Returns $YESNO_YES if $word means "yes", $YESNO_EMPTY if $word is empty
+ # and $YESNO_NO otherwise (if $word probably means "no").
+ #
+ # ~int str_trim ( *args )
+ # Removes whitespace at the beginning and end of a string and replaces
+ # any whitespace sequence within the string with a single space char.
+ # Passes the return value of the underlying sed command.
+ #
+ # ~int str_upper ( *args )
+ # Echoes the uppercase variant of stdin or *args.
+ # Passes tr's return value.
+ #
+ # ~int str_lower ( *args )
+ # Echoes the lowercase variant of stdin or *args.
+ # Passes tr's return value.
+ #
+ # ~int str_field ( fieldspec, *args, **FIELD_SEPARATOR=' ' )
+ # Echoes the requested fields of stdin or *args.
+ # Passes cut's return value.
+ #
+ #
+ # --- int ---
+ #
+ # int is_int ( word )
+ # Returns 0 if $word is an integer, else 1.
+ #
+ # int is_natural ( word )
+ # Returns 0 if $word is an integer >= 0, else 1.
+ #
+ # int is_positive ( word )
+ # Returns 0 if $word is an integer >= 1, else 1.
+ #
+ # int is_negative ( word )
+ # Returns 0 if $word is an integer < 0, else 1.
+ #
+
+
++++++++++++++++++++++
+ Adding a hook event
++++++++++++++++++++++
+
+in roverlay
+
+.. code-block:: python
+
+ # import hook module
+ import roverlay.hook
+
+ # ...
+ # then:
+ roverlay.hook.run ( "<event>" )
+ # ...
+ roverlay.hook.run ( "<another event>" )
+
+
+in additions dir (assuming that mux.sh is used)
+
+.. code-block:: text
+
+ <somehow create> ${ADDITIONS_DIR}/<hook>.<event>
+ <somehow create> ${ADDITIONS_DIR}/<hook>/<event>.sh
^ permalink raw reply related [flat|nested] 41+ messages in thread
* [gentoo-commits] proj/R_overlay:gsoc13/next commit in: doc/rst/
@ 2013-06-18 14:12 André Erdmann
2013-06-18 14:12 ` [gentoo-commits] proj/R_overlay:master " André Erdmann
0 siblings, 1 reply; 41+ messages in thread
From: André Erdmann @ 2013-06-18 14:12 UTC (permalink / raw
To: gentoo-commits
commit: 846a4c66dabd31bbb4b9c2e17394ad53cc086d3d
Author: André Erdmann <dywi <AT> mailerd <DOT> de>
AuthorDate: Tue Jun 18 12:54:18 2013 +0000
Commit: André Erdmann <dywi <AT> mailerd <DOT> de>
CommitDate: Tue Jun 18 12:54:18 2013 +0000
URL: http://git.overlays.gentoo.org/gitweb/?p=proj/R_overlay.git;a=commit;h=846a4c66
doc/rst: r_suggests USE_EXPAND variable
---
doc/rst/usage.rst | 190 +++++++++++++++++++++++++++++++++++++++++++++---------
1 file changed, 161 insertions(+), 29 deletions(-)
diff --git a/doc/rst/usage.rst b/doc/rst/usage.rst
index ab2c563..8d06bab 100644
--- a/doc/rst/usage.rst
+++ b/doc/rst/usage.rst
@@ -302,6 +302,18 @@ have reasonable defaults if *roverlay* has been installed using *emerge*:
Example: FIELD_DEFINITION = ~/roverlay/config/description_fields.conf
+ USE_EXPAND_DESC
+ A file that contains USE_EXPAND flag descriptions. This option is not
+ required.
+
+ Example: USE_EXPAND_DESC = ~/roverlay/config/useflag/useflag_desc
+
+ USE_EXPAND_RENAME
+ The value of this option should point to a USE flag remap file which
+ can be used to rename USE_EXPAND flags. This option is not required.
+
+ Example: USE_EXPAND_RENAME = ~/roverlay/config/useflag_rename
+
OVERLAY_ECLASS
This option lists eclass files that should be imported into the overlay
(into *OVERLAY_DIR*/eclass/) and inherited in all ebuilds.
@@ -557,6 +569,10 @@ These are the steps that *roverlay* performs:
6. write the overlay
+ * write/update the profiles dir
+
+ * *roverlay* respects manual changes to USE_EXPAND description files
+
* write all ebuilds and apply patches to them
(supports threads on a per package basis)
@@ -588,6 +604,8 @@ eclass file are used, the result should look like:
<overlay root>/profiles/categories
<overlay root>/profiles/repo_name
<overlay root>/profiles/use.desc
+ <overlay root>/profiles/desc
+ <overlay root>/profiles/desc/r_suggests.desc
<overlay root>/sci-R/<many directories per R package>
<overlay root>/sci-R/seewave/
<overlay root>/sci-R/seewave/Manifest
@@ -604,19 +622,22 @@ Ebuild Template
.. code-block:: text
<ebuild header>
+
+ EAPI=<EAPI>
+
inherit <eclass(es)>
DESCRIPTION="<the R package's description>"
SRC_URI="<src uri for the R package>"
- IUSE="${IUSE:-}
- R_suggests
+ IUSE="${IUSE-}
+ r_suggests_<flag> r_suggests_<another flag> ...
"
- R_SUGGESTS="<R package suggestions>"
+ R_SUGGESTS="r_suggests_<flag>? ( <optional dependency> ) ..."
DEPEND="<build time dependencies for the R package>"
- RDEPEND="${DEPEND:-}
+ RDEPEND="${DEPEND-}
<runtime dependencies>
- R_suggests? ( ${R_SUGGESTS} )
+ ${R_SUGGESTS-}
"
_UNRESOLVED_PACKAGES=(<unresolvable, but optional dependencies>)
@@ -628,6 +649,9 @@ Ebuild Template
.. code-block:: text
<ebuild header>
+
+ EAPI=<EAPI>
+
inherit <eclass(es)>
SRC_URI="<src uri for the R package>"
@@ -638,7 +662,7 @@ Example: seewave 1.6.4 from CRAN:
.. code-block:: sh
- # Copyright 1999-2012 Gentoo Foundation
+ # Copyright 1999-2013 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
# $Header: $
@@ -648,11 +672,13 @@ Example: seewave 1.6.4 from CRAN:
DESCRIPTION="Time wave analysis and graphical representation"
SRC_URI="http://cran.r-project.org/src/contrib/seewave_1.6.4.tar.gz"
- IUSE="${IUSE:-}
- R_suggests
+ IUSE="${IUSE-}
+ r_suggests_sound
+ r_suggests_audio
"
- R_SUGGESTS="sci-R/sound
- sci-R/audio
+ R_SUGGESTS="
+ r_suggests_sound? ( sci-R/sound )
+ r_suggests_audio? ( sci-R/audio )
"
DEPEND="sci-R/fftw
sci-R/tuneR
@@ -660,11 +686,11 @@ Example: seewave 1.6.4 from CRAN:
sci-R/rpanel
sci-R/rgl
"
- RDEPEND="${DEPEND:-}
+ RDEPEND="${DEPEND-}
media-libs/flac
sci-libs/fftw
media-libs/libsndfile
- R_suggests? ( ${R_SUGGESTS} )
+ ${R_SUGGESTS-}
"
Example: MetaPCA 0.1.3 from CRAN's archive:
@@ -673,7 +699,7 @@ Example: MetaPCA 0.1.3 from CRAN's archive:
.. code-block:: sh
- # Copyright 1999-2012 Gentoo Foundation
+ # Copyright 1999-2013 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
# $Header: $
@@ -683,21 +709,29 @@ Example: MetaPCA 0.1.3 from CRAN's archive:
DESCRIPTION="MetaPCA: Meta-analysis in the Di... (see metadata)"
SRC_URI="http://cran.r-project.org/src/contrib/Archive/MetaPCA/MetaPCA_0.1.3.tar.gz"
- IUSE="${IUSE:-}
- R_suggests
+ IUSE="${IUSE-}
+ r_suggests_domc
+ r_suggests_affy
+ r_suggests_ellipse
+ r_suggests_pcapp
+ r_suggests_mass
+ r_suggests_impute
+ r_suggests_dosmp
+ r_suggests_geoquery
"
- R_SUGGESTS="sci-R/doMC
- sci-R/affy
- sci-R/ellipse
- sci-R/pcaPP
- sci-R/MASS
- sci-R/impute
- sci-R/doSMP
- sci-R/GEOquery
+ R_SUGGESTS="
+ r_suggests_domc? ( sci-R/doMC )
+ r_suggests_affy? ( sci-R/affy )
+ r_suggests_ellipse? ( sci-R/ellipse )
+ r_suggests_pcapp? ( sci-R/pcaPP )
+ r_suggests_mass? ( sci-R/MASS )
+ r_suggests_impute? ( sci-R/impute )
+ r_suggests_dosmp? ( sci-R/doSMP )
+ r_suggests_geoquery? ( sci-R/GEOquery )
"
DEPEND="sci-R/foreach"
- RDEPEND="${DEPEND:-}
- R_suggests? ( ${R_SUGGESTS} )
+ RDEPEND="${DEPEND-}
+ ${R_SUGGESTS-}
"
_UNRESOLVED_PACKAGES=('hgu133plus2.db')
@@ -1847,6 +1881,11 @@ RSYNC_BWLIMIT
ADDITIONS_DIR:
Alias to OVERLAY_ADDITIONS_DIR_.
+.. _BACKUP_DESC:
+
+BACKUP_DESC
+ Alias to OVERLAY_BACKUP_DESC_.
+
.. _DISTDIR:
DISTDIR
@@ -1862,6 +1901,11 @@ DISTDIR_FLAT
DISTDIR_STRATEGY
Alias to OVERLAY_DISTDIR_STRATEGY_.
+.. _EBUILD_USE_EXPAND_NAME:
+
+EBUILD_USE_EXPAND_NAME
+ Name of the R_SUGGESTS USE_EXPAND variable. Defaults to *R_SUGGESTS*.
+
.. _ECLASS:
ECLASS
@@ -1880,6 +1924,14 @@ OVERLAY_ADDITIONS_DIR
Defaults to <not set>, which disables this feature.
+.. _OVERLAY_BACKUP_DESC:
+
+OVERLAY_BACKUP_DESC
+ A *bool* that indicates whether the description file of the *R_SUGGESTS*
+ USE_EXPAND variable should be backed up before (over-)writing it.
+
+ Defaults to *true*.
+
.. _OVERLAY_CATEGORY:
OVERLAY_CATEGORY
@@ -1997,6 +2049,11 @@ OVERLAY_NAME
Defaults to *R_Overlay*.
+.. _USE_EXPAND_NAME:
+
+USE_EXPAND_NAME:
+ Alias to EBUILD_USE_EXPAND_NAME_.
+
--------------------
other config files
--------------------
@@ -2011,6 +2068,21 @@ reasons:
The paths to these files have to be listed in the main config file and
can be overridden with the appropriate command line options.
+.. _EBUILD_USE_EXPAND_DESC:
+
+EBUILD_USE_EXPAND_DESC
+ Path to a flag description file (for the *R_SUGGESTS* USE_EXPAND variable).
+ The syntax of this file is identical to USE_EXPAND description files
+ (``<overlay root>/profiles/desc/r_suggests.desc``).
+
+ Defaults to <not set>, which disables this option.
+
+.. _EBUILD_USE_EXPAND_RENAME:
+
+EBUILD_USE_EXPAND_RENAME
+ Path to a file that lists alternative names for a flag in the *R_SUGGESTS*
+ USE_EXPAND variable.
+
.. _FIELD_DEFINITION:
FIELD_DEFINITION
@@ -2066,6 +2138,16 @@ SIMPLE_RULES_FILE
SIMPLE_RULES_FILES
Alias to SIMPLE_RULES_FILE_.
+.. _USE_EXPAND_DESC:
+
+USE_EXPAND_DESC
+ Alias to EBUILD_USE_EXPAND_DESC_.
+
+.. _USE_EXPAND_RENAME:
+
+USE_EXPAND_RENAME
+ Alias to EBUILD_USE_EXPAND_RENAME_.
+
---------
logging
---------
@@ -2210,11 +2292,61 @@ LOG_FILE_UNRESOLVABLE
Defaults to <not set>, which disables this feature.
+====================
+ Other config files
+====================
+
+-----------------------------
+ USE_EXPAND flag rename file
+-----------------------------
+
+The USE_EXPAND_RENAME_ file contains dictionary-like entries that assign
+*effective* flag names to flag names generated at runtime.
+
+The syntax is as follows:
+
+.. code-block:: text
+
+ # comments start with '#'
+
+ <effective flag> <runtime flag> [<another runtime flag>...]
+
+ # a '=' can be used as separator to improve readability
+ <effective flag> = <runtime flag> [<another runtime flag>...]
+
+ # the previous line can be continued with leading whitespace
+ <effective flag> = <runtime flag>
+ [<another runtime flag>...]
+
+
+Example:
+
+.. code-block:: text
+
+ # rename 'audio' and 'snd' to 'sound'
+ sound = audio snd
+
+
+Each flag is renamed at most once, so the following example renames 'sound'
+to media, but 'audio' to 'sound':;
+
+.. code-block:: text
+
+ sound = audio snd
+ media = sound video
+
+
+.. Caution::
+
+ Assigning more than one *effective flag* to a *runtime flag* leads to
+ unpredictable results.
+
+
.. _Field Definition:
-=========================
+-------------------------
Field Definition Config
-=========================
+-------------------------
The field definition file uses ConfigParser_ syntax and defines
how an R package's DESCRIPTION file is read.
@@ -2319,9 +2451,9 @@ Known field flags:
.. _default field definition file:
---------------------------------------------
+++++++++++++++++++++++++++++++++++++++++++++
Example: The default field definition file
---------------------------------------------
+++++++++++++++++++++++++++++++++++++++++++++
This is the default field definition file (without any ignored fields):
^ permalink raw reply related [flat|nested] 41+ messages in thread* [gentoo-commits] proj/R_overlay:master commit in: doc/rst/
2013-06-18 14:12 [gentoo-commits] proj/R_overlay:gsoc13/next " André Erdmann
@ 2013-06-18 14:12 ` André Erdmann
0 siblings, 0 replies; 41+ messages in thread
From: André Erdmann @ 2013-06-18 14:12 UTC (permalink / raw
To: gentoo-commits
commit: 846a4c66dabd31bbb4b9c2e17394ad53cc086d3d
Author: André Erdmann <dywi <AT> mailerd <DOT> de>
AuthorDate: Tue Jun 18 12:54:18 2013 +0000
Commit: André Erdmann <dywi <AT> mailerd <DOT> de>
CommitDate: Tue Jun 18 12:54:18 2013 +0000
URL: http://git.overlays.gentoo.org/gitweb/?p=proj/R_overlay.git;a=commit;h=846a4c66
doc/rst: r_suggests USE_EXPAND variable
---
doc/rst/usage.rst | 190 +++++++++++++++++++++++++++++++++++++++++++++---------
1 file changed, 161 insertions(+), 29 deletions(-)
diff --git a/doc/rst/usage.rst b/doc/rst/usage.rst
index ab2c563..8d06bab 100644
--- a/doc/rst/usage.rst
+++ b/doc/rst/usage.rst
@@ -302,6 +302,18 @@ have reasonable defaults if *roverlay* has been installed using *emerge*:
Example: FIELD_DEFINITION = ~/roverlay/config/description_fields.conf
+ USE_EXPAND_DESC
+ A file that contains USE_EXPAND flag descriptions. This option is not
+ required.
+
+ Example: USE_EXPAND_DESC = ~/roverlay/config/useflag/useflag_desc
+
+ USE_EXPAND_RENAME
+ The value of this option should point to a USE flag remap file which
+ can be used to rename USE_EXPAND flags. This option is not required.
+
+ Example: USE_EXPAND_RENAME = ~/roverlay/config/useflag_rename
+
OVERLAY_ECLASS
This option lists eclass files that should be imported into the overlay
(into *OVERLAY_DIR*/eclass/) and inherited in all ebuilds.
@@ -557,6 +569,10 @@ These are the steps that *roverlay* performs:
6. write the overlay
+ * write/update the profiles dir
+
+ * *roverlay* respects manual changes to USE_EXPAND description files
+
* write all ebuilds and apply patches to them
(supports threads on a per package basis)
@@ -588,6 +604,8 @@ eclass file are used, the result should look like:
<overlay root>/profiles/categories
<overlay root>/profiles/repo_name
<overlay root>/profiles/use.desc
+ <overlay root>/profiles/desc
+ <overlay root>/profiles/desc/r_suggests.desc
<overlay root>/sci-R/<many directories per R package>
<overlay root>/sci-R/seewave/
<overlay root>/sci-R/seewave/Manifest
@@ -604,19 +622,22 @@ Ebuild Template
.. code-block:: text
<ebuild header>
+
+ EAPI=<EAPI>
+
inherit <eclass(es)>
DESCRIPTION="<the R package's description>"
SRC_URI="<src uri for the R package>"
- IUSE="${IUSE:-}
- R_suggests
+ IUSE="${IUSE-}
+ r_suggests_<flag> r_suggests_<another flag> ...
"
- R_SUGGESTS="<R package suggestions>"
+ R_SUGGESTS="r_suggests_<flag>? ( <optional dependency> ) ..."
DEPEND="<build time dependencies for the R package>"
- RDEPEND="${DEPEND:-}
+ RDEPEND="${DEPEND-}
<runtime dependencies>
- R_suggests? ( ${R_SUGGESTS} )
+ ${R_SUGGESTS-}
"
_UNRESOLVED_PACKAGES=(<unresolvable, but optional dependencies>)
@@ -628,6 +649,9 @@ Ebuild Template
.. code-block:: text
<ebuild header>
+
+ EAPI=<EAPI>
+
inherit <eclass(es)>
SRC_URI="<src uri for the R package>"
@@ -638,7 +662,7 @@ Example: seewave 1.6.4 from CRAN:
.. code-block:: sh
- # Copyright 1999-2012 Gentoo Foundation
+ # Copyright 1999-2013 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
# $Header: $
@@ -648,11 +672,13 @@ Example: seewave 1.6.4 from CRAN:
DESCRIPTION="Time wave analysis and graphical representation"
SRC_URI="http://cran.r-project.org/src/contrib/seewave_1.6.4.tar.gz"
- IUSE="${IUSE:-}
- R_suggests
+ IUSE="${IUSE-}
+ r_suggests_sound
+ r_suggests_audio
"
- R_SUGGESTS="sci-R/sound
- sci-R/audio
+ R_SUGGESTS="
+ r_suggests_sound? ( sci-R/sound )
+ r_suggests_audio? ( sci-R/audio )
"
DEPEND="sci-R/fftw
sci-R/tuneR
@@ -660,11 +686,11 @@ Example: seewave 1.6.4 from CRAN:
sci-R/rpanel
sci-R/rgl
"
- RDEPEND="${DEPEND:-}
+ RDEPEND="${DEPEND-}
media-libs/flac
sci-libs/fftw
media-libs/libsndfile
- R_suggests? ( ${R_SUGGESTS} )
+ ${R_SUGGESTS-}
"
Example: MetaPCA 0.1.3 from CRAN's archive:
@@ -673,7 +699,7 @@ Example: MetaPCA 0.1.3 from CRAN's archive:
.. code-block:: sh
- # Copyright 1999-2012 Gentoo Foundation
+ # Copyright 1999-2013 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
# $Header: $
@@ -683,21 +709,29 @@ Example: MetaPCA 0.1.3 from CRAN's archive:
DESCRIPTION="MetaPCA: Meta-analysis in the Di... (see metadata)"
SRC_URI="http://cran.r-project.org/src/contrib/Archive/MetaPCA/MetaPCA_0.1.3.tar.gz"
- IUSE="${IUSE:-}
- R_suggests
+ IUSE="${IUSE-}
+ r_suggests_domc
+ r_suggests_affy
+ r_suggests_ellipse
+ r_suggests_pcapp
+ r_suggests_mass
+ r_suggests_impute
+ r_suggests_dosmp
+ r_suggests_geoquery
"
- R_SUGGESTS="sci-R/doMC
- sci-R/affy
- sci-R/ellipse
- sci-R/pcaPP
- sci-R/MASS
- sci-R/impute
- sci-R/doSMP
- sci-R/GEOquery
+ R_SUGGESTS="
+ r_suggests_domc? ( sci-R/doMC )
+ r_suggests_affy? ( sci-R/affy )
+ r_suggests_ellipse? ( sci-R/ellipse )
+ r_suggests_pcapp? ( sci-R/pcaPP )
+ r_suggests_mass? ( sci-R/MASS )
+ r_suggests_impute? ( sci-R/impute )
+ r_suggests_dosmp? ( sci-R/doSMP )
+ r_suggests_geoquery? ( sci-R/GEOquery )
"
DEPEND="sci-R/foreach"
- RDEPEND="${DEPEND:-}
- R_suggests? ( ${R_SUGGESTS} )
+ RDEPEND="${DEPEND-}
+ ${R_SUGGESTS-}
"
_UNRESOLVED_PACKAGES=('hgu133plus2.db')
@@ -1847,6 +1881,11 @@ RSYNC_BWLIMIT
ADDITIONS_DIR:
Alias to OVERLAY_ADDITIONS_DIR_.
+.. _BACKUP_DESC:
+
+BACKUP_DESC
+ Alias to OVERLAY_BACKUP_DESC_.
+
.. _DISTDIR:
DISTDIR
@@ -1862,6 +1901,11 @@ DISTDIR_FLAT
DISTDIR_STRATEGY
Alias to OVERLAY_DISTDIR_STRATEGY_.
+.. _EBUILD_USE_EXPAND_NAME:
+
+EBUILD_USE_EXPAND_NAME
+ Name of the R_SUGGESTS USE_EXPAND variable. Defaults to *R_SUGGESTS*.
+
.. _ECLASS:
ECLASS
@@ -1880,6 +1924,14 @@ OVERLAY_ADDITIONS_DIR
Defaults to <not set>, which disables this feature.
+.. _OVERLAY_BACKUP_DESC:
+
+OVERLAY_BACKUP_DESC
+ A *bool* that indicates whether the description file of the *R_SUGGESTS*
+ USE_EXPAND variable should be backed up before (over-)writing it.
+
+ Defaults to *true*.
+
.. _OVERLAY_CATEGORY:
OVERLAY_CATEGORY
@@ -1997,6 +2049,11 @@ OVERLAY_NAME
Defaults to *R_Overlay*.
+.. _USE_EXPAND_NAME:
+
+USE_EXPAND_NAME:
+ Alias to EBUILD_USE_EXPAND_NAME_.
+
--------------------
other config files
--------------------
@@ -2011,6 +2068,21 @@ reasons:
The paths to these files have to be listed in the main config file and
can be overridden with the appropriate command line options.
+.. _EBUILD_USE_EXPAND_DESC:
+
+EBUILD_USE_EXPAND_DESC
+ Path to a flag description file (for the *R_SUGGESTS* USE_EXPAND variable).
+ The syntax of this file is identical to USE_EXPAND description files
+ (``<overlay root>/profiles/desc/r_suggests.desc``).
+
+ Defaults to <not set>, which disables this option.
+
+.. _EBUILD_USE_EXPAND_RENAME:
+
+EBUILD_USE_EXPAND_RENAME
+ Path to a file that lists alternative names for a flag in the *R_SUGGESTS*
+ USE_EXPAND variable.
+
.. _FIELD_DEFINITION:
FIELD_DEFINITION
@@ -2066,6 +2138,16 @@ SIMPLE_RULES_FILE
SIMPLE_RULES_FILES
Alias to SIMPLE_RULES_FILE_.
+.. _USE_EXPAND_DESC:
+
+USE_EXPAND_DESC
+ Alias to EBUILD_USE_EXPAND_DESC_.
+
+.. _USE_EXPAND_RENAME:
+
+USE_EXPAND_RENAME
+ Alias to EBUILD_USE_EXPAND_RENAME_.
+
---------
logging
---------
@@ -2210,11 +2292,61 @@ LOG_FILE_UNRESOLVABLE
Defaults to <not set>, which disables this feature.
+====================
+ Other config files
+====================
+
+-----------------------------
+ USE_EXPAND flag rename file
+-----------------------------
+
+The USE_EXPAND_RENAME_ file contains dictionary-like entries that assign
+*effective* flag names to flag names generated at runtime.
+
+The syntax is as follows:
+
+.. code-block:: text
+
+ # comments start with '#'
+
+ <effective flag> <runtime flag> [<another runtime flag>...]
+
+ # a '=' can be used as separator to improve readability
+ <effective flag> = <runtime flag> [<another runtime flag>...]
+
+ # the previous line can be continued with leading whitespace
+ <effective flag> = <runtime flag>
+ [<another runtime flag>...]
+
+
+Example:
+
+.. code-block:: text
+
+ # rename 'audio' and 'snd' to 'sound'
+ sound = audio snd
+
+
+Each flag is renamed at most once, so the following example renames 'sound'
+to media, but 'audio' to 'sound':;
+
+.. code-block:: text
+
+ sound = audio snd
+ media = sound video
+
+
+.. Caution::
+
+ Assigning more than one *effective flag* to a *runtime flag* leads to
+ unpredictable results.
+
+
.. _Field Definition:
-=========================
+-------------------------
Field Definition Config
-=========================
+-------------------------
The field definition file uses ConfigParser_ syntax and defines
how an R package's DESCRIPTION file is read.
@@ -2319,9 +2451,9 @@ Known field flags:
.. _default field definition file:
---------------------------------------------
+++++++++++++++++++++++++++++++++++++++++++++
Example: The default field definition file
---------------------------------------------
+++++++++++++++++++++++++++++++++++++++++++++
This is the default field definition file (without any ignored fields):
^ permalink raw reply related [flat|nested] 41+ messages in thread
* [gentoo-commits] proj/R_overlay:gsoc13/next commit in: doc/rst/
@ 2013-06-12 21:10 André Erdmann
2013-06-13 16:34 ` [gentoo-commits] proj/R_overlay:master " André Erdmann
0 siblings, 1 reply; 41+ messages in thread
From: André Erdmann @ 2013-06-12 21:10 UTC (permalink / raw
To: gentoo-commits
commit: b882ecc22d31458fc64949fd458931ec79a88456
Author: André Erdmann <dywi <AT> mailerd <DOT> de>
AuthorDate: Wed Jun 12 21:09:03 2013 +0000
Commit: André Erdmann <dywi <AT> mailerd <DOT> de>
CommitDate: Wed Jun 12 21:09:03 2013 +0000
URL: http://git.overlays.gentoo.org/gitweb/?p=proj/R_overlay.git;a=commit;h=b882ecc2
doc/rst: additions dir
---
doc/rst/usage.rst | 132 +++++++++++++++++++++++++++++++++++++++++++++++++-----
1 file changed, 120 insertions(+), 12 deletions(-)
diff --git a/doc/rst/usage.rst b/doc/rst/usage.rst
index adf26da..ab2c563 100644
--- a/doc/rst/usage.rst
+++ b/doc/rst/usage.rst
@@ -52,19 +52,20 @@ This document is targeted at
* *roverlay* maintainers who **control and test overlay creation**,
e.g. configure which R packages will be part of the generated overlay
- Depending on what you want to configure, chapters 5-8 are relevant,
- namely `Repositories / Getting Packages`_, `Dependency Rules`_,
- `Configuration Reference`_ and `Field Definition Config`_.
+ Depending on what you want to configure, chapters 5-10 are relevant,
+ namely `Repositories / Getting Packages`_, `Additions Directory`,
+ `Dependency Rules`_, `Package Rules`_, `Configuration Reference`_
+ and `Field Definition Config`_.
There is another chapter that is only interesting for testing, the
- `Dependency Resolution Console`_ (10), which can be used to interactively
+ `Dependency Resolution Console`_ (11), which can be used to interactively
test dependency rules.
* *roverlay* code maintainers who want to know **how roverlay works** for
code improvements etc.
- The most important chapter is `Implementation Overview`_ (11) which has
- references to other chapters (4-9) where required.
+ The most important chapter is `Implementation Overview`_ (12) which has
+ references to other chapters (4-10) where required.
Expected prior knowlegde:
@@ -279,13 +280,19 @@ have reasonable defaults if *roverlay* has been installed using *emerge*:
Example: REPO_CONFIG = ~/roverlay/config/repo.list
- PACKAGE_RULES:
+ PACKAGE_RULES
A list of files and/or directories with package rules.
Package rules can be used to control overlay/ebuild creation.
This option is not required.
Example: PACKAGE_RULES = ~/roverlay/config/packagerules.d
+ ADDITIONS_DIR
+ Directory with an overlay-like structure that contains extra files, e.g.
+ ebuild patches and hand-written ebuilds.
+
+ Example: ADDITIONS_DIR = ~/roverlay/additions
+
FIELD_DEFINITION
The value of this option should point to a field definition file which
controls how an R package's DESCRIPTION file is read.
@@ -330,6 +337,7 @@ There is another option that is useful for creating new dependency rules,
LOG_FILE_UNRESOLVABLE_, which will write all unresolvable dependencies
to the specified file (one dependency string per line).
+
+++++++++++++++++++++++++++++++++++++++++++++++++
Extended Configuration / Where to go from here?
+++++++++++++++++++++++++++++++++++++++++++++++++
@@ -510,7 +518,9 @@ These are the steps that *roverlay* performs:
2. scan the R Overlay directory (if it exists) for valid ebuilds
-3. **add** - queue all R packages for ebuild creation
+3. import ebuilds from the additions dir
+
+4. **add** - queue all R packages for ebuild creation
* all repositories are asked to list their packages which are then added
to a queue
@@ -522,7 +532,7 @@ These are the steps that *roverlay* performs:
See also: `Package Rules`_
-4. **create** - process each package *p* from the package queue
+5. **create** - process each package *p* from the package queue
(thread-able on a per package basis)
* read *p*'s DESCRIPTION file that contains information fields
@@ -545,9 +555,9 @@ These are the steps that *roverlay* performs:
* **done** with *p* - the overlay creator takes control over *p*
and may decide to write *p*'s ebuild now (or later)
-5. write the overlay
+6. write the overlay
- * write all ebuilds
+ * write all ebuilds and apply patches to them
(supports threads on a per package basis)
* write the *metadata.xml* files
@@ -957,6 +967,91 @@ A local directory will never be modified.
websync_pkglist_.
+=====================
+ Additions Directory
+=====================
+
+The *additions directory* is a directory with overlay-like structure that
+contains extra files for overlay creation. Currently, ebuild patches and
+ebuild files are supported.
+
+To give an idea of how this directory could
+
+
+
+------------------
+ Patching ebuilds
+------------------
+
+Patches can apply to a **specific version** or to **all versions** of a
+package.
+
+The naming convention for patches is (full filesystem paths relative to the
+additions dir):
+
+.. code:: text
+
+ # version-specific patches
+ ${CATEGORY}/${PN}/${PF}[-dddd].patch
+
+ # version-agnostic patches
+ ${CATEGORY}/${PN}/${PN}[-dddd].patch
+
+
+The *-dddd* part is optional and can be used to apply more than one patch to
+an ebuild in the specified order. *d* should be a digit (0..9) and exactly
+4 digits are expected. The not-numbered patch is always applied first.
+So, in theory, up to 10001 patches per ebuild are supported.
+
+The *default* (version-agnostic) patches are only applied to ebuilds for
+which no version-specific patches exist.
+
+Exempting a specific ebuild from being patched can be achieved by creating
+an empty patch file (or a symlink to /dev/null). This is only necessary
+if *default* patches are available, else it adds some overhead.
+
+.. Caution::
+
+ Don't try to patch the (R)DEPEND variables of an ebuild.
+ It will *randomly* break because *roverlay* uses unordered data structures
+ for collecting dependencies.
+
+Example:
+
+.. code:: text
+
+ <additions dir>/sci-CRAN/R_oo/R_oo-1.9.8.patch
+ <additions dir>/sci-CRAN/R_oo/R_oo-1.9.8-0000.patch
+ <additions dir>/sci-CRAN/R_oo/R_oo-1.9.8-0001.patch
+ <additions dir>/sci-R/seewave/seewave-1.6.7.patch
+ <additions dir>/sci-R/seewave/seewave.patch
+
+
+-------------------
+ Importing ebuilds
+-------------------
+
+Foreign ebuilds can be imported into the overlay by simple putting them into
+the additions directory.
+
+The naming convention is similar to ebuild patches and identical to the
+portage tree:
+
+.. code::
+
+ ${CATEGORY}/${PN}/${PF}.ebuild
+
+
+Ebuilds imported that way can not be overwritten by generated ebuilds and
+benefit from most overlay creation features, e.g. Manifest file creation.
+However, they cannot be used for metadata creation.
+
+.. Important::
+
+ Importing ebuilds is only supported by the default Manifest implementation
+ (*ebuildmanifest*).
+
+
==================
Dependency Rules
==================
@@ -1747,6 +1842,11 @@ RSYNC_BWLIMIT
overlay options
-----------------
+.. _ADDITIONS_DIR:
+
+ADDITIONS_DIR:
+ Alias to OVERLAY_ADDITIONS_DIR_.
+
.. _DISTDIR:
DISTDIR
@@ -1772,6 +1872,14 @@ ECLASS
MANIFEST_IMPLEMENTATION
Alias to OVERLAY_MANIFEST_IMPLEMENTATION_.
+.. _OVERLAY_ADDITIONS_DIR:
+
+OVERLAY_ADDITIONS_DIR
+ Directory with an overlay-like structure that contains extra files, e.g.
+ ebuild patches and hand-written ebuilds. This option is not required.
+
+ Defaults to <not set>, which disables this feature.
+
.. _OVERLAY_CATEGORY:
OVERLAY_CATEGORY
@@ -1790,7 +1898,7 @@ OVERLAY_DIR
.. _OVERLAY_DISTDIR_FLAT:
OVERLAY_DISTDIR_FLAT
- A *bool* that controls the overall layout of _OVERLAY_DISTDIR_ROOT.
+ A *bool* that controls the overall layout of OVERLAY_DISTDIR_ROOT_.
A flat distdir is a single directory with all package files or package
file links in it. A nested distdir contains per-package directories.
^ permalink raw reply related [flat|nested] 41+ messages in thread* [gentoo-commits] proj/R_overlay:master commit in: doc/rst/
2013-06-12 21:10 [gentoo-commits] proj/R_overlay:gsoc13/next " André Erdmann
@ 2013-06-13 16:34 ` André Erdmann
0 siblings, 0 replies; 41+ messages in thread
From: André Erdmann @ 2013-06-13 16:34 UTC (permalink / raw
To: gentoo-commits
commit: b882ecc22d31458fc64949fd458931ec79a88456
Author: André Erdmann <dywi <AT> mailerd <DOT> de>
AuthorDate: Wed Jun 12 21:09:03 2013 +0000
Commit: André Erdmann <dywi <AT> mailerd <DOT> de>
CommitDate: Wed Jun 12 21:09:03 2013 +0000
URL: http://git.overlays.gentoo.org/gitweb/?p=proj/R_overlay.git;a=commit;h=b882ecc2
doc/rst: additions dir
---
doc/rst/usage.rst | 132 +++++++++++++++++++++++++++++++++++++++++++++++++-----
1 file changed, 120 insertions(+), 12 deletions(-)
diff --git a/doc/rst/usage.rst b/doc/rst/usage.rst
index adf26da..ab2c563 100644
--- a/doc/rst/usage.rst
+++ b/doc/rst/usage.rst
@@ -52,19 +52,20 @@ This document is targeted at
* *roverlay* maintainers who **control and test overlay creation**,
e.g. configure which R packages will be part of the generated overlay
- Depending on what you want to configure, chapters 5-8 are relevant,
- namely `Repositories / Getting Packages`_, `Dependency Rules`_,
- `Configuration Reference`_ and `Field Definition Config`_.
+ Depending on what you want to configure, chapters 5-10 are relevant,
+ namely `Repositories / Getting Packages`_, `Additions Directory`,
+ `Dependency Rules`_, `Package Rules`_, `Configuration Reference`_
+ and `Field Definition Config`_.
There is another chapter that is only interesting for testing, the
- `Dependency Resolution Console`_ (10), which can be used to interactively
+ `Dependency Resolution Console`_ (11), which can be used to interactively
test dependency rules.
* *roverlay* code maintainers who want to know **how roverlay works** for
code improvements etc.
- The most important chapter is `Implementation Overview`_ (11) which has
- references to other chapters (4-9) where required.
+ The most important chapter is `Implementation Overview`_ (12) which has
+ references to other chapters (4-10) where required.
Expected prior knowlegde:
@@ -279,13 +280,19 @@ have reasonable defaults if *roverlay* has been installed using *emerge*:
Example: REPO_CONFIG = ~/roverlay/config/repo.list
- PACKAGE_RULES:
+ PACKAGE_RULES
A list of files and/or directories with package rules.
Package rules can be used to control overlay/ebuild creation.
This option is not required.
Example: PACKAGE_RULES = ~/roverlay/config/packagerules.d
+ ADDITIONS_DIR
+ Directory with an overlay-like structure that contains extra files, e.g.
+ ebuild patches and hand-written ebuilds.
+
+ Example: ADDITIONS_DIR = ~/roverlay/additions
+
FIELD_DEFINITION
The value of this option should point to a field definition file which
controls how an R package's DESCRIPTION file is read.
@@ -330,6 +337,7 @@ There is another option that is useful for creating new dependency rules,
LOG_FILE_UNRESOLVABLE_, which will write all unresolvable dependencies
to the specified file (one dependency string per line).
+
+++++++++++++++++++++++++++++++++++++++++++++++++
Extended Configuration / Where to go from here?
+++++++++++++++++++++++++++++++++++++++++++++++++
@@ -510,7 +518,9 @@ These are the steps that *roverlay* performs:
2. scan the R Overlay directory (if it exists) for valid ebuilds
-3. **add** - queue all R packages for ebuild creation
+3. import ebuilds from the additions dir
+
+4. **add** - queue all R packages for ebuild creation
* all repositories are asked to list their packages which are then added
to a queue
@@ -522,7 +532,7 @@ These are the steps that *roverlay* performs:
See also: `Package Rules`_
-4. **create** - process each package *p* from the package queue
+5. **create** - process each package *p* from the package queue
(thread-able on a per package basis)
* read *p*'s DESCRIPTION file that contains information fields
@@ -545,9 +555,9 @@ These are the steps that *roverlay* performs:
* **done** with *p* - the overlay creator takes control over *p*
and may decide to write *p*'s ebuild now (or later)
-5. write the overlay
+6. write the overlay
- * write all ebuilds
+ * write all ebuilds and apply patches to them
(supports threads on a per package basis)
* write the *metadata.xml* files
@@ -957,6 +967,91 @@ A local directory will never be modified.
websync_pkglist_.
+=====================
+ Additions Directory
+=====================
+
+The *additions directory* is a directory with overlay-like structure that
+contains extra files for overlay creation. Currently, ebuild patches and
+ebuild files are supported.
+
+To give an idea of how this directory could
+
+
+
+------------------
+ Patching ebuilds
+------------------
+
+Patches can apply to a **specific version** or to **all versions** of a
+package.
+
+The naming convention for patches is (full filesystem paths relative to the
+additions dir):
+
+.. code:: text
+
+ # version-specific patches
+ ${CATEGORY}/${PN}/${PF}[-dddd].patch
+
+ # version-agnostic patches
+ ${CATEGORY}/${PN}/${PN}[-dddd].patch
+
+
+The *-dddd* part is optional and can be used to apply more than one patch to
+an ebuild in the specified order. *d* should be a digit (0..9) and exactly
+4 digits are expected. The not-numbered patch is always applied first.
+So, in theory, up to 10001 patches per ebuild are supported.
+
+The *default* (version-agnostic) patches are only applied to ebuilds for
+which no version-specific patches exist.
+
+Exempting a specific ebuild from being patched can be achieved by creating
+an empty patch file (or a symlink to /dev/null). This is only necessary
+if *default* patches are available, else it adds some overhead.
+
+.. Caution::
+
+ Don't try to patch the (R)DEPEND variables of an ebuild.
+ It will *randomly* break because *roverlay* uses unordered data structures
+ for collecting dependencies.
+
+Example:
+
+.. code:: text
+
+ <additions dir>/sci-CRAN/R_oo/R_oo-1.9.8.patch
+ <additions dir>/sci-CRAN/R_oo/R_oo-1.9.8-0000.patch
+ <additions dir>/sci-CRAN/R_oo/R_oo-1.9.8-0001.patch
+ <additions dir>/sci-R/seewave/seewave-1.6.7.patch
+ <additions dir>/sci-R/seewave/seewave.patch
+
+
+-------------------
+ Importing ebuilds
+-------------------
+
+Foreign ebuilds can be imported into the overlay by simple putting them into
+the additions directory.
+
+The naming convention is similar to ebuild patches and identical to the
+portage tree:
+
+.. code::
+
+ ${CATEGORY}/${PN}/${PF}.ebuild
+
+
+Ebuilds imported that way can not be overwritten by generated ebuilds and
+benefit from most overlay creation features, e.g. Manifest file creation.
+However, they cannot be used for metadata creation.
+
+.. Important::
+
+ Importing ebuilds is only supported by the default Manifest implementation
+ (*ebuildmanifest*).
+
+
==================
Dependency Rules
==================
@@ -1747,6 +1842,11 @@ RSYNC_BWLIMIT
overlay options
-----------------
+.. _ADDITIONS_DIR:
+
+ADDITIONS_DIR:
+ Alias to OVERLAY_ADDITIONS_DIR_.
+
.. _DISTDIR:
DISTDIR
@@ -1772,6 +1872,14 @@ ECLASS
MANIFEST_IMPLEMENTATION
Alias to OVERLAY_MANIFEST_IMPLEMENTATION_.
+.. _OVERLAY_ADDITIONS_DIR:
+
+OVERLAY_ADDITIONS_DIR
+ Directory with an overlay-like structure that contains extra files, e.g.
+ ebuild patches and hand-written ebuilds. This option is not required.
+
+ Defaults to <not set>, which disables this feature.
+
.. _OVERLAY_CATEGORY:
OVERLAY_CATEGORY
@@ -1790,7 +1898,7 @@ OVERLAY_DIR
.. _OVERLAY_DISTDIR_FLAT:
OVERLAY_DISTDIR_FLAT
- A *bool* that controls the overall layout of _OVERLAY_DISTDIR_ROOT.
+ A *bool* that controls the overall layout of OVERLAY_DISTDIR_ROOT_.
A flat distdir is a single directory with all package files or package
file links in it. A nested distdir contains per-package directories.
^ permalink raw reply related [flat|nested] 41+ messages in thread
* [gentoo-commits] proj/R_overlay:gsoc13/next commit in: doc/rst/
@ 2013-06-05 18:08 André Erdmann
2013-06-13 16:34 ` [gentoo-commits] proj/R_overlay:master " André Erdmann
0 siblings, 1 reply; 41+ messages in thread
From: André Erdmann @ 2013-06-05 18:08 UTC (permalink / raw
To: gentoo-commits
commit: c73dd7909d2969d35451bd0945be545ddf0a155b
Author: André Erdmann <dywi <AT> mailerd <DOT> de>
AuthorDate: Wed Jun 5 18:05:14 2013 +0000
Commit: André Erdmann <dywi <AT> mailerd <DOT> de>
CommitDate: Wed Jun 5 18:05:14 2013 +0000
URL: http://git.overlays.gentoo.org/gitweb/?p=proj/R_overlay.git;a=commit;h=c73dd790
doc/rst, package rules: set/rename actions
---
doc/rst/usage.rst | 110 +++++++++++++++++++++++++++++++++++++++++++++++++-----
1 file changed, 100 insertions(+), 10 deletions(-)
diff --git a/doc/rst/usage.rst b/doc/rst/usage.rst
index 34ce7ac..e74c091 100644
--- a/doc/rst/usage.rst
+++ b/doc/rst/usage.rst
@@ -405,6 +405,10 @@ to know in detail what *roverlay* does before running it.
Summary: Expect 300 (slow) or 600MB (fast) memory consumption when using
the default package repositories.
+--fixup-category-move, --fixup-category-move-reverse
+ Remove ebuilds that have been moved to a different category.
+ See `Action Blocks`_ in `Package Rules`_ for details.
+
--config file, -c file
Path to the config file
@@ -1223,6 +1227,7 @@ Rule Stubs
[<keychar>]<short dependency>
+
===============
Package Rules
===============
@@ -1315,7 +1320,12 @@ These *match keywords* are recognized:
| package_name | *implicit* | package file name without version |
| | | and file extension, e.g. *seewave* |
+---------------+------------------+--------------------------------------+
- | name | *implicit* | *alias to package_name* |
+ | ebuild_name | *implicit* | ebuild name ``${PN}``, which is the |
+ | | | package_name with special chars |
+ | | | removed or replaced (e.g., |
+ | | | *R.oo* (pkg) => *R_oo* (ebuild)) |
+ +---------------+------------------+--------------------------------------+
+ | name | *implicit* | *alias to ebuild_name* |
+---------------+------------------+--------------------------------------+
Note the **implicit operator**. It will be used whenever no explicit operator
@@ -1478,21 +1488,60 @@ control *where*) and the number of values they accept:
| keywords | ebuild variables | >= 1 | set per-package |
| | | | ``KEYWORDS`` |
+----------------+------------------+-------------+------------------------+
- | trace | package rules | none | marks a package as |
+ | trace | package rules | none | mark a package as |
| | | | modified |
+ + +-------------+------------------------+
- | | | 1 | adds the stored string |
+ | | | 1 | add the stored string |
| | | | to a package's |
| | | | *modified* variable |
| | | | whenever this action |
| | | | is applied |
+----------------+------------------+-------------+------------------------+
+ | set | package | 2 | set package |
+ +----------------+ metadata, +-------------+ information |
+ | set_<key> | overlay creaton | 1 | |
+ +----------------+------------------+-------------+------------------------+
+ | rename | package | 2 | modify package |
+ +----------------+ metadata, +-------------+ information with |
+ | rename_<key> | overlay creation | 1 | sed-like |
+ | | | | *s/expr/repl/* |
+ | | | | statements |
+ +----------------+------------------+-------------+------------------------+
+
+The two-arg form of the set/rename keywords expect a <key> as first and
+a value / sed expression as second arg. The one-arg form expects the latter
+one only. The "/" delimitier in the sed expression can be any character.
+The following *info keys* can be set and/or modified:
+
+.. table:: info keys for set/rename
+
+ +--------------+---------------------+-------------------------------------+
+ | info key | supports set/rename | description |
+ +==============+=====================+=====================================+
+ | name | yes / yes | rename the ebuild |
+ +--------------+---------------------+-------------------------------------+
+ | category | yes / **no** | set package category |
+ +--------------+---------------------+-------------------------------------+
+ | destfile | yes / yes | rename ebuild destfile by using the |
+ | | | '->' operator in ``${SRC_URI}`` |
+ +--------------+---------------------+-------------------------------------+
+
+.. Caution::
+
+ Category moves are not handled automatically. In incremental mode, overlay
+ creation has to be called with either ``--fixup-category-move`` or
+ ``--fixup-category-move-reverse``, depending on whether the package(s)
+ have been moved away from the default category or back to the default
+ category ("reverse"). Configuring both category move types at once requires
+ a full recreation of the overlay, that is ``rm -rf <overlay dir>``
+ followed by ``roverlay create``.
.. Note::
- Applying the same (non-incremental) ebuild variable action more than once
- is possible, but only the last one will have an effect on ebuild creation.
+ Applying the same (non-incremental) ebuild variable, set or rename action
+ more than once is possible, but only the last one will have an effect
+ on ebuild creation.
Extended Action Block Syntax
@@ -1562,6 +1611,46 @@ if the package is from BIOC/experiment, and otherwise to ``-x86 amd64``:
END;
+A rule that assigns all packages from BIOC-2.10/bioc to sci-bioc:
+
+.. code::
+
+ MATCH:
+ repo == BIOC-2.10/bioc
+ ACTION:
+ set category sci-bioc
+ END;
+
+ # alternatively:
+ MATCH:
+ repo == BIOC-2.10/bioc
+ ACTION:
+ set_category sci-bioc
+ END;
+
+
+The following example prefixes all *yaml* packages with *Rpkg_*:
+
+.. code::
+
+ MATCH:
+ ebuild_name ,= yaml
+ ACTION:
+ rename destfile s/^/Rpkg_/
+ END;
+
+
+Moving such packages to a "R-package" sub directory would be possible, too:
+
+.. code::
+
+ MATCH:
+ name ,= yaml
+ ACTION:
+ rename_destfile s=^=R-package=
+ END;
+
+
=========================
Configuration Reference
=========================
@@ -1639,8 +1728,9 @@ DISTROOT
EBUILD_PROG
Name or path of the ebuild executables that is required for (external)
- Manifest file creation. A wrong value will cause ebuild creation late,
- which is a huge time loss, so make sure that this option is properly set.
+ Manifest file creation. A wrong value will cause ebuild creation to fail
+ late, which is a huge time loss, so make sure that this option is properly
+ set.
Defaults to *ebuild*, which should be fine in most cases.
@@ -2340,9 +2430,9 @@ functionality:
* *sync* all repos and *nosync* all repos (offline mode)
* create *PackageInfo* instances for R packages from all repositories
-++++++++++++++
- Repositories
-++++++++++++++
+++++++++++++
+ Repository
+++++++++++++
The functionality described above is an abstraction layer that calls the
respective function for each repository and collects the result.
^ permalink raw reply related [flat|nested] 41+ messages in thread* [gentoo-commits] proj/R_overlay:master commit in: doc/rst/
2013-06-05 18:08 [gentoo-commits] proj/R_overlay:gsoc13/next " André Erdmann
@ 2013-06-13 16:34 ` André Erdmann
0 siblings, 0 replies; 41+ messages in thread
From: André Erdmann @ 2013-06-13 16:34 UTC (permalink / raw
To: gentoo-commits
commit: c73dd7909d2969d35451bd0945be545ddf0a155b
Author: André Erdmann <dywi <AT> mailerd <DOT> de>
AuthorDate: Wed Jun 5 18:05:14 2013 +0000
Commit: André Erdmann <dywi <AT> mailerd <DOT> de>
CommitDate: Wed Jun 5 18:05:14 2013 +0000
URL: http://git.overlays.gentoo.org/gitweb/?p=proj/R_overlay.git;a=commit;h=c73dd790
doc/rst, package rules: set/rename actions
---
doc/rst/usage.rst | 110 +++++++++++++++++++++++++++++++++++++++++++++++++-----
1 file changed, 100 insertions(+), 10 deletions(-)
diff --git a/doc/rst/usage.rst b/doc/rst/usage.rst
index 34ce7ac..e74c091 100644
--- a/doc/rst/usage.rst
+++ b/doc/rst/usage.rst
@@ -405,6 +405,10 @@ to know in detail what *roverlay* does before running it.
Summary: Expect 300 (slow) or 600MB (fast) memory consumption when using
the default package repositories.
+--fixup-category-move, --fixup-category-move-reverse
+ Remove ebuilds that have been moved to a different category.
+ See `Action Blocks`_ in `Package Rules`_ for details.
+
--config file, -c file
Path to the config file
@@ -1223,6 +1227,7 @@ Rule Stubs
[<keychar>]<short dependency>
+
===============
Package Rules
===============
@@ -1315,7 +1320,12 @@ These *match keywords* are recognized:
| package_name | *implicit* | package file name without version |
| | | and file extension, e.g. *seewave* |
+---------------+------------------+--------------------------------------+
- | name | *implicit* | *alias to package_name* |
+ | ebuild_name | *implicit* | ebuild name ``${PN}``, which is the |
+ | | | package_name with special chars |
+ | | | removed or replaced (e.g., |
+ | | | *R.oo* (pkg) => *R_oo* (ebuild)) |
+ +---------------+------------------+--------------------------------------+
+ | name | *implicit* | *alias to ebuild_name* |
+---------------+------------------+--------------------------------------+
Note the **implicit operator**. It will be used whenever no explicit operator
@@ -1478,21 +1488,60 @@ control *where*) and the number of values they accept:
| keywords | ebuild variables | >= 1 | set per-package |
| | | | ``KEYWORDS`` |
+----------------+------------------+-------------+------------------------+
- | trace | package rules | none | marks a package as |
+ | trace | package rules | none | mark a package as |
| | | | modified |
+ + +-------------+------------------------+
- | | | 1 | adds the stored string |
+ | | | 1 | add the stored string |
| | | | to a package's |
| | | | *modified* variable |
| | | | whenever this action |
| | | | is applied |
+----------------+------------------+-------------+------------------------+
+ | set | package | 2 | set package |
+ +----------------+ metadata, +-------------+ information |
+ | set_<key> | overlay creaton | 1 | |
+ +----------------+------------------+-------------+------------------------+
+ | rename | package | 2 | modify package |
+ +----------------+ metadata, +-------------+ information with |
+ | rename_<key> | overlay creation | 1 | sed-like |
+ | | | | *s/expr/repl/* |
+ | | | | statements |
+ +----------------+------------------+-------------+------------------------+
+
+The two-arg form of the set/rename keywords expect a <key> as first and
+a value / sed expression as second arg. The one-arg form expects the latter
+one only. The "/" delimitier in the sed expression can be any character.
+The following *info keys* can be set and/or modified:
+
+.. table:: info keys for set/rename
+
+ +--------------+---------------------+-------------------------------------+
+ | info key | supports set/rename | description |
+ +==============+=====================+=====================================+
+ | name | yes / yes | rename the ebuild |
+ +--------------+---------------------+-------------------------------------+
+ | category | yes / **no** | set package category |
+ +--------------+---------------------+-------------------------------------+
+ | destfile | yes / yes | rename ebuild destfile by using the |
+ | | | '->' operator in ``${SRC_URI}`` |
+ +--------------+---------------------+-------------------------------------+
+
+.. Caution::
+
+ Category moves are not handled automatically. In incremental mode, overlay
+ creation has to be called with either ``--fixup-category-move`` or
+ ``--fixup-category-move-reverse``, depending on whether the package(s)
+ have been moved away from the default category or back to the default
+ category ("reverse"). Configuring both category move types at once requires
+ a full recreation of the overlay, that is ``rm -rf <overlay dir>``
+ followed by ``roverlay create``.
.. Note::
- Applying the same (non-incremental) ebuild variable action more than once
- is possible, but only the last one will have an effect on ebuild creation.
+ Applying the same (non-incremental) ebuild variable, set or rename action
+ more than once is possible, but only the last one will have an effect
+ on ebuild creation.
Extended Action Block Syntax
@@ -1562,6 +1611,46 @@ if the package is from BIOC/experiment, and otherwise to ``-x86 amd64``:
END;
+A rule that assigns all packages from BIOC-2.10/bioc to sci-bioc:
+
+.. code::
+
+ MATCH:
+ repo == BIOC-2.10/bioc
+ ACTION:
+ set category sci-bioc
+ END;
+
+ # alternatively:
+ MATCH:
+ repo == BIOC-2.10/bioc
+ ACTION:
+ set_category sci-bioc
+ END;
+
+
+The following example prefixes all *yaml* packages with *Rpkg_*:
+
+.. code::
+
+ MATCH:
+ ebuild_name ,= yaml
+ ACTION:
+ rename destfile s/^/Rpkg_/
+ END;
+
+
+Moving such packages to a "R-package" sub directory would be possible, too:
+
+.. code::
+
+ MATCH:
+ name ,= yaml
+ ACTION:
+ rename_destfile s=^=R-package=
+ END;
+
+
=========================
Configuration Reference
=========================
@@ -1639,8 +1728,9 @@ DISTROOT
EBUILD_PROG
Name or path of the ebuild executables that is required for (external)
- Manifest file creation. A wrong value will cause ebuild creation late,
- which is a huge time loss, so make sure that this option is properly set.
+ Manifest file creation. A wrong value will cause ebuild creation to fail
+ late, which is a huge time loss, so make sure that this option is properly
+ set.
Defaults to *ebuild*, which should be fine in most cases.
@@ -2340,9 +2430,9 @@ functionality:
* *sync* all repos and *nosync* all repos (offline mode)
* create *PackageInfo* instances for R packages from all repositories
-++++++++++++++
- Repositories
-++++++++++++++
+++++++++++++
+ Repository
+++++++++++++
The functionality described above is an abstraction layer that calls the
respective function for each repository and collects the result.
^ permalink raw reply related [flat|nested] 41+ messages in thread
* [gentoo-commits] proj/R_overlay:master commit in: doc/rst/
@ 2013-04-25 16:44 André Erdmann
0 siblings, 0 replies; 41+ messages in thread
From: André Erdmann @ 2013-04-25 16:44 UTC (permalink / raw
To: gentoo-commits
commit: ef32760f4053d3fce73fb0e11e9bdf6e960c75ad
Author: André Erdmann <dywi <AT> mailerd <DOT> de>
AuthorDate: Tue Apr 23 09:34:04 2013 +0000
Commit: André Erdmann <dywi <AT> mailerd <DOT> de>
CommitDate: Tue Apr 23 09:34:04 2013 +0000
URL: http://git.overlays.gentoo.org/gitweb/?p=proj/R_overlay.git;a=commit;h=ef32760f
doc/rst: package rules
Added the following:
* package rules 'trace' action
* roverlay.py 'apply_rules' command
* Note about apply identical evar actions more than once
---
doc/rst/usage.rst | 44 ++++++++++++++++++++++++++++----------------
1 files changed, 28 insertions(+), 16 deletions(-)
diff --git a/doc/rst/usage.rst b/doc/rst/usage.rst
index bd541f8..34ce7ac 100644
--- a/doc/rst/usage.rst
+++ b/doc/rst/usage.rst
@@ -57,14 +57,14 @@ This document is targeted at
`Configuration Reference`_ and `Field Definition Config`_.
There is another chapter that is only interesting for testing, the
- `Dependency Resolution Console`_ (9), which can be used to interactively
+ `Dependency Resolution Console`_ (10), which can be used to interactively
test dependency rules.
* *roverlay* code maintainers who want to know **how roverlay works** for
code improvements etc.
- The most important chapter is `Implementation Overview`_ (10) which has
- references to other chapters (4-8) where required.
+ The most important chapter is `Implementation Overview`_ (11) which has
+ references to other chapters (4-9) where required.
Expected prior knowlegde:
@@ -467,6 +467,15 @@ depres_console, depres
More information can be found in the `DepRes Console`_ section.
+apply_rules
+ Applies the package rules to all available packages and reports what has
+ been done, either to stdout or to ``--dump-file <file>``.
+
+ Meant for testing.
+
+ This command implies the **sync** command unless the *--nosync* option
+ is specified.
+
----------------------------
Providing a package mirror
----------------------------
@@ -1469,6 +1478,21 @@ control *where*) and the number of values they accept:
| keywords | ebuild variables | >= 1 | set per-package |
| | | | ``KEYWORDS`` |
+----------------+------------------+-------------+------------------------+
+ | trace | package rules | none | marks a package as |
+ | | | | modified |
+ + + +-------------+------------------------+
+ | | | 1 | adds the stored string |
+ | | | | to a package's |
+ | | | | *modified* variable |
+ | | | | whenever this action |
+ | | | | is applied |
+ +----------------+------------------+-------------+------------------------+
+
+
+.. Note::
+
+ Applying the same (non-incremental) ebuild variable action more than once
+ is possible, but only the last one will have an effect on ebuild creation.
Extended Action Block Syntax
@@ -1529,12 +1553,7 @@ if the package is from BIOC/experiment, and otherwise to ``-x86 amd64``:
* package_name ~ x86_64
* package_name ~ amd64
ACTION:
- MATCH:
- NOR
- * repo == BIOC/experiment
- ACTION:
- keywords "-x86 amd64"
- END;
+ keywords "-x86 amd64"
MATCH:
repo == BIOC/experiment
ACTION:
@@ -1543,13 +1562,6 @@ if the package is from BIOC/experiment, and otherwise to ``-x86 amd64``:
END;
-.. Caution::
-
- Applying the same action more than once per package is not supported.
- That is why the example above uses another nested rule with a *NOR*-match
- instead of simply specifying the desired action.
- This limitation will be removed soon.
-
=========================
Configuration Reference
=========================
^ permalink raw reply related [flat|nested] 41+ messages in thread* [gentoo-commits] proj/R_overlay:master commit in: doc/rst/
@ 2013-03-05 11:27 André Erdmann
0 siblings, 0 replies; 41+ messages in thread
From: André Erdmann @ 2013-03-05 11:27 UTC (permalink / raw
To: gentoo-commits
commit: 1b419a02a273495ee1480c2920103c58a7515bb6
Author: André Erdmann <dywi <AT> mailerd <DOT> de>
AuthorDate: Tue Mar 5 11:21:03 2013 +0000
Commit: André Erdmann <dywi <AT> mailerd <DOT> de>
CommitDate: Tue Mar 5 11:21:03 2013 +0000
URL: http://git.overlays.gentoo.org/gitweb/?p=proj/R_overlay.git;a=commit;h=1b419a02
doc/rst: Package Rule Files, Configuration
This commit adds a new chapter, Package Rules, which explains how to write
rules that control overlay/ebuild creation.
Additionally, the configuration reference has been updated to reflect the
changes recently made, namely the DISTDIR config options.
---
doc/rst/usage.rst | 553 +++++++++++++++++++++++++++++++++++++++++++++++------
1 files changed, 496 insertions(+), 57 deletions(-)
diff --git a/doc/rst/usage.rst b/doc/rst/usage.rst
index 45457f3..bd541f8 100644
--- a/doc/rst/usage.rst
+++ b/doc/rst/usage.rst
@@ -88,7 +88,7 @@ Expected prior knowlegde:
* for Manifest creation:
- * *ebuild* from portage
+ * portage (*ebuild* and/or the *portage libs* directly)
* *true* or *echo* from coreutils or busybox for preventing
package downloads during Manifest creation (optional)
@@ -98,7 +98,7 @@ Expected prior knowlegde:
disk
* 50-55GB disk space for the R packages
- * a filesystem that supports symbolic links
+ * a filesystem that supports symbolic or hard links
* there will be many small-sized files (ebuilds),
so a filesystem with lots of inodes and a small block size
may be advantageous
@@ -109,7 +109,9 @@ Expected prior knowlegde:
using a slower one.
time
- Expect 3-6h execution time, depending on computing and I/O speed.
+ Expect 3-6h execution time for the first run, depending on computing
+ and I/O speed. *roverlay* is able to work in incremental mode,
+ thus making subsequent runs need a lot less time.
---------------------
via emerge (Gentoo)
@@ -180,7 +182,7 @@ as the *R Overlay src directory* from now on.
#!/bin/sh
# /usr/local/bin/roverlay.sh
# example wrapper script for roverlay
- cd ${ROVERLAY_SRC:-~/roverlay/src} && ./roverlay.py $*
+ cd ${ROVERLAY_SRC:-~/roverlay/src} && ./roverlay.py "$@"
==================
@@ -222,13 +224,14 @@ The following options should be set before running *roverlay*:
This option is **required** and can be overridden on the command line
via ``--distroot <directory>``.
- .. Note::
+ Example: DISTFILES = ~/roverlay/distfiles
- This directory will also contain a directory *__tmp__*
- with symlinks to all packages which can be used as package mirror,
- see `Providing a package mirror`_.
+ DISTDIR
+ This sets the directory that contains symbolic or hard links to
+ all package files for which an ebuild could be created. It is used
+ for Manifest file creation and can serve as package mirror directory.
- Example: DISTFILES = ~/roverlay/distfiles
+ Example: DISTDIR = ~/roverlay/distdir
LOG_FILE
This sets the log file. An empty value disables file logging.
@@ -246,7 +249,7 @@ The following options should be set before running *roverlay*:
Be careful with low log levels, especially *DEBUG*.
They produce a lot of messages that help to track ebuild creation of
- the R packages, but increase the size of log files dramatically.
+ the R packages, but increase the log file size dramatically.
LOG_LEVEL_CONSOLE
This sets the console log level.
@@ -272,10 +275,17 @@ have reasonable defaults if *roverlay* has been installed using *emerge*:
REPO_CONFIG
A list with one or more files that list repositories.
This option is **required** and can be overridden on the command line
- via one or more ``repo-config <file>`` statements.
+ via one or more ``--repo-config <file>`` statements.
Example: REPO_CONFIG = ~/roverlay/config/repo.list
+ PACKAGE_RULES:
+ A list of files and/or directories with package rules.
+ Package rules can be used to control overlay/ebuild creation.
+ This option is not required.
+
+ Example: PACKAGE_RULES = ~/roverlay/config/packagerules.d
+
FIELD_DEFINITION
The value of this option should point to a field definition file which
controls how an R package's DESCRIPTION file is read.
@@ -294,6 +304,28 @@ have reasonable defaults if *roverlay* has been installed using *emerge*:
Example: OVERLAY_ECLASS = ~/roverlay/eclass/R-packages.eclass
+ DISTDIR_STRATEGY
+ A list of methods that define how to create the DISTDIR. The methods
+ will be tried in the specified order, until the first one succeeds.
+ The available methods are *symlink*, *hardlink*, *copy* and *tmpdir*.
+ This option is **required**.
+
+ Example: DISTDIR_STRATEGY = "hardlink symlink"
+
+ Try hard links first, then fall back to symbolic ones. This is the
+ default value for this option.
+
+ DISTDIR_FLAT
+ This option controls whether DISTDIR will contain per-package
+ subdirectories with links to the package files ("not flat") or all
+ links/files in a single directory ("flat"). This option is ignored
+ if DISTDIR_STRATEGY is *tmpdir*.
+ Leaving this option as-is (*enabled*) is recommended if you want to use
+ DISTDIR as package mirror.
+
+ Example: DISTDIR_FLAT = yes
+
+
There is another option that is useful for creating new dependency rules,
LOG_FILE_UNRESOLVABLE_, which will write all unresolvable dependencies
to the specified file (one dependency string per line).
@@ -314,6 +346,9 @@ Dependency Rules
See `Dependency Rules`_, which explains the dependency rule syntax amd how
they work.
+Package Rules
+ See `Package Rules`_, which explains how to control *ebuild creation*.
+
Main Config
See `Configuration Reference`_ for all main config options like log file
rotation and assistance for writing new *dependency rules*.
@@ -402,10 +437,13 @@ For **testing** *roverlay*, these **options** may be convenient:
Repo config file to use. Can be specified more than once.
This disables all repo files configured in the main config file.
---distdir directory, --from directory
+--local-distdir directory, --from directory
Create an overlay using the packages found in *directory*. This disables
all other repositories. The *SRC_URI* ebuild variable will be invalid!
+--print-package-rules, --ppr
+ Print package rules to stdout after parsing them and exit.
+
--overlay directory, -O directory
Create the overlay at the given position.
@@ -433,11 +471,16 @@ depres_console, depres
Providing a package mirror
----------------------------
-No recommendations available at this time.
-The current ManifestCreation implementation creates a directory
-*<distfiles root>/__tmp__* with symlinks to all packages, which could be used
-for providing packages, but this may change in near future since external
-Manifest creation is too slow (takes >60% of overlay creation time).
+DISTDIR_ with a non-temporary strategy can be used to create a directory
+containing all package files (as symbolic/hard links or as files).
+You have to set up a *data service*, e.g. an http server, that makes this
+directory accessible.
+
+The default configuration will create hard links to all package files for
+which an ebuild could be created in a single directory. It will fall back
+to symbolic links if hard links are not supported. This should be fine in
+most cases, but fine-tuning can be done via OVERLAY_DISTDIR_STRATEGY_ and
+OVERLAY_DISTDIR_FLAT_.
===============================
Basic Implementation Overview
@@ -454,7 +497,7 @@ These are the steps that *roverlay* performs:
2. scan the R Overlay directory (if it exists) for valid ebuilds
-3. queue all R packages for ebuild creation
+3. **add** - queue all R packages for ebuild creation
* all repositories are asked to list their packages which are then added
to a queue
@@ -462,6 +505,10 @@ These are the steps that *roverlay* performs:
* packages may be declined by the overlay creator if they already have
an ebuild
+ * packages may be declined or manipulated by package rules
+
+ See also: `Package Rules`_
+
4. **create** - process each package *p* from the package queue
(thread-able on a per package basis)
@@ -488,15 +535,16 @@ These are the steps that *roverlay* performs:
5. write the overlay
* write all ebuilds
- (thread-able on a per package basis)
+ (supports threads on a per package basis)
* write the *metadata.xml* files
- (thread-able on a per package basis)
+ (supports threads on a per package basis)
* this uses the latest created ebuild available for a package
* write the *Manifest* files
- (not thread-able)
+ (does not support threads by default / supports threads on a per package
+ basis when using *portage* directly)
* this uses all ebuilds availabe for a package
@@ -1166,6 +1214,341 @@ Rule Stubs
[<keychar>]<short dependency>
+===============
+ Package Rules
+===============
+
+Package Rules can be used to control both overlay and ebuild creation.
+Each package rule consists of conditions, e.g. *package name contains amd64*,
+and actions, e.g. *set KEYWORDS="-x86 amd64"*.
+The actions of a rule will only be applied if a package meets all conditions,
+otherwise the rule does nothing.
+Moreover, rules can contain further rules which will only take effect if all
+enclosing rules match a given package.
+
+--------------------------
+ Package Rule File Syntax
+--------------------------
+
+As stated above, each rule has two parts, a *match block* that lists the
+rule's conditions and an *action block* that defines which actions and
+nested rules are applied to a package if the rule matches it, i.e. if all
+conditions are met.
+
+A rule file contains zero or more package rules.
+Each rule has to declare one *match* and one *action statement* at least.
+The basic syntax for a rule with 1..m *match* and 1..n *action statements* is
+
+.. code::
+
+ MATCH:
+ <match statement 1>
+ <match statement 2>
+ ...
+ <match statement m>
+ ACTION:
+ <action statement 1>
+ <action statement 2>
+ ...
+ <action statement n>
+ END;
+
+
+A rule is introduced by the ``MATCH:`` keyword, which starts the
+*match block* and is followed by one or more *match statements*, one per line.
+The *match block* ends with the ``ACTION:`` keyword, which also starts the
+*action block* and is followed by one or more *action statements*
+(again, one per line). Finally, the rule is terminated by the ``END;`` keyword.
+
+Indention is purely optional, leading and ending whitespace will be discarded.
+Lines starting with ``#`` or ``;`` are considered to be comments and will be
+ignored.
+
+++++++++++++++
+ Match Blocks
+++++++++++++++
+
+The *match block* lists one or more conditions, which all must evaluate to
+*true* for a certain package, otherwise no actions will be applied.
+There are two types of conditions, *trivial* conditions,
+e.g. *always true/false* or *random - flip a coin*, and *non-trivial* ones
+that depend on the information a package has, e.g. its repository or name.
+
+Only *non-trivial* conditions can be defined in *match statements*.
+The consist of a **match keyword** that defines *what* should be matched, an
+**accepted value** to compare against and an **operator** that defines the
+relation *accepted value - package's information*, i.e. *how* to compare.
+The operator can be omitted, in which case it is determined by the
+*match keyword*.
+
+The *match statement* syntax is
+
+.. code::
+
+ <match keyword> [<operator>] <accepted value>
+
+
+These *match keywords* are recognized:
+
+.. table:: match statement keywords
+
+ +---------------+------------------+--------------------------------------+
+ | match keyword | default operator | matches |
+ +===============+==================+======================================+
+ | repo | nocase-string | *alias to repo_name* |
+ +---------------+------------------+--------------------------------------+
+ | repo_name | nocase-string | name of the repo, e.g. *CRAN* |
+ +---------------+------------------+--------------------------------------+
+ | package | *implicit* | package file name with version |
+ | | | but without the file extension, e.g. |
+ | | | *rpart.plot_1.3-0* |
+ +---------------+------------------+--------------------------------------+
+ | package_name | *implicit* | package file name without version |
+ | | | and file extension, e.g. *seewave* |
+ +---------------+------------------+--------------------------------------+
+ | name | *implicit* | *alias to package_name* |
+ +---------------+------------------+--------------------------------------+
+
+Note the **implicit operator**. It will be used whenever no explicit operator
+has been specified in the match statement and the match keyword does not
+define a default one. Four explicit operators are available:
+
+.. table:: match statement operators
+
+ +---------------+-------------+--------------------------------------------+
+ | operator name | operator(s) | description |
+ +===============+=============+============================================+
+ | exact-string | \=\= \= | exact string match |
+ +---------------+-------------+--------------------------------------------+
+ | nocase-string | ,= =, | case-insensitive string match |
+ +---------------+-------------+--------------------------------------------+
+ | exact-regex | ~= =~ | exact regex match *^<expression>$* |
+ +---------------+-------------+--------------------------------------------+
+ | regex | ~~ ~ | partial regex match |
+ +---------------+-------------+--------------------------------------------+
+ | *implicit* | *none* | *exact-regex* operator if *accepted value* |
+ | | | has any wildcard characters (?, \*), else |
+ | | | *exact-string*. Wildcard chars will |
+ | | | be replaced with their regex equivalents. |
+ +---------------+-------------+--------------------------------------------+
+
+The *accepted value* is a simple string or a regular expression,
+which depends on the operator.
+
+
+Extended Match Block Syntax
+---------------------------
+
+Sometimes, a rule should apply its actions to a package if it matches *any*
+condition, e.g. *package from CRAN or BIOC*, or if it does not match a certain
+condition, e.g. *package not from BIOC/experiment*.
+
+This is supported by special *match keywords* that represent
+*boolean functions*. Such a *match statement* is introduced by the keyword,
+followed by one or more *match statements* that are indented by one asterisk
+``*`` or dash ``-`` character for each *boolean function* that is currently
+active. These characters are important and indicate the *match depth*.
+A depth of 0 means that no function is active.
+
+Syntax Example:
+
+.. code::
+
+ MATCH:
+ <match statement 1, match depth 0>
+ ...
+ <boolean function>
+ * <match statement 1, match depth 1>
+ * <match statement 2, match depth 1>
+ * ...
+ * <match statement m, match depth 1>
+ ...
+ <match statement n, match depth 0>
+ ACTION:
+ ...
+ END;
+
+
+For reference, the following table lists the *boolean functions* available,
+their *match keywords* and their meaning:
+
+.. table:: boolean functions
+
+ +------------------+-------------+----------------------------------------+
+ | boolean function | match | description |
+ | | keyword(s) | |
+ +==================+=============+========================================+
+ | AND | and all && | all listed conditions must match |
+ +------------------+-------------+----------------------------------------+
+ | OR | or \|\| | any |
+ | | | of the listed conditions must match |
+ +------------------+-------------+----------------------------------------+
+ | XOR1 | xor1 xor ^^ | exactly one |
+ | | | of the listed conditions must match |
+ +------------------+-------------+----------------------------------------+
+ | NOR | nor none | none |
+ | | | of the listed conditions must match |
+ +------------------+-------------+----------------------------------------+
+
+
+In other words, a (boolean) match keyword starts a *nested match block*
+at any position in the current one and increases the *match depth* by one.
+The nested block is terminated by indenting out, i.e. decreasing the
+*match depth* by one. The (extended) match statement syntax then becomes:
+
+.. code::
+
+ <'*'^<match_depth>> <(basic) match statement>
+
+
+.. Note::
+
+ The extended match statement syntax does not support boolean functions
+ with a fixed number of conditions, e.g. 1. This is why there is no
+ *NOT* function. The definition for more than one condition would be
+ ambiguous, either *NOR* or *NAND*.
+
+ Correspondingly, the logic for the top-level match block is *AND* by
+ convention.
+
+
+Using this syntax, match blocks can be nested indefinitely (minus technical
+limitations):
+
+.. code::
+
+ MATCH:
+ <match statement 1, depth 0>
+ <boolean function 2, depth 0>
+ * <match statement 1, depth 1>
+ * <match statement 2, depth 1>
+ * ...
+ * <match statement k-1, depth 1>
+ * <boolean function k, depth 1>
+ ** <match statement 1, depth 2>
+ ** ...
+ ** <match statement o, depth 2>
+ * <match statement k+1, depth 1>
+ * ...
+ * <match statement m, depth 1>
+ ...
+ <match statement n, depth 0>
+ ACTION:
+ ...
+ END;
+
+
++++++++++++++++
+ Action Blocks
++++++++++++++++
+
+The action block syntax is quite simple. Each *action statement* starts
+with an *action keyword*, optionally followed by one or more *values*.
+
+Action statement syntax:
+
+.. code::
+
+ <action keyword> [<value>]*
+
+
+The value(s) can be enclosed by quotation characters (``"``, ``'``).
+
+The following table lists all *action keywords*, their impact (*what* they
+control *where*) and the number of values they accept:
+
+.. table:: action keywords
+
+ +----------------+------------------+-------------+------------------------+
+ | action keyword | affects | # of values | description |
+ +================+==================+=============+========================+
+ | ignore | | | ignore package, |
+ +----------------+ overlay creation | none | do not try to create |
+ | do-not-process | | | an ebuild for it |
+ +----------------+------------------+-------------+------------------------+
+ | keywords | ebuild variables | >= 1 | set per-package |
+ | | | | ``KEYWORDS`` |
+ +----------------+------------------+-------------+------------------------+
+
+
+Extended Action Block Syntax
+----------------------------
+
+A mentioned before, action blocks can contain *nested rules*. The syntax
+is exactly the same as for the normal package rules:
+
+.. code::
+
+ MATCH:
+ # top-level rule, match block
+ ...
+ ACTION:
+ # top-level rule, action block
+ ...
+ MATCH:
+ # nested rule, match block
+ ...
+ ACTION:
+ # nested rule, action block
+ ...
+ END;
+ # top-level rule, action block continues
+ ...
+ END;
+
+Rules can be nested indefinitely, whitespace indention is optional.
+A *nested rule* only becomes active, i.e. tries to match a package, if its
+enclosing rule already matched it. This can be used to reduce the number of
+checks necessary for a given package.
+
++++++++++++++++++++++++
+ Package Rule Examples
++++++++++++++++++++++++
+
+A rule that ignores the 'yaqcaffy' package from CRAN, which is also available
+from BIOC:
+
+.. code::
+
+ MATCH:
+ repo == CRAN
+ package_name == yaqcaffy
+ ACTION:
+ do-not-process
+ END;
+
+
+A more complex example that sets the ``KEYWORDS`` ebuild variable for
+all packages whose name contains *amd64* or *x86_64* to ``-x86 ~amd64``
+if the package is from BIOC/experiment, and otherwise to ``-x86 amd64``:
+
+.. code::
+
+ MATCH:
+ or
+ * package_name ~ x86_64
+ * package_name ~ amd64
+ ACTION:
+ MATCH:
+ NOR
+ * repo == BIOC/experiment
+ ACTION:
+ keywords "-x86 amd64"
+ END;
+ MATCH:
+ repo == BIOC/experiment
+ ACTION:
+ keywords "-x86 ~amd64"
+ END;
+ END;
+
+
+.. Caution::
+
+ Applying the same action more than once per package is not supported.
+ That is why the example above uses another nested rule with a *NOR*-match
+ instead of simply specifying the desired action.
+ This limitation will be removed soon.
=========================
Configuration Reference
@@ -1262,6 +1645,21 @@ RSYNC_BWLIMIT
overlay options
-----------------
+.. _DISTDIR:
+
+DISTDIR
+ Alias to OVERLAY_DISTDIR_ROOT_.
+
+.. _DISTDIR_FLAT:
+
+DISTDIR_FLAT
+ Alias to OVERLAY_DISTDIR_FLAT_.
+
+.. _DISTDIR_STRATEGY:
+
+DISTDIR_STRATEGY
+ Alias to OVERLAY_DISTDIR_STRATEGY_.
+
.. _ECLASS:
ECLASS
@@ -1287,6 +1685,61 @@ OVERLAY_DIR
This option is **required**.
+.. _OVERLAY_DISTDIR_FLAT:
+
+OVERLAY_DISTDIR_FLAT
+ A *bool* that controls the overall layout of _OVERLAY_DISTDIR_ROOT.
+
+ A flat distdir is a single directory with all package files or package
+ file links in it. A nested distdir contains per-package directories.
+
+ Defaults to *true*.
+ This option has no effect if the distdir strategy is *tmpdir*.
+
+.. _OVERLAY_DISTDIR_ROOT:
+
+OVERLAY_DISTDIR_ROOT
+ Sets the DISTDIR root directory. It is used for Manifest file
+ creation, but can serve as package mirror directory as well.
+
+ The actual appearance of this directory is up to the distdir strategy
+ (OVERLAY_DISTDIR_STRATEGY_) and OVERLAY_DISTDIR_FLAT_.
+ Basically, it contains all package files that have a valid ebuild.
+
+ .. Note::
+
+ This directory will not be cleaned up, only broken symbolic links
+ will be removed. On the one hand, it is absolutely guaranteed that
+ package files will not disappear unless replaced by a new file with
+ the same name, but on the other hand, the directory may get bloated
+ over time.
+
+.. _OVERLAY_DISTDIR_STRATEGY:
+
+OVERLAY_DISTDIR_STRATEGY
+ The distdir strategy defines *how* package files are created.
+ It is a list of methods that will be tried in the specified order, until
+ the first one succeeds.
+
+ .. table:: distdir creation methods
+
+ +----------+-----------------------------------------------------------+
+ | method | description |
+ +==========+===========================================================+
+ | symlink | use symbolic links |
+ +----------+-----------------------------------------------------------+
+ | hardlink | use hard links |
+ +----------+-----------------------------------------------------------+
+ | copy | copy package files |
+ | | Obviously, this will need much more disk space. |
+ +----------+-----------------------------------------------------------+
+ | tmpdir | use a temporary DISTDIR that will be deleted at exit. |
+ | | This method is not compatible with any of the above. |
+ +----------+-----------------------------------------------------------+
+
+ This option is **required**, but has a reasonable value in the default
+ config file, "hardlink symlink".
+
.. _OVERLAY_ECLASS:
OVERLAY_ECLASS
@@ -1311,9 +1764,11 @@ OVERLAY_KEEP_NTH_LATEST
.. _OVERLAY_MANIFEST_IMPLEMENTATION:
OVERLAY_MANIFEST_IMPLEMENTATION
- Sets the implementation to use for Manifest file writing.
- Available choices are 'external:ebuild', 'default' and 'none'.
- Defaults to 'default'.
+ Sets the implementation that will be used for Manifest file writing.
+ Available choices are *ebuild*, *portage*, *default* and *none*.
+ Defaults to *default* (-> *ebuild*).
+ *portage* is highly experimental and therefore not recommended
+ for production usage.
.. Note::
@@ -1332,37 +1787,6 @@ OVERLAY_NAME
Defaults to *R_Overlay*.
-.. _OVERLAY_SYMLINK_DISTROOT_ROOT:
-
-OVERLAY_SYMLINK_DISTROOT_ROOT
- Root directory where per-package (name) symlink directories will be
- created during Manifest file creation.
-
- Defaults to <not set>, which is only valid if OVERLAY_SYMLINK_DISTROOT_TMP_
- is set to true, in which case a directory in the user's $TMPDIR will be
- used.
-
-.. _OVERLAY_SYMLINK_DISTROOT_TMP:
-
-OVERLAY_SYMLINK_DISTROOT_TMP
- A *bool* that sets whether the symlink distroot is a temporary (true)
- or persistent (false) directory.
-
- A temporary directory will be wiped at exit
- whereas a persistent one will only be cleaned up
- (remove broken symlinks, ...).
-
-
-.. _SYMLINK_DISTROOT:
-
-SYMLINK_DISTROOT
- Alias to OVERLAY_SYMLINK_DISTROOT_ROOT_.
-
-.. _SYMLINK_DISTROOT_TMP:
-
-SYMLINK_DISTROOT_TMP
- Alias to OVERLAY_SYMLINK_DISTROOT_TMP_.
-
--------------------
other config files
--------------------
@@ -1390,6 +1814,17 @@ FIELD_DEFINITION
FIELD_DEFINITION_FILE
Alias to FIELD_DEFINITION_.
+.. _PACKAGE_RULES:
+
+PACKAGE_RULES
+ Alias to PACKAGE_RULE_FILES_.
+
+.. _PACKAGE_RULE_FILES:
+
+PACKAGE_RULE_FILES
+ A list of files and directories with package rules.
+ Directories will be recursively scanned for rule files.
+
.. _REPO_CONFIG:
REPO_CONFIG
@@ -1990,6 +2425,7 @@ The other available repository types have an internal-only implementation:
Repository types also need an entry in the repository config loader in order
to be accessible.
+
---------
Overlay
---------
@@ -2063,6 +2499,8 @@ It does the following steps:
#. The *DESCRIPTION* and *SRC_URI* variables are created
+#. Add any ebuild variables created by package rules, e.g. *KEYWORDS*
+
#. **done** - Generate the ebuild as text, add it to *p* and mark *p*
as *ebuild successfully created*
@@ -2083,7 +2521,8 @@ to an *Ebuilder* that automatically sorts them and creates the ebuild.
Overlay creation is the process of creating an overlay for many R packages
and *roverlay*'s main task. More specifically, *OverlayCreation* is an
*R packages -> Overlay (-> overlay in filesystem)* interface.
-It accepts *PackageInfo* objects as input, tries to reserve a slot in the
+It accepts *PackageInfo* objects as input, applies package rules to them,
+which possibly filters some packages out, tries to reserve a slot in the
overlay for them, and, if successful, adds them to the work queue.
The work queue is processed by *OverlayWorkers* that run ebuild creation
@@ -2152,7 +2591,7 @@ Try other dependency types
document, such property is indicated by *TRY_OTHER* and
*<preferred dependency type> first*, e.g. *package first*.
-*Mandatory* and *Option* are mutually exclusive.
+*Mandatory* and *Optional* are mutually exclusive.
The *dependency type* of a *dependency string* is determined by its origin,
i.e. info field in the package's DESCRIPTION file.
^ permalink raw reply related [flat|nested] 41+ messages in thread* [gentoo-commits] proj/R_overlay:master commit in: doc/rst/
@ 2013-01-28 23:54 André Erdmann
0 siblings, 0 replies; 41+ messages in thread
From: André Erdmann @ 2013-01-28 23:54 UTC (permalink / raw
To: gentoo-commits
commit: 8c08fcc1d159d64c89f12415eab6dd487e5cd3b2
Author: André Erdmann <dywi <AT> mailerd <DOT> de>
AuthorDate: Mon Jan 28 23:27:31 2013 +0000
Commit: André Erdmann <dywi <AT> mailerd <DOT> de>
CommitDate: Mon Jan 28 23:37:04 2013 +0000
URL: http://git.overlays.gentoo.org/gitweb/?p=proj/R_overlay.git;a=commit;h=8c08fcc1
doc/rst: Configuration Reference
added SYMLINK_DISTROOT, SYMLINK_DISTROOT_TMP options
---
doc/rst/usage.rst | 31 +++++++++++++++++++++++++++++++
1 files changed, 31 insertions(+), 0 deletions(-)
diff --git a/doc/rst/usage.rst b/doc/rst/usage.rst
index 46e5e00..45457f3 100644
--- a/doc/rst/usage.rst
+++ b/doc/rst/usage.rst
@@ -1332,6 +1332,37 @@ OVERLAY_NAME
Defaults to *R_Overlay*.
+.. _OVERLAY_SYMLINK_DISTROOT_ROOT:
+
+OVERLAY_SYMLINK_DISTROOT_ROOT
+ Root directory where per-package (name) symlink directories will be
+ created during Manifest file creation.
+
+ Defaults to <not set>, which is only valid if OVERLAY_SYMLINK_DISTROOT_TMP_
+ is set to true, in which case a directory in the user's $TMPDIR will be
+ used.
+
+.. _OVERLAY_SYMLINK_DISTROOT_TMP:
+
+OVERLAY_SYMLINK_DISTROOT_TMP
+ A *bool* that sets whether the symlink distroot is a temporary (true)
+ or persistent (false) directory.
+
+ A temporary directory will be wiped at exit
+ whereas a persistent one will only be cleaned up
+ (remove broken symlinks, ...).
+
+
+.. _SYMLINK_DISTROOT:
+
+SYMLINK_DISTROOT
+ Alias to OVERLAY_SYMLINK_DISTROOT_ROOT_.
+
+.. _SYMLINK_DISTROOT_TMP:
+
+SYMLINK_DISTROOT_TMP
+ Alias to OVERLAY_SYMLINK_DISTROOT_TMP_.
+
--------------------
other config files
--------------------
^ permalink raw reply related [flat|nested] 41+ messages in thread* [gentoo-commits] proj/R_overlay:master commit in: doc/rst/
@ 2013-01-09 19:15 André Erdmann
0 siblings, 0 replies; 41+ messages in thread
From: André Erdmann @ 2013-01-09 19:15 UTC (permalink / raw
To: gentoo-commits
commit: 033e97680c88f8094d3c6e74a875ee0ddf8523f2
Author: André Erdmann <dywi <AT> mailerd <DOT> de>
AuthorDate: Wed Jan 9 18:51:52 2013 +0000
Commit: André Erdmann <dywi <AT> mailerd <DOT> de>
CommitDate: Wed Jan 9 18:51:52 2013 +0000
URL: http://git.overlays.gentoo.org/gitweb/?p=proj/R_overlay.git;a=commit;h=033e9768
doc/rst: update user config file location
---
doc/rst/usage.rst | 4 ++--
1 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/doc/rst/usage.rst b/doc/rst/usage.rst
index 39cdac8..31c81f0 100644
--- a/doc/rst/usage.rst
+++ b/doc/rst/usage.rst
@@ -192,11 +192,11 @@ as the *R Overlay src directory* from now on.
------------------------------
*roverlay* needs a configuration file to run.
-If roverlay has been installed with *emerge*, it will for the config file in
+If roverlay has been installed with *emerge*, it will look for the config file in
that order:
1. *<current directory>/R-overlay.conf*
-#. *~/.R-overlay.conf*
+#. *~/roverlay/R-overlay.conf*
#. */etc/roverlay/R-overlay.conf*,
which is part of the installation but has to be modified.
^ permalink raw reply related [flat|nested] 41+ messages in thread* [gentoo-commits] proj/R_overlay:master commit in: doc/rst/
@ 2012-08-20 11:16 André Erdmann
0 siblings, 0 replies; 41+ messages in thread
From: André Erdmann @ 2012-08-20 11:16 UTC (permalink / raw
To: gentoo-commits
commit: 07234a05091a688bea219ccab3a7f5ea96b09fa3
Author: André Erdmann <dywi <AT> mailerd <DOT> de>
AuthorDate: Mon Aug 20 11:13:58 2012 +0000
Commit: André Erdmann <dywi <AT> mailerd <DOT> de>
CommitDate: Mon Aug 20 11:13:58 2012 +0000
URL: http://git.overlays.gentoo.org/gitweb/?p=proj/R_overlay.git;a=commit;h=07234a05
doc: implementation overview
---
doc/rst/usage.rst | 385 +++++++++++++++++++++++++++--------------------------
1 files changed, 195 insertions(+), 190 deletions(-)
diff --git a/doc/rst/usage.rst b/doc/rst/usage.rst
index 2963d3a..87e4ed6 100644
--- a/doc/rst/usage.rst
+++ b/doc/rst/usage.rst
@@ -1535,8 +1535,8 @@ how an R package's DESCRIPTION file is read.
See the next section, `default field definition file`_, for an example.
Each information field has its own section which declares a set of options
-and flags. Flags are case-insensivitve options
-without a value - they're enabled by listing them.
+and flags. Flags are case-insensitive options without a value - they're
+enabled by listing them.
.. _field option:
.. _field options:
@@ -1573,7 +1573,8 @@ Known field options:
.. _field option\: alias_nocase:
alias_nocase
- Same as `field option: alias`_, but aliases are case-insensitive.
+ Same as `field option: alias`_, but the listed aliases are
+ case-insensitive.
.. _field option\: flags:
@@ -1806,8 +1807,10 @@ Example Session:
Implementation Overview
=========================
-<< there's also in-code documentation for which html files can be generated
-using pydoc >>
+This chapter gives an in-depth overview of how roverlay works.
+Code documentation is also available and html pages for it can be created
+with ``make pydoc`` in the *R Overlay src directory* or by using pydoc
+directly.
-------------
PackageInfo
@@ -1823,7 +1826,7 @@ After initialization it contains data like
* the SRC_URI
* the package name, version
-Not all of these data are really existent, some are calculated. *SRC_URI*,
+Not all of these are really existent, some are calculated. *SRC_URI*,
for example, can often be calculated by combining the origin's "root" src uri
with the package file.
@@ -1835,6 +1838,168 @@ the overlay. The overlay may refuse to accept *p* if there's already an ebuild
for it. Otherwise, *p* is now part of the overlay and has to pass
*ebuild creation*.
+
+-----------------------
+ Repository Management
+-----------------------
+
+Repositories are managed in a list-like object, *RepoList*. Its task is to
+provide R package input for overlay creation and implements the following
+functionality:
+
+* load repository config from file(s)
+* directly add a directory as *local repository*
+* *sync* all repos and *nosync* all repos (offline mode)
+* create *PackageInfo* instances for R packages from all repositories
+
+++++++++++++++
+ Repositories
+++++++++++++++
+
+The functionality described above is an abstraction layer that calls the
+respective function for each repository and collects the result.
+So, while the *RepoList* object knows *what* to do for all repositories,
+a repository object *repo* extends this by:
+
+* data
+
+ * repository *type*
+
+ * filesystem directory *distdir* where *repo*'s R packages are stored
+
+ * the *root src_uri*, which is used to determine the *SRC_URI* ebuild
+ variable for all packages from *repo*:
+
+ *SRC_URI* = *root src_uri* + '/' + <path of R package relative to *distdir*>
+
+ * other data like the sync status, repository name
+
+* functionality
+
+ * sync/nosync
+ * create *PackageInfo* instances for all packages from *repo*
+ * status indicators, e.g. if sync was successful
+
+The actual functionality depends on the *repository type*, i.e. the
+implementing class. The most basic implementation that provides all common
+data, status indicator functions and *PackageInfo* creation is called
+*BasicRepo*. It also implements a rather abstract sync function that calls
+subclass-specifc *_sync()*/*_nosync()* functions if available.
+*BasicRepo*s are used to realize *local repositories*. The other available
+repository types, *rsync*, *websync_repo* and *websync_pkglist* derive from
+*BasicRepo*.
+
+
+Adding new repository types
+---------------------------
+
+Adding new repository types is best done by creating a new repo class
+that inherits *BasicRepo*. The table below shows *BasicRepo*'s subclass
+awareness (concerning sync) and what may be changed if required.
+Most repository types want to define their own sync functionality and
+can do so by implementing *_dosync()*:
+
+.. table:: deriving repository types from BasicRepo
+
+ +-------------------+--------------------------------------------------------+
+ | function/method | description |
+ +===================+========================================================+
+ | _dosync() | sync packages using a remote, has to return True/False |
+ +-------------------+--------------------------------------------------------+
+ | _nosync() | sync packages in offline mode (returns True/False) |
+ +-------------------+--------------------------------------------------------+
+ | sync (*online?*) | implemented by *BasicRepo*, calls _dosync()/_nosync() |
+ | | if available, else checks whether *distdir* exists |
+ +-------------------+--------------------------------------------------------+
+ | scan_distdir(...) | *BasicRepo*: creates *PackageInfo* instances for all |
+ | | R packages in *distdir*. Derived classes can override |
+ | | this e.g. if they want to expose only synced packages |
+ +-------------------+--------------------------------------------------------+
+ | ready() | tells whether _dosync()/_nosync() was successful, |
+ | | used by *RepoList* to decide whether to call |
+ | | scan_distdir() or not. Properly implemented by |
+ | | *BasicRepo* when using its sync() method, else needs |
+ | | to be overridden. |
+ +-------------------+--------------------------------------------------------+
+ | __init__() | has to be implemented if the new class has additional |
+ | | data. Refer to in-code documentation and examples. |
+ +-------------------+--------------------------------------------------------+
+
+
+The *RsyncRepo*, for example, extends *BasicRepo* by rsync-specific data, e.g.
+the uri used for rsync, and has its own *__init__()* method. It also
+implements *_dosync()*, which calls the *rsync* executable in a filtered
+environment that contains only variables like USER, PATH and RSYNC_PROXY.
+The other available repository types have an internal-only implementation:
+
+.. table::
+
+ +-----------------+--------------------+----------------------------------+
+ | repository type | repository class | _dosync() implementation |
+ +=================+====================+==================================+
+ | local | BasicRepo | *not applicable* |
+ +-----------------+--------------------+----------------------------------+
+ | rsync | RsyncRepo | **external**, using *rsync* in |
+ | | | a filtered environment |
+ +-----------------+--------------------+----------------------------------+
+ | websync_repo | WebsyncRepo | internal, using *urllib* |
+ | websync_pkglist | WebsyncPackageList | |
+ +-----------------+--------------------+----------------------------------+
+
+Repository types also need an entry in the repository config loader in order
+to be accessible.
+
+---------
+ Overlay
+---------
+
+The *overlay* is roverlay's central data structure that represents a *portage
+overlay*. It's organized in a tree structure (overlay root, categories,
+package directories) and implements all overlay-related functionality:
+
+* Scan the *portage overlay* for existing ebuilds
+
+* Add *PackageInfo* objects to the overlay. Packages can be declined if
+ they already exist as ebuild (incremental overlay).
+ Adding multiple packages at once is **thread-safe**, but overlay writing
+ is not.
+
+* List all known packages (filesystem and runtime/memory)
+
+* Write the overlay to its filesystem location
+
+ * initialize the overlay (write the *profiles/* directory,
+ import eclass files)
+ * Write ebuilds; all *PackageInfo* instances with an ebuild will be written
+ * Generate and write metadata
+ * Write Manifest files
+
+* Features like `OVERLAY_KEEP_NTH_LATEST`_ make use of ebuild deletion,
+ but unconditional ebuild deletion is only available on the package directory
+ level
+
++++++++++++++++++++
+ Metadata Creation
++++++++++++++++++++
+
+*metadata.xml* files are created with a tree structure that contains *metadata
+nodes*, e.g. '<pkgmetadata>...</pkgmetadata>' and '<use>...</use>' are *nodes*.
+The current implementation writes the R package's full description
+('Title' and 'Description') into the metadata file.
+Metadata creation uses the latest package, i.e. highest version,
+for which an ebuild has been created.
+
++++++++++++++++++++
+ Manifest Creation
++++++++++++++++++++
+
+Manifest files are created by calling the *ebuild* executable for each
+package directory in a filtered environment where FETCHCOMMAND and
+RESUMECOMMAND are set to no-operation. The directories that contain the R
+package files are passed in the PORTAGE_RO_DISTDIRS variable and DISTDIR
+is set to `DISTFILES_ROOT`_/__tmp__.
+
+
-----------------
Ebuild Creation
-----------------
@@ -1842,6 +2007,8 @@ for it. Otherwise, *p* is now part of the overlay and has to pass
Ebuild creation is the process centered around one *PackageInfo* instance *p*
that tries to create an ebuild for it.
+It does the following steps:
+
#. Read the DESCRIPTION file of *p* R package tarball and stores the
data in an associative array ('DESCRIPTION field' -> 'data')
@@ -1868,6 +2035,22 @@ class. An *EbuildVar* defines its position in the ebuild and how its text
output should look like. Ebuild text is created by adding ebuild variables
to an *Ebuilder* that automatically sorts them and creates the ebuild.
+------------------
+ Overlay Creation
+------------------
+
+Overlay creation is the process of creating an overlay for many R packages
+and *roverlay*'s main task. More specifically, *OverlayCreation* is an
+*R packages -> Overlay (-> overlay in filesystem)* interface.
+It accepts *PackageInfo* objects as input, tries to reserve a slot in the
+overlay for them, and, if successful, adds them to the work queue.
+
+The work queue is processed by *OverlayWorkers* that run ebuild creation
+for a *PackageInfo* object and inform the *OverlayCreation* about the result
+afterwards. Overlay creation doesn't stop if an ebuild cannot be created,
+instead the event is logged. Running more than one *OverlayWorker* in parallel
+is possible.
+
-----------------------
Dependency Resolution
-----------------------
@@ -1897,7 +2080,7 @@ From the ebuild creation perspective, dependency resolution works like this:
Details about dependency resolution like how *channels* work can be found
-in the following sections.
+in the following subsections.
++++++++++++++++++
Dependency types
@@ -1923,10 +2106,10 @@ System Dependency
Try other dependency types
This declares that the *dependency string* can be resolved by ignoring its
- dependency type partially. This property allows to resolve
- package dependencies as system dependencies and vice versa.
- Throughout this guide, such property is indicated by *<preferred dependency
- type> first*, e.g. *package first*.
+ dependency type partially. This property allows to resolve package
+ dependencies as system dependencies and vice versa. Throughout this
+ document, such property is indicated by *TRY_OTHER* and
+ *<preferred dependency type> first*, e.g. *package first*.
*Mandatory* and *Option* are mutually exclusive.
@@ -2069,7 +2252,7 @@ available, *channels* and *listeners*.
Listener modules
Listener modules are used to react on certain dependency resolution
- *events*, e.g. if a *dependency environment* has been found unresolvable <<lang!!>>.
+ *events*, e.g. if a *dependency environment* is unresolvable.
They usually have access to the *event* and the *dependency environment*.
However, they cannot begin communication with the *dependency resolver*.
@@ -2142,181 +2325,3 @@ Its mode of operation of operation is best described in pseudo-code:
The dependency resolver can be **run as thread**, in which case the while loop
becomes "loop until resolver closes".
-
-
------------------------
- Repository Management
------------------------
-
-Repositories are managed in a list-like object, *RepoList*. Its task is to
-provide R package input for overlay creation and implements the following
-functionality:
-
-* load repository config from file(s)
-* directly add a directory as *local repository*
-* *sync* all repos and *nosync* all repos (offline mode)
-* create *PackageInfo* instances for R packages from all repositories
-
-++++++++++++++
- Repositories
-++++++++++++++
-
-The functionality described above is an abstraction layer that calls the
-respective function for each repository and collects the result.
-So, while the *RepoList* object knows *what* to do for all repositories,
-a repository object *repo* extends this by:
-
-* data
-
- * repository *type*
-
- * filesystem directory *distdir* where *repo*'s R packages are stored
-
- * the *root src_uri*, which is used to determine the *SRC_URI* ebuild
- variable for all packages from *repo*:
-
- *SRC_URI* = *root src_uri* + '/' + <path of R package relative to *distdir*>
-
- * other data like the sync status, repository name
-
-* functionality
-
- * sync/nosync
- * create *PackageInfo* instances for all packages from *repo*
- * status indicators, e.g. if sync was successful
-
-The actual functionality depends on the *repository type*, i.e. the
-implementing class. The most basic implementation that provides all common
-data, status indicator functions and *PackageInfo* creation is called
-*BasicRepo*. It also implements a rather abstract sync function that calls
-subclass-specifc *_sync()*/*_nosync()* functions if available.
-*BasicRepo*s are used to realize *local repositories*. The other available
-repository types, *rsync*, *websync_repo* and *websync_pkglist* derive from
-*BasicRepo*.
-
-
-Adding new repository types
----------------------------
-
-This is best done by creating a new repo class that inherits *BasicRepo*.
-The table below shows *BasicRepo*'s subclass awareness (concerning sync)
-and what may be changed if required. Most repository types want to define
-their own sync functionality and can do so by implementing *_dosync()*:
-
-.. table:: deriving repository types from BasicRepo
-
- +-------------------+--------------------------------------------------------+
- | function/method | description |
- +===================+========================================================+
- | _dosync() | sync packages using a remote, has to return True/False |
- +-------------------+--------------------------------------------------------+
- | _nosync() | sync packages in offline mode (returns True/False) |
- +-------------------+--------------------------------------------------------+
- | sync (*online?*) | implemented by *BasicRepo*, calls _dosync()/_nosync() |
- | | if available, else checks whether *distdir* exists |
- +-------------------+--------------------------------------------------------+
- | scan_distdir(...) | *BasicRepo*: creates *PackageInfo* instances for all |
- | | R packages in *distdir*. Derived classes can override |
- | | this e.g. if they want to expose only synced packages |
- +-------------------+--------------------------------------------------------+
- | ready() | tells whether _dosync()/_nosync() was successful, |
- | | used by *RepoList* to decide whether to call |
- | | scan_distdir() or not. Properly implemented by |
- | | *BasicRepo* when using its sync() method, else needs |
- | | to be overridden. |
- +-------------------+--------------------------------------------------------+
- | __init__() | has to be implemented if the new class has additional |
- | | data. Refer to in-code documentation and examples. |
- +-------------------+--------------------------------------------------------+
-
-
-The *RsyncRepo*, for example, extends *BasicRepo* by rsync-specific data, e.g.
-the uri used for rsync, and has its own *__init__()* method. It also
-implements *_dosync()*, which calls the *rsync* executable in a filtered
-environment that contains only variables like USER, PATH and RSYNC_PROXY.
-The other available repository types have an internal-only implementation:
-
-.. table::
-
- +-----------------+--------------------+----------------------------------+
- | repository type | repository class | _dosync() implementation |
- +=================+====================+==================================+
- | local | BasicRepo | *not applicable* |
- +-----------------+--------------------+----------------------------------+
- | rsync | RsyncRepo | **external**, using *rsync* in |
- | | | a filtered environment |
- +-----------------+--------------------+----------------------------------+
- | websync_repo | WebsyncRepo | internal, using *urllib* |
- | websync_pkglist | WebsyncPackageList | |
- +-----------------+--------------------+----------------------------------+
-
-Repository types also need an entry in the repository config loader in order
-to be accessible.
-
----------
- Overlay
----------
-
-*overlay* is roverlay's central data structure that represents a *portage
-overlay*. It's organized in a tree structure (overlay root, categories,
-package directories) and implements all overlay-related functionality:
-
-* Scan the *portage overlay* for existing ebuilds
-
-* Add *PackageInfo* objects to the overlay. Packages can be declined if
- they already exist as ebuild (incremental overlay).
- Adding multiple packages at once is **thread-safe**.
-
-* List all known packages (filesystem and runtime/memory)
-
-* Write the overlay to its filesystem location
-
- * initialize the overlay (write the *profiles/* directory,
- import eclass files)
- * Write ebuilds; all *PackageInfo* instances with an ebuild will be written
- * Generate and write metadata
- * Write Manifest files
-
-* Features like `OVERLAY_KEEP_NTH_LATEST`_ make use of ebuild deletion,
- but unconditional ebuild deletion is only available on the package directory
- level
-
-+++++++++++++++++++
- Metadata Creation
-+++++++++++++++++++
-
-*metadata.xml* files are created with a tree structure that contains *metadata
-nodes*, e.g. '<pkgmetadata>...</pkgmetadata>' and '<use>...</use>' are *nodes*.
-The current implementation writes the R package's full description
-('Title' and 'Description') into the metadata file.
-Metadata creation uses the latest package, i.e. highest version,
-for which an ebuild has been created.
-
-+++++++++++++++++++
- Manifest Creation
-+++++++++++++++++++
-
-Manifest files are created by calling the *ebuild* executable for each
-package directory in a filtered environment where FETCHCOMMAND and
-RESUMECOMMAND are set to no-operation. The directories that contain the R
-package files are passed in the PORTAGE_RO_DISTDIRS variable and DISTDIR
-is set to `DISTFILES_ROOT`_/__tmp__.
-
-------------------
- Overlay Creation
-------------------
-
-Overlay creation is the process of creating an overlay for many R packages
-and *roverlay*'s main task. More specifically, *OverlayCreation* is an
-*R packages -> Overlay (-> overlay in filesystem)* interface.
-It accepts *PackageInfo* objects as input, tries to reserve a slot in the
-overlay for them, and, if successful, adds them to the work queue.
-
-The work queue is processed by *OverlayWorkers* that run ebuild creation
-for a *PackageInfo* object and inform the *OverlayCreation* about the result
-afterwards. Overlay creation doesn't stop if an ebuild cannot be created,
-instead the event is logged. Running more than one *OverlayWorker* in parallel
-is possible.
-
-
-
^ permalink raw reply related [flat|nested] 41+ messages in thread* [gentoo-commits] proj/R_overlay:master commit in: doc/rst/
@ 2012-08-17 17:26 André Erdmann
0 siblings, 0 replies; 41+ messages in thread
From: André Erdmann @ 2012-08-17 17:26 UTC (permalink / raw
To: gentoo-commits
commit: 23d49744b5d100f5e7add1f24fe666c6a81754c6
Author: André Erdmann <dywi <AT> mailerd <DOT> de>
AuthorDate: Fri Aug 17 17:26:27 2012 +0000
Commit: André Erdmann <dywi <AT> mailerd <DOT> de>
CommitDate: Fri Aug 17 17:26:27 2012 +0000
URL: http://git.overlays.gentoo.org/gitweb/?p=proj/R_overlay.git;a=commit;h=23d49744
doc: implementation overview
---
doc/rst/usage.rst | 243 +++++++++++++++++++++++++++++++++++++++--------------
1 files changed, 181 insertions(+), 62 deletions(-)
diff --git a/doc/rst/usage.rst b/doc/rst/usage.rst
index 3550522..2963d3a 100644
--- a/doc/rst/usage.rst
+++ b/doc/rst/usage.rst
@@ -1809,16 +1809,12 @@ Example Session:
<< there's also in-code documentation for which html files can be generated
using pydoc >>
----------------------------
- << ~ package job steps >>
----------------------------
-
-+++++++++++++
+-------------
PackageInfo
-+++++++++++++
+-------------
*PackageInfo* is the data object that contains all information about an
-R package and is created by the owing repository.
+R package and is created by the owning repository.
After initialization it contains data like
@@ -1831,7 +1827,7 @@ Not all of these data are really existent, some are calculated. *SRC_URI*,
for example, can often be calculated by combining the origin's "root" src uri
with the package file.
-Initializion may fail if the package's name cannot be understood, which is
+Initialization may fail if the package's name cannot be understood, which is
most likely due to unsupported versioning schemes.
It is then checked whether the newly created *PackageInfo p* can be part of
@@ -1839,20 +1835,38 @@ the overlay. The overlay may refuse to accept *p* if there's already an ebuild
for it. Otherwise, *p* is now part of the overlay and has to pass
*ebuild creation*.
-+++++++++++++++++
+-----------------
Ebuild Creation
-+++++++++++++++++
+-----------------
+
+Ebuild creation is the process centered around one *PackageInfo* instance *p*
+that tries to create an ebuild for it.
+
+#. Read the DESCRIPTION file of *p* R package tarball and stores the
+ data in an associative array ('DESCRIPTION field' -> 'data')
+
+#. Call `dependency resolution`_
-<<
-reads package tarball;uses depres;uses Ebuilder
+#. If dependency resolution was successful, dependency ebuild variables are
+ created (*DEPEND*, *RDEPEND* and *R_SUGGESTS*, also *IUSE*, *MISSINGDEPS*).
+ Otherwise **ebuild creation stops** and *p* is marked as
+ *ebuild uncreatable*. The overlay creation may decide to remove *p* in
+ order to save memory etc.
-adds data to a PackageInfo instance
+#. The *DESCRIPTION* and *SRC_URI* variables are created
-* DESCRIPTION data parsed from the package tarball
-* ebuild by using (a) direct desc data access for description, (b) depres
- and (c) already available information (SRC_URI)
+#. **done** - Generate the ebuild as text, add it to *p* and mark *p*
+ as *ebuild successfully created*
->>
+
+++++++++++++++++++
+ Ebuild Variables
+++++++++++++++++++
+
+Each ebuild variable is an object whose class is derived from the *EbuildVar*
+class. An *EbuildVar* defines its position in the ebuild and how its text
+output should look like. Ebuild text is created by adding ebuild variables
+to an *Ebuilder* that automatically sorts them and creates the ebuild.
-----------------------
Dependency Resolution
@@ -2134,70 +2148,175 @@ becomes "loop until resolver closes".
Repository Management
-----------------------
-<<
+Repositories are managed in a list-like object, *RepoList*. Its task is to
+provide R package input for overlay creation and implements the following
+functionality:
+
+* load repository config from file(s)
+* directly add a directory as *local repository*
+* *sync* all repos and *nosync* all repos (offline mode)
+* create *PackageInfo* instances for R packages from all repositories
+
+++++++++++++++
+ Repositories
+++++++++++++++
-+-----------------+-------------------------------------+
-| repository type | underlying implementation |
-+=================+=====================================+
-| rsync | **external** (filtered environment) |
-+-----------------+-------------------------------------+
-| websync_* | **internal** (using *urllib*) |
-+-----------------+-------------------------------------+
-| local | **none** (nothing to sync) |
-+-----------------+-------------------------------------+
+The functionality described above is an abstraction layer that calls the
+respective function for each repository and collects the result.
+So, while the *RepoList* object knows *what* to do for all repositories,
+a repository object *repo* extends this by:
->>
+* data
-------------------
- Overlay Creation
-------------------
+ * repository *type*
+
+ * filesystem directory *distdir* where *repo*'s R packages are stored
-<<
- dimension: per-package (Ebuild Creation) -> _all_ packages ($this)
+ * the *root src_uri*, which is used to determine the *SRC_URI* ebuild
+ variable for all packages from *repo*:
- also implements the overlay module, that provides ebuild writing,
- metadata/Manifest creation/writing, incremental overlay creation,...
+ *SRC_URI* = *root src_uri* + '/' + <path of R package relative to *distdir*>
->>
+ * other data like the sync status, repository name
-<<
- overlay module
+* functionality
- .. caution::
+ * sync/nosync
+ * create *PackageInfo* instances for all packages from *repo*
+ * status indicators, e.g. if sync was successful
- Never deep-copy an overlay object. It leads to infinite recursion
- due to double-linkage beetween PackageInfo and PackageDir.
+The actual functionality depends on the *repository type*, i.e. the
+implementing class. The most basic implementation that provides all common
+data, status indicator functions and *PackageInfo* creation is called
+*BasicRepo*. It also implements a rather abstract sync function that calls
+subclass-specifc *_sync()*/*_nosync()* functions if available.
+*BasicRepo*s are used to realize *local repositories*. The other available
+repository types, *rsync*, *websync_repo* and *websync_pkglist* derive from
+*BasicRepo*.
->>
+
+Adding new repository types
+---------------------------
+
+This is best done by creating a new repo class that inherits *BasicRepo*.
+The table below shows *BasicRepo*'s subclass awareness (concerning sync)
+and what may be changed if required. Most repository types want to define
+their own sync functionality and can do so by implementing *_dosync()*:
+
+.. table:: deriving repository types from BasicRepo
+
+ +-------------------+--------------------------------------------------------+
+ | function/method | description |
+ +===================+========================================================+
+ | _dosync() | sync packages using a remote, has to return True/False |
+ +-------------------+--------------------------------------------------------+
+ | _nosync() | sync packages in offline mode (returns True/False) |
+ +-------------------+--------------------------------------------------------+
+ | sync (*online?*) | implemented by *BasicRepo*, calls _dosync()/_nosync() |
+ | | if available, else checks whether *distdir* exists |
+ +-------------------+--------------------------------------------------------+
+ | scan_distdir(...) | *BasicRepo*: creates *PackageInfo* instances for all |
+ | | R packages in *distdir*. Derived classes can override |
+ | | this e.g. if they want to expose only synced packages |
+ +-------------------+--------------------------------------------------------+
+ | ready() | tells whether _dosync()/_nosync() was successful, |
+ | | used by *RepoList* to decide whether to call |
+ | | scan_distdir() or not. Properly implemented by |
+ | | *BasicRepo* when using its sync() method, else needs |
+ | | to be overridden. |
+ +-------------------+--------------------------------------------------------+
+ | __init__() | has to be implemented if the new class has additional |
+ | | data. Refer to in-code documentation and examples. |
+ +-------------------+--------------------------------------------------------+
+
+
+The *RsyncRepo*, for example, extends *BasicRepo* by rsync-specific data, e.g.
+the uri used for rsync, and has its own *__init__()* method. It also
+implements *_dosync()*, which calls the *rsync* executable in a filtered
+environment that contains only variables like USER, PATH and RSYNC_PROXY.
+The other available repository types have an internal-only implementation:
+
+.. table::
+
+ +-----------------+--------------------+----------------------------------+
+ | repository type | repository class | _dosync() implementation |
+ +=================+====================+==================================+
+ | local | BasicRepo | *not applicable* |
+ +-----------------+--------------------+----------------------------------+
+ | rsync | RsyncRepo | **external**, using *rsync* in |
+ | | | a filtered environment |
+ +-----------------+--------------------+----------------------------------+
+ | websync_repo | WebsyncRepo | internal, using *urllib* |
+ | websync_pkglist | WebsyncPackageList | |
+ +-----------------+--------------------+----------------------------------+
+
+Repository types also need an entry in the repository config loader in order
+to be accessible.
+
+---------
+ Overlay
+---------
+
+*overlay* is roverlay's central data structure that represents a *portage
+overlay*. It's organized in a tree structure (overlay root, categories,
+package directories) and implements all overlay-related functionality:
+
+* Scan the *portage overlay* for existing ebuilds
+
+* Add *PackageInfo* objects to the overlay. Packages can be declined if
+ they already exist as ebuild (incremental overlay).
+ Adding multiple packages at once is **thread-safe**.
+
+* List all known packages (filesystem and runtime/memory)
+
+* Write the overlay to its filesystem location
+
+ * initialize the overlay (write the *profiles/* directory,
+ import eclass files)
+ * Write ebuilds; all *PackageInfo* instances with an ebuild will be written
+ * Generate and write metadata
+ * Write Manifest files
+
+* Features like `OVERLAY_KEEP_NTH_LATEST`_ make use of ebuild deletion,
+ but unconditional ebuild deletion is only available on the package directory
+ level
+++++++++++++++++++
Metadata Creation
+++++++++++++++++++
-<<
-TODO = Fr;
-metadata creation uses the latest package that has
-an ebuild and uses its description data (Title and Description) to create
-a metadata tree structure that is string-exportable
->>
-
-Metadata Tree Structure
------------------------
-
-<<TODO = Fr;>>
+*metadata.xml* files are created with a tree structure that contains *metadata
+nodes*, e.g. '<pkgmetadata>...</pkgmetadata>' and '<use>...</use>' are *nodes*.
+The current implementation writes the R package's full description
+('Title' and 'Description') into the metadata file.
+Metadata creation uses the latest package, i.e. highest version,
+for which an ebuild has been created.
+++++++++++++++++++
Manifest Creation
+++++++++++++++++++
-<< TODO = Fr;
-not much to say here, "ebuild *<ebuild file>* digest" is called in a safe
-environment (with FETCHCOMMAND/RESUMECOMMAND set to no-op) and creates
-the Manifest file
->>
+Manifest files are created by calling the *ebuild* executable for each
+package directory in a filtered environment where FETCHCOMMAND and
+RESUMECOMMAND are set to no-operation. The directories that contain the R
+package files are passed in the PORTAGE_RO_DISTDIRS variable and DISTDIR
+is set to `DISTFILES_ROOT`_/__tmp__.
+
+------------------
+ Overlay Creation
+------------------
+
+Overlay creation is the process of creating an overlay for many R packages
+and *roverlay*'s main task. More specifically, *OverlayCreation* is an
+*R packages -> Overlay (-> overlay in filesystem)* interface.
+It accepts *PackageInfo* objects as input, tries to reserve a slot in the
+overlay for them, and, if successful, adds them to the work queue.
+
+The work queue is processed by *OverlayWorkers* that run ebuild creation
+for a *PackageInfo* object and inform the *OverlayCreation* about the result
+afterwards. Overlay creation doesn't stop if an ebuild cannot be created,
+instead the event is logged. Running more than one *OverlayWorker* in parallel
+is possible.
+
-++++++++++++++++
- Ebuild Writing
-++++++++++++++++
-<<TODO = Fr;>>
^ permalink raw reply related [flat|nested] 41+ messages in thread* [gentoo-commits] proj/R_overlay:master commit in: doc/rst/
@ 2012-08-16 16:28 André Erdmann
0 siblings, 0 replies; 41+ messages in thread
From: André Erdmann @ 2012-08-16 16:28 UTC (permalink / raw
To: gentoo-commits
commit: 4711e9841406fb24dc738ec54c71a1197575156c
Author: André Erdmann <dywi <AT> mailerd <DOT> de>
AuthorDate: Thu Aug 16 16:27:49 2012 +0000
Commit: André Erdmann <dywi <AT> mailerd <DOT> de>
CommitDate: Thu Aug 16 16:27:49 2012 +0000
URL: http://git.overlays.gentoo.org/gitweb/?p=proj/R_overlay.git;a=commit;h=4711e984
doc: Introduction, Dependency Resolution
---
doc/rst/usage.rst | 325 +++++++++++++++++++++++++++++++++++++++++++----------
1 files changed, 263 insertions(+), 62 deletions(-)
diff --git a/doc/rst/usage.rst b/doc/rst/usage.rst
index 8be11e5..3550522 100644
--- a/doc/rst/usage.rst
+++ b/doc/rst/usage.rst
@@ -9,6 +9,8 @@
.. _roverlay git repo:
http://git.overlays.gentoo.org/gitweb/?p=proj/R_overlay.git;a=summary
+.. _git repository: `roverlay git repo`_
+
.. _omegahat's PACKAGES file:
http://www.omegahat.org/R/src/contrib/PACKAGES
@@ -26,47 +28,51 @@
Introduction
==============
-<<
-~audience~
+*roverlay* is an application that aims to provide integration of R packages
+in Gentoo by creating a portage overlay for them.
+Naturally, this also requires proper dependency resolution, especially on the
+system level which cannot be done by *install.packages()* in R.
-"system" / using roverlay:
- Chapter 2 + 3, `Installation`_ and `Running Roverlay`_
- maybe Chapter 4, `Basic Implementation Overview` that describes what to
- expect from roverlay
+The project associated with *roverlay* is called
+*Automatically generated overlay of R packages*, the initial work has been
+done during the *Google Summer of Code 2012*.
-"maintenance" / controlling overlay creation:
- Chapters 5-8, maybe chapter 9 for testing dependency rules
+At its current state, *roverlay* is capable of creating a complete overlay
+with metadata and Manifest files by reading R packages.
+It's also able to work incrementally, i.e. update an existing *R Overlay*.
+Most aspects of overlay creation are configurable with text files.
-"testing"
- Chapters: variable (depends on what to test to what extend)
- basically chapters 2-9 (with the `DepRes Console`_ section that is only
- interesting for testing)
+*roverlay* is written in python. There's no homepage available, only a
+`git repository`_ that contains the source code.
- * overlay maintainers. Those who use *roverlay* to generate the overlay
- and provide an R package mirror.
- * *roverlay* maintainers
+This document is targeted at
-"architecture doc"
- Mainly chapter 10, maybe 4,5,6,8
+ * overlay maintainers who **use roverlay** to create the R Overlay
-"end-users" / using the created overlay
- not covered by this doc
+ The most relevant chapters are `Installation`_ (2) and
+ `Running Roverlay`_ (3). Additionally, have a look at
+ `Basic Implementation Overview`_ (4) if you want to know what *roverlay*
+ does and what to expect from the generated overlay.
-expected prior knowledge
+ * *roverlay* maintainers who **control and test overlay creation**,
+ e.g. configure which R packages will be part of the generated overlay
- * portage overlays
- * what an R package is (except for "system")
+ Depending on what you want to configure, chapters 5-8 are relevant,
+ namely `Repositories / Getting Packages`_, `Dependency Rules`_,
+ `Configuration Reference`_ and `Field Definition Config`_.
->>
+ There's another chapter that is only interesting for testing, the
+ `Dependency Resolution Console`_ (9), which can be used to interactively
+ test dependency rules.
+ * *roverlay* code maintainers who want to know **how roverlay works** for
+ code improvements etc.
-*roverlay* is
-Automatically Generated Overlay of R packages;;
-GSoC Project;;
-`roverlay git repo`_ (no project homepage available);;
-<>;;
-
+ The most important chapter is `Implementation Overview`_ (10) which has
+ references to other chapters (4-8) where required.
+It's expected that you already know about *R packages* (basically a tarball)
+and some portage basics, e.g. *Depend Atoms* and what an overlay is.
==============
@@ -663,6 +669,9 @@ Example: seewave from CRAN
</longdescription>
</pkgmetadata>
+
+.. _repositories:
+
=================================
Repositories / Getting Packages
=================================
@@ -1797,6 +1806,54 @@ Example Session:
Implementation Overview
=========================
+<< there's also in-code documentation for which html files can be generated
+using pydoc >>
+
+---------------------------
+ << ~ package job steps >>
+---------------------------
+
++++++++++++++
+ PackageInfo
++++++++++++++
+
+*PackageInfo* is the data object that contains all information about an
+R package and is created by the owing repository.
+
+After initialization it contains data like
+
+* the path to the R package file
+* the origin (repository)
+* the SRC_URI
+* the package name, version
+
+Not all of these data are really existent, some are calculated. *SRC_URI*,
+for example, can often be calculated by combining the origin's "root" src uri
+with the package file.
+
+Initializion may fail if the package's name cannot be understood, which is
+most likely due to unsupported versioning schemes.
+
+It is then checked whether the newly created *PackageInfo p* can be part of
+the overlay. The overlay may refuse to accept *p* if there's already an ebuild
+for it. Otherwise, *p* is now part of the overlay and has to pass
+*ebuild creation*.
+
++++++++++++++++++
+ Ebuild Creation
++++++++++++++++++
+
+<<
+reads package tarball;uses depres;uses Ebuilder
+
+adds data to a PackageInfo instance
+
+* DESCRIPTION data parsed from the package tarball
+* ebuild by using (a) direct desc data access for description, (b) depres
+ and (c) already available information (SRC_URI)
+
+>>
+
-----------------------
Dependency Resolution
-----------------------
@@ -1904,6 +1961,57 @@ Whether and how this variable is used is up to the eclass file(s).
The default *R-packages eclass* reports unresolvable,
but optional dependencies during the *pkg_postinst()* ebuild phase.
++++++++++++++++++++++++++
+ Dependency Environments
++++++++++++++++++++++++++
+
+A *dependency environment* is an object that reflects the whole dependency
+resolution process of a single *dependency string*. It usually contains
+the *dependency string*, its *dependency type*, information about its
+resolution state (*resolved*, *unresolvable*, *to be processed*) and the
+resolving resolving *dependency*, if any.
+
+It is initialized by the communication *channel* and processed by the
+*dependency resolver*.
+
++++++++++++++++++++
+ EbuildJob Channel
++++++++++++++++++++
+
+The *EbuildJob Channel* is used by the ebuild creation to communicate with
+the *dependency resolver*. It is initialized by an ebuild creation process and
+realizes a greedy *string to string* dependency resolution.
+
+Its mode of operation is
+
+#. Accept *dependency strings*, create *dependency environments* for them
+ and add them to the registered *dependency resolver*.
+ The *dependency resolver* may already be resolving the added dependencies.
+
+ Leave this state if the ebuild creation signalizes that all *dependency
+ strings* have been added.
+
+#. Tell the *dependency resolver* that this channel is waiting for results.
+
+ The channel using a *blocking queue* for waiting. It expects that the
+ *dependency resolver* sends processed *dependency environments* though this
+ channel, whether successful or not.
+
+#. Process each received *dependency environment* until all dependencies have
+ been resolved or waiting doesn't make sense anymore, i.e. at least one
+ *required* dependency could not be resolved.
+
+ * add successful resolved dependencies to the "resolved" list
+ * add unresolved, but optional dependencies to the "unresolvable" list
+ * any other unresolved dependency is interpreted as "channel cannot satisfy
+ the request", the **channel stops waiting** for remaining results.
+
+#. The *channel* returns the result to the ebuild creation.
+
+ It's either a 2-tuple of resolved and unresolvable dependencies or
+ "nothing" if the request is not satisfiable, i.e. one or more required
+ dependencies are unresolvable.
+
+++++++++++++++++++++++
Dependency Rule Pools
@@ -1938,65 +2046,158 @@ This pool contains rules for all known R packages and is able to resolve
R package dependencies.
By convention, it will never resolve any system dependencies.
++++++++++++++++++++++++++++++
+ Dependency Resolver Modules
++++++++++++++++++++++++++++++
+The dependency resolver can be extended by modules. Two base types are
+available, *channels* and *listeners*.
+Listener modules
+ Listener modules are used to react on certain dependency resolution
+ *events*, e.g. if a *dependency environment* has been found unresolvable <<lang!!>>.
+ They usually have access to the *event* and the *dependency environment*.
+ However, they cannot begin communication with the *dependency resolver*.
-<<
-Dependency resolution is split into several components.
-Each package *p* has zero or more dependencies,......
+ In the current *roverlay* implementation, a listener module is used to
+ report all unresolvable dependencies to a separate file.
+Channel modules
+ Channel modules interact with the resolver, e.g. queue dependencies for
+ resolution, wait for results, and send them to the other end.
-Dependency Rules
- x
+ The previously described `EbuildJob Channel`_ is such a channel.
++++++++++++++++++++++
+ Dependency Resolver
++++++++++++++++++++++
-Dependency Resolver
- This is the
+The dependency resolver puts all parts of dependency resolution together,
+*rule pools*, *listeners* and *channels*. It's main task is a loop that
+processes all queued *dependency environments* and sends the result back to
+their origin (an *EbuildJob channel*).
->>
+Besides that, it also offers functionality to register new channels, add
+listeners, load rule pools from one or more files etc..
+A dependency resolver has to be explicitly closed in which case it will stop
+working and notify all listeners about that.
-<<
-EbuildJobChannel "work flow"
+Its mode of operation of operation is best described in pseudo-code:
+
+.. code-block:: text
+
+ while <dependencies queued for resolution>
+
+ depenv <= get the next dependency environment
+ resolved <= False
+
+ # try to resolve depenv
+
+ if <depenv's type contains PACKAGE> and
+ <the dynamic selfdep rule pool resolves depenv>
-1. accept *dependency strings* and create *dependency environments* for them
- until the ebuild creation signalizes that all dep strs have been added
- (by calling *<channel>.satisfy_request()*)
+ resolved <= True
-2. tell the *dependency resolver* that this channel is waiting for results
+ else
+ if <a rule pool accepts depenv's type and resolves depenv>
+ resolved <= True
-3. wait for a (partial) result from the resolver
+ else if <depenv's type contains TRY_OTHER>
- * **stop waiting** if a *required* dependency could not be resolved.
- The channel returns "unresolvable" to the ebuild creation and
- closes afterwards.
+ if <a rule pool supports TRY_OTHER and doesn't accept depenv's type
+ and resolves depenv>
- * add unresolvable, but optional deps to the list of unresolvable deps
- * add resolved deps to the list of resolved deps
+ resolved <= True
+ end if
+ end if
-4. repeat 3. until all dependencies have been processed (or one req dep could
- not be resolved)
-5. return the resolved and unresolvable deps to the ebuild creation
+ # send the result to depenv's channel
+
+ if resolved
+ mark depenv as resolved, add the resolving dependency to it
+
+ else
+ mark depenv as unresolvable
+
+ end if
+
+ send depenv to its channel
+
+ end while
+
+The dependency resolver can be **run as thread**, in which case the while loop
+becomes "loop until resolver closes".
+
+
+-----------------------
+ Repository Management
+-----------------------
+
+<<
+
++-----------------+-------------------------------------+
+| repository type | underlying implementation |
++=================+=====================================+
+| rsync | **external** (filtered environment) |
++-----------------+-------------------------------------+
+| websync_* | **internal** (using *urllib*) |
++-----------------+-------------------------------------+
+| local | **none** (nothing to sync) |
++-----------------+-------------------------------------+
>>
+------------------
+ Overlay Creation
+------------------
<<
-Dependency Resolver "work flow"
+ dimension: per-package (Ebuild Creation) -> _all_ packages ($this)
-threaded executation
+ also implements the overlay module, that provides ebuild writing,
+ metadata/Manifest creation/writing, incremental overlay creation,...
-.. code-block:: text
+>>
+
+<<
+ overlay module
+
+ .. caution::
- LOOP UNTIL RESOLVER CLOSED:
+ Never deep-copy an overlay object. It leads to infinite recursion
+ due to double-linkage beetween PackageInfo and PackageDir.
- wait for a dependency environment d_e
+>>
- search in all rule pools
- if rule pool matches d_e
- d_e is resolved - tell d_e the resolving dependency and send
- d_e back to its channel
++++++++++++++++++++
+ Metadata Creation
++++++++++++++++++++
- d_e is unresolvable - send it back to its channel
+<<
+TODO = Fr;
+metadata creation uses the latest package that has
+an ebuild and uses its description data (Title and Description) to create
+a metadata tree structure that is string-exportable
+>>
+
+Metadata Tree Structure
+-----------------------
+<<TODO = Fr;>>
+
++++++++++++++++++++
+ Manifest Creation
++++++++++++++++++++
+
+<< TODO = Fr;
+not much to say here, "ebuild *<ebuild file>* digest" is called in a safe
+environment (with FETCHCOMMAND/RESUMECOMMAND set to no-op) and creates
+the Manifest file
>>
+
+++++++++++++++++
+ Ebuild Writing
+++++++++++++++++
+
+<<TODO = Fr;>>
^ permalink raw reply related [flat|nested] 41+ messages in thread* [gentoo-commits] proj/R_overlay:master commit in: doc/rst/
@ 2012-08-15 16:14 André Erdmann
0 siblings, 0 replies; 41+ messages in thread
From: André Erdmann @ 2012-08-15 16:14 UTC (permalink / raw
To: gentoo-commits
commit: a5e90d3d5031ba52b9d010fdf8cd989dc2cba981
Author: André Erdmann <dywi <AT> mailerd <DOT> de>
AuthorDate: Wed Aug 15 16:14:22 2012 +0000
Commit: André Erdmann <dywi <AT> mailerd <DOT> de>
CommitDate: Wed Aug 15 16:14:22 2012 +0000
URL: http://git.overlays.gentoo.org/gitweb/?p=proj/R_overlay.git;a=commit;h=a5e90d3d
usage doc
---
doc/rst/usage.rst | 1125 ++++++++++++++++++++++++++++++++---------------------
1 files changed, 682 insertions(+), 443 deletions(-)
diff --git a/doc/rst/usage.rst b/doc/rst/usage.rst
index 4f77f44..8be11e5 100644
--- a/doc/rst/usage.rst
+++ b/doc/rst/usage.rst
@@ -1,6 +1,6 @@
.. |date| date:: %b %d %Y
-.. header:: Automatically Generated Overlay of R package - Manual (|date|)
+.. header:: Automatically Generated Overlay of R packages - Manual (|date|)
.. _roverlay-9999.ebuild:
@@ -26,6 +26,40 @@
Introduction
==============
+<<
+~audience~
+
+"system" / using roverlay:
+ Chapter 2 + 3, `Installation`_ and `Running Roverlay`_
+ maybe Chapter 4, `Basic Implementation Overview` that describes what to
+ expect from roverlay
+
+"maintenance" / controlling overlay creation:
+ Chapters 5-8, maybe chapter 9 for testing dependency rules
+
+"testing"
+ Chapters: variable (depends on what to test to what extend)
+ basically chapters 2-9 (with the `DepRes Console`_ section that is only
+ interesting for testing)
+
+ * overlay maintainers. Those who use *roverlay* to generate the overlay
+ and provide an R package mirror.
+ * *roverlay* maintainers
+
+"architecture doc"
+ Mainly chapter 10, maybe 4,5,6,8
+
+"end-users" / using the created overlay
+ not covered by this doc
+
+expected prior knowledge
+
+ * portage overlays
+ * what an R package is (except for "system")
+
+>>
+
+
*roverlay* is
Automatically Generated Overlay of R packages;;
GSoC Project;;
@@ -57,32 +91,30 @@ GSoC Project;;
* for generating documentation files: python docutils >= 0.9
-* hardware requirements (when the default configuration):
+* hardware requirements (when using the default configuration):
disk
+ * 50-55GB disk space for the R packages
* a filesystem that supports symbolic links
- * 50-55GB disk space for the R packages when using the default
- R package hosts (CRAN with archive, BIOC, R-Forge and Omegahat)
- * <a lot of inodes for PORTAGE_RO_DISTDIR/__tmp__ and the overlay>
+ * there will be many small-sized files (ebuilds),
+ so a filesystem with lots of inodes and a small block size
+ may be advantageous
memory
up to 600MB which depends on the amount of processed packages and the
write mechanism in use. The amount can be halved (approximately) when
using a slower one.
- other
- No other hardware requirements. <as a measurement for computing speed,
- i/o requirements:
- overlay creation takes 3-4h on modern desktop hardware with overlay
- writing to a tmpfs, most time is spent for Manifest creation>
+ time
+ Expect 3-6h execution time, depending on computing and I/O speed.
---------------------
via emerge (Gentoo)
---------------------
A live ebuild is available, `roverlay-9999.ebuild`_.
-Add it to your local overlay and run ``emerge roverlay``, which will also
-install all config files into */etc/roverlay*.
+Add it to your local overlay and run ``emerge roverlay``, which also installs
+all necessary config files into */etc/roverlay*.
---------------------
Manual Installation
@@ -107,8 +139,8 @@ install *roverlay* and its python modules:
* *PYMOD_FILE_LIST*, which lists the installed python module files
and defaults to './roverlay_files.list'
-* *PYTHON*, name of path of the python interpreter that will run 'setup.py',
- defaults to 'python'
+* *PYTHON*, name or path of the python interpreter that is used to run
+ 'setup.py', defaults to 'python'
*roverlay* can later be uninstalled with ``make uninstall``.
@@ -123,20 +155,20 @@ install *roverlay* and its python modules:
.. Warning::
- Support for this installation type is limited - it won't install/create
+ Support for this installation type is limited - it doesn't create/install
any config files!
---------------------------------------
Using *roverlay* without installation
---------------------------------------
-This is possible, too. Make sure to meet the dependencies as listed
-in Prerequisites_.
-Then, simply clone the git repository to a local directory that
-will later be referenced as the *R Overlay src directory*.
+This is possible, too.
+Make sure to meet the dependencies as listed in Prerequisites_.
+Then, simply clone the git repository to a local directory that is referenced
+as the *R Overlay src directory* from now on.
.. Note::
- You'll have to cd into the *R Overlay src directory* before running
+ You have to *cd* into the *R Overlay src directory* before running
*roverlay* to ensure that the python modules can be imported correctly.
You can work around this by setting up a wrapper script:
@@ -158,7 +190,6 @@ will later be referenced as the *R Overlay src directory*.
------------------------------
*roverlay* needs a configuration file to run.
-
If you've installed *roverlay* with *emerge*, it will look for the config
file in that order:
@@ -199,7 +230,7 @@ The following options should be set before running *roverlay*:
Example: DISTFILES = ~/roverlay/distfiles
LOG_FILE
- This sets the log file.
+ This sets the log file. An empty value disables file logging.
Example: LOG_FILE = ~/roverlay/log/roverlay.log
@@ -231,16 +262,14 @@ have reasonable defaults if *roverlay* has been installed using *emerge*:
SIMPLE_RULES_FILE
This option lists dependency rule files and/or directories with
- such files that should be used for dependency resolution (see
- `Dependency Rules`).
+ such files that should be used for dependency resolution.
Although not required, this option is **recommended** since ebuild
creation without dependency rules fails for most R packages.
Example: SIMPLE_RULES_FILE = ~/roverlay/config/simple-deprules.d
REPO_CONFIG
- A list with one or more files that list repositories
- (see `Repositories / Getting Packages`_).
+ A list with one or more files that list repositories.
This option is **required** and can be overridden on the command line
via one or more ``repo-config <file>`` statements.
@@ -268,7 +297,31 @@ There's another option that is useful if you want to create new dependency
rules, LOG_FILE_UNRESOLVABLE_, which will write all unresolvable dependencies
to the specified file (one dependency string per line).
-For details and a full configuration reference, see `Configuration Reference`_.
++++++++++++++++++++++++++++++++++++++++++++++++++
+ Extended Configuration / Where to go from here?
++++++++++++++++++++++++++++++++++++++++++++++++++
+
+Proceed with `Running it`_ if you're fine with the default configuration
+and the changes you've already made, otherwise the following chapters are
+relevant and should provide you with the knowledge to determine the ideal
+configuration.
+
+Repositories
+ See `Repositories / Getting Packages`_, which describes how repositories
+ can be configured.
+
+Dependency Rules
+ See `Dependency Rules`_, which explains how dependency rules work and
+ how they're written.
+
+Main Config
+ See `Configuration Reference`_ for all main config options like log file
+ rotation and assistance for writing new *dependency rules*.
+
+Field Definition
+ Refer to `Field Definition`_ in case you want to change *how* R packages
+ are being read, e.g. if you want the 'Depents' information field (obviously
+ a typo) to be understood as 'Depends'.
------------
Running it
@@ -294,7 +347,8 @@ as described in `Required configuration steps`_.
The default command is *create*, which downloads the R packages (unless
explicitly forbidden to do so) and generates the overlay. This is the
desired behavior in most cases, so simply running ``roverlay`` should be
-fine.
+fine. See `Basic Implementation Overview`_ if you'd rather like to know
+in detail what *roverlay* does before running it.
*roverlay* also accepts some **options**, most notably:
@@ -313,7 +367,7 @@ fine.
kept until all packages have been processed.
Test results show that memory consumption increases by factor 2 when using
the faster write mechanism (at ca. 95% ebuild creation success rate),
- <while ebuild write time decreases by ???>.
+ <<while ebuild write time decreases by ???>>.
Summary: Expect 300 (slow) or 600MB (fast) memory consumption when using
the default package repositories.
@@ -381,20 +435,19 @@ depres_console, depres
Providing a package mirror
----------------------------
-<No recommendations at this time. The current ManifestCreation implementation
-creates a directory *<distfiles root>/__tmp__* with symlinks to all packages,
-which could be used for providing packages, but this may change
-in near future since external Manifest creation is too slow
-(takes >60% of overlay creation time).>
-
+No recommendations available at this time.
+The current ManifestCreation implementation creates a directory
+*<distfiles root>/__tmp__* with symlinks to all packages, which could be used
+for providing packages, but this may change in near future since external
+Manifest creation is too slow (takes >60% of overlay creation time).
-=========================
- Implementation Overview
-=========================
+===============================
+ Basic Implementation Overview
+===============================
---------------------------------
- How *roverlay* basically works
---------------------------------
+----------------------
+ How *roverlay* works
+----------------------
These are the steps that *roverlay* performs:
@@ -450,163 +503,6 @@ These are the steps that *roverlay* performs:
* this uses all ebuilds availabe for a package
------------------------
- Dependency Resolution
------------------------
-
-Each ebuild creation process has access to the *dependency resolver* that
-accepts *dependency strings*, tries to resolve them and returns the result,
-either "unresolvable" or the resolving *dependency* as
-*Dynamic DEPEND*/*DEPEND Atom*.
-
-The ebuild creation uses *channels* for communication with the *dependency
-resolver*, so-called *EbuildJobChannels* that handle the 'high-level'
-string-to-string dependency resolution for a set of *dependency strings*.
-Typically, one *channel* is used per ebuild variable (DEPEND, RDEPEND and
-R_SUGGESTS).
-
-From the ebuild creation perspective, dependency resolution works like this:
-
-#. Collect the *dependency strings* from the DESCRIPTION data and add them
- to the communication *channels* (up to 3 will be used)
-
-#. Wait until all channels are *done*
-
-#. **Stop ebuild creation** if a channel reports that it couldn't resolve
- all *required dependencies*. No ebuild can be created in that case.
-
-#. **Successfully done** - transfer the channel results to ebuild variables
-
-
-Details about dependency resolution like how *channels* work can be found
-in the following sections.
-
-++++++++++++++++++
- Dependency types
-++++++++++++++++++
-
-Every *dependency string* has a *dependency type* that declares how a
-dependency should be resolved. It has one or more of these properties:
-
-Mandatory
- Ebuild creation fails if the *dependency string* in question cannot
- be resolved.
-
-Optional
- The opposite of *Mandatory*, ebuild creation keeps going even if the
- *dependency string* is unresolvable.
-
-Package Dependency
- This declares that the *dependency string* could be another R package.
-
-System Dependency
- This declares that the *dependency string* could be a system dependency,
- e.g. a scientific library or a video encoder.
-
-Try other dependency types
- This declares that the *dependency string* can be resolved by ignoring its
- dependency type partially. This property allows to resolve
- package dependencies as system dependencies and vice versa.
- Throughout this guide, such property is indicated by *<preferred dependency
- type> first*, e.g. *package first*.
-
-*Mandatory* and *Option* are mutually exclusive.
-
-The *dependency type* of a *dependency string* is determined by the its origin,
-i.e. info field in the package's DESCRIPTION file.
-The *Suggests* field, for example, gets the
-*"package dependency only and optional"* type, whereas the *SystemRequirements*
-gets *"system dependency, but try others, and mandatory"*.
-
-
-DESCRIPTION file dependency fields
-----------------------------------
-
-The DESCRIPTION file of an R package contains several fields that list
-dependencies on R packages or other software like scientific libraries.
-This section describes which *dependency fields* are used and how.
-
-.. table:: R package dependency fields
-
- +--------------------+----------------------+------------------+-----------+
- | dependency field | ebuild variable | dependency type | required |
- +====================+======================+==================+===========+
- | Depends | DEPEND | package first | *yes* |
- +--------------------+ + + +
- | Imports | | | |
- +--------------------+----------------------+------------------+ +
- | LinkingTo | RDEPEND | package first | |
- +--------------------+ +------------------+ +
- | SystemRequirements | | system first | |
- +--------------------+----------------------+------------------+-----------+
- | Suggests | R_SUGGESTS | package **only** | **no** |
- + +----------------------+------------------+-----------+
- | | _UNRESOLVED_PACKAGES | *unresolvable* | *n/a* |
- +--------------------+----------------------+------------------+-----------+
-
-A non-empty *R_SUGGESTS* ebuild variable will enable the *R_suggests* USE
-flag. *R_SUGGESTS* is a runtime dependency (a *Dynamic DEPEND* in *RDEPEND*).
-
-Ebuild creation keeps going if an optional dependency cannot be resolved.
-This isn't desirable for most *dependency fields*, but extremely
-useful for R package suggestions that often link to other repositories or
-private homepages.
-Such unresolvable dependencies go into the *_UNRESOLVED_PACKAGES* ebuild
-variable.
-Whether and how this variable is used is up to the eclass file(s).
-The default *R-packages eclass* reports unresolvable,
-but optional dependencies during the *pkg_postinst()* ebuild phase.
-
-
-+++++++++++++++++++++++
- Dependency Rule Pools
-+++++++++++++++++++++++
-
-The *dependency resolver* doesn't know *how* to resolve a *dependency string*.
-Instead, it searches through a list of *dependency rule pools* that may be
-able to do this.
-
-A *dependency rule pool* combines a list of *dependency rules* with a
-*dependency type* and is able to determine whether it accepts the type
-of a *dependency string* or not.
-
-*Dependency rules* are objects with a "matches" function that returns the
-*resolving dependency* if it matches the given *dependency string*, else
-it returns "cannot resolve". Note the difference between
-"a rule cannot resolve a dep string" and "dep string is unresolvable",
-which means that no rule can resolve a particular *dependency string*.
-
-See `Dependency Rules`_ for the concrete rules available.
-
-Rule pools are normally created by reading rule files, but some rule pools
-consist of rules that exist in memory only.
-These are called **Dynamic Rule Pools** and use runtime data like "all known
-R packages" to create rules.
-
-
-.. _Dynamic Selfdep Rule Pool:
-
-*roverlay* uses one dynamic rule pool, the **Dynamic Selfdep Rule Pool**.
-This pool contains rules for all known R packages and is able to resolve
-R package dependencies.
-By convention, it will never resolve any system dependencies.
-
-
-
-
-<
-Dependency resolution is split into several components.
-Each package *p* has zero or more dependencies,......
-
-
-Dependency Rules
- x
-
-
-Dependency Resolver
- This is the
-
-
--------------------------------------------------------------
Expected Overlay Result / Structure of the generated overlay
--------------------------------------------------------------
@@ -630,14 +526,151 @@ the *R-packages* eclass file, the result should look like:
<overlay root>/sci-R/seewave/seewave-1.5.9.ebuild
<overlay root>/sci-R/seewave/seewave-1.6.4.ebuild
+
+++++++++++++++++++++++++
+ Expected Ebuild Result
+++++++++++++++++++++++++
+
+Ebuild Template
+ .. code-block:: text
+
+ <ebuild header>
+ inherit <eclass(es)>
+
+ DESCRIPTION="<the R package's description>"
+ SRC_URI="<src uri for the R package>"
+
+ IUSE="${IUSE:-}
+ R_suggests
+ "
+ R_SUGGESTS="<R package suggestions>"
+ DEPEND="<build time dependencies for the R package>"
+ RDEPEND="${DEPEND:-}
+ <runtime dependencies>
+ R_suggests? ( ${R_SUGGESTS} )
+ "
+
+ _UNRESOLVED_PACKAGES=(<unresolvable, but optional dependencies>)
+
+ Some of the variables may be missing if they're not needed.
+
+ A really minimal ebuild would look like:
+
+ .. code-block:: text
+
+ <ebuild header>
+ inherit <eclass(es)>
+
+ SRC_URI="<src uri for the R package>"
+
+Example: seewave 1.6.4 from CRAN:
+ The default ebuild header (which cannot be changed) automatically sets
+ the ebuild's copyright year to 1999-*<current year>*.
+
+ .. code-block:: sh
+
+ # Copyright 1999-2012 Gentoo Foundation
+ # Distributed under the terms of the GNU General Public License v2
+ # $Header: $
+
+ EAPI=4
+ inherit R-packages
+
+ DESCRIPTION="Time wave analysis and graphical representation"
+ SRC_URI="http://cran.r-project.org/src/contrib/seewave_1.6.4.tar.gz"
+
+ IUSE="${IUSE:-}
+ R_suggests
+ "
+ R_SUGGESTS="sci-R/sound
+ sci-R/audio
+ "
+ DEPEND="sci-R/fftw
+ sci-R/tuneR
+ >=dev-lang/R-2.15.0
+ sci-R/rpanel
+ sci-R/rgl
+ "
+ RDEPEND="${DEPEND:-}
+ media-libs/flac
+ sci-libs/fftw
+ media-libs/libsndfile
+ R_suggests? ( ${R_SUGGESTS} )
+ "
+
+Example: MetaPCA 0.1.3 from CRAN's archive:
+ Note the shortened *DESCRIPTION* variable that points to the *metadata.xml*
+ file. This happens if the description is too long.
+
+ .. code-block:: sh
+
+ # Copyright 1999-2012 Gentoo Foundation
+ # Distributed under the terms of the GNU General Public License v2
+ # $Header: $
+
+ EAPI=4
+ inherit R-packages
+
+ DESCRIPTION="MetaPCA: Meta-analysis in the Di... (see metadata)"
+ SRC_URI="http://cran.r-project.org/src/contrib/Archive/MetaPCA/MetaPCA_0.1.3.tar.gz"
+
+ IUSE="${IUSE:-}
+ R_suggests
+ "
+ R_SUGGESTS="sci-R/doMC
+ sci-R/affy
+ sci-R/ellipse
+ sci-R/pcaPP
+ sci-R/MASS
+ sci-R/impute
+ sci-R/doSMP
+ sci-R/GEOquery
+ "
+ DEPEND="sci-R/foreach"
+ RDEPEND="${DEPEND:-}
+ R_suggests? ( ${R_SUGGESTS} )
+ "
+
+ _UNRESOLVED_PACKAGES=('hgu133plus2.db')
+
+
+++++++++++++++++++++++++++++++++
+ Expected *metadata.xml* Result
+++++++++++++++++++++++++++++++++
+
+The *metadata.xml* will contain the full description for the latest version
+of a package.
+
+Example: seewave from CRAN
+ Note the ' // ' delimiter. It will be used to separate description strings
+ if a package has more than one, e.g. one in its *Title* and one in its
+ *Description* information field.
+
+ .. code-block:: xml
+
+ <?xml version="1.0" encoding="UTF-8"?>
+ <!DOCTYPE pkgmetadata SYSTEM "http://www.gentoo.org/dtd/metadata.dtd">
+ <pkgmetadata>
+ <longdescription>
+ Time wave analysis and graphical representation // seewave
+ provides functions for analysing, manipulating, displaying,
+ editing and synthesizing time waves (particularly sound). This
+ package processes time analysis (oscillograms and envelopes),
+ spectral content, resonance quality factor, entropy, cross
+ correlation and autocorrelation, zero-crossing, dominant
+ frequency, analytic signal, frequency coherence, 2D and 3D
+ spectrograms and many other analyses.
+ </longdescription>
+ </pkgmetadata>
+
=================================
Repositories / Getting Packages
=================================
*roverlay* is capable of downloading R packages via rsync and http,
-and is able to use any packages locally available. The concrete method used
-to get and use the packages is determined by the concrete
-**type of a repository** and that's what this section is about.
+and is able to use any packages locally available. The method used to get and
+use the packages is determined by the concrete **type of a repository** and
+that's what this section is about. It also covers repository configuration.
.. _repo config:
@@ -661,19 +694,22 @@ Such options are declared with ``<option> = <value>`` in the repo entry.
The common options for repository entries are:
-* *type*, which declares the repository type. Available types are:
+* *type* (optional), which declares the repository type.
+ Available types are:
- * rsync_
- * websync_repo_
- * websync_pkglist_
- * local_
+ * rsync_
+ * websync_repo_
+ * websync_pkglist_
+ * local_
- Defaults to *rsync*
+ Defaults to *rsync*.
-* *src_uri*, which declares how ebuilds can download the packages. Some repo
- types use this for downloading, too.
+* *src_uri* (**required**),
+ which declares how ebuilds can download the packages.
+ Some repo types use this for downloading, too.
-* *directory*, which explicitly sets the package directory to use.
+* *directory* (optional),
+ which explicitly sets the package directory to use.
The default behavior is to use `DISTFILES_ROOT`_/<repo name>
@@ -688,7 +724,7 @@ The common options for repository entries are:
-------------
Runs *rsync* to download packages. Automatic sync retries are supported if
-*rsync*'s exit codes indicates chances of success.
+*rsync*'s exit code indicates chances of success.
For example, up to 3 retries are granted if *rsync* returns
*Partial transfer due to vanished source files* which likely happens when
syncing big repositories like CRAN.
@@ -812,7 +848,7 @@ one package uri per line:
http://www.omegahat.org/R/src/contrib/Aspell_0.2-0.tar.gz
...
-Comments are not supported. Assuming that such a package list exists as <at?>
+Comments are not supported. Assuming that such a package list exists at
*~/roverlay/config/http_packages.list*, an example entry in the repo config
file would be:
@@ -869,8 +905,8 @@ A local directory will never be modified.
*Simple dependency rules* use a dictionary and string transformations
to resolve dependencies. *Fuzzy simple dependency rules* extend these by
-a set of regexes, which allows to resolve many dependency strings that
-minimally differ (e.g. only in the required version and/or comments:
+a set of regular expressions, which allows to resolve many dependency strings
+that minimally differ (e.g. only in the required version and/or comments:
`R (>= 2.10)` and `R [2.14] from http://www.r-project.org/`) with a single
dictionary entry.
@@ -932,7 +968,7 @@ Fuzzy Rules
++++++++++++++++++++
This sections lists some rule file examples.
-See `Rule File Syntax`_ for a formal<precise,..?> description.
+See `Rule File Syntax`_ for a formal description.
Example 1 - *default* fuzzy rule
@@ -958,7 +994,7 @@ Example 2 - *default* simple rule stub
R Package rules are dynamically created at runtime and therefore not
needed. Write them only if certain R package dependencies cannot
- be resolved.
+ be resolved. See *Selfdep* in `Rule File Syntax`_ for details.
Example 3 - *default* simple rule
A rule that matches several *dependency strings* and resolves them
@@ -1067,7 +1103,7 @@ Single line rules
Multi line rules
resolve several *dependency strings*.
Their rule block begins with '{' + newline, followed by one
- *dependency string* per line, and ends with '}'.
+ *dependency string* per line, and ends with '}' in a separate line.
Syntax:
.. code-block:: text
@@ -1078,34 +1114,46 @@ Multi line rules
...
}
+ .. Note::
+
+ Technically, this rule syntax can be used to specify rules with
+ zero or more *dependency strings*. An empty rule doesn't make much sense,
+ though.
+
+Comments
+ start with **#**. There are a few exceptions to that, the *#deptype* and
+ *#! NOPARSE* keywords. Comments inside rule blocks are not allowed and
+ will be read as normal *dependency strings*.
+
Selfdep
- This is another name for *dependency strings* that are resolved by an
- R package with the same name, which is also part of the overlay being
- created.
+ This is another name for *dependency strings* that are resolved by an
+ R package with the same name, which is also part of the overlay being
+ created.
- Example: *zoo* is resolved as *sci-R/zoo* (if OVERLAY_CATEGORY is *sci-R*)
+ Example: *zoo* is resolved as *sci-R/zoo*, assuming that `OVERLAY_CATEGORY`_
+ is set to *sci-R*
- Writing selfdep rules is not necessary since *roverlay* automatically
- creates rules for all R known packages (see `Dynamic Selfdep Rule Pool`_
- for details).
+ Writing selfdep rules is not necessary since *roverlay* automatically
+ creates rules for all known R packages (see `Dynamic Selfdep Rule Pool`_
+ for details).
- There are a few exceptions to that in which case selfdep rules have to
- be written:
+ There are a few exceptions to that in which case selfdep rules have to
+ be written:
- * The *dependency string* is assumed to be a system dependency (not an
- R package). This is likely a "bug" in the DESCRIPTION file of the
- R package being processed.
+ * The *dependency string* is assumed to be a system dependency (not an
+ R package). This is likely a "bug" in the DESCRIPTION file of the
+ R package being processed.
- * The R package name is not ebuild-name compliant (e.g. contains the '.'
- char, which is remapped to '_'.).
- Most *char remap* cases are handled properly, but there may be a few
- exceptions.
+ * The R package name is not ebuild-name compliant (e.g. contains the '.'
+ char, which is remapped to '_'.).
+ Most *char remap* cases are handled properly, but there may be a few
+ exceptions.
- .. Caution::
+ .. Caution::
- Writing unnecessary selfdep rules slows dependency resolution down!
- Each rule will exist twice, once as *dynamic* rule and once as
- the written one.
+ Writing unnecessary selfdep rules slows dependency resolution down!
+ Each rule will exist twice, once as *dynamic* rule and once as
+ the written one.
Rule Stubs
@@ -1120,178 +1168,35 @@ Rule Stubs
[<keychar>]<short dependency>
-Comments
- start with **#**. There are a few exceptions to that, the *#deptype* and
- *#! NOPARSE* keywords. Comments inside rule blocks are not allowed and
- will be read as normal *dependency strings*.
+=========================
+ Configuration Reference
+=========================
-================
- DepRes Console
-================
+The main config file uses '<option> = <value>' syntax, comment lines start
+with **#**. Variable substitution ("${X}") is not supported. Quotes around
+the value are optional and allow to span long values over multiple lines.
+Whitespace is ignored, file **paths must not contain whitespace**.
-As previously stated, the *DepRes Console* is only meant for **testing**.
-It's an interactive console with the following features:
+Some options have value type restrictions. These *value types* are used:
-* resolve dependencies
-* create new dependency rules (**only single line rules**)
-* create dependency rules for each R package found in a directory
-* load rules from files
-* save rules to a file
+log_level
+ Value has to be a log level. Available choise are *DEBUG*, *INFO*, *WARN*,
+ *WARNING*, *ERROR* and *CRITICAL*.
-Rules are managed in a set. These so-called *rule pools* are organized in
-a *first-in-first-out* data structure that allows
-to create or remove them easily at runtime.
+bool
+ Value is a string that represents a boolean value.
-Running ``roverlay depres_console`` will print a welcome message that
-lists all available commands and a few usage hints.
+ This table illustrates which value strings are accepted:
-For reference, these commands are currently available:
-
-+---------------------+----------------------------------------------------+
-| command | description |
-+=====================+====================================================+
-| help, | lists all commands |
-| h | |
-+---------------------+----------------------------------------------------+
-| help --list, | lists all help topics for which help is available |
-| h --list | |
-+---------------------+----------------------------------------------------+
-| help *<cmd>*, | prints a command-specific help message |
-| h *<cmd>* | |
-+---------------------+----------------------------------------------------+
-| load *<file|dir>*, | loads a rule file or a directory with rule files |
-| l *<file|dir>* | into a new *rule pool* |
-+---------------------+----------------------------------------------------+
-| load_conf, | loads the rule files listed in the config file |
-| lc | into a new *rule pool* |
-+---------------------+----------------------------------------------------+
-| addrule *<rule>* | creates a new rule and adds it to the topmost, |
-| + *<rule>* | i.e. latest *rule pool*. This command uses |
-| | `Rule File Syntax`_, but multi line rules are |
-| | not supported. |
-+---------------------+----------------------------------------------------+
-| add_pool, | creates a new *rule pool* |
-| << | |
-+---------------------+----------------------------------------------------+
-| unwind, | removes the topmost *rule pool* and all of its |
-| >> | rules |
-+---------------------+----------------------------------------------------+
-| resolve *<dep>*, | tries to resolve the given dependency string and |
-| ? *<dep>* | prints the result |
-+---------------------+----------------------------------------------------+
-| print, p | prints the rules of the topmost *rule pool* |
-+---------------------+----------------------------------------------------+
-| print all, p all | prints the rules of all *rule pools* |
-+---------------------+----------------------------------------------------+
-| write *<file>*, | writes the rules of the topmost *rule pool* into |
-| w *<file>* | *<file>* |
-+---------------------+----------------------------------------------------+
-| cd *<dir>* | changes the working directory, also creates it if |
-| | necessary |
-+---------------------+----------------------------------------------------+
-| scandir *<dir>*, | creates dependency rules for each R package found |
-| sd *<dir>* | in *<dir>* |
-+---------------------+----------------------------------------------------+
-| set, unset | prints the status of currently (in)active modes |
-+---------------------+----------------------------------------------------+
-| set *<mode>*, | set or unsets *<mode>*. There's only one mode to |
-| unset *<mode>* | control, the *shlex mode* which controls how |
-| | command arguments are parsed |
-+---------------------+----------------------------------------------------+
-| mkhelp | verifies that each accessible command has a help |
-| | message |
-+---------------------+----------------------------------------------------+
-| exit, qq, q | exit the *DepRes Console* |
-+---------------------+----------------------------------------------------+
-
-
-
-Example Session:
- .. code-block::
-
- == depres console ==
- Run 'help' to list all known commands
- More specifically, 'help <cmd>' prints a help message for the given
- command, and 'help --list' lists all help topics available
- Use 'load_conf' or 'lc' to load the configured rule files
-
- commands: load, unwind, set, help, >>, load_conf, <<, cd, mkhelp,
- resolve, lc, add_pool, addrule, h, +, l, li, write, p, r, ?, w, print,
- sd, unset, scandir
- exit-commands: q, qq, exit
-
- cmd % + ~dev-lang/R :: R language
- new rules:
- ~dev-lang/R :: R language
- --- ---
- command succeeded.
-
- cmd % ? R language
- Trying to resolve ('R language',).
- Resolved as: ('dev-lang/R',)
-
- cmd % ? R language [ 2.15 ]
- Trying to resolve ('R language [ 2.15 ]',).
- Resolved as: ('>=dev-lang/R-2.15',)
-
- cmd % ? R
- Trying to resolve ('R',).
- Channel returned None. At least one dep couldn't be resolved.
-
- cmd % p
- ~dev-lang/R :: R language
-
- cmd % >>
- Pool removed from resolver.
-
- cmd % p
-
- cmd % ? R language
- Trying to resolve ('R language',).
- Channel returned None. At least one dep couldn't be resolved.
-
- cmd % exit
-
-
-=========================
- Configuration Reference
-=========================
-
---------------
- Repositories
---------------
-
-<merge with basic..overview::repo>
-
--------------
-Main Config
--------------
-
-The main config file uses "<option> = <value>" syntax, comment lines start
-with **#**. Variable substitution ("${X}") is not supported. Quotes around
-the value are optional and allow to span long values over multiple lines.
-Whitespace is ignored, file **paths must not contain whitespace**.
-
-Some options have value type restrictions. These *value types* are used:
-
-log_level
- Value has to be a log level. Available choise are *DEBUG*, *INFO*, *WARN*,
- *WARNING*, *ERROR* and *CRITICAL*.
-
-bool
- Value is a string that represents a boolean value.
-
- This table illustrates which values strings are accepted:
-
- +--------------------------------+----------------------+
- | string value | boolean value |
- +================================+======================+
- | y, yes, on, 1, true, enabled | *True* |
- +--------------------------------+----------------------+
- | n, no, off, 0, false, disabled | *False* |
- +--------------------------------+----------------------+
- | *<any other value>* | **not allowed** |
- +--------------------------------+----------------------+
+ +--------------------------------+----------------------+
+ | string value | boolean value |
+ +================================+======================+
+ | y, yes, on, 1, true, enabled | *True* |
+ +--------------------------------+----------------------+
+ | n, no, off, 0, false, disabled | *False* |
+ +--------------------------------+----------------------+
+ | *<any other value>* | **not allowed** |
+ +--------------------------------+----------------------+
There are also some implicit *value types*:
@@ -1313,9 +1218,9 @@ file, dir
The following sections will list all config entries.
-++++++++++++++
+--------------
misc options
-++++++++++++++
+--------------
.. _DISTFILES:
@@ -1337,9 +1242,9 @@ DISTROOT
Alias for DISTFILES_ROOT_.
-+++++++++++++++++
+-----------------
overlay options
-+++++++++++++++++
+-----------------
.. _ECLASS:
@@ -1391,9 +1296,9 @@ OVERLAY_NAME
Defaults to *R_Overlay*.
-++++++++++++++++++++
+--------------------
other config files
-++++++++++++++++++++
+--------------------
Some config config options are split from the main config file for various
reasons:
@@ -1449,18 +1354,16 @@ SIMPLE_RULES_FILE
SIMPLE_RULES_FILES
Alias to SIMPLE_RULES_FILE_.
-+++++++++
+---------
logging
-+++++++++
+---------
.. _LOG_DATE_FORMAT:
LOG_DATE_FORMAT
- The date format used in log messages.
+ The date format (ISO8601) used in log messages.
Defaults to *%F %H:%M:%S*.
- (<<explain the date format by referencing to date(1)
- or python's logging->? module>>)
.. _LOG_ENABLED:
@@ -1481,9 +1384,9 @@ LOG_LEVEL
Defaults to <not set> - all log targets will use their own defaults
-
-console logging
----------------
++++++++++++++++++
+ console logging
++++++++++++++++++
.. _LOG_CONSOLE:
@@ -1506,9 +1409,9 @@ LOG_LEVEL_CONSOLE
Defaults to *INFO*.
-
-file logging
-------------
+++++++++++++++
+ file logging
+++++++++++++++
.. _LOG_FILE:
@@ -1573,9 +1476,9 @@ LOG_FILE_ROTATE_COUNT
Defaults to *3* and has no effect if LOG_FILE_ROTATE_ is disabled.
-++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+--------------------------------------------------------------------
options for debugging, manual dependency rule creation and testing
-++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+--------------------------------------------------------------------
.. _DESCRIPTION_DIR:
@@ -1611,12 +1514,16 @@ RSYNC_BWLIMIT
Defaults to <not set>, which disables bandwidth limitation.
-------------------
- Field Definition
-------------------
-The field definition file uses ConfigParser_ syntax. For an example, see
-`default field definition file`_.
+.. _Field Definition:
+
+=========================
+ Field Definition Config
+=========================
+
+The field definition file uses ConfigParser_ syntax and defines
+how an R package's DESCRIPTION file is read.
+See the next section, `default field definition file`_, for an example.
Each information field has its own section which declares a set of options
and flags. Flags are case-insensivitve options
@@ -1681,8 +1588,8 @@ Known field flags:
.. _field flag\: isList:
isList
- Declares that a field's value is a list whose values are separated by
- by ',' or ';'.
+ Declares that a field's value is a list whose values are separated
+ by ',' and/or ';'.
.. _field flag\: isWhitespaceList:
@@ -1706,27 +1613,19 @@ Known field flags:
ignore
Declares that a field is known but entirely ignored. Unknown fields
- are ignored, too, the main difference is the log message.
+ are ignored, too, the main difference is the emitted log message if
+ such a field is found.
.. Note::
It won't be checked whether a flag is known or not.
-==========
- Appendix
-==========
-
------------------
- ebuild examples
------------------
-<required? this section would contain a minimal (DESCRIPTION, SRC_URI)
-and a 'bloated' (all vars, +DEPEND,RDEPEND,IUSE,R_SUGGEST,_UNRESOLVED_PACKAGES)
-ebuild>
+.. _default field definition file:
--------------------------------
- Default Field Definition File
--------------------------------
+--------------------------------------------
+ Example: The default field definition file
+--------------------------------------------
This is the default field definition file (without any ignored fields):
@@ -1761,3 +1660,343 @@ This is the default field definition file (without any ignored fields):
[OS_Type]
alias_nocase = OS_TYPE
allowed_values = unix
+
+
+
+.. _DepRes Console:
+
+===============================
+ Dependency Resolution Console
+===============================
+
+As previously stated, the *DepRes Console* is only meant for **testing**.
+It's an interactive console with the following features:
+
+* resolve dependencies
+* create new dependency rules (**only single line rules**)
+* create dependency rules for each R package found in a directory
+* load rules from files
+* save rules to a file
+
+Rules are managed in a set. These so-called *rule pools* are organized in
+a *first-in-first-out* data structure that allows
+to create or remove them easily at runtime.
+
+Running ``roverlay depres_console`` will print a welcome message that
+lists all available commands and a few usage hints.
+
+For reference, these commands are currently available:
+
++---------------------+----------------------------------------------------+
+| command | description |
++=====================+====================================================+
+| help, | lists all commands |
+| h | |
++---------------------+----------------------------------------------------+
+| help --list, | lists all help topics for which help is available |
+| h --list | |
++---------------------+----------------------------------------------------+
+| help *<cmd>*, | prints a command-specific help message |
+| h *<cmd>* | |
++---------------------+----------------------------------------------------+
+| load *<file|dir>*, | loads a rule file or a directory with rule files |
+| l *<file|dir>* | into a new *rule pool* |
++---------------------+----------------------------------------------------+
+| load_conf, | loads the rule files listed in the config file |
+| lc | into a new *rule pool* |
++---------------------+----------------------------------------------------+
+| addrule *<rule>* | creates a new rule and adds it to the topmost, |
+| + *<rule>* | i.e. latest *rule pool*. This command uses |
+| | `Rule File Syntax`_, but multi line rules are |
+| | not supported. |
++---------------------+----------------------------------------------------+
+| add_pool, | creates a new *rule pool* |
+| << | |
++---------------------+----------------------------------------------------+
+| unwind, | removes the topmost *rule pool* and all of its |
+| >> | rules |
++---------------------+----------------------------------------------------+
+| resolve *<dep>*, | tries to resolve the given dependency string and |
+| ? *<dep>* | prints the result |
++---------------------+----------------------------------------------------+
+| print, p | prints the rules of the topmost *rule pool* |
++---------------------+----------------------------------------------------+
+| print all, p all | prints the rules of all *rule pools* |
++---------------------+----------------------------------------------------+
+| write *<file>*, | writes the rules of the topmost *rule pool* into |
+| w *<file>* | *<file>* |
++---------------------+----------------------------------------------------+
+| cd *<dir>* | changes the working directory, also creates it if |
+| | necessary |
++---------------------+----------------------------------------------------+
+| scandir *<dir>*, | creates dependency rules for each R package found |
+| sd *<dir>* | in *<dir>* |
++---------------------+----------------------------------------------------+
+| set, unset | prints the status of currently (in)active modes |
++---------------------+----------------------------------------------------+
+| set *<mode>*, | sets or unsets *<mode>*. There's only one mode to |
+| unset *<mode>* | control, the *shlex mode* which controls how |
+| | command arguments are parsed |
++---------------------+----------------------------------------------------+
+| mkhelp | verifies that each accessible command has a help |
+| | message |
++---------------------+----------------------------------------------------+
+| exit, qq, q | exits the *DepRes Console* |
++---------------------+----------------------------------------------------+
+
+
+
+Example Session:
+ .. code-block::
+
+ == depres console ==
+ Run 'help' to list all known commands
+ More specifically, 'help <cmd>' prints a help message for the given
+ command, and 'help --list' lists all help topics available
+ Use 'load_conf' or 'lc' to load the configured rule files
+
+ commands: load, unwind, set, help, >>, load_conf, <<, cd, mkhelp,
+ resolve, lc, add_pool, addrule, h, +, l, li, write, p, r, ?, w, print,
+ sd, unset, scandir
+ exit-commands: q, qq, exit
+
+ cmd % + ~dev-lang/R :: R language
+ new rules:
+ ~dev-lang/R :: R language
+ --- ---
+ command succeeded.
+
+ cmd % ? R language
+ Trying to resolve ('R language',).
+ Resolved as: ('dev-lang/R',)
+
+ cmd % ? R language [ 2.15 ]
+ Trying to resolve ('R language [ 2.15 ]',).
+ Resolved as: ('>=dev-lang/R-2.15',)
+
+ cmd % ? R
+ Trying to resolve ('R',).
+ Channel returned None. At least one dep couldn't be resolved.
+
+ cmd % p
+ ~dev-lang/R :: R language
+
+ cmd % >>
+ Pool removed from resolver.
+
+ cmd % p
+
+ cmd % ? R language
+ Trying to resolve ('R language',).
+ Channel returned None. At least one dep couldn't be resolved.
+
+ cmd % exit
+
+
+=========================
+ Implementation Overview
+=========================
+
+-----------------------
+ Dependency Resolution
+-----------------------
+
+Each ebuild creation process has access to the *dependency resolver* that
+accepts *dependency strings*, tries to resolve them and returns the result,
+either "unresolvable" or the resolving *dependency* as
+*Dynamic DEPEND*/*DEPEND Atom*.
+
+The ebuild creation uses *channels* for communication with the *dependency
+resolver*, so-called *EbuildJobChannels* that handle the 'high-level'
+string-to-string dependency resolution for a set of *dependency strings*.
+Typically, one *channel* is used per ebuild variable (DEPEND, RDEPEND and
+R_SUGGESTS).
+
+From the ebuild creation perspective, dependency resolution works like this:
+
+#. Collect the *dependency strings* from the DESCRIPTION data and add them
+ to the communication *channels* (up to 3 will be used)
+
+#. Wait until all channels are *done*
+
+#. **Stop ebuild creation** if a channel reports that it couldn't resolve
+ all *required dependencies*. No ebuild can be created in that case.
+
+#. **Successfully done** - transfer the channel results to ebuild variables
+
+
+Details about dependency resolution like how *channels* work can be found
+in the following sections.
+
+++++++++++++++++++
+ Dependency types
+++++++++++++++++++
+
+Every *dependency string* has a *dependency type* that declares how a
+dependency should be resolved. It has one or more of these properties:
+
+Mandatory
+ Ebuild creation fails if the *dependency string* in question cannot
+ be resolved.
+
+Optional
+ The opposite of *Mandatory*, ebuild creation keeps going even if the
+ *dependency string* is unresolvable.
+
+Package Dependency
+ This declares that the *dependency string* could be another R package.
+
+System Dependency
+ This declares that the *dependency string* could be a system dependency,
+ e.g. a scientific library or a video encoder.
+
+Try other dependency types
+ This declares that the *dependency string* can be resolved by ignoring its
+ dependency type partially. This property allows to resolve
+ package dependencies as system dependencies and vice versa.
+ Throughout this guide, such property is indicated by *<preferred dependency
+ type> first*, e.g. *package first*.
+
+*Mandatory* and *Option* are mutually exclusive.
+
+The *dependency type* of a *dependency string* is determined by its origin,
+i.e. info field in the package's DESCRIPTION file.
+The *Suggests* field, for example, gets the
+*"package dependency only and optional"* type, whereas the *SystemRequirements*
+gets *"system dependency, but try others, and mandatory"*.
+
+
+DESCRIPTION file dependency fields
+----------------------------------
+
+The DESCRIPTION file of an R package contains several fields that list
+dependencies on R packages or other software like scientific libraries.
+This section describes which *dependency fields* are used and how.
+
+.. table:: R package dependency fields
+
+ +--------------------+----------------------+------------------+-----------+
+ | dependency field | ebuild variable | dependency type | required |
+ +====================+======================+==================+===========+
+ | Depends | DEPEND | package first | *yes* |
+ +--------------------+ + + +
+ | Imports | | | |
+ +--------------------+----------------------+------------------+ +
+ | LinkingTo | RDEPEND | package first | |
+ +--------------------+ +------------------+ +
+ | SystemRequirements | | system first | |
+ +--------------------+----------------------+------------------+-----------+
+ | Suggests | R_SUGGESTS | package **only** | **no** |
+ + +----------------------+------------------+-----------+
+ | | _UNRESOLVED_PACKAGES | *unresolvable* | *n/a* |
+ +--------------------+----------------------+------------------+-----------+
+
+A non-empty *R_SUGGESTS* ebuild variable will enable the *R_suggests* USE
+flag. *R_SUGGESTS* is a runtime dependency (a *Dynamic DEPEND* in *RDEPEND*).
+
+Ebuild creation keeps going if an optional dependency cannot be resolved.
+This isn't desirable for most *dependency fields*, but extremely
+useful for R package suggestions that often link to other repositories or
+private homepages.
+Such unresolvable dependencies go into the *_UNRESOLVED_PACKAGES* ebuild
+variable.
+Whether and how this variable is used is up to the eclass file(s).
+The default *R-packages eclass* reports unresolvable,
+but optional dependencies during the *pkg_postinst()* ebuild phase.
+
+
++++++++++++++++++++++++
+ Dependency Rule Pools
++++++++++++++++++++++++
+
+The *dependency resolver* doesn't know *how* to resolve a *dependency string*.
+Instead, it searches through a list of *dependency rule pools* that may be
+able to do this.
+
+A *dependency rule pool* combines a list of *dependency rules* with a
+*dependency type* and is able to determine whether it accepts the type
+of a *dependency string* or not.
+
+*Dependency rules* are objects with a "matches" function that returns the
+*resolving dependency* if it matches the given *dependency string*, else
+it returns "cannot resolve". Note the difference between
+"a rule cannot resolve a dep string" and "dep string is unresolvable",
+which means that no rule can resolve a particular *dependency string*.
+
+See `Dependency Rules`_ for the concrete rules available.
+
+Rule pools are normally created by reading rule files, but some rule pools
+consist of rules that exist in memory only.
+These are called **Dynamic Rule Pools** and use runtime data like "all known
+R packages" to create rules.
+
+
+.. _Dynamic Selfdep Rule Pool:
+
+*roverlay* uses one dynamic rule pool, the **Dynamic Selfdep Rule Pool**.
+This pool contains rules for all known R packages and is able to resolve
+R package dependencies.
+By convention, it will never resolve any system dependencies.
+
+
+
+
+<<
+Dependency resolution is split into several components.
+Each package *p* has zero or more dependencies,......
+
+
+Dependency Rules
+ x
+
+
+Dependency Resolver
+ This is the
+
+>>
+
+<<
+EbuildJobChannel "work flow"
+
+1. accept *dependency strings* and create *dependency environments* for them
+ until the ebuild creation signalizes that all dep strs have been added
+ (by calling *<channel>.satisfy_request()*)
+
+2. tell the *dependency resolver* that this channel is waiting for results
+
+3. wait for a (partial) result from the resolver
+
+ * **stop waiting** if a *required* dependency could not be resolved.
+ The channel returns "unresolvable" to the ebuild creation and
+ closes afterwards.
+
+ * add unresolvable, but optional deps to the list of unresolvable deps
+ * add resolved deps to the list of resolved deps
+
+4. repeat 3. until all dependencies have been processed (or one req dep could
+ not be resolved)
+
+5. return the resolved and unresolvable deps to the ebuild creation
+
+>>
+
+
+<<
+Dependency Resolver "work flow"
+
+threaded executation
+
+.. code-block:: text
+
+ LOOP UNTIL RESOLVER CLOSED:
+
+ wait for a dependency environment d_e
+
+ search in all rule pools
+ if rule pool matches d_e
+ d_e is resolved - tell d_e the resolving dependency and send
+ d_e back to its channel
+
+ d_e is unresolvable - send it back to its channel
+
+>>
^ permalink raw reply related [flat|nested] 41+ messages in thread* [gentoo-commits] proj/R_overlay:master commit in: doc/rst/
@ 2012-08-14 15:24 André Erdmann
0 siblings, 0 replies; 41+ messages in thread
From: André Erdmann @ 2012-08-14 15:24 UTC (permalink / raw
To: gentoo-commits
commit: c46dfb64a22330de7846e4fcc719a59123b1839f
Author: André Erdmann <dywi <AT> mailerd <DOT> de>
AuthorDate: Tue Aug 14 15:24:34 2012 +0000
Commit: André Erdmann <dywi <AT> mailerd <DOT> de>
CommitDate: Tue Aug 14 15:24:34 2012 +0000
URL: http://git.overlays.gentoo.org/gitweb/?p=proj/R_overlay.git;a=commit;h=c46dfb64
usage.rst: dependency resolution
---
doc/rst/usage.rst | 380 +++++++++++++++++++++++++++++++++++++++--------------
1 files changed, 278 insertions(+), 102 deletions(-)
diff --git a/doc/rst/usage.rst b/doc/rst/usage.rst
index 3099e83..4f77f44 100644
--- a/doc/rst/usage.rst
+++ b/doc/rst/usage.rst
@@ -1,3 +1,8 @@
+.. |date| date:: %b %d %Y
+
+.. header:: Automatically Generated Overlay of R package - Manual (|date|)
+
+
.. _roverlay-9999.ebuild:
http://git.overlays.gentoo.org/gitweb/?p=proj/R_overlay.git;a=blob;f=roverlay-9999.ebuild;hb=refs/heads/master
@@ -13,6 +18,8 @@
.. sectnum::
.. contents::
+ :backlinks: top
+
==============
@@ -22,9 +29,12 @@
*roverlay* is
Automatically Generated Overlay of R packages;;
GSoC Project;;
+`roverlay git repo`_ (no project homepage available);;
<>;;
+
+
==============
Installation
==============
@@ -45,7 +55,7 @@ GSoC Project;;
* *true* or *echo* from coreutils or busybox for preventing
package downloads during Manifest creation (optional)
-* for generating documentation files: python docutils > 0.8.1
+* for generating documentation files: python docutils >= 0.9
* hardware requirements (when the default configuration):
@@ -90,7 +100,7 @@ install *roverlay* and its python modules:
``make install`` also accepts some variables, namely:
-* *DESTDIR* defaults to ''
+* *DESTDIR*
* *BINDIR*, defaults to *DESTDIR*/usr/local/bin
@@ -200,7 +210,7 @@ The following options should be set before running *roverlay*:
Example: LOG_LEVEL = WARNING
- .. caution::
+ .. Note::
Be careful with low log levels, especially *DEBUG*.
They produce a lot of messages that you probably don't want to see
@@ -222,7 +232,7 @@ have reasonable defaults if *roverlay* has been installed using *emerge*:
SIMPLE_RULES_FILE
This option lists dependency rule files and/or directories with
such files that should be used for dependency resolution (see
- `Dependency Rules / Resolving Dependencies`_).
+ `Dependency Rules`).
Although not required, this option is **recommended** since ebuild
creation without dependency rules fails for most R packages.
@@ -382,9 +392,11 @@ in near future since external Manifest creation is too slow
Implementation Overview
=========================
-This section gives a basic overview of how *roverlay* works.
+--------------------------------
+ How *roverlay* basically works
+--------------------------------
-<how *roverlay* basically works:>
+These are the steps that *roverlay* performs:
1. **sync** - get R packages using various methods
(rsync, http, local directory)
@@ -414,6 +426,8 @@ This section gives a basic overview of how *roverlay* works.
* **immediately stop** processing *p* if a *required* dependency
cannot be resolved in which case a valid ebuild cannot be created
+ See also: `Dependency Resolution`_
+
* create an ebuild for *p* by using the dependency resolution results
and a few information fields like 'Description'
@@ -435,9 +449,190 @@ This section gives a basic overview of how *roverlay* works.
* this uses all ebuilds availabe for a package
----------------------------------
+
+-----------------------
+ Dependency Resolution
+-----------------------
+
+Each ebuild creation process has access to the *dependency resolver* that
+accepts *dependency strings*, tries to resolve them and returns the result,
+either "unresolvable" or the resolving *dependency* as
+*Dynamic DEPEND*/*DEPEND Atom*.
+
+The ebuild creation uses *channels* for communication with the *dependency
+resolver*, so-called *EbuildJobChannels* that handle the 'high-level'
+string-to-string dependency resolution for a set of *dependency strings*.
+Typically, one *channel* is used per ebuild variable (DEPEND, RDEPEND and
+R_SUGGESTS).
+
+From the ebuild creation perspective, dependency resolution works like this:
+
+#. Collect the *dependency strings* from the DESCRIPTION data and add them
+ to the communication *channels* (up to 3 will be used)
+
+#. Wait until all channels are *done*
+
+#. **Stop ebuild creation** if a channel reports that it couldn't resolve
+ all *required dependencies*. No ebuild can be created in that case.
+
+#. **Successfully done** - transfer the channel results to ebuild variables
+
+
+Details about dependency resolution like how *channels* work can be found
+in the following sections.
+
+++++++++++++++++++
+ Dependency types
+++++++++++++++++++
+
+Every *dependency string* has a *dependency type* that declares how a
+dependency should be resolved. It has one or more of these properties:
+
+Mandatory
+ Ebuild creation fails if the *dependency string* in question cannot
+ be resolved.
+
+Optional
+ The opposite of *Mandatory*, ebuild creation keeps going even if the
+ *dependency string* is unresolvable.
+
+Package Dependency
+ This declares that the *dependency string* could be another R package.
+
+System Dependency
+ This declares that the *dependency string* could be a system dependency,
+ e.g. a scientific library or a video encoder.
+
+Try other dependency types
+ This declares that the *dependency string* can be resolved by ignoring its
+ dependency type partially. This property allows to resolve
+ package dependencies as system dependencies and vice versa.
+ Throughout this guide, such property is indicated by *<preferred dependency
+ type> first*, e.g. *package first*.
+
+*Mandatory* and *Option* are mutually exclusive.
+
+The *dependency type* of a *dependency string* is determined by the its origin,
+i.e. info field in the package's DESCRIPTION file.
+The *Suggests* field, for example, gets the
+*"package dependency only and optional"* type, whereas the *SystemRequirements*
+gets *"system dependency, but try others, and mandatory"*.
+
+
+DESCRIPTION file dependency fields
+----------------------------------
+
+The DESCRIPTION file of an R package contains several fields that list
+dependencies on R packages or other software like scientific libraries.
+This section describes which *dependency fields* are used and how.
+
+.. table:: R package dependency fields
+
+ +--------------------+----------------------+------------------+-----------+
+ | dependency field | ebuild variable | dependency type | required |
+ +====================+======================+==================+===========+
+ | Depends | DEPEND | package first | *yes* |
+ +--------------------+ + + +
+ | Imports | | | |
+ +--------------------+----------------------+------------------+ +
+ | LinkingTo | RDEPEND | package first | |
+ +--------------------+ +------------------+ +
+ | SystemRequirements | | system first | |
+ +--------------------+----------------------+------------------+-----------+
+ | Suggests | R_SUGGESTS | package **only** | **no** |
+ + +----------------------+------------------+-----------+
+ | | _UNRESOLVED_PACKAGES | *unresolvable* | *n/a* |
+ +--------------------+----------------------+------------------+-----------+
+
+A non-empty *R_SUGGESTS* ebuild variable will enable the *R_suggests* USE
+flag. *R_SUGGESTS* is a runtime dependency (a *Dynamic DEPEND* in *RDEPEND*).
+
+Ebuild creation keeps going if an optional dependency cannot be resolved.
+This isn't desirable for most *dependency fields*, but extremely
+useful for R package suggestions that often link to other repositories or
+private homepages.
+Such unresolvable dependencies go into the *_UNRESOLVED_PACKAGES* ebuild
+variable.
+Whether and how this variable is used is up to the eclass file(s).
+The default *R-packages eclass* reports unresolvable,
+but optional dependencies during the *pkg_postinst()* ebuild phase.
+
+
++++++++++++++++++++++++
+ Dependency Rule Pools
++++++++++++++++++++++++
+
+The *dependency resolver* doesn't know *how* to resolve a *dependency string*.
+Instead, it searches through a list of *dependency rule pools* that may be
+able to do this.
+
+A *dependency rule pool* combines a list of *dependency rules* with a
+*dependency type* and is able to determine whether it accepts the type
+of a *dependency string* or not.
+
+*Dependency rules* are objects with a "matches" function that returns the
+*resolving dependency* if it matches the given *dependency string*, else
+it returns "cannot resolve". Note the difference between
+"a rule cannot resolve a dep string" and "dep string is unresolvable",
+which means that no rule can resolve a particular *dependency string*.
+
+See `Dependency Rules`_ for the concrete rules available.
+
+Rule pools are normally created by reading rule files, but some rule pools
+consist of rules that exist in memory only.
+These are called **Dynamic Rule Pools** and use runtime data like "all known
+R packages" to create rules.
+
+
+.. _Dynamic Selfdep Rule Pool:
+
+*roverlay* uses one dynamic rule pool, the **Dynamic Selfdep Rule Pool**.
+This pool contains rules for all known R packages and is able to resolve
+R package dependencies.
+By convention, it will never resolve any system dependencies.
+
+
+
+
+<
+Dependency resolution is split into several components.
+Each package *p* has zero or more dependencies,......
+
+
+Dependency Rules
+ x
+
+
+Dependency Resolver
+ This is the
+
+
+--------------------------------------------------------------
+ Expected Overlay Result / Structure of the generated overlay
+--------------------------------------------------------------
+
+Assuming that you're using the default configuration (where possible) and
+the *R-packages* eclass file, the result should look like:
+
+.. code-block:: text
+
+ <overlay root>/
+ <overlay root>/eclass
+ <overlay root>/eclass/R-packages.eclass
+ <overlay root>/profiles
+ <overlay root>/profiles/categories
+ <overlay root>/profiles/repo_name
+ <overlay root>/profiles/use.desc
+ <overlay root>/sci-R/<many directories per R package>
+ <overlay root>/sci-R/seewave/
+ <overlay root>/sci-R/seewave/Manifest
+ <overlay root>/sci-R/seewave/metadata.xml
+ <overlay root>/sci-R/seewave/seewave-1.5.9.ebuild
+ <overlay root>/sci-R/seewave/seewave-1.6.4.ebuild
+
+=================================
Repositories / Getting Packages
----------------------------------
+=================================
*roverlay* is capable of downloading R packages via rsync and http,
and is able to use any packages locally available. The concrete method used
@@ -446,9 +641,9 @@ to get and use the packages is determined by the concrete
.. _repo config:
-++++++++++++++++++++++++++++++++
+--------------------------------
A word about repo config files
-++++++++++++++++++++++++++++++++
+--------------------------------
Repo config files use ConfigParser_ syntax (known from ini files).
@@ -488,9 +683,9 @@ The common options for repository entries are:
.. _rsync:
-+++++++++++++
+-------------
Rsync repos
-+++++++++++++
+-------------
Runs *rsync* to download packages. Automatic sync retries are supported if
*rsync*'s exit codes indicates chances of success.
@@ -536,9 +731,9 @@ Examples:
.. _websync_repo:
-++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+------------------------------------------------------------
Getting packages from a repository that supports http only
-++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+------------------------------------------------------------
This is your best bet if the remote is a repository but doesn't offer
rsync access. Basic digest verification is supported (MD5).
@@ -599,9 +794,9 @@ None of these options are required.
.. _websync_pkglist:
-+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+---------------------------------------------------------------------
Getting packages from several remotes using http and a package list
-+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+---------------------------------------------------------------------
This is not a real repository type, instead it creates a *local* repository
by downloading single R packages from different remotes.
@@ -635,9 +830,9 @@ This repo type extends the default options by:
.. _local:
-+++++++++++++++++++++++++
+-------------------------
Using local directories
-+++++++++++++++++++++++++
+-------------------------
Using local package directories is possible, too.
@@ -663,49 +858,14 @@ A local directory will never be modified.
you should consider using one of the **websync** repo types,
websync_repo_ and websync_pkglist_.
--------------------------------------------
- Dependency Rules / Resolving Dependencies
--------------------------------------------
-
-++++++++++++++++++
- Dependency types
-++++++++++++++++++
-
-Every *dependency string* has a *dependency type* that declares how a
-dependency should be resolved. It has one or more of these properties:
-
-Mandatory
- Ebuild creation fails if the *dependency string* in question cannot
- be resolved.
-
-Optional
- The opposite of *Mandatory*, ebuild creation keeps going even if the
- *dependency string* is unresolvable.
-
-Package Dependency
- This declares that the *dependency string* could be another R package.
-
-System Dependency
- This declares that the *dependency string* could be a system dependency,
- e.g. a scientific library or a video encoder.
-
-Try other dependency types
- This declares that the *dependency string* can be resolved by ignoring its
- dependency type partially. This property allows to resolve
- package dependencies as system dependencies and vice versa.
-
-*Mandatory* and *Option* are mutually exclusive.
-
-The *dependency type* of a *dependency string* is determined by the its origin,
-i.e. info field in the package's DESCRIPTION file.
-The *Suggests* field, for example,
-gets the *"package dependency and optional"* type,
-whereas the *SystemRequirements* gets *"system dependency and mandatory"*.
+==================
+ Dependency Rules
+==================
-+++++++++++++++++++++++++
+-------------------------
Simple Dependency Rules
-+++++++++++++++++++++++++
+-------------------------
*Simple dependency rules* use a dictionary and string transformations
to resolve dependencies. *Fuzzy simple dependency rules* extend these by
@@ -716,8 +876,9 @@ dictionary entry.
This is the only rule implementation currently available.
-Rule Variants
--------------
++++++++++++++++
+ Rule Variants
++++++++++++++++
default
The expected behavior of a dictionary-based rule: It matches one or more
@@ -727,8 +888,9 @@ ignore
This variant will ignore *dependency strings*. Technically, it will
resolve them as **nothing**.
-Rule types
-----------
+++++++++++++
+ Rule types
+++++++++++++
Simple Rules
A simple rule resolves **exact string matches** (case-insensitive).
@@ -765,8 +927,9 @@ Fuzzy Rules
* "R ( !2.10 )" as "( dev-lang/R !=dev-lang/R-2.10 )"
-Rule File Examples
-------------------
+++++++++++++++++++++
+ Rule File Examples
+++++++++++++++++++++
This sections lists some rule file examples.
See `Rule File Syntax`_ for a formal<precise,..?> description.
@@ -791,6 +954,12 @@ Example 2 - *default* simple rule stub
zoo
+ .. Note::
+
+ R Package rules are dynamically created at runtime and therefore not
+ needed. Write them only if certain R package dependencies cannot
+ be resolved.
+
Example 3 - *default* simple rule
A rule that matches several *dependency strings* and resolves them
as "sci-libs/gdal and sci-libs/proj":
@@ -827,8 +996,9 @@ else in *<R Overlay src directory>/simple-deprules.d*.
.. _Dependency Rule File Syntax:
-Rule File Syntax
-----------------
+++++++++++++++++++
+ Rule File Syntax
+++++++++++++++++++
Simple dependency rule files have a special syntax. Each rule is introduced
with the resolving *dependency* prefixed by a *keychar* that sets the rule
@@ -908,47 +1078,53 @@ Multi line rules
...
}
+Selfdep
+ This is another name for *dependency strings* that are resolved by an
+ R package with the same name, which is also part of the overlay being
+ created.
+
+ Example: *zoo* is resolved as *sci-R/zoo* (if OVERLAY_CATEGORY is *sci-R*)
+
+ Writing selfdep rules is not necessary since *roverlay* automatically
+ creates rules for all R known packages (see `Dynamic Selfdep Rule Pool`_
+ for details).
+
+ There are a few exceptions to that in which case selfdep rules have to
+ be written:
+
+ * The *dependency string* is assumed to be a system dependency (not an
+ R package). This is likely a "bug" in the DESCRIPTION file of the
+ R package being processed.
+
+ * The R package name is not ebuild-name compliant (e.g. contains the '.'
+ char, which is remapped to '_'.).
+ Most *char remap* cases are handled properly, but there may be a few
+ exceptions.
+
+ .. Caution::
+
+ Writing unnecessary selfdep rules slows dependency resolution down!
+ Each rule will exist twice, once as *dynamic* rule and once as
+ the written one.
+
+
Rule Stubs
- There's a shorter syntax for dependencies that are resolved within the
- created overlay. For example, if your OVERLAY_CATEGORY is
- *sci-R*, *zoo* should be resolved as *sci-R/zoo*.
- This rule can be written as a single word, *zoo*. Such stubs are called
- **selfdeps**.
+ There's a shorter syntax for selfdeps.
+ For example, if your OVERLAY_CATEGORY is *sci-R*,
+ *zoo* should be resolved as *sci-R/zoo*.
+ This rule can be written as a single word, *zoo*.
Syntax:
.. code:: text
[<keychar>]<short dependency>
+
Comments
start with **#**. There are a few exceptions to that, the *#deptype* and
*#! NOPARSE* keywords. Comments inside rule blocks are not allowed and
will be read as normal *dependency strings*.
---------------------------------------------------------------
- Expected Overlay Result / Structure of the generated overlay
---------------------------------------------------------------
-
-Assuming that you're using the default configuration (where possible) and
-the *R-packages* eclass file, the result should look like:
-
-.. code-block:: text
-
- <overlay root>/
- <overlay root>/eclass
- <overlay root>/eclass/R-packages.eclass
- <overlay root>/profiles
- <overlay root>/profiles/categories
- <overlay root>/profiles/repo_name
- <overlay root>/profiles/use.desc
- <overlay root>/sci-R/<many directories per R package>
- <overlay root>/sci-R/seewave/
- <overlay root>/sci-R/seewave/Manifest
- <overlay root>/sci-R/seewave/metadata.xml
- <overlay root>/sci-R/seewave/seewave-1.5.9.ebuild
- <overlay root>/sci-R/seewave/seewave-1.6.4.ebuild
-
-
================
DepRes Console
================
@@ -1081,13 +1257,6 @@ Example Session:
Configuration Reference
=========================
-------------------
- Dependency Rules
-------------------
-
-<merge with basic..overview::deprules>
-
-
--------------
Repositories
--------------
@@ -1548,6 +1717,13 @@ Known field flags:
Appendix
==========
+-----------------
+ ebuild examples
+-----------------
+<required? this section would contain a minimal (DESCRIPTION, SRC_URI)
+and a 'bloated' (all vars, +DEPEND,RDEPEND,IUSE,R_SUGGEST,_UNRESOLVED_PACKAGES)
+ebuild>
+
-------------------------------
Default Field Definition File
-------------------------------
^ permalink raw reply related [flat|nested] 41+ messages in thread* [gentoo-commits] proj/R_overlay:master commit in: doc/rst/
@ 2012-08-10 17:54 André Erdmann
0 siblings, 0 replies; 41+ messages in thread
From: André Erdmann @ 2012-08-10 17:54 UTC (permalink / raw
To: gentoo-commits
commit: 967e8bce9969a16836464981d4ba47aef9524432
Author: André Erdmann <dywi <AT> mailerd <DOT> de>
AuthorDate: Fri Aug 10 17:54:34 2012 +0000
Commit: André Erdmann <dywi <AT> mailerd <DOT> de>
CommitDate: Fri Aug 10 17:54:34 2012 +0000
URL: http://git.overlays.gentoo.org/gitweb/?p=proj/R_overlay.git;a=commit;h=967e8bce
usage guide: numbered sections
---
doc/rst/usage.rst | 1 +
1 files changed, 1 insertions(+), 0 deletions(-)
diff --git a/doc/rst/usage.rst b/doc/rst/usage.rst
index c444685..ebf8aa0 100644
--- a/doc/rst/usage.rst
+++ b/doc/rst/usage.rst
@@ -10,6 +10,7 @@
.. _ConfigParser:
http://docs.python.org/library/configparser.html
+.. sectnum::
.. contents::
^ permalink raw reply related [flat|nested] 41+ messages in thread* [gentoo-commits] proj/R_overlay:master commit in: doc/rst/
@ 2012-08-10 15:16 André Erdmann
0 siblings, 0 replies; 41+ messages in thread
From: André Erdmann @ 2012-08-10 15:16 UTC (permalink / raw
To: gentoo-commits
commit: 141a7dfa757cafe54a3fbb33121c16ad417cf721
Author: André Erdmann <dywi <AT> mailerd <DOT> de>
AuthorDate: Fri Aug 10 15:10:33 2012 +0000
Commit: André Erdmann <dywi <AT> mailerd <DOT> de>
CommitDate: Fri Aug 10 15:11:05 2012 +0000
URL: http://git.overlays.gentoo.org/gitweb/?p=proj/R_overlay.git;a=commit;h=141a7dfa
usage guide (incomplete)
---
doc/rst/usage.rst | 1103 +++++++++++++++++++++++++++++++++++++++++++++++++++++
1 files changed, 1103 insertions(+), 0 deletions(-)
diff --git a/doc/rst/usage.rst b/doc/rst/usage.rst
new file mode 100644
index 0000000..c444685
--- /dev/null
+++ b/doc/rst/usage.rst
@@ -0,0 +1,1103 @@
+.. _roverlay-9999.ebuild:
+ http://git.overlays.gentoo.org/gitweb/?p=proj/R_overlay.git;a=blob;f=roverlay-9999.ebuild;hb=refs/heads/master
+
+.. _roverlay git repo:
+ http://git.overlays.gentoo.org/gitweb/?p=proj/R_overlay.git;a=summary
+
+.. _omegahat's PACKAGES file:
+ http://www.omegahat.org/R/src/contrib/PACKAGES
+
+.. _ConfigParser:
+ http://docs.python.org/library/configparser.html
+
+
+.. contents::
+
+
+==============
+ Introduction
+==============
+
+*roverlay* is
+Automatically Generated Overlay of R packages;;
+GSoC Project;;
+<>;;
+
+
+==============
+ Installation
+==============
+
+---------------
+ Prerequisites
+---------------
+
+* python >= 2.7 (tested with python 2.7 and 3.2)
+
+* argparse (http://code.google.com/p/argparse)
+
+* rsync (for using rsync repositories)
+
+* for Manifest creation:
+
+ * *ebuild* from portage
+ * *true* or *echo* from coreutils or busybox for preventing
+ package downloads during Manifest creation (optional)
+
+* for generating documentation files: python docutils > 0.8.1
+
+* hardware requirements (when the default configuration):
+
+ disk
+ * a filesystem that supports symbolic links
+ * 50-55GB disk space for the R packages when using the default
+ R package hosts (CRAN with archive, BIOC, R-Forge and Omegahat)
+ * <a lot of inodes for PORTAGE_RO_DISTDIR/__tmp__ and the overlay>
+
+ memory
+ up to 600MB which depends on the amount of processed packages and the
+ write mechanism in use. The amount can be halved (approximately) when
+ using a slower one.
+
+ other
+ No other hardware requirements. <as a measurement for computing speed,
+ i/o requirements:
+ overlay creation takes 3-4h on modern desktop hardware with overlay
+ writing to a tmpfs, most time is spent for Manifest creation>
+
+---------------------
+ via emerge (Gentoo)
+---------------------
+
+A live ebuild is available, `roverlay-9999.ebuild`_.
+Add it to your local overlay and run ``emerge roverlay``, which will also
+install all config files into */etc/roverlay*.
+
+---------------------
+ Manual Installation
+---------------------
+
+After installing the dependencies as listed in `Prerequisites`_,
+clone the `roverlay git repo`_ and then
+install *roverlay* and its python modules:
+
+.. code-block:: sh
+
+ git clone git://git.overlays.gentoo.org/proj/R_overlay.git
+
+ cd R_overlay && make install
+
+``make install`` also accepts some variables, namely:
+
+* *DESTDIR* defaults to ''
+
+* *BINDIR*, defaults to *DESTDIR*/usr/local/bin
+
+* *PYMOD_FILE_LIST*, which lists the installed python module files
+ and defaults to './roverlay_files.list'
+
+* *PYTHON*, name of path of the python interpreter that will run 'setup.py',
+ defaults to 'python'
+
+
+*roverlay* can later be uninstalled with ``make uninstall``.
+
+.. Note::
+
+ Make sure to include ``--record <somewhere>/roverlay_files.list``
+ when running ``./setup.py install`` manually,
+ which can later be used to safely remove the python module files with
+ ``xargs rm -vrf < <somewhere>/roverlay_files.list``.
+ The *make* targets take care of this.
+
+.. Warning::
+
+ Support for this installation type is limited - it won't install/create
+ any config files!
+
+---------------------------------------
+ Using *roverlay* without installation
+---------------------------------------
+
+This is possible, too. Make sure to meet the dependencies as listed
+in Prerequisites_.
+Then, simply clone the git repository to a local directory that
+will later be referenced as the *R Overlay src directory*.
+
+.. Note::
+ You'll have to cd into the *R Overlay src directory* before running
+ *roverlay* to ensure that the python modules can be imported correctly.
+
+ You can work around this by setting up a wrapper script:
+
+ .. code-block:: sh
+
+ #!/bin/sh
+ # /usr/local/bin/roverlay.sh
+ # example wrapper script for roverlay
+ cd ${ROVERLAY_SRC:-~/roverlay/src} && ./roverlay.py $*
+
+
+==================
+ Running Roverlay
+==================
+
+------------------------------
+ Required configuration steps
+------------------------------
+
+*roverlay* needs a configuration file to run.
+
+If you've installed *roverlay* with *emerge*, it will look for the config
+file in that order:
+
+1. *<current directory>/R-overlay.conf*
+2. *~/.R-overlay.conf*
+3. */etc/roverlay/R-overlay.conf*,
+ which is part of the installation but has to be modified.
+
+Otherwise, an example config file is available in the *R Overlay src directory*
+and *roverlay* will only look for *R-overlay.conf* in the current directory.
+
+The config file is a text file with '<option> = <value>' syntax.
+Some options accept multiple values (e.g. <option> = file1, file2), in which
+case the values have to be enclosed
+with quotes (-> ``<option> = "file1 file2"``).
+
+
+The following options should be set before running *roverlay*:
+
+ OVERLAY_DIR
+ This sets the directory of the overlay that will be created.
+ This option is **required** and can be overridden on the command line
+ via ``--overlay <directory>``.
+
+ Example: OVERLAY_DIR = ~/roverlay/overlay
+
+ DISTFILES
+ This sets the root directory of all per-repo package directories.
+ This option is **required** and can be overridden on the command line
+ via ``--distroot <directory>``.
+
+ .. Note::
+
+ This directory will also contain a directory *__tmp__*
+ with symlinks to all packages which can be used as package mirror,
+ see `Providing a package mirror`_.
+
+ Example: DISTFILES = ~/roverlay/distfiles
+
+ LOG_FILE
+ This sets the log file.
+
+ Example: LOG_FILE = ~/roverlay/log/roverlay.log
+
+ LOG_LEVEL
+ This sets the global log level, which is used for all log formats
+ that don't override this option. Valid log levels are
+ ``DEBUG``, ``INFO``, ``WARN``/``WARNING``, ``ERROR`` and ``CRITICAL``.
+
+ Example: LOG_LEVEL = WARNING
+
+ LOG_LEVEL_CONSOLE
+ This sets the console log level.
+
+ Example: LOG_LEVEL_CONSOLE = INFO
+
+ LOG_LEVEL_FILE
+ This sets the log level for file logging.
+
+ Example: LOG_LEVEL_FILE = ERROR
+
+The following options should also be set (most of them are required), but
+have reasonable defaults if *roverlay* has been installed using *emerge*:
+
+ SIMPLE_RULES_FILE
+ This option lists the dependency rules files that should be used
+ for dependency resolution (see
+ `Dependency Rules / Resolving Dependencies`_).
+ Although not required, this option is **recommended** since ebuild
+ creation without dependency rules fails for most R packages.
+
+ Example: SIMPLE_RULES_FILE = ~/roverlay/config/simple-deprules.d
+
+ REPO_CONFIG
+ A list with one or more files that list repositories
+ (see `Repositories / Getting Packages`_).
+ This option is **required** and can be overridden on the command line
+ via one or more ``repo-config <file>`` statements.
+
+ Example: REPO_CONFIG = ~/roverlay/config/repo.list
+
+ FIELD_DEFINITION
+ The value of this option should point to a field definition file which
+ controls how an R package's DESCRIPTION file is read.
+ The file supplied by default should be fine.
+ This option is **required** and can be overridden on the command line
+ via ``--field-definition <file>``.
+
+ Example: FIELD_DEFINITION = ~/roverlay/config/description_fields.conf
+
+ OVERLAY_ECLASS
+ This option lists eclass files that should be imported into the overlay
+ (into *OVERLAY_DIR*/eclass/) and inherited in all ebuilds.
+ Specifying an eclass file that implements the ebuild phase functions
+ (e.g. *src_install()*) is highly **recommended**. A default file
+ named *R-packages.eclass* should be part of your installation.
+
+ Example: OVERLAY_ECLASS = ~/roverlay/eclass/R-packages.eclass
+
+
+For details and a full configuration reference, see `Configuration Reference`_.
+
+------------
+ Running it
+------------
+
+If you've installed *roverlay*, you can run it with ``roverlay``, otherwise
+you'll have to cd into the *R overlay src directory* and run ``./roverlay.py``.
+
+In any case, the basic *roverlay* script usage is
+
+.. code::
+
+ roverlay --config <config file> [<options>] [<commands>]
+
+or
+
+.. code::
+
+ roverlay [<options>] [<commands>]
+
+which will search for the config file
+as described in `Required configuration steps`_.
+The default command is *create*, which downloads the R packages (unless
+explicitly forbidden to do so) and generates the overlay. This is the
+desired behavior in most cases, so simply running ``roverlay`` should be
+fine.
+
+*roverlay* also accepts some **options**, most notably:
+
+--nosync, --no-sync
+ Don't download R packages.
+
+--no-incremental
+ Force recreation of existing ebuilds
+
+--immediate-ebuild-writes
+ Immediately write ebuilds when they're ready.
+
+ The default behavior is
+ to wait for all ebuilds and then write them using ebuild write threads.
+ The latter one is faster, but consumes more memory since ebuilds must be
+ kept until all packages have been processed.
+ Test results show that memory consumption increases by factor 2 when using
+ the faster write mechanism (at ca. 95% ebuild creation success rate),
+ <while ebuild write time decreases by ???>.
+
+ Summary: Expect 300 (slow) or 600MB (fast) memory consumption when using
+ the default package repositories.
+
+--config file, -c file
+ Path to the config file
+
+--help, -h
+ Show all options
+
+
+.. Note::
+ *--no-incremental* doesn't delete an existing overlay, it will merely
+ ignores and, potentially, overwrites existing ebuilds.
+ Use *rm -rf <overlay>* to do that.
+
+
+For **testing** *roverlay*, these **options** may be convenient:
+
+--no-manifest
+ Skip Manifest file creation.
+
+ This saves a considerable amount of time
+ (>100min when using the default package repositories) at the expense of
+ an overlay that is not suitable for production usage.
+
+--no-write
+ Don't write the overlay
+
+--show
+ Print all ebuilds and metadata to console
+
+--repo-config file, -R file
+ Repo config file to use. Can be specified more than once.
+ This disables all repo files configured in the main config file.
+
+--distdir directory, --from directory
+ Create an overlay using the packages found in *directory*. This disables
+ all other repositories. The *SRC_URI* ebuild variable will be invalid!
+
+--overlay directory, -O directory
+ Create the overlay at the given position.
+
+For reference, these **commands** are accepted by *roverlay*:
+
+create
+ As described above, this will run ebuild, metadata creation, including
+ overlay and Manifest file writing.
+ This command implies the **sync** command unless the *--nosync* option
+ is specified.
+
+sync
+ This will download all packages from the configured remotes.
+
+depres_console, depres
+ Starts an interactive dependency resolution console that supports rule
+ creation/deletion, loading rules from files, writing rules to files
+ and resolving dependencies.
+
+ Meant for **testing only**.
+
+----------------------------
+ Providing a package mirror
+----------------------------
+
+<No recommendations at this time. The current ManifestCreation implementation
+creates a directory *<distfiles root>/__tmp__* with symlinks to all packages,
+which could be used for providing packages, but this may change
+in near future since external Manifest creation is too slow
+(takes >60% of overlay creation time).>
+
+
+=========================
+ Implementation Overview
+=========================
+
+This section gives a basic overview of how *roverlay* works.
+
+<how *roverlay* basically works:>
+
+1. **sync** - get R packages using various methods
+ (rsync, http, local directory)
+
+2. scan the R Overlay directory (if it exists) for valid ebuilds
+
+3. queue all R packages for ebuild creation
+
+ * all repositories are asked to list their packages which are then added
+ to a queue
+
+ * packages may be declined by the overlay creator if they already have
+ an ebuild
+
+4. **create** - process each package *p* from the package queue
+ (thread-able on a per package basis)
+
+ * read *p*'s DESCRIPTION file that contains information fields
+ like 'Depends', 'Description' and 'Suggests'
+
+ * resolve *p*'s dependencies
+
+ * differentiate between *required* and *optional* dependencies
+ (for example, dependencies from the 'Depends' field are required,
+ while those from 'Suggests' are optional)
+
+ * **immediately stop** processing *p* if a *required* dependency
+ cannot be resolved in which case a valid ebuild cannot be created
+
+ * create an ebuild for *p* by using the dependency resolution results
+ and a few information fields like 'Description'
+
+ * **done** with *p* - the overlay creator takes control over *p*
+ and may decide to write *p*'s ebuild now (or later)
+
+5. write the overlay
+
+ * write all ebuilds
+ (thread-able on a per package basis)
+
+ * write the *metadata.xml* files
+ (thread-able on a per package basis)
+
+ * this uses the latest created ebuild available for a package
+
+ * write the *Manifest* files
+ (not thread-able)
+
+ * this uses all ebuilds availabe for a package
+
+---------------------------------
+ Repositories / Getting Packages
+---------------------------------
+
+*roverlay* is capable of downloading R packages via rsync and http,
+and is able to use any packages locally available. The concrete method used
+to get and use the packages is determined by the concrete
+**type of a repository** and that's what this section is about.
+
+++++++++++++++++++++++++++++++++
+ A word about repo config files
+++++++++++++++++++++++++++++++++
+
+Repo config files use ConfigParser_ syntax (known from ini files).
+
+Each repo entry section is introduced with ``[<repo name>]`` and defines
+
+* how *roverlay* can download the R packages from a repo
+ and where they should be stored
+* how ebuilds can download the packages (-> *SRC_URI*)
+* repo type specific options, e.g. whether the repo supports package file
+ verification
+
+Such options are declared with ``<option> = <value>`` in the repo entry.
+
+The common options for repository entries are:
+
+* *type*, which declares the repository type. Available types are:
+
+ * rsync_
+ * websync_repo_
+ * websync_pkglist_
+ * local_
+
+ Defaults to *rsync*
+
+* *src_uri*, which declares how ebuilds can download the packages. Some repo
+ types use this for downloading, too.
+
+* *directory*, which explicitly sets the package directory to use.
+ The default behavior is to use `DISTFILES_ROOT`_/<repo name>
+
+
+.. Hint::
+ Repo names are allowed contain slashes, which will be respected when
+ creating the default directory.
+
+.. _rsync:
+
++++++++++++++
+ Rsync repos
++++++++++++++
+
+Runs *rsync* to download packages. Automatic sync retries are supported if
+*rsync*'s exit codes indicates chances of success.
+For example, up to 3 retries are granted if *rsync* returns
+*Partial transfer due to vanished source files* which likely happens when
+syncing big repositories like CRAN.
+
+This repo type extends the default options by:
+
+* *rsync_uri* (**required**), which specifies the uri used for syncing
+
+* *recursive* (optional), which passes ``--recursive`` to *rsync* if set to
+ 'yes'
+
+* *extra_rsync_opts* (optional), which passes arbitrary options to *rsync*.
+ This can be used include/exclude files or to show progress during transfer.
+ Options with whitespace are not supported.
+
+Examples:
+
+* CRAN
+
+ .. code-block:: ini
+
+ [CRAN]
+ type = rsync
+ rsync_uri = cran.r-project.org::CRAN/src/contrib
+ src_uri = http://cran.r-project.org/src/contrib
+ extra_rsync_opts = --progress --exclude=PACKAGES --exclude=PACKAGES.gz
+
+
+* CRAN's archive:
+
+ .. code-block:: ini
+
+ [CRAN-Archive]
+ type = rsync
+ rsync_uri = cran.r-project.org::CRAN/src/contrib/Archive
+ src_uri = http://cran.r-project.org/src/contrib/Archive
+ extra_rsync_opts = --progress
+ recursive = yes
+
+
+.. _websync_repo:
+
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+ Getting packages from a repository that supports http only
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+
+This is your best bet if the remote is a repository but doesn't offer
+rsync access. Basic digest verification is supported (MD5).
+The remote has to have a package list file, typically named
+*PACKAGES*, with a special syntax (debian control file syntax).
+
+A package list example,
+excerpt from `omegahat's PACKAGES file`_ (as of Aug 2012):
+
+.. code-block:: control
+
+ ...
+ Package: CGIwithR
+ Version: 0.73-0
+ Suggests: GDD
+ License: GPL (>= 2)
+ MD5sum: 50b1f48209c9e66909c7afb3a4b8af5e
+
+ Package: CodeDepends
+ Version: 0.2-1
+ Depends: methods
+ Imports: codetools, XML
+ Suggests: graph, Rgraphviz
+ License: GPL
+ MD5sum: e2ec3505f9db1a96919a72f07673a2d8
+ ...
+
+An example repo config entry for omegahat:
+
+.. code-block:: ini
+
+ [omegahat]
+ type = websync_repo
+ src_uri = http://www.omegahat.org/R/src/contrib
+ digest = md5
+ #digest = none
+
+
+This repo type extends the default options by:
+
+* *digest*, which declares that the remote supports digest based package file
+ verification. Accepted values are 'md5' and 'none'. Defaults to 'none',
+ which disables verification.
+
+* *pkglist_file*, which sets the name of the package list file and defaults
+ to PACKAGES
+
+* *pkglist_uri*, which explicitly sets the uri of the package list file.
+ Defaults to *src_uri*/*pkglist_file*
+
+None of these options are required.
+
+
+.. Note::
+
+ The content type of the remote's package list file has to be exactly
+ *text/plain*, compressed files are not supported.
+
+.. _websync_pkglist:
+
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+ Getting packages from several remotes using http and a package list
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+
+This is not a real repository type, instead it creates a *local* repository
+by downloading single R packages from different remotes.
+Its only requirement is that a package is downloadable via http.
+Apart from an entry in the repo config file, it also needs a file that lists
+one package uri per line:
+
+.. code-block:: text
+
+ ...
+ http://cran.r-project.org/src/contrib/seewave_1.6.4.tar.gz
+ http://download.r-forge.r-project.org/src/contrib/zoo_1.7-7.tar.gz
+ http://www.omegahat.org/R/src/contrib/Aspell_0.2-0.tar.gz
+ ...
+
+Comments are not supported. Assuming that such a package list exists as <at?>
+*~/roverlay/config/http_packages.list*, an example entry in the repo config
+file would be:
+
+.. code-block:: ini
+
+ [http-packages]
+ type = websync_pkglist
+ pkglist = ~/roverlay/config/http_packages.list
+
+
+This repo type extends the default options by:
+
+* *pkglist*, which sets the package list file. This option is **required**.
+
+
+.. _local:
+
++++++++++++++++++++++++++
+ Using local directories
++++++++++++++++++++++++++
+
+Using local package directories is possible, too.
+
+Example:
+
+.. code-block:: ini
+
+ [local-packages]
+ type = local
+ directory = /var/local/R-packages
+ src_uri = http://localhost/R-packages
+
+This will use all packages from */var/local/R-packages* and assumes
+that they're available via *http://localhost/R-packages*.
+
+A local directory will never be modified.
+
+.. Important::
+
+ Using this repo type is **not recommended for production usage** because
+ the *SRC_URI* variable in created ebuilds will be invalid unless you've
+ downloaded all packages from the same remote in which case
+ you should consider using one of the **websync** repo types,
+ websync_repo_ and websync_pkglist_.
+
+-------------------------------------------
+ Dependency Rules / Resolving Dependencies
+-------------------------------------------
+
+++++++++++++++++++
+ Dependency types
+++++++++++++++++++
+
+Every *dependency string* has a *dependency type* that declares how a
+dependency should be resolved. It has one or more of these properties:
+
+Mandatory
+ Ebuild creation fails if the *dependency string* in question cannot
+ be resolved.
+
+Optional
+ The opposite of *Mandatory*, ebuild creation keeps going even if the
+ *dependency string* is unresolvable.
+
+Package Dependency
+ This declares that the *dependency string* could be another R package.
+
+System Dependency
+ This declares that the *dependency string* could be a system dependency,
+ e.g. a scientific library or a video encoder.
+
+Try other dependency types
+ This declares that the *dependency string* can be resolved by ignoring its
+ dependency type partially. This property allows to resolve
+ package dependencies as system dependencies and vice versa.
+
+*Mandatory* and *Option* are mutually exclusive.
+
+The *dependency type* of a *dependency string* is determined by the its origin,
+i.e. info field in the package's DESCRIPTION file.
+The *Suggests* field, for example,
+gets the *"package dependency and optional"* type,
+whereas the *SystemRequirements* gets *"system dependency and mandatory"*.
+
+
++++++++++++++++++++++++++
+ Simple Dependency Rules
++++++++++++++++++++++++++
+
+*Simple dependency rules* use a dictionary and string transformations
+to resolve dependencies. *Fuzzy simple dependency rules* extend these by
+a set of regexes, which allows to resolve many dependency strings that
+minimally differ (e.g. only in the required version and/or comments:
+`R (>= 2.10)` and `R [2.14] from http://www.r-project.org/`) with a single
+dictionary entry.
+
+This is the only rule implementation currently available.
+
+Rule Variants
+-------------
+
+default
+ The expected behavior of a dictionary-based rule: It matches one or more
+ *dependency strings* and resolves them as a *dependency*
+
+ignore
+ This variant will ignore *dependency strings*. Technically, it will
+ resolve them as **nothing**.
+
+Rule types
+----------
+
+Simple Rules
+ A simple rule resolves **exact string matches** (case-insensitive).
+
+ Example:
+ Given a rule *R* that says "resolve 'R' and 'the R programming language'
+ as 'dev-lang/R'", any of these *dependency strings* will be resolved
+ as dev-lang/R:
+
+ * r
+ * THE R PROGRAMMING LanGuAgE
+ * R
+
+Fuzzy Rules
+ Fuzzy Rules are **extended Simple Rules**. If the basic lookup
+ as described above fails for a *dependency string*,
+ they will *try* to resolve it as a **version-relative match**.
+
+ To do this, the *dependency string* will be split into components like
+ *dependency name*, *dependency version* and useless comments, which are
+ discarded.
+ Then, if the *dependency name* matches a dictionary entry, a resolving
+ *dependency* will be created.
+
+ Example:
+ Given the same rule as in the Simple Rules example, but as fuzzy rule
+ "fuzzy-resolve 'R' and 'the R programming language' as 'dev-lang/R'",
+ it will resolve any of these *dependency strings*:
+
+ * "r" as "dev-lang/R"
+ * "R 2.12" as ">=dev-lang/R-2.12"
+ * "The R PROGRAMMING LANGUAGE [<2.14] from http://www.r-project.org/"
+ as "<dev-lang/R-2.14"
+ * "R ( !2.10 )" as "( dev-lang/R !=dev-lang/R-2.10 )"
+
+
+Rule File Examples
+------------------
+
+This sections lists some rule file examples.
+See `Rule File Syntax`_ for a formal<precise,..?> description.
+
+
+Example 1 - *default* fuzzy rule
+ A rule that matches many dependencies on dev-lang/R, for example
+ "r 2.12", "R(>= 2.14)", "R [<2.10]", "r{ !2.12 }", and "R", and
+ resolves them as '>=dev-lang/R-2.12', '>=dev-lang/R-2.14',
+ '<dev-lang/R-2.10', etc.:
+
+ .. code:: text
+
+ ~dev-lang/R :: R
+
+
+Example 2 - *default* simple rule stub
+ A rule that case-insensitively matches 'zoo' and resolves it as 'sci-R/zoo',
+ assuming the OVERLAY_CATEGORY is set to 'sci-R':
+
+ .. code:: text
+
+ zoo
+
+Example 3 - *default* simple rule
+ A rule that matches several *dependency strings* and resolves them
+ as "sci-libs/gdal and sci-libs/proj":
+
+ .. code-block:: text
+
+ ( sci-libs/gdal sci-libs/proj ) {
+ for building from source: GDAL >= 1.3.1 && GDAL < 1.6.0 (until tested) library and PROJ.4 (proj >= 4.4.9)
+ for building from source: GDAL >= 1.3.1 library and PROJ.4 (proj >= 4.4.9)
+ for building from source: GDAL >= 1.3.1 library and PROJ.4(proj >= 4.4.9)
+ for building from source: GDAL >= 1.6.0 library and PROJ.4(proj >= 4.4.9)
+ }
+
+Example 4 - *ignore* simple rule
+ A rule that matches text that should be ignored.
+ This is a good way to deal with free-style text found
+ in some R package DESCRIPTION files.
+
+ .. code-block:: text
+
+ ! {
+ see README
+ read INSTALL
+ Will use djmrgl or rgl packages for rendering if present
+ }
+
+
+Please see the default rule files for more extensive examples that cover
+other aspects like limiting a rule to certain dependency types.
+They're found in */etc/roverlay/simple-deprules.d*
+if you've installed *roverlay* with *emerge*,
+else in *<R Overlay src directory>/simple-deprules.d*.
+
+Rule File Syntax
+----------------
+
+Simple dependency rule files have a special syntax. Each rule is introduced
+with the resolving *dependency* prefixed by a *keychar* that sets the rule
+type if required. The *dependency strings* resolved by this rule are listed
+after a rule separator or within a rule block. Leading/trailing whitespace
+is ignored.
+
+Ignore rules
+ have only a keychar but no *dependency*.
+
+Keychars
+ set the rule type.
+
+ * **!** introduces a *ignore* simple rule
+ * **~** introduces a *default* fuzzy rule
+ * **%** introduces a *ignore* fuzzy rule
+
+ Anything else is not treated as keychar and thus introduces a *default*
+ simple rule.
+
+Keywords
+ There are two keywords that control how a rule file is read.
+
+ The important one is the *#deptype <dependency type>* directive that
+ defines that all rules until the next *deptype* directory or end of file,
+ whatever comes first, will only match *dependency strings*
+ with the specified *dependency type*.
+
+ Available dependency types choices are
+
+ * *all* (no type restrictions)
+ * *pkg* (resolve only R package dependencies)
+ * *sys* (resolve only system dependencies)
+
+ The other keyword is *#! NOPARSE* which stops parsing of a rule file.
+
+Dependencies
+ are strings that are recognized by portage as **Dynamic DEPENDs**
+ (see the ebuild(5) man page).
+
+ Examples:
+
+ * dev-lang/R
+ * ( media-libs/tiff >=sci-libs/fftw-3 )
+ * >=x11-drivers/nvidia-drivers-270
+
+
+ .. Note::
+
+ The fuzzy rule types support **DEPEND Atom Bases** only.
+
+ .. Warning::
+
+ Dependency strings cannot start with *~* as it is a keychar.
+ Use braces *( ~... )* to work around that.
+
+
+Single line rules
+ resolve exactly one *dependency string*. Their rule separator is ' :: '.
+
+ Syntax:
+ .. code:: text
+
+ [<keychar>]<dependency> :: <dependency string>
+
+Multi line rules
+ resolve several *dependency strings*.
+ Their rule block begins with '{' + newline, followed by one
+ *dependency string* per line, and ends with '}'.
+
+ Syntax:
+ .. code-block:: text
+
+ [<keychar>]<dependency> {
+ <dependency string>
+ [<dependency string>]
+ ...
+ }
+
+Rule Stubs
+ There's a shorter syntax for dependencies that are resolved within the
+ created overlay. For example, if your OVERLAY_CATEGORY is
+ *sci-R*, *zoo* should be resolved as *sci-R/zoo*.
+ This rule can be written as a single word, *zoo*. Such stubs are called
+ **selfdeps**.
+
+ Syntax:
+ .. code:: text
+
+ [<keychar>]<short dependency>
+
+Comments
+ start with **#**. There are a few exceptions to that, the *#deptype* and
+ *#! NOPARSE* keywords. Comments inside rule blocks are not allowed and
+ will be read as normal *dependency strings*.
+
+--------------------------------------------------------------
+ Expected Overlay Result / Structure of the generated overlay
+--------------------------------------------------------------
+
+Assuming that you're using the default configuration (where possible) and
+the *R-packages* eclass file, the result should look like:
+
+.. code-block:: text
+
+ <overlay root>/
+ <overlay root>/eclass
+ <overlay root>/eclass/R-packages.eclass
+ <overlay root>/profiles
+ <overlay root>/profiles/categories
+ <overlay root>/profiles/repo_name
+ <overlay root>/profiles/use.desc
+ <overlay root>/sci-R/<many directories per R package>
+ <overlay root>/sci-R/seewave/
+ <overlay root>/sci-R/seewave/Manifest
+ <overlay root>/sci-R/seewave/metadata.xml
+ <overlay root>/sci-R/seewave/seewave-1.5.9.ebuild
+ <overlay root>/sci-R/seewave/seewave-1.6.4.ebuild
+
+
+=========================
+ Configuration Reference
+=========================
+
+------------------
+ Dependency Rules
+------------------
+
+<merge with basic..overview::deprules>
+
+
+--------------
+ Repositories
+--------------
+
+<merge with basic..overview::repo>
+
+-------------
+Main Config
+-------------
+
+.. _DISTFILES_ROOT:
+
+DISTFILES_ROOT
+ <>
+
+The main config file uses shell syntax.
+
+------------------
+ Field Definition
+------------------
+
+The field definition file uses ConfigParser_ syntax. For an example, see
+`default field definition file`_.
+
+Each information field has its own section which declares a set of options
+and flags. Flags are case-insensivitve options
+without a value - they're enabled by listing them.
+
+.. _field option:
+.. _field options:
+
+Known field options:
+
+ .. _field option\: default_value:
+
+ default_value
+ Sets the default value for a field, which implies that any read
+ DESCRIPTION file will contain this field, either with the value read
+ from the file or (as fallback) the default value.
+ Disables the `'mandatory' field flag`_.
+
+ .. _field option\: allowed_value:
+
+ allowed_value
+ Declares that a field has a value whitelist and adds the value to that
+ list (preserves whitespace).
+
+ .. _field option\: allowed_values:
+
+ allowed_values
+ Declares that a field has a value whitelist and adds the values to
+ that list (values are separated by whitespace).
+
+ .. _field option\: alias_withcase:
+ .. _field option\: alias:
+
+ alias_withcase, alias
+ Declares case-sensitive field name aliases. This can be used to fix
+ 'typos', e.g. *Suggest* and *Suggests* both mean *Suggests*.
+
+ .. _field option\: alias_nocase:
+
+ alias_nocase
+ Same as `field option: alias`_, but aliases are case-insensitive.
+
+ .. _field option\: flags:
+
+ flags
+ List of `field flags`_. Note that any option without a value is treated
+ as flag.
+
+.. _field flags:
+.. _field flag:
+
+Known field flags:
+
+ .. _field flag\: joinValues:
+
+ joinValues
+ Declares that a field's value is one string even if it spans over
+ multiple lines.
+ The lines will be joined with a single space character ' '.
+ The default behavior is to merge lines.
+
+ .. _field flag\: isList:
+
+ isList
+ Declares that a field's value is a list whose values are separated by
+ by ',' or ';'.
+
+ .. _field flag\: isWhitespaceList:
+
+ isWhitespaceList
+ Declares that a field's value is a list whose values are separated by
+ whitespace. Has no effect if `field flag: isList` is set.
+
+ .. _field flag\: mandatory:
+ .. _'mandatory' field flag:
+
+ mandatory
+ Declares that a field is required in *all* DESCRIPTION files.
+ This means that R packages without that field are considered as unusable,
+ i.e. ebuild creation fails early.
+ This flag is (effectively) useless in conjunction with
+ `field option: default_value`_ unless the default value evaluates to
+ False (e.g. is an empty string).
+
+
+ .. _field flag\: ignore:
+
+ ignore
+ Declares that a field is known but entirely ignored. Unknown fields
+ are ignored, too, the main difference is the log message.
+
+.. Note::
+
+ It won't be checked whether a flag is known or not.
+
+
+==========
+ Appendix
+==========
+
+-------------------------------
+ Default Field Definition File
+-------------------------------
+
+This is the default field definition file (without any ignored fields):
+
+.. code-block:: ini
+
+ [Description]
+ joinValues
+
+ [Title]
+ joinValues
+
+ [Suggests]
+ alias_nocase = Suggests, Suggest, %Suggests, Suggets, Recommends
+ isList
+
+ [Depends]
+ alias_nocase = Depends, Dependencies, Dependes, %Depends, Depents, Require, Requires
+ isList
+
+ [Imports]
+ alias_nocase = Imports, Import
+ isList
+
+ [LinkingTo]
+ alias_nocase = LinkingTo, LinkingdTo, LinkinTo
+ isList
+
+ [SystemRequirements]
+ alias_nocase = SystemRequirements, SystemRequirement
+ isList
+
+ [OS_Type]
+ alias_nocase = OS_TYPE
+ allowed_values = unix
^ permalink raw reply related [flat|nested] 41+ messages in thread
end of thread, other threads:[~2014-08-23 19:03 UTC | newest]
Thread overview: 41+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2014-07-18 16:20 [gentoo-commits] proj/R_overlay:master commit in: doc/rst/ André Erdmann
2014-07-18 14:55 ` [gentoo-commits] proj/R_overlay:wip/addition_control " André Erdmann
-- strict thread matches above, loose matches on Subject: below --
2014-08-23 19:03 [gentoo-commits] proj/R_overlay:master " André Erdmann
2014-07-29 18:29 André Erdmann
2014-08-23 19:03 ` André Erdmann
2014-06-05 22:09 André Erdmann
2013-11-14 18:24 André Erdmann
2013-09-23 15:30 André Erdmann
2013-09-23 12:34 André Erdmann
2013-09-16 8:56 André Erdmann
2013-09-03 14:32 André Erdmann
2013-08-27 15:39 André Erdmann
2013-08-23 13:52 André Erdmann
2013-08-19 15:42 André Erdmann
2013-08-09 15:27 André Erdmann
2013-08-07 16:10 André Erdmann
2013-08-06 16:02 André Erdmann
2013-08-02 14:30 André Erdmann
2013-08-01 17:14 André Erdmann
2013-07-23 18:34 [gentoo-commits] proj/R_overlay:gsoc13/next " André Erdmann
2013-07-23 18:34 ` [gentoo-commits] proj/R_overlay:master " André Erdmann
2013-07-23 14:57 [gentoo-commits] proj/R_overlay:gsoc13/next " André Erdmann
2013-07-23 14:57 ` [gentoo-commits] proj/R_overlay:master " André Erdmann
2013-07-16 16:35 [gentoo-commits] proj/R_overlay:gsoc13/next " André Erdmann
2013-07-16 16:36 ` [gentoo-commits] proj/R_overlay:master " André Erdmann
2013-07-15 22:31 [gentoo-commits] proj/R_overlay:gsoc13/next " André Erdmann
2013-07-16 16:36 ` [gentoo-commits] proj/R_overlay:master " André Erdmann
2013-07-03 10:05 André Erdmann
2013-07-03 10:04 [gentoo-commits] proj/R_overlay:gsoc13/next " André Erdmann
2013-07-03 10:05 ` [gentoo-commits] proj/R_overlay:master " André Erdmann
2013-06-26 17:29 [gentoo-commits] proj/R_overlay:gsoc13/next " André Erdmann
2013-06-30 15:58 ` [gentoo-commits] proj/R_overlay:master " André Erdmann
2013-06-25 21:10 [gentoo-commits] proj/R_overlay:gsoc13/next " André Erdmann
2013-06-30 15:58 ` [gentoo-commits] proj/R_overlay:master " André Erdmann
2013-06-18 14:12 [gentoo-commits] proj/R_overlay:gsoc13/next " André Erdmann
2013-06-18 14:12 ` [gentoo-commits] proj/R_overlay:master " André Erdmann
2013-06-12 21:10 [gentoo-commits] proj/R_overlay:gsoc13/next " André Erdmann
2013-06-13 16:34 ` [gentoo-commits] proj/R_overlay:master " André Erdmann
2013-06-05 18:08 [gentoo-commits] proj/R_overlay:gsoc13/next " André Erdmann
2013-06-13 16:34 ` [gentoo-commits] proj/R_overlay:master " André Erdmann
2013-04-25 16:44 André Erdmann
2013-03-05 11:27 André Erdmann
2013-01-28 23:54 André Erdmann
2013-01-09 19:15 André Erdmann
2012-08-20 11:16 André Erdmann
2012-08-17 17:26 André Erdmann
2012-08-16 16:28 André Erdmann
2012-08-15 16:14 André Erdmann
2012-08-14 15:24 André Erdmann
2012-08-10 17:54 André Erdmann
2012-08-10 15:16 André Erdmann
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox