[gentoo-commits] proj/pms-test-suite:master commit in: doc/

[gentoo-commits] proj/pms-test-suite:master commit in: doc/

[Michał Górny]

commit:     98fed8f726aaf5d41a59b775f412eeed7a31ee0c
Author:     Michał Górny <mgorny <AT> gentoo <DOT> org>
AuthorDate: Sun Jun 19 10:33:08 2011 +0000
Commit:     Michał Górny <mgorny <AT> gentoo <DOT> org>
CommitDate: Sun Jun 19 10:33:08 2011 +0000
URL:        http://git.overlays.gentoo.org/gitweb/?p=proj/pms-test-suite.git;a=commit;h=98fed8f7

Basic docs on the library format.

---
 doc/library-format.md |   32 ++++++++++++++++++++++++++++++++
 1 files changed, 32 insertions(+), 0 deletions(-)

diff --git a/doc/library-format.md b/doc/library-format.md
new file mode 100644
index 0000000..a2fa056
--- /dev/null
+++ b/doc/library-format.md
@@ -0,0 +1,32 @@
+PMS Test Suite -- test library format
+=====================================
+
+Basic concepts
+--------------
+
+The basic unit of a test suite is a _test case_. A test case forms a single
+Python class. That class is then instantiated into one or more test case
+instances.
+
+A _test case instance_ corresponds to a single actual test. Each instance
+provides functions to:
+
+1. merge any number of files to the test repository,
+2. clean up the system before and after performing the tests,
+3. merge the tests,
+4. check the test results.
+
+A number of tests builds up a test library. Each _test library_ consists of one
+or more Python packages, providing a number of Python modules. Those modules
+shall contain at least the test case classes, and a single `TestLibrary`
+subclass, providing a complete list of test cases.
+
+
+More information
+----------------
+
+1. `pydoc PMSTestSuite.library.library`
+2. `pydoc PMSTestSuite.library.case`
+3. `pydoc PMSTestSuite.library.depend_case`
+
+<!-- vim:se syn=markdown :-->

[gentoo-commits] proj/pms-test-suite:master commit in: doc/

[Michał Górny]

commit:     6de2d47900a83189dcf9b699348c1f9e2d53c251
Author:     Michał Górny <mgorny <AT> gentoo <DOT> org>
AuthorDate: Mon Jun 20 11:32:38 2011 +0000
Commit:     Michał Górny <mgorny <AT> gentoo <DOT> org>
CommitDate: Mon Jun 20 11:32:38 2011 +0000
URL:        http://git.overlays.gentoo.org/gitweb/?p=proj/pms-test-suite.git;a=commit;h=6de2d479

Describe the main library module.

---
 doc/library-format.md |   38 +++++++++++++++++++++++++++++++++++++-
 1 files changed, 37 insertions(+), 1 deletions(-)

diff --git a/doc/library-format.md b/doc/library-format.md
index a2fa056..14d6e00 100644
--- a/doc/library-format.md
+++ b/doc/library-format.md
@@ -22,10 +22,46 @@ shall contain at least the test case classes, and a single `TestLibrary`
 subclass, providing a complete list of test cases.
 
 
+The main library module
+-----------------------
+
+Each library has to supply a _main library module_. Such a module has to exist
+in the `PMSTestSuite.library` namespace. The module name (with that prefix
+stripped) is considered the _test library name_ and is passed as the `-l`
+argument.
+
+For example, `-l foobar` would correspond to a module called
+`PMSTestSuite.library.foobar`.
+
+The main library module has to declare a subclass of
+`PMSTestSuite.library.TestLibrary` class. The subclass has to override
+the `test_names` property with a list of test class names with module paths
+relative to the main module.
+
+Example main module of a test library:
+
+	from PMSTestSuite.library import TestLibrary
+
+	class FoobarTestLibrary(TestLibrary):
+		"""
+		Foo bar?
+		"""
+
+		test_names=[
+			'foo.BarTest',
+			'bar.FooTest'
+		]
+
+The above example would load two tests:
+
+- `BarTest` from `PMSTestSuite.library.foobar.foo`,
+- and `FooTest` from `PMSTestSuite.library.foobar.bar`.
+
+
 More information
 ----------------
 
-1. `pydoc PMSTestSuite.library.library`
+1. `pydoc PMSTestSuite.library`
 2. `pydoc PMSTestSuite.library.case`
 3. `pydoc PMSTestSuite.library.depend_case`
 

[gentoo-commits] proj/pms-test-suite:master commit in: doc/

[Michał Górny]

commit:     bf06933c9d33ee68241646630d85b051e7f5febe
Author:     Michał Górny <mgorny <AT> gentoo <DOT> org>
AuthorDate: Mon Jun 20 19:19:32 2011 +0000
Commit:     Michał Górny <mgorny <AT> gentoo <DOT> org>
CommitDate: Mon Jun 20 19:19:32 2011 +0000
URL:        http://git.overlays.gentoo.org/gitweb/?p=proj/pms-test-suite.git;a=commit;h=bf06933c

Add docs about the TestCase.

---
 doc/library-format.md |   26 +++++++++++++++++++++++++-
 1 files changed, 25 insertions(+), 1 deletions(-)

diff --git a/doc/library-format.md b/doc/library-format.md
index 14d6e00..203c6c4 100644
--- a/doc/library-format.md
+++ b/doc/library-format.md
@@ -58,11 +58,35 @@ The above example would load two tests:
 - and `FooTest` from `PMSTestSuite.library.foobar.bar`.
 
 
+The test class
+--------------
+
+Each _test case_ has to be a subclass of `PMSTestSuite.library.case.TestCase`.
+That class represents a most general test case with an interface described
+above. A particular subclass needs to override the following methods:
+
+- `get_output_files()` returning a dict representing files which should be
+	output to the test repository (keys being filenames relative
+	to the repository root and values being file contents),
+- `clean()` removing any merged test data before and after performing the test,
+- `start()` starting (scheduling) the actual test,
+- `check_result()` checking the test results. It should return True if the test
+	succeeded and False otherwise.
+
+The three latter functions will be passed a `PackageManager` instance, which
+methods `merge()`, `unmerge()` and `lookup_vardb()` can be used. For more
+information, please see the docs of `PMSTestSuite.pm` module.
+
+In most cases, though, you won't be subclassing `TestCase` directly but using
+one of its subclasses defined in the `PMSTestSuite.library` submodules.
+
+
 More information
 ----------------
 
 1. `pydoc PMSTestSuite.library`
 2. `pydoc PMSTestSuite.library.case`
-3. `pydoc PMSTestSuite.library.depend_case`
+3. `pydoc PMSTestSuite.pm.PackageManager`
+4. `pydoc PMSTestSuite.library.depend_case`
 
 <!-- vim:se syn=markdown :-->

[gentoo-commits] proj/pms-test-suite:master commit in: doc/

[Michał Górny]

commit:     91a841278220d7fc3012dd28faec32e1e40b46a1
Author:     Michał Górny <mgorny <AT> gentoo <DOT> org>
AuthorDate: Wed Jun 29 17:35:25 2011 +0000
Commit:     Michał Górny <mgorny <AT> gentoo <DOT> org>
CommitDate: Wed Jun 29 17:35:25 2011 +0000
URL:        http://git.overlays.gentoo.org/gitweb/?p=proj/pms-test-suite.git;a=commit;h=91a84127

Update docs to mention supported_eapis.

---
 doc/ebuild-test-case.md |    8 ++++++--
 1 files changed, 6 insertions(+), 2 deletions(-)

diff --git a/doc/ebuild-test-case.md b/doc/ebuild-test-case.md
index 458dcd4..4078cf1 100644
--- a/doc/ebuild-test-case.md
+++ b/doc/ebuild-test-case.md
@@ -24,7 +24,8 @@ An example use case would be like:
 	class RandomExampleTest(EbuildTestCase):
 		""" An absolutely random test. """
 
-		relevant_eapis = (0, 1, 2, 4)
+		supported_eapis = (2, 3, 4)
+		relevant_eapis = (3, 4)
 		expect_failure = True
 
 		ebuild_vars = {
@@ -43,8 +44,11 @@ Where:
   here),
 - the docstring is used to form the ebuild `DESCRIPTION` (it can be overrode
   using `ebuild_vars['DESCRIPTION']`,
+- `supported_eapis` (_iterable of strings_) specifies for which EAPIs the test
+  gives predictible (useful) results. It defaults to all supported EAPIs,
 - `relevant_eapis` (_iterable of strings_) specifies for which EAPIs the test
-  will be performed,
+  will be performed by default (i.e. for which the results are expected
+  to change). It defaults to a random EAPI from `supported_eapis`,
 - `expect_failure` (_bool_, default: _False_) specifies whether the ebuild is
   supposed to fail merge,
 - `ebuild_vars` (_dict of str -> str_) specifies additional global ebuild