summaryrefslogtreecommitdiff
path: root/source4/selftest/README
diff options
context:
space:
mode:
Diffstat (limited to 'source4/selftest/README')
-rw-r--r--source4/selftest/README81
1 files changed, 69 insertions, 12 deletions
diff --git a/source4/selftest/README b/source4/selftest/README
index e8e87c8b3f..f8be20a569 100644
--- a/source4/selftest/README
+++ b/source4/selftest/README
@@ -3,15 +3,70 @@
This directory contains test scripts that are useful for running a
bunch of tests all at once.
-===============
-Available tests
-===============
-The available tests are obtained from a script, usually
-selftest/samba{3,4}_tests.sh. This script should for each test output
+Available testsuites
+====================
+The available testsuites are obtained from a script, usually
+selftest/samba{3,4}_tests.sh. This script should for each testsuite output
the name of the test, the command to run and the environment that should be
-provided.
+provided. Use the included "plantest" function to generate the required output.
+
+Testsuite behaviour
+================================
+
+Exit code
+------------
+The testsuites should exit with a non-zero exit code if at least one
+test failed. Skipped tests should not influence the exit code.
+
+Output format
+-------------
+Testsuites can simply use the exit code to indicate whether all of their
+tests have succeeded or one or more have failed. It is also possible to
+provide more granular information using the Subunit protocol.
+
+This protocol works by writing simple messages to standard output. Any
+messages that can not be interpreted by this protocol are considered comments
+for the last announced test.
+
+Accepted commands are:
+
+test
+~~~~~~~~~~~~
+test: <NAME>
+
+Announce that a new test with the specified name is starting
+
+success
+~~~~~~~~~~~~~~~
+success: <NAME>
+
+Announce that the test with the specified name is done and ran successfully.
+
+failure
+~~~~~~~~~~~~~~~
+failure: <NAME>
+failure: <NAME> [ REASON ]
+
+Announce that the test with the specified name failed. Optionally, it is
+possible to specify a reason it failed.
+
+skip
+~~~~~~~~~~~~
+skip: <NAME>
+skip: <NAME> [ REASON ]
+
+Announce that the test with the specified name was skipped. Optionally a
+reason can be specified.
+
+knownfail
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+knownfail: <NAME>
+knownfail: <NAME> [ REASON ]
+
+Announce that the test with the specified name was run and failed as expected.
+Alternatively it is also possible to simply return "failure:" here but
+specify in the samba4-knownfailures file that it is failing.
-============
Environments
============
Tests often need to run against a server with particular things set up,
@@ -23,6 +78,7 @@ The following environments are currently available:
- none: No server set up, no variables set.
- dc: Domain controller set up. The following environment variables will
be set:
+
* USERNAME: Administrator user name
* PASSWORD: Administrator password
* DOMAIN: Domain name
@@ -34,6 +90,7 @@ The following environments are currently available:
- member: Domain controller and member server that is joined to it set up. The
following environment variables will be set:
+
* USERNAME: Domain administrator user name
* PASSWORD: Domain administrator password
* DOMAIN: Domain name
@@ -41,22 +98,22 @@ The following environments are currently available:
* SERVER: Name of the member server
-=============
Running tests
=============
-To run all the tests use:
+To run all the tests use::
make test
-To run a quick subset (aiming for about 1 minute of testing) run:
+To run a quick subset (aiming for about 1 minute of testing) run::
make quicktest
-To run a specific test, use this syntax
+To run a specific test, use this syntax::
make test TESTS=testname
-for example
+for example::
make test TESTS=samba4.BASE-DELETE
+