From 9062214d0931b1bf6eae2fd4d382d7c5a00b307e Mon Sep 17 00:00:00 2001
From: Jelmer Vernooij <jelmer@samba.org>
Date: Tue, 15 Apr 2008 22:42:52 +0200
Subject: Add some documentation on test behaviour in selftest. (This used to
 be commit e8e9f2cc7b0503e30b211094f452829395a73eec)

---
 source4/selftest/README | 41 +++++++++++++++++++++++++++++------------
 1 file changed, 29 insertions(+), 12 deletions(-)

(limited to 'source4/selftest/README')

diff --git a/source4/selftest/README b/source4/selftest/README
index e8e87c8b3f..fdf04f8278 100644
--- a/source4/selftest/README
+++ b/source4/selftest/README
@@ -3,15 +3,32 @@
 This directory contains test scripts that are useful for running a
 bunch of tests all at once. 
 
-===============
-Available tests
-===============
-The available tests are obtained from a script, usually 
-selftest/samba{3,4}_tests.sh. This script should for each test output 
+Available testsuites
+====================
+The available testsuites are obtained from a script, usually 
+selftest/samba{3,4}_tests.sh. This script should for each testsuite output 
 the name of the test, the command to run and the environment that should be 
-provided.
+provided. Use the included "plantest" function to generate the required output.
+
+Testsuite behaviour
+================================
+
+Exit code
+------------
+The testsuites should exit with a non-zero exit code if at least one 
+test failed. 
+
+Output format
+-------------
+
+test: <NAME>
+success: <NAME> 
+failure: <NAME> 
+error: <NAME>
+skip: <NAME>
+
+failure: <NAME> [ EXTENDED DESCRIPTION ]
 
-============
 Environments
 ============
 Tests often need to run against a server with particular things set up, 
@@ -41,22 +58,22 @@ The following environments are currently available:
 	 * SERVER: Name of the member server
 
 
-=============
 Running tests
 =============
 
-To run all the tests use:
+To run all the tests use::
 
    make test
 
-To run a quick subset (aiming for about 1 minute of testing) run:
+To run a quick subset (aiming for about 1 minute of testing) run::
 
    make quicktest
 
-To run a specific test, use this syntax
+To run a specific test, use this syntax::
 
    make test TESTS=testname
 
-for example
+for example::
 
    make test TESTS=samba4.BASE-DELETE
+
-- 
cgit 


From 8170956b6875cc4826087d06c2573f42bed60a98 Mon Sep 17 00:00:00 2001
From: Jelmer Vernooij <jelmer@samba.org>
Date: Tue, 15 Apr 2008 23:21:43 +0200
Subject: Add some documentation about the expected testsuite output. (This
 used to be commit a24651d649d24589a6c65e7bc4be9c49fdc80b2a)

---
 source4/selftest/README | 46 +++++++++++++++++++++++++++++++++++++++++++---
 1 file changed, 43 insertions(+), 3 deletions(-)

(limited to 'source4/selftest/README')

diff --git a/source4/selftest/README b/source4/selftest/README
index fdf04f8278..f8be20a569 100644
--- a/source4/selftest/README
+++ b/source4/selftest/README
@@ -16,18 +16,56 @@ Testsuite behaviour
 Exit code
 ------------
 The testsuites should exit with a non-zero exit code if at least one 
-test failed. 
+test failed. Skipped tests should not influence the exit code.
 
 Output format
 -------------
+Testsuites can simply use the exit code to indicate whether all of their 
+tests have succeeded or one or more have failed. It is also possible to 
+provide more granular information using the Subunit protocol. 
 
+This protocol works by writing simple messages to standard output. Any 
+messages that can not be interpreted by this protocol are considered comments 
+for the last announced test.
+
+Accepted commands are:
+
+test
+~~~~~~~~~~~~
 test: <NAME>
+
+Announce that a new test with the specified name is starting
+
+success
+~~~~~~~~~~~~~~~
 success: <NAME> 
+
+Announce that the test with the specified name is done and ran successfully.
+
+failure
+~~~~~~~~~~~~~~~
 failure: <NAME> 
-error: <NAME>
+failure: <NAME> [ REASON ]
+
+Announce that the test with the specified name failed. Optionally, it is 
+possible to specify a reason it failed.
+
+skip
+~~~~~~~~~~~~
 skip: <NAME>
+skip: <NAME> [ REASON ]
+
+Announce that the test with the specified name was skipped. Optionally a 
+reason can be specified.
 
-failure: <NAME> [ EXTENDED DESCRIPTION ]
+knownfail
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+knownfail: <NAME>
+knownfail: <NAME> [ REASON ]
+
+Announce that the test with the specified name was run and failed as expected.
+Alternatively it is also possible to simply return "failure:" here but 
+specify in the samba4-knownfailures file that it is failing.
 
 Environments
 ============
@@ -40,6 +78,7 @@ The following environments are currently available:
  - none: No server set up, no variables set.
  - dc: Domain controller set up. The following environment variables will 
    be set:
+
      * USERNAME: Administrator user name
 	 * PASSWORD: Administrator password
 	 * DOMAIN: Domain name
@@ -51,6 +90,7 @@ The following environments are currently available:
 
  - member: Domain controller and member server that is joined to it set up. The
    following environment variables will be set:
+
      * USERNAME: Domain administrator user name
 	 * PASSWORD: Domain administrator password
 	 * DOMAIN: Domain name
-- 
cgit