1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
|
==============================================================================
CFEngine testsuite
==============================================================================
CFEngine has an extensive testsuite which covers lot of functionality which can
be tested as a series of cf-agent runs.
You are encouraged to run this testsuite on any new software/hardware
configuration, in order to
* verify that CFEngine functions correctly
* provide developers with reproducible way to fix any problems encountered in
new configurations / environment
In case you find a bug you are encouraged to create tests in format of testsuite
which demonstrate bug found, so the test could be added to this testsuite and
checked for in the future.
------------------------------------------------------------------------------
Preparing for running tests
------------------------------------------------------------------------------
* Compile and install CFEngine.
- It is advised to use Tokyo Cabinet as it gives much better performance in
test suite over Berkeley DB.
- You might want to run CFEngine from source tree. Then you need to provide it
with symlink from $workdir/bin/cf-promises to built src/cf-promises, as
cf-agent looks there for cf-promises. Building with --disable-shared in this
case might save a lot of headaches too.
* Make sure there is no failsafe.cf in $workdir/inputs directory, otherwise some
tests will be errorneously marked as failed.
* Make sure user you are running testsuite from has sudo rights. This is
necessary to test functionality which is only available for root user.
------------------------------------------------------------------------------
Running testsuite
------------------------------------------------------------------------------
Currently, all tests are assumed to be run by root. All tests ought only to
create files and directories in /tmp, and ought not to modify other files.
Run
./testall --agent=$workdir/bin/cf-agent
e.g.
./testall --agent=/var/cfengine/bin/cf-agent
Testing will start. For every test case the name and result (failed / passed)
will be produced. At the end testing summary will be provided.
Test runner creates the following log files:
* test.log contains detailed information about each test case (name, standard
output/error contents, return code, and test status).
* summary.log contains summary information, like the one displayed during
testsuite run.
Also a directory .succeeded will be created, containing stamps for each passed
test case, so test cases which passed before and failing in subsequent testsuite
run will be additionally marked in output as "(UNEXPECTED FAILURE)".
You might run a subset of tests by passing either filenames:
./testall --agent=$workdir/bin/cf-agent 01_vars/01_basic/001.cf 01_vars/01_basic/002x.cf
or directories to 'testall':
./testall --agent=$workdir/bin/cf-agent 01_vars
------------------------------------------------------------------------------
Creating/editing test cases
------------------------------------------------------------------------------
Each test should be 100% standalone, and must contain at least 4 main bundles:
init setup, create initial and hoped-for final states
test the actual test code
check the comparison of expected and actual results
fini cleanup, restore to pre-test state
Look in default.cf for some standard check and fini bundles (for example, to
compare files when testing file edits, or for cleaning up temporary files).
Tests should be named with only digits (e.g., "001.cf") unless they are expected
to crash (that is, if they contan syntax errors or other faults), in which case
the filename should end in an 'x' (e.g., "001x.cf").
Tests which use Nova-only features should have a name which ends in an 'n' if
they are expected to pass, and 'nx' if they are expected to crash (e.g.,
"007n.cf" or "007nx.cf").
Tests which are not expected to pass yet (e.g. there is a bug in code which
prevents tests from passing) should be placed in 'staging' subdirectory in the
test directory where they belong. Such test cases will be only run if --staging
argument to ./testall is passed.
Tests which need network connectivity should be placed to 'network'
subdirectories. Those tests may be disabled by passing --no-network option to
'testall'.
NOTE: Since the class 'ok' is used in most tests, never create a persistent
class called 'ok' in any test. Persistent classes are cleaned up between test
runs, but better safe than sorry.
------------------------------------------------------------------------------
Glossary
------------------------------------------------------------------------------
For purposes of testing, here is what our terms mean:
Pass: the test did what we expected (whether that was setting a variable,
editing a file, killing or starting a process, or correctly failing to do
these actions in the light of existing conditions or attributes). Note that
in the case of tests that end in an 'x', a Pass is generated when the test
abnormally terminates and we wanted it to do that.
FAIL: not doing what we wanted: either test finished and returned "FAIL" from
check bundle, or something went wrong - cf-agent might have dropped core,
cf-promises may have denied execution of the promises, etc.
FAILed to crash: test was expected to crash, but did not. This is another kind
of failure, split into separate kind due to low impact.
Skipped: test is skipped due to be either explicitly disabled or being
Nova-specific and being run on Community cf-agent.
|
