1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
|
There's a booth-test package which contains two types of tests.
It installs the necessary files into `/usr/share/booth/tests`.
=== Live tests (booth operation)
BEWARE: Run this with _test_ clusters only!
The live testing utility tests booth operation using the given
`booth.conf`:
$ /usr/share/booth/tests/test/live_test.sh booth.conf
It is possible to run only specific tests. Run the script without
arguments to see usage and the list of tests and netem network
emulation functions.
There are some restrictions on how booth.conf is formatted.
There may be several tickets defined and all of them will be
tested, one after another (they will be tested separately).
The tickets must have expire and timeout parameters configured.
Example booth.conf:
------------
transport="UDP"
port="9929"
arbitrator="10.2.12.53"
arbitrator="10.2.13.82"
site="10.2.12.101"
site="10.2.13.101"
site="10.121.187.99"
ticket="ticket-A"
expire = 30
timeout = 3
retries = 3
before-acquire-handler = /usr/share/booth/service-runnable d-src1
------------
A split brain condition is also tested. For that to work, all
sites need `iptables` installed. The supplied script `booth_path`
is used to manipulate iptables rules.
==== Pacemaker configuration
This is a sample pacemaker configuration for a single-node
cluster:
primitive booth ocf:pacemaker:booth-site
primitive d-src1 ocf:heartbeat:Dummy
rsc_ticket global-d-src1 ticket-A: d-src1
Additionally, you may also add an ocf:booth:sharedrsc resource to
also check that the ticket is granted always to only one site:
primitive shared ocf:booth:sharedrsc \
params dir="10.2.13.82:/var/tmp/boothtestdir"
rsc_ticket global-shared ticket-A: shared
Please adjust to your environment.
==== Network environment emulation
To introduce packet loss or network delays, set the NETEM_ENV
environment variable. There are currently three netem network
emulation settings supported:
- loss: all servers emulate packet loss (30% by default)
- single_loss: the first site in the configuration emulates
packet loss (30% by default)
- net_delay: all servers emulate packet delay (100ms by default
with random variation of 10%)
The settings can be supplied by adding ':' to the emulator name.
For instance:
# NETEM_ENV=loss:50 /usr/share/booth/tests/test/live_test.sh booth.conf
It is not necessary to run the test script on one of the sites.
Just copy the script and make the test `booth.conf` available
locally:
$ scp testsite:/usr/share/booth/tests/test/live_test.sh .
$ scp testsite:/etc/booth/booth.conf .
$ sh live_test.sh booth.conf
You need at least two sites and one arbitrator.
The configuration can contain just one ticket.
It is not necessary to configure the `before-acquire-handler`.
Notes:
- (BEWARE!) the supplied configuration files is copied to
/etc/booth/booth.conf to all sites/arbitrators thus overwriting
any existing configuration
- the utility uses ssh to manage booth at all sites/arbitrators
and logs in as user `root`
- it is required that ssh public authentication works without
providing the passphrase (otherwise it is impractical)
- the log file is ./test_booth.log (it is actually a shell trace,
with timestamps if you're running bash)
- in case one of the tests fail, hb_report is created
If you want to open a bug report, please attach all hb_reports
and `test_booth.log`.
=== Simple tests (commandline, config file)
Run (as non-root)
# make check
or
# make test/runtests.py
# python test/runtests.py
to run the tests written in python.
It is also possible to run the tests as a root when
"--allow-root-user" parameter is used or if
the BOOTH_RUNTESTS_ROOT_USER environment variable is defined.
By default tests uses TCP port based on current PID in range
from 9929 to 10937 to allow running multiple instances in parallel.
It is possible to use "--single-instance" parameter or define
BOOTH_RUNTESTS_SINGLE_INSTANCE environment variable to make tests use
only single port (9929), but parallel instances will fail.
# vim: set ft=asciidoc :
|
