1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
|
Benchmarks concepts
=====================
The same FunkLaod test can be turned into a load test, just by invoking the
bench runner ``fl-run-bench``.
Page
~~~~
A page is an http get/post request with associated sub requests like
redirects, images or links (css, js files). This is what users see as a
single page.
Note that an xmlrpc call or a put/delete is also taken in account as a page.
Test
~~~~
A test is made with 3 methods: setUp/test_name/tearDown. During the test_name
method each get/post request is called a page.
::
[setUp][page 1] [page 2] ... [page n] [tearDown]
======================================================> time
<----------------------------------> test method
<--> sleeptime_min to sleeptime_max
<-----> page 1 connection time
Cycle
~~~~~
A cycle is a load of n concurrents test during a 'duration' period.
Threads are launched every 'startupdelay' seconds, each thread executes
test in a loop.
Once all threads have been started we start to record stats.
Only tests that end during the 'duration' period are taken into account
for the test stats (in the representation below test like [---X are not
take into account).
Only pages and requests that finish during the 'duration' are taken into
account for the request and pages statistic
Before a cycle a setUpCycle method is called, after a cycle a tearDownCycle
method is called, you can use these methods to test differents server
configuration for each cycle.
::
Threads
^
|
|
|n [---test--] [--------] [--|---X
|...
| | |
|2 [------|--] [--------] [-------] |
| | |
|1 [------X | [--------] [-------] [--|--X
| | |
|[setUpCycle] | | [tearDownCycle]
===========================================================> time
<------ cycle duration ------->
<----- staging -----> <---- staging ----->
<-> startupdelay <---> sleeptime
Cycles
~~~~~~
FunkLoad_ can execute many cycles with different number of CUs
(Concurrent Users), this way you can find easily the maximum number of
users that your application can handle.
Running n cycles with the same CUs is a good way to see how the application
handles a writing test over time.
Running n cycles with the same CUs with a reading test and a setUpCycle that
change the application configuration will help you to find the right tuning.
::
cvus = [n1, n2, ...]
Threads
^
|
|
|n2 __________
| / \
| / \
|n1 _________ / \
| / \ / \
| / \ / \
| / \ / \
==================================================> time
<-------> duration <-------->
<-----> cycle sleep time
Tips
~~~~~
Here are few remarks/advices to obtain workable metrics.
* Since it may use significant CPU resources, make sure that
performance limits are not hit by FunkLoad before your server's
limit is reached. Check this by launching a bench from another host.
* Having a cycle with one user gives a usefull reference.
* Run multiple cycles for a bench.
* A bench is composed of a benching test (or scenario) run many
times. A good benching test should not be too long so you have a
higher testing rate (that is, more benching tests can come to their
end).
* The cycle duration for the benching test should be long enough.
Around 5 times the duration of a single benching test is a value
that is usually a safe bet. You can obtain this duration of a single
benching test by running ``fl-run-test myfile.py
MyTestCase.testSomething``.
Rationale : Normally a cycle duration of a single benching test
should be enough. But from the testing platform side if there are
more than one concurrent user, there are many threads to start and
it takes some time. And on from the tested platform side it is
common that a benching test will last longer and longer as the
server is used by more and more users.
* You should use many cycles with the same step interval to produce
readable charts (1:10:20:30:40:50:60 vs 1:10:100)
* A benching test must have the same number of page and in the same
order.
* Use a Makefile to make reproductible bench.
* There is no debug option while doing a bench (since this would be
illegible with all the threads). So, if a bench fails (that is using
`fl-run-bench`), use ``fl-run-test -d`` to debug.
.. _FunkLoad: http://funkload.nuxeo.org/
|
