1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
|
=================
OS-Faults + Rally
=================
Combination of OS-Faults and `Rally`_ gives a powerful tool to test OpenStack high availability
and fail-over under the load.
Fault injection is implemented with help of Rally `Fault Injection Hook`_. Following
is an example of Rally scenario performing Keystone authentication with restart of one of
Memcached services:
.. code-block:: yaml
---
Authenticate.keystone:
-
runner:
type: "constant_for_duration"
duration: 30
concurrency: 5
context:
users:
tenants: 1
users_per_tenant: 1
hooks:
-
name: fault_injection
args:
action: restart memcached service on one node
trigger:
name: event
args:
unit: iteration
at: [100]
The moment of fault injection can be specified as iteration number or in time relative
to the beginning of the test:
.. code-block:: yaml
trigger:
name: event
args:
unit: time
at: [10]
Parameter `action` contains fault specification in human-friendly format, see
:ref:`Human API` for details.
More on reliability testing of OpenStack:
* `Reliability Test Plan`_ in OpenStack performance documentation
* `Keystone authentication with restart memcached report`_ collected in OpenStack deployed by Fuel
* `Introduction into reliability metrics`_ video cast
.. references:
.. _Rally: http://rally.readthedocs.io
.. _Fault Injection Hook: http://docs.xrally.xyz/projects/openstack/en/0.10.0/plugins/plugin_reference.html?highlight=fault_injection#fault-injection-hook-action
.. _Reliability Test Plan: https://docs.openstack.org/performance-docs/latest/test_plans/reliability/version_2/plan.html
.. _Keystone authentication with restart memcached report: https://docs.openstack.org/performance-docs/latest/test_results/reliability/version_2/reports/keystone/authenticate_with_restart_memcached_service_on_one_node/index.html
.. _Introduction into reliability metrics: https://youtu.be/MIj4clkKtfY
|
