1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
|
Reviewing Tempest Code
======================
To start read the `OpenStack Common Review Checklist
<https://docs.openstack.org/infra/manual/developers.html#peer-review>`_
Ensuring code is executed
-------------------------
For any new or change to a test it has to be verified in the gate. This means
that the first thing to check with any change is that a gate job actually runs
it. Tests which aren't executed either because of configuration or skips should
not be accepted.
If a new test is added that depends on a new config option (like a feature
flag), the commit message must reference a change in DevStack or DevStack-Gate
that enables the execution of this newly introduced test. This reference could
either be a `Cross-Repository Dependency <https://docs.openstack.org/infra/
manual/developers.html#cross-repository-dependencies>`_ or a simple link
to a Gerrit review.
Execution time
--------------
While checking in the job logs that a new test is actually executed, also
pay attention to the execution time of that test. Keep in mind that each test
is going to be executed hundreds of time each day, because Tempest tests
run in many OpenStack projects. It's worth considering how important/critical
the feature under test is with how costly the new test is.
Unit Tests
----------
For any change that adds new functionality to either common functionality or an
out-of-band tool unit tests are required. This is to ensure we don't introduce
future regressions and to test conditions which we may not hit in the gate runs.
Tests, and service clients aren't required to have unit tests since they should
be self verifying by running them in the gate.
API Stability
-------------
Tests should only be added for published stable APIs. If a patch contains
tests for an API which hasn't been marked as stable or for an API which
doesn't conform to the `API stability guidelines
<https://wiki.openstack.org/wiki/Governance/Approved/APIStability>`_ then it
should not be approved.
Reject Copy and Paste Test Code
-------------------------------
When creating new tests that are similar to existing tests it is tempting to
simply copy the code and make a few modifications. This increases code size and
the maintenance burden. Such changes should not be approved if it is easy to
abstract the duplicated code into a function or method.
Tests overlap
-------------
When a new test is being proposed, question whether this feature is not already
tested with Tempest. Tempest has more than 1200 tests, spread amongst many
directories, so it's easy to introduce test duplication. For example, testing
volume attachment to a server could be a compute test or a volume test, depending
on how you see it. So one must look carefully in the entire code base for possible
overlap. As a rule of thumb, the older a feature is, the more likely it's
already tested.
Being explicit
--------------
When tests are being added that depend on a configurable feature or extension,
polling the API to discover that it is enabled should not be done. This will
just result in bugs being masked because the test can be skipped automatically.
Instead the config file should be used to determine whether a test should be
skipped or not. Do not approve changes that depend on an API call to determine
whether to skip or not.
Configuration Options
---------------------
With the introduction of the Tempest external test plugin interface we needed
to provide a stable contract for Tempest's configuration options. This means
we can no longer simply remove a configuration option when it's no longer used.
Patches proposed that remove options without a deprecation cycle should not
be approved. Similarly when changing default values with configuration we need
to similarly be careful that we don't break existing functionality. Also, when
adding options, just as before, we need to weigh the benefit of adding an
additional option against the complexity and maintenance overhead having it
costs.
Test Documentation
------------------
When a new test is being added refer to the :ref:`TestDocumentation` section in
hacking to see if the requirements are being met. With the exception of a class
level docstring linking to the API ref doc in the API tests and a docstring for
scenario tests this is up to the reviewers discretion whether a docstring is
required or not.
Test Removal and Refactoring
----------------------------
Make sure that any test that is renamed, relocated (e.g. moved to another
class), or removed does not belong to the `interop`_ testing suite -- which
includes a select suite of Tempest tests for the purposes of validating that
OpenStack vendor clouds are interoperable -- or a project's `whitelist`_ or
`blacklist`_ files.
It is of critical importance that no interop, whitelist or blacklist test
reference be broken by a patch set introduced to Tempest that renames,
relocates or removes a referenced test.
Please check the existence of code which references Tempest tests with:
http://codesearch.openstack.org/
Interop
^^^^^^^
Make sure that modifications to an `interop`_ test are backwards-compatible.
This means that code modifications to tests should not undermine the quality of
the validation currently performed by the test or significantly alter the
behavior of the test.
Removal
^^^^^^^
Reference the :ref:`test-removal` guidelines for understanding best practices
associated with test removal.
.. _interop: https://www.openstack.org/brand/interop
.. _whitelist: https://docs.openstack.org/tempest/latest/run.html#test-selection
.. _blacklist: https://docs.openstack.org/tempest/latest/run.html#test-selection
Release Notes
-------------
Release notes are how we indicate to users and other consumers of Tempest what
has changed in a given release. Since Tempest 10.0.0 we've been using `reno`_
to manage and build the release notes. There are certain types of changes that
require release notes and we should not approve them without including a release
note. These include but aren't limited to, any addition, deprecation or removal
from the lib interface, any change to configuration options (including
deprecation), CLI additions or deprecations, major feature additions, and
anything backwards incompatible or would require a user to take note or do
something extra.
.. _reno: https://docs.openstack.org/reno/latest/
Deprecated Code
---------------
Sometimes we have some bugs in deprecated code. Basically, we leave it. Because
we don't need to maintain it. However, if the bug is critical, we might need to
fix it. When it will happen, we will deal with it on a case-by-case basis.
When to approve
---------------
* Every patch needs two +2s before being approved.
* It's ok to hold off on an approval until a subject matter expert reviews it
* If a patch has already been approved but requires a trivial rebase to merge,
you do not have to wait for a second +2, since the patch has already had
two +2s.
|
