1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
|
# FAQ for package maintainers
## Why should I use autopkgtest when my build runs the test suite?
autopkgtest is intended to test packages in their *installed* state, and that
is not the same as testing the code that is in the source tree. During the build,
the tests run against the code in the source tree. Depending on the package,
that could be enough, but *also* running those same tests against the installed
package can catch a host of problems that can't be caught when running against
the source tree, e.g.:
* ABI breakages and other types of problems that are "magically" fixed with a
rebuild.
* Missing dependencies (as opposed to build dependencies)
* Some files that are essential for the software to function correctly were not
included in the binary package.
Another practical reason to have autopkgtest tests in your package is that
debci runs autopkgtest more frequently than packages are built. For example,
when any of your dependencies is updated, your package is tested again against
that new version.
Now, besides running tests that normally run during the package build under
autopkgtest as well, it is also a good idea to add tests that are specific to
the package as installed and are more "black box", i.e. tests that don't assume
or have access to implementation details of the package (those are usually
called integration tests). For example:
* if a package provides a service, add tests that exercises that service as a
client, and check the expected results of that interaction.
* if a package provides a program, add tests that check that the invocations of
that program with different parameters and inputs produce the corresponding
expected results.
This type of test can also, and most often that not should, be contributed
upstream.
## If autopkgtest can rebuild the package, why won't debci do that?
The primary reason is because we want to test the packages that are actually in
the archive, because that is what our users get from us.
Weaker, but practical reasons, are:
* Rebuilding every package would add significant load to the system.
* The buildd network already builds packages and we don't want to duplicate
that function.
## How can I reproduce the test run locally?
**NOTE:** if you intend to run tests frequently, you should consider installing
`apt-cacher-ng` before anything else. `debci` will notice the running proxy and
will setup the testbed to use it, so you won't have to wait for the download of
each package more than once.
### Using no virtualization backend
This is the fastest way to run tests, but does not reproduce the continuous
integration environment faithfully and your local environment may make the
results unreliable. It's useful nevertheless when initially writing the tests,
or for quick runs against your main host system.
To run the test suite from **the root of a source package** against the
currently installed packages, run:
```
$ autopkgtest --output-dir /tmp/output-dir -B . -- null
```
For more details, see the documentation for the `autopkgtest` package.
### Using the lxc backend
The first step is installing debci and autopkgtest:
```
$ sudo apt install debci autopkgtest
```
You will also need permissions to run the `lxc-*` commands as root, preserving
your environment. An easy way to do that is to add yourself to the debci group.
```
$ sudo adduser YOUR_USERNAME debci
```
You're now ready to create the lxc container:
```
$ sudo debci setup
```
This might take a few minutes since it will create a fresh container from
scratch.
You can use an alternative mirror using, for example:
```
$ sudo env debci_mirror=http://my.local.mirror/debian debci setup
```
Now to actually run tests, we'll use `autopkgtest` directly. The following
examples assume your architecture is amd64, replace it by your actual
architecture if that is not the case.
To run the test suite **from a source package in the archive**, just pass the
_source package name_ to autopkgtest:
```
$ autopkgtest --user debci --output-dir /tmp/output-dir SOURCEPACKAGE \
-- lxc --sudo autopkgtest-unstable-amd64
```
To run the test suite against **a locally-built source package**, using the
test suite from that source package and the binary packages you just built, you
can pass the `.changes` file to autopkgtest:
```
$ autopkgtest --user debci --output-dir /tmp/output-dir \
/path/to/PACKAGE_x.y-z_amd64.changes \
-- lxc --sudo autopkgtest-unstable-amd64
```
For more details, see the documentation for the `autopkgtest` package.
## How do I get my package to have its test suite executed?
Test suites must be included in source packages as defined in
the [DEP-8 specification](http://dep.debian.net/deps/dep8/). In short.
* The fact that the package has a test suite must be declared in the source
control file. With dpkg-source version 1.17.11 or later this is added
automatically when debian/tests/control is present.
* if the package is built with dpkg earlier than 1.17.11, you need to
add an `Testsuite: autopkgtest` entry to the source stanza in
`debian/control`.
* if the package is built with dpkg earlier than 1.17.6, you need to use
`XS-Testsuite: autopkgtest` instead.
* tests are declared in `debian/tests/control`.
Please refer to the DEP-8 spec for details on how to declare your tests.
## How exactly is the test suite executed?
Test suites are executed by
[autopkgtest](http://packages.debian.org/autopkgtest). The version of
autopkgtest used to execute the tests is shown in the log file for each test
run.
## How often are test suites executed?
The test suite for a source package will be executed:
* when any package in the dependency chain of its binary packages changes;
* when the package itself changes;
* when 1 month is passed since the test suite was run for the last time.
[edit this page](https://salsa.debian.org/ci-team/debci/edit/master/docs/MAINTAINERS.md)
|
