# Deploying debci to your own infrastructure

This document describes how to install debci for production usage. If you are
trying to setup a development environment, see
{file:docs/HACKING.md How to setup a development environment} instead.

## Architecture

The following picture represents the debci architecture:

![architecture](architecture.svg)

* The whole system communicates using an AMQP server, in this case `rabbitmq`,
  as a message queue.
* Test jobs are injected into a queue by the `debci-enqueue` command line tool,
  or by the web UI/API.
* Test jobs are picked up from the test queue by `debci-worker` daemons. They
  run the tests, and push the results back into a different queue.
* `debci-collector` receives the test results, processes them, feeds the
  database with the results, and produces the HTML pages, atom feeds etc that
  in the web interface.
* Each component (`debci-collector`, `debci-worker`, `rabbitmq`, and the command
  line tools) can live in different hosts if necessary, as long as all of the
  debci components can connect to the `rabbitmq` server.
* You can have as many `debci-worker` nodes in the system as you want, but
  there must be only one `debci-collector`.

## Deployment procedure

Install the debci-collector:

```
$ sudo apt install debci-collector
```

`debci-collector` recommends `rabbitmq-server`, and debci-collector will use a
locally-installed rabbitmq server by default. If you want to use a remote
rabbitmq, you need to add a configuration file with the `.conf` extension to
`/etc/debci/conf.d` with something like this:

```
debci_amqp_server=amqp://MYRABBITMQSERVER
```

Note that if `MYRABBITMQSERVER` is network accessible, it has to have the
proper ACLs configured. Check the rabbitmq documentation for details.

On each worker node, install `apt-cacher-ng` to cache package downloads, and
`debci-worker` itself:

```
$ sudo apt install apt-cacher-ng debci-worker
```

As with `debci-collector`, `debci-worker` will connect to a local rabbitmq by
default. To make it connect to a remote rabbitmq-server you can do the same as
above.

Note that when first installed, `debci-worker` will first build a testbed (a
container or a virtual machine image, depending on the selected backend), and
only after that is finished the worker will be able to start processing test
jobs.

## Submitting test jobs

On any host that can connect to `rabbitmq-server`, first install `debci`:

```
$ sudo apt install debci
```

As usual, you will prompted for the address of the AMQP server. If the
`rabbitmq-server` is on the same host, just leave it blank.

Say you want to run the tests for the `ruby-defauts` package. It is as easy as

```
$ debci enqueue ruby-defaults
```

## Scheduling job submission

By default, `debci` will not submit any jobs. You need to decide how, and how
often, you want to submit jobs. A simple way of doing that is using `cron`.

**Example: schedule tests for a set of packages once a day**

```bash
# /etc/cron.d/debci
0 7 * * * debci debci enqueue rake rubygems-integration ruby-defaults
```

You can also automate calls to `debci enqueue` in any other way you want.

## Multiple debci-worker processes per node

If you have a system which has a lot of CPUs and a large amount of RAM
available you can run multiple debci-worker daemons at once to maximise
the resources being used.

By default there is only a single instance running, however it is possible to
increase this. Example to have 4 simultaneous instances:

```
$ systemctl enable --now debci-worker@{1..4}.service
```

If we want to check on the status of them:

```
$ systemctl status debci-worker@{1..4}.service
$ journalctl -u debci-worker@2.service
```

However, the above only supports the same default backend.
As a starting point to be able to support multiple backends, you could create a
systemd override `/etc/systemd/system/debci-worker@.service.d/override.conf`:

```
[Service]
ExecStart=
ExecStart=/bin/bash -c 'v=%i; b="$${v%%%%_*}"; exec debci worker --backend "$$b"'
```

Then disable any previously created debci-workers, before creating any new ones:

```
$ systemctl --type=service \
  | awk '/debci-worker@/ {print $1}' \
  | while read x; do
    systemctl disable --now $x
  done
$ systemctl enable --now debci-worker@lxc_{1..4}.service
$ systemctl enable --now debci-worker@qemu_{1..2}.service
```

It is possible to start/stop all debci-workers at once rather than individually by:

```
$ systemctl enable --now debci-worker
```

Recap of services:

* `debci-worker.service` which is all debci-worker's, allowing them all to be
  controlled at once
* `debci-worker.service@X.service` is the individual daemons *(replacing X with
  the instant)*

## Setting up backends

debci supports various different backends such as lxc and qemu. As the
default is `lxc`, it will be recommend with `debci-worker`:

```
$ sudo apt install debci-worker
```

Upon first install, `debci-worker` will first build a testbed based on the
selected backend (environment variable: `debci_backend`). It is possible to
create additional testbeds powered by different backends by doing:

```
$ sudo -u debci debci setup --backend qemu
```

For another example see {file:docs/MAINTAINERS.md MAINTAINERS.md} and for
more information about fake backend see {file:docs/HACKING.md HACKING.md}.

## Setting up PostgreSQL database

debci requires a database, and supports either: PostgreSQL and SQLite. The
default is SQLite which is stored in `data/debci.sqlite3`. However the database
can be changed using the environment variable, `database_url`.

First, make sure PostgreSQL is installed by:

```
$ sudo apt install postgresql
```

Afterwards, write a file to debci's configuration directory, such as
`/etc/debci/conf.d/database.conf`, with the following contents to use `debci`
as the database name:

```
debci_database_url='postgresql:///debci'
```

After switching, you may wish to initialize the database:

```
$ sudo -u debci debci migrate
```

*Note, this does not copy any data, but creates fresh.*

Check the PostgreSQL documentation for securing the service.

## Accessing Self Service (Web UI)

It is possible to access `debci-collector` web interface to perform certain actions.
The authentication uses [Debian Salsa](https://salsa.debian.org/) (which is a
[self-hosted GitLab](https://wiki.debian.org/gitlab/) instance) for OAuth.

First, create a [new application](https://salsa.debian.org/oauth/applications)
with the necessary values:

* Name: Example Project
* Redirect URI: `https://ci.example.org/user/auth/gitlab/callback` (Replace:
  `ci.example.org` with your public domain)
* Scope: `read_user`

Afterwards, write a file to debci's configuration directory, such as
`/etc/debci/conf.d/oauth.conf`, with the following contents:

* debci's environment variable salsa_client_id: GitLab's Application ID
* debci's environment variable salsa_client_secret: GitLab's Secret

Below is an example format:

```
debci_salsa_client_id="0000000000000000000000000000000000000000000000000000000000000000"
debci_salsa_client_secret="0000000000000000000000000000000000000000000000000000000000000000"
```

You will then need to restart the `debci-collector` service to reload the
configuration changes.

## Creating API tokens

It is possible to self-create a single API token from `debci-collector`'s
web UI. Alternatively, the command line can be used, which also gives more
options such as being able to specify the username:

```
$ sudo -u debci debci api
[...]
$ sudo -u debci debci api setkey USER1
00000000-0000-0000-0000-000000000000
```

*Note, it is not possible at this stage to revoke tokens via `debci api`
without manually altering the database.*