sccache distributed compilation quickstart
==========================================

This is a quick start guide to getting distributed compilation working with sccache. This guide primarily covers Linux clients.
macOS and Windows clients are supported but have seen significantly less testing.

Get sccache binaries
--------------------

Either [install pre-built sccache binaries](https://github.com/mozilla/sccache#installation), or build sccache locally with the `dist-client` and `dist-server` features enabled:
```
cargo build --release --features="dist-client dist-server"
```

The `target/release/sccache` binary will be used on the client, and the `target/release/sccache-dist` binary will be used on the scheduler and build server.

If you're only planning to use the client, it is enabled by default, so just `cargo install sccache` should do the trick.

Configure a scheduler
---------------------

If you're adding a server to a cluster that has already been set up, skip ahead to [configuring a build server](#configure-a-build-server).

The scheduler is a daemon that manages compile request from clients and parcels them out to build servers. You only need one of these per sccache setup. Currently only Linux is supported for running the scheduler.

Create a scheduler.conf file to configure client/server authentication. A minimal example looks like:
```toml
# The socket address the scheduler will listen on. It's strongly recommended
# to listen on localhost and put a HTTPS server in front of it.
public_addr = "127.0.0.1:10600"

[client_auth]
type = "token"
token = "my client token"

[server_auth]
type = "jwt_hs256"
secret_key = "my secret key"
```

Mozilla build servers will typically require clients to be authenticated with the
[Mozilla identity system](https://github.com/mozilla-iam/mozilla-iam).

To configure for scheduler for this, the `client_auth` section should be as follows
so any client tokens are validated with the Mozilla service:

```
[client_auth]
type = "mozilla"
required_groups = ["group_name"]
```

Where `group_name` is a Mozilla LDAP group. Users will be required to belong to this group to successfully authenticate with the scheduler.

Start the scheduler by running:
```
sccache-dist scheduler --config scheduler.conf
```

Like the local server, the scheduler process will daemonize itself unless `SCCACHE_NO_DAEMON=1` is set. If the scheduler fails to start you may need to set `SCCACHE_LOG=trace` when starting it to get useful diagnostics.

Configure a build server
------------------------

A build server communicates with the scheduler and executes compiles requested by clients. Only Linux is supported for running a build server, but executing cross-compile requests from macOS/Windows clients is supported. You can also run a build server on FreeBSD, please see [distributed sccache on FreeBSD](DistributedFreeBSD.md).

The build server requires [bubblewrap](https://github.com/projectatomic/bubblewrap) to sandbox execution, at least version 0.3.0. Verify your version of bubblewrap *before* attempting to run the server. On Ubuntu 18.10+ you can `apt install bubblewrap` to install it. If you build from source you will need to first install your distro's equivalent of the `libcap-dev` package.

Create a server.conf file to configure authentication, storage locations, network addresses and the path to bubblewrap. A minimal example looks like:
```toml
# This is where client toolchains will be stored.
cache_dir = "/tmp/toolchains"
# The maximum size of the toolchain cache, in bytes.
# If unspecified the default is 10GB.
# toolchain_cache_size = 10737418240
# A public IP address and port that clients will use to connect to this builder.
public_addr = "192.168.1.1:10501"
# The URL used to connect to the scheduler (should use https, given an ideal
# setup of a HTTPS server in front of the scheduler)
scheduler_url = "https://192.168.1.1"

[builder]
type = "overlay"
# The directory under which a sandboxed filesystem will be created for builds.
build_dir = "/tmp/build"
# The path to the bubblewrap version 0.3.0+ `bwrap` binary.
bwrap_path = "/usr/bin/bwrap"

[scheduler_auth]
type = "jwt_token"
# This will be generated by the `generate-jwt-hs256-server-token` command or
# provided by an administrator of the sccache cluster.
token = "my server's token"
```

Due to bubblewrap requirements currently the build server *must* be run as root. Start the build server by running:
```
sudo sccache-dist server --config server.conf
```

As with the scheduler, if the build server fails to start you may need to set `SCCACHE_LOG=trace` to get useful diagnostics.

Configure a client
------------------

A client uses `sccache` to wrap compile commands, communicates with the scheduler to find available build servers, and communicates with build servers to execute the compiles and receive the results.

Clients that are not targeting linux64 require the `icecc-create-env` script or should be provided with an archive. `icecc-create-env` is part of `icecream` for packaging toolchains. You can install icecream to get this script (`apt install icecc` on Ubuntu), or download it from the git repository and place it in your `PATH`: `curl https://raw.githubusercontent.com/icecc/icecream/master/client/icecc-create-env.in > icecc-create-env && chmod +x icecc-create-env`. See [using custom toolchains](#using-custom-toolchains).

Create a client config file in `~/.config/sccache/config` (on Linux), `~/Library/Application Support/Mozilla.sccache/config` (on macOS), or `%APPDATA%\Mozilla\sccache\config\config` (on Windows). A minimal example looks like:
```toml
[dist]
# The URL used to connect to the scheduler (should use https, given an ideal
# setup of a HTTPS server in front of the scheduler)
scheduler_url = "https://192.168.1.1"
# Used for mapping local toolchains to remote cross-compile toolchains. Empty in
# this example where the client and build server are both Linux.
toolchains = []
# Size of the local toolchain cache, in bytes (5GB here, 10GB if unspecified).
toolchain_cache_size = 5368709120

[dist.auth]
type = "token"
# This should match the `client_auth` section of the scheduler config.
token = "my client token"
```

Clients using Mozilla build servers should configure their `dist.auth` section as follows:

```
[dist.auth]
type = "mozilla"
```

And retrieve a token from the Mozilla identity service by running `sccache --dist-auth`
and following the instructions. Completing this process will retrieve and cache a token
valid for 7 days.

Make sure to run `sccache --stop-server` and `sccache --start-server` if sccache was
running before changing the configuration.

You can check the status with `sccache --dist-status`, it should say something like:

```
$ sccache --dist-status
{"SchedulerStatus":["https://sccache1.corpdmz.ber3.mozilla.com/",{"num_servers":3,"num_cpus":56,"in_progress":24}]}
```

Using custom toolchains
-----------------------

Since Windows and macOS cannot automatically package toolchains, it is important to be
able to manually specify toolchains for distribution. This functionality is also available
on Linux.

Using custom toolchains involves adding a `dist.toolchains` section to your client config
file (you can add it multiple times to specify multiple toolchains).

On Linux and macOS:

```
[[dist.toolchains]]
type = "path_override"
compiler_executable = "/home/me/.mozbuild/clang/bin/clang"
archive = "/home/me/.mozbuild/toolchains/33d92fcd79ffef6e-clang-dist-toolchain.tar.xz"
archive_compiler_executable = "/builds/worker/toolchains/clang/bin/clang"
```

On Windows:

```
[[dist.toolchains]]
type = "path_override"
compiler_executable = "C:/clang/bin\\clang-cl.exe"
archive = "C:/toolchains/33d92fcd79ffef6e-clang-dist-toolchain.tar.xz"
archive_compiler_executable = "/builds/worker/toolchains/clang/bin/clang"
```

Where:
 - `compiler_executable` identifies the path that sccache will match against to activate
   this configuration (you need to be careful on Windows - paths can have slashes in both
   directions, and you may need to escape backslashes, as in the example)
 - `archive` is the compressed tar archive containing the compiler toolchain to distribute
   when `compiler_executable` is matched
 - `archive_compiler_executable` is the path within the archive the distributed
   compilation should invoke

A toolchain archive should be a Gzip compressed TAR archive, containing a filesystem
sufficient to run the compiler without relying on any external files. If you have archives
compatible with icecream (created with `icecc-create-env`, like
[these ones](https://github.com/jyavenard/mozilla-icecream) for macOS), they should also work
with sccache. To create a Windows toolchain, it is recommended that you download the [Clang
binaries for Ubuntu 16.04](http://releases.llvm.org/download.html) and extract them,
package up the toolchain using the extracted `bin/clang` file (requires
[PR #321](https://github.com/mozilla/sccache/pull/321)) and then insert `bin/clang-cl` at
the appropriate path as a symlink to the `bin/clang` binary.

Considerations when distributing from macOS
-------------------------------------------

When distributing from a macOS client, additional flags and configuration
may be required:

- An explicit target should be passed to the compiler, for instance by adding
  `--target=x86_64-apple-darwin16.0.0` to your build system's `CFLAGS`.
- An explicit toolchain archive will need to be configured, as described above.
  In case rust is being cached, the same version of `rustc` will need to be used
  for local compiles as is found in the distributed archive.
- The client config will be read from `~/Library/Application Support/Mozilla.sccache/config`,
  not `~/.config/sccache/config`.
- Some cross compilers may not understand some intrinsics used in more recent macOS
  SDKs. The 10.11 SDK is known to work.

Making a build server start at boot time
----------------------------------------

It is very easy with a systemd service to spawn the server on boot.

You can create a service file like `/etc/systemd/system/sccache-server.service`
with the following contents:

```ini
[Unit]
Description=sccache-dist server
Wants=network-online.target
After=network-online.target

[Service]
ExecStart=/path/to/sccache-dist server --config /path/to/server.conf

[Install]
WantedBy=multi-user.target
```

**Note** that if the `sccache-dist` binary is in a user's home directory, and
you're in a distro with SELinux enabled (like Fedora), you may need to use an
`ExecStart` line like:

```ini
ExecStart=/bin/bash -c "/home/<user>/path/to/sccache-dist server --config /home/<user>/path/to/server.conf"
```

This is because SELinux by default prevents services from running binaries in
home directories, for some reason. Using a shell works around that. An
alternative would be to move the `sccache-dist` binary to somewhere like
`/usr/local/bin`, but then you need to remember to update it manually.

After creating that file, you can ensure it's working and enable it by default
like:

```
# systemctl daemon-reload
# systemctl start sccache-server
# systemctl status # And check it's fine.
# systemctl enable sccache-server # This enables the service on boot
```