1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180
|
# Building multi-node KAS environment using Omnibus
This document describes how to build multi-node KAS environment using Omnibus.
It is mainly for testing purpose.
[[_TOC_]]
## Overview
It creates total of five VM instances in GCP (Google Cloud Platform).
- Two nodes with GitLab monolith + KAS
- One database node
- One Redis node
- One Gitaly node
## Building the environment
### Create VM instances
Go to GCP and create total of five instances.
We can follow steps described in [Installing GitLab on Google Cloud Platform](https://docs.gitlab.com/ee/install/google_cloud_platform/).
In this document, we use Debian as an OS installed in the VM instances.
- `n2-standard-2` for machine type
- `50GB` for boot disk size
Next, let's create new firewall rule to allow all ports.
Go to "Create firewall rule" and set the following:
- `0.0.0.0/0` for Source IP ranges
- `tcp:0-65535` for Protocols and ports
- Ingress for Direction of traffic
After creating the firewall rule, update the VM instances to allow all ports.
Go to "Edit" and add the firewall rule to the "Network tags".
Now, all the VM instances can communicate with each other.
### Database node setup
Follow these steps to setup PostgreSQL instance: [Standalone PostgreSQL using the Linux package](https://docs.gitlab.com/ee/administration/reference_architectures/2k_users.html#standalone-postgresql-using-the-linux-package).
### Redis node setup
Follow [Standalone Redis using the Linux package](https://docs.gitlab.com/ee/administration/reference_architectures/2k_users.html#standalone-redis-using-the-linux-package)
### Gitaly node
Follow [Configure Gitaly](https://docs.gitlab.com/ee/administration/reference_architectures/2k_users.html#configure-gitaly).
### GitLab monolith + KAS nodes setup
First, we follow [Configure GitLab Rails](https://docs.gitlab.com/ee/administration/reference_architectures/2k_users.html#configure-gitlab-rails)
and install Omnibus package in the two nodes.
#### `gitlab.rb` configuration
Create `/etc/gitlab/gitlab.rb` file and copy the configurations below.
This is an example setup used for verifying KAS->KAS TLS communication.
You need to replace the host and password for the database and Redis:
- `gitlab_rails['db_host']`
- `gitlab_rails['db_password']`
- `gitlab_rails['redis_host']`
- `gitlab_rails['redis_password']`
- `gitlab_kas['api_secret_key']`
- `gitlab_kas['private_api_secret_key']`
- URLs from `gitlab.example.com` to your domain.
```ruby
external_url 'https://gitlab.example.com'
## PostgreSQL connection details
gitlab_rails['db_adapter'] = 'postgresql'
gitlab_rails['db_encoding'] = 'unicode'
gitlab_rails['db_host'] = '<Database node IP address>' # IP/hostname of database server
gitlab_rails['db_password'] = '<Database password>'
### GitLab Redis settings
gitlab_rails['redis_port'] = '6379'
gitlab_rails['redis_host'] = '<Redis node IP address>' # IP/hostname of Redis server
gitlab_rails['redis_password'] = '<Redis password>'
##! Settings used by the GitLab application
gitlab_rails['gitlab_kas_enabled'] = true
gitlab_rails['gitlab_kas_external_url'] = 'wss://gitlab.example.com/-/kubernetes-agent/'
gitlab_rails['gitlab_kas_internal_url'] = 'grpc://gitlab.example.com:8153'
gitlab_rails['gitlab_kas_external_k8s_proxy_url'] = 'https://gitlab.example.com/-/kubernetes-agent/k8s-proxy/'
##! Define to enable GitLab KAS
gitlab_kas_external_url "wss://gitlab.example.com/-/kubernetes-agent/"
gitlab_kas['enable'] = true
##! Shared secret used for authentication between KAS and GitLab
gitlab_kas['api_secret_key'] = '<32_bytes_long_base64_encoded_value>'
##! Shared secret used for authentication between different KAS instances in a multi-node setup
gitlab_kas['private_api_secret_key'] = '<32_bytes_long_base64_encoded_value>'
##! Listen configuration for GitLab KAS
gitlab_kas['listen_address'] = '0.0.0.0:8150'
gitlab_kas['internal_api_listen_address'] = '0.0.0.0:8153'
gitlab_kas['private_api_listen_address'] = '0.0.0.0:8155'
gitlab_kas['private_api_certificate_file'] = "/etc/gitlab/ssl/gitlab.example.com.crt" # Necessary for KAS->KAS TLS communication
gitlab_kas['private_api_key_file'] = "/etc/gitlab/ssl/gitlab.example.com.key" # Necessary for KAS->KAS TLS communication
gitlab_kas['log_level'] = 'debug'
##! Environment variables for GitLab KAS
gitlab_kas['env'] = {
'SSL_CERT_DIR' => "/etc/gitlab/ssl",
'OWN_PRIVATE_API_URL' => 'grpcs://<IP address of this instance>:8155', # Use `grpcs` for KAS->KAS TLS communication
'OWN_PRIVATE_API_HOST' => 'gitlab.example.com',
'GRPC_GO_LOG_VERBOSITY_LEVEL' => '99',
'GRPC_GO_LOG_SEVERITY_LEVEL' => 'debug'
}
##! Redis settings for GitLab KAS
gitlab_kas['redis_host'] = '<Redis node IP address>'
gitlab_kas['redis_port'] = '6379'
gitlab_kas['redis_password'] = '<Redis password>'
# Gitaly
gitlab_rails['gitaly_token'] = 'gitaly-secret'
gitlab_shell['secret_token'] = 'secret-token'
git_data_dirs({
'default' => { 'gitaly_address' => 'tcp://<IP address>' },
})
gitaly['enable'] = false
```
For more information, refer to configurations for Rails from [Configure GitLab Rails](https://docs.gitlab.com/ee/administration/reference_architectures/2k_users.html#configure-gitlab-rails)
and [KAS: Enable on multiple nodes](https://docs.gitlab.com/ee/administration/clusters/kas.html).
Be noted `OWN_PRIVATE_API_URL` needs to be the IP address or hostname of the node you're configuring.
#### SSL Cert file
Omnibus will create SSL cert files using Let'sEncrypt automatically unless you add `letsencrypt['enable'] = false`.
So, after you set up your first Rails monolith + KAS instance, copy cert files from the first instance
and copy them to the second instance.
```shell
$ gcloud compute scp --zone "us-central1-a" --project "<YOUR-PROJECT>" \
--recurse your-1st-rails-node:/etc/gitlab/ssl ./
$ gcloud compute scp --zone "us-central1-a" --project "<YOUR-PROJECT>" \
--recurse ./ssl your-2nd-rails-node:/etc/gitlab/ssl
```
In the second instance, you can add these settings to disable Let'sEncrypt and use copied cert files.
This is necessary if you want to use the same domain for multiple instances.
Otherwise, Let'sEncrypt will fail to issue a certificate for the same domain.
```ruby
letsencrypt['enable'] = false
nginx['ssl_certificate'] = "/etc/gitlab/ssl/gitlab.example.com.crt"
nginx['ssl_certificate_key'] = "/etc/gitlab/ssl/gitlab.example.com.key"
```
## Troubleshooting
### Database connection
If your Gitlab Rails + KAS node fails to talk to the database, then check `/var/opt/gitlab/postgresql/data/pg_hba.conf`
in the database node.
The Gitlab Rails + KAS node IP address needs be allowed to communicate to the database.
### Check Gitaly connection
```shell
$ gitlab-rake gitlab:gitaly:check
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
```
### Check generated KAS config
The generated KAS config is located at `/var/opt/gitlab/gitlab-kas/gitlab-kas-config.yml` in the KAS node.
|