1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
|
# Building multi-node KAS environment using Omnibus
This document describes how to build multi-node KAS environment using Omnibus.
It is mainly for testing purpose.
[[_TOC_]]
## Overview
It creates total of five VM instances in GCP (Google Cloud Platform).
- Two nodes with GitLab monolith + KAS
- One database node
- One Redis node
- One Gitaly node
## Building the environment
### Create VM instances
Go to GCP and create total of five instances.
We can follow steps described in [Installing GitLab on Google Cloud Platform](https://docs.gitlab.com/ee/install/google_cloud_platform/).
In this document, we use Debian as an OS installed in the VM instances.
- `n2-standard-2` for machine type
- `50GB` for boot disk size
Next, let's create new firewall rule to allow all ports.
Go to "Create firewall rule" and set the following:
- `0.0.0.0/0` for Source IP ranges
- `tcp:0-65535` for Protocols and ports
- Ingress for Direction of traffic
After creating the firewall rule, update the VM instances to allow all ports.
Go to "Edit" and add the firewall rule to the "Network tags".
Now, all the VM instances can communicate with each other.
### Database node setup
Follow these steps to setup PostgreSQL instance: [Standalone PostgreSQL using the Linux package](https://docs.gitlab.com/ee/administration/reference_architectures/2k_users.html#standalone-postgresql-using-the-linux-package).
### Redis node setup
Follow [Standalone Redis using the Linux package](https://docs.gitlab.com/ee/administration/reference_architectures/2k_users.html#standalone-redis-using-the-linux-package)
### Gitaly node
Follow [Configure Gitaly](https://docs.gitlab.com/ee/administration/reference_architectures/2k_users.html#configure-gitaly).
### GitLab monolith + KAS nodes setup
First, we follow [Configure GitLab Rails](https://docs.gitlab.com/ee/administration/reference_architectures/2k_users.html#configure-gitlab-rails)
and install Omnibus package in the two nodes.
#### `gitlab.rb` configuration
Create `/etc/gitlab/gitlab.rb` file and copy the configurations below.
This is an example setup used for verifying KAS->KAS TLS communication.
You need to replace the host and password for the database and Redis:
- `gitlab_rails['db_host']`
- `gitlab_rails['db_password']`
- `gitlab_rails['redis_host']`
- `gitlab_rails['redis_password']`
- `gitlab_kas['api_secret_key']`
- `gitlab_kas['private_api_secret_key']`
- URLs from `gitlab.example.com` to your domain.
```ruby
external_url 'https://gitlab.example.com'
## PostgreSQL connection details
gitlab_rails['db_adapter'] = 'postgresql'
gitlab_rails['db_encoding'] = 'unicode'
gitlab_rails['db_host'] = '<Database node IP address>' # IP/hostname of database server
gitlab_rails['db_password'] = '<Database password>'
### GitLab Redis settings
gitlab_rails['redis_port'] = '6379'
gitlab_rails['redis_host'] = '<Redis node IP address>' # IP/hostname of Redis server
gitlab_rails['redis_password'] = '<Redis password>'
##! Settings used by the GitLab application
gitlab_rails['gitlab_kas_enabled'] = true
gitlab_rails['gitlab_kas_external_url'] = 'wss://gitlab.example.com/-/kubernetes-agent/'
gitlab_rails['gitlab_kas_internal_url'] = 'grpc://gitlab.example.com:8153'
gitlab_rails['gitlab_kas_external_k8s_proxy_url'] = 'https://gitlab.example.com/-/kubernetes-agent/k8s-proxy/'
##! Define to enable GitLab KAS
gitlab_kas_external_url "wss://gitlab.example.com/-/kubernetes-agent/"
gitlab_kas['enable'] = true
##! Shared secret used for authentication between KAS and GitLab
gitlab_kas['api_secret_key'] = '<32_bytes_long_base64_encoded_value>'
##! Shared secret used for authentication between different KAS instances in a multi-node setup
gitlab_kas['private_api_secret_key'] = '<32_bytes_long_base64_encoded_value>'
##! Listen configuration for GitLab KAS
gitlab_kas['listen_address'] = '0.0.0.0:8150'
gitlab_kas['internal_api_listen_address'] = '0.0.0.0:8153'
gitlab_kas['private_api_listen_address'] = '0.0.0.0:8155'
gitlab_kas['private_api_certificate_file'] = "/etc/gitlab/ssl/gitlab.example.com.crt" # Necessary for KAS->KAS TLS communication
gitlab_kas['private_api_key_file'] = "/etc/gitlab/ssl/gitlab.example.com.key" # Necessary for KAS->KAS TLS communication
gitlab_kas['log_level'] = 'debug'
##! Environment variables for GitLab KAS
gitlab_kas['env'] = {
'SSL_CERT_DIR' => "/etc/gitlab/ssl",
'OWN_PRIVATE_API_URL' => 'grpcs://<IP address of this instance>:8155', # Use `grpcs` for KAS->KAS TLS communication
'OWN_PRIVATE_API_HOST' => 'gitlab.example.com',
'GRPC_GO_LOG_VERBOSITY_LEVEL' => '99',
'GRPC_GO_LOG_SEVERITY_LEVEL' => 'debug'
}
##! Redis settings for GitLab KAS
gitlab_kas['redis_host'] = '<Redis node IP address>'
gitlab_kas['redis_port'] = '6379'
gitlab_kas['redis_password'] = '<Redis password>'
# Gitaly
gitlab_rails['gitaly_token'] = 'gitaly-secret'
gitlab_shell['secret_token'] = 'secret-token'
git_data_dirs({
'default' => { 'gitaly_address' => 'tcp://<IP address>' },
})
gitaly['enable'] = false
```
For more information, refer to configurations for Rails from [Configure GitLab Rails](https://docs.gitlab.com/ee/administration/reference_architectures/2k_users.html#configure-gitlab-rails)
and [KAS: Enable on multiple nodes](https://docs.gitlab.com/ee/administration/clusters/kas.html).
Be noted `OWN_PRIVATE_API_URL` needs to be the IP address or hostname of the node you're configuring.
#### SSL Cert file
Omnibus will create SSL cert files using Let'sEncrypt automatically unless you add `letsencrypt['enable'] = false`.
So, after you set up your first Rails monolith + KAS instance, copy cert files from the first instance
and copy them to the second instance.
```shell
$ gcloud compute scp --zone "us-central1-a" --project "<YOUR-PROJECT>" \
--recurse your-1st-rails-node:/etc/gitlab/ssl ./
$ gcloud compute scp --zone "us-central1-a" --project "<YOUR-PROJECT>" \
--recurse ./ssl your-2nd-rails-node:/etc/gitlab/ssl
```
In the second instance, you can add these settings to disable Let'sEncrypt and use copied cert files.
This is necessary if you want to use the same domain for multiple instances.
Otherwise, Let'sEncrypt will fail to issue a certificate for the same domain.
```ruby
letsencrypt['enable'] = false
nginx['ssl_certificate'] = "/etc/gitlab/ssl/gitlab.example.com.crt"
nginx['ssl_certificate_key'] = "/etc/gitlab/ssl/gitlab.example.com.key"
```
## Troubleshooting
### Database connection
If your Gitlab Rails + KAS node fails to talk to the database, then check `/var/opt/gitlab/postgresql/data/pg_hba.conf`
in the database node.
The Gitlab Rails + KAS node IP address needs be allowed to communicate to the database.
### Check Gitaly connection
```shell
$ gitlab-rake gitlab:gitaly:check
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
```
### Check generated KAS config
The generated KAS config is located at `/var/opt/gitlab/gitlab-kas/gitlab-kas-config.yml` in the KAS node.
|
