1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
|
---
title: "Coordinating database migrations"
layout: default
canonical: "/puppetdb/latest/migration_coordination.html"
---
# Coordinating database migrations
[config]: ./configure.markdown
By default at startup PuppetDB will attempt to perform any database
updates that might be needed. If there is a possibiity that multiple
PuppetDB instances could run at the same time, it's important to
coordinate the update process so that only one server attempts to perform
updates. The [configuration documentation][config#coordinating-database-migrations]
explains how to arrange that, and the broader context, but not the
detailed process, which is described here.
Assuming all of the PuppetDB instances have been configured
appropriately, then only one will have `[database] migrate` set to
true, so that it will be the only one that should attempt a migration.
That will guard against concurrent migration attempts, but doesn't
help if the servers are misconfigured, and it doesn't do anything to
protect against another concern, the possibility that PuppetDB
instances might try to operate on a database that's at the incorrect
migration level during normal operations.
This might happen because, for example, a newer version of PuppetDB
has migrated the database to a version they don't understand, or
because they're not configured to perform a migration and the database
is too old. And note that these concerns apply all of the time, not
just at startup, since the database might accidentally be migrated
while older PuppetDB instances are still running.
In an attempt to guard against all of these possibilities, PuppetDB
does the following:
When acting as a migrator (`migrate = true`)
-------------------------------------------------
* Connects to the database as the
[migrator-username][config#migrator-username] using a completely
independent connection pool.
* Begins a transaction.
* Grabs an exclusive migration lock.
* Checks to see if any migrations are required, and if not, ends the
transaction, releasing the lock, and then proceeds with normal
operations.
* If migrations *are* required, proceeds as follows.
* Revokes connection privileges from the normal
[`database` username][config#database], and if it's different, the
[`read-database` username][config#read-database]. This prevents any
new connections from being established.
* Terminates any existing database connections from the normal
[`database` username][config#database], and if it's different, the
[`read-database` username][config#read-database]. The revocation
above may not affect them.
* [Changes its role](https://www.postgresql.org/docs/11/sql-set-role.html)
to the normal [username][config#username] so that all new database objects
will be owned by the normal user. If nothing else, this ensures
the normal user will be able to drop those partitions during
routine garbage collection (e.g. for reports).
* Performs all the required migrations.
* Commits the transaction, releasing the lock.
* Restores the normal user's connection privileges.
* Resumes normal operations.
All PuppetDB instances, including a migrator after migration
------------------------------------------------------------
* Always connects to the database as the normal
[`database` username][config#database], and if it's different, the
[`read-database` username][config#read-database] using the routine
read and write pools.
* Sets a [HikariCP connectionInitSql](https://github.com/brettwooldridge/HikariCP#infrequently-used)
guard that prevents any new connections to a database that isn't at
the correct migration level by examining the levels in the
schema_migrations table.
* Establishes a [periodic check][config#schema-check-interval] that
will shut PuppetDB down if it detects a database that's either newer
or older than it's prepared to handle. This also makes sure that
PuppetDB won't continue using any existing (pool) connections to the
database (since the `connectionInitSql` check above only prevents
new connections).
When a version mismatch is detected, exits with a with a status of
77 (ASCII `M`) if the migration level is too new, and 109 (ASCII
`m`) if it's too old. (Note: in versions before 6.10 the exit
statuses are only required to be non-zero, not necessarily `m` or
`M`).
|
