1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
|
---
stage: Data Stores
group: Database
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments
---
# Troubleshooting
DETAILS:
**Tier:** Free, Premium, Ultimate
**Offering:** Self-managed
<!-- Linked from lib/gitlab/database/migrations/batched_background_migration_helpers.rb -->
## Database migrations failing because of batched background migration not finished
When updating to GitLab version 14.2 or later, database migrations might fail with a message like:
```plaintext
StandardError: An error has occurred, all later migrations canceled:
Expected batched background migration for the given configuration to be marked as 'finished', but it is 'active':
{:job_class_name=>"CopyColumnUsingBackgroundMigrationJob",
:table_name=>"push_event_payloads",
:column_name=>"event_id",
:job_arguments=>[["event_id"],
["event_id_convert_to_bigint"]]
}
```
First, check if you have followed the [version-specific upgrade instructions for 14.2](https://docs.gitlab.com/17.3/ee/update/versions/gitlab_14_changes.html#1420).
If you have, you can [manually finish the batched background migration](background_migrations.md#finish-a-failed-migration-manually)).
If you haven't, choose one of the following methods:
1. [Rollback and upgrade](#roll-back-and-follow-the-required-upgrade-path) through one of the required
versions before updating to 14.2+.
1. [Roll forward](#roll-forward-and-finish-the-migrations-on-the-upgraded-version), staying on the current
version and manually ensuring that the batched migrations complete successfully.
### Roll back and follow the required upgrade path
1. [Rollback and restore the previously installed version](../administration/backup_restore/index.md)
1. Update to either 14.0.5 or 14.1 **before** updating to 14.2+
1. [Check the status](background_migrations.md#check-the-status-of-batched-background-migrations) of the batched background migrations and
make sure they are all marked as finished before attempting to upgrade again. If any remain marked as active,
you can [manually finish them](background_migrations.md#finish-a-failed-migration-manually).
### Roll forward and finish the migrations on the upgraded version
#### For a deployment with downtime
To run all the batched background migrations, it can take a significant amount of time
depending on the size of your GitLab installation.
1. [Check the status](background_migrations.md#check-the-status-of-batched-background-migrations) of the batched background migrations in the
database, and [manually run them](background_migrations.md#finish-a-failed-migration-manually) with the appropriate
arguments until the status query returns no rows.
1. When the status of all of all them is marked as complete, re-run migrations for your installation.
1. [Complete the database migrations](../administration/raketasks/maintenance.md#run-incomplete-database-migrations) from your GitLab upgrade:
```plaintext
sudo gitlab-rake db:migrate
```
1. Run a reconfigure:
```plaintext
sudo gitlab-ctl reconfigure
```
1. Finish the upgrade for your installation.
#### For a no-downtime deployment
As the failing migrations are post-deployment migrations, you can remain on a running instance of the upgraded
version and wait for the batched background migrations to finish.
1. [Check the status](background_migrations.md#check-the-status-of-batched-background-migrations) of the batched background migration from
the error message, and make sure it is listed as finished. If it is still active, either wait until it is done,
or [manually finish it](background_migrations.md#finish-a-failed-migration-manually).
1. Re-run migrations for your installation, so the remaining post-deployment migrations finish.
## Background migrations remain in the Sidekiq queue
WARNING:
The following operations can disrupt your GitLab performance. They run a number of Sidekiq jobs that perform various database or file updates.
Run the following check. If it returns non-zero and the count does not decrease over time, follow the rest of the steps in this section.
```shell
# For Linux package installations:
sudo gitlab-rails runner -e production 'puts Gitlab::BackgroundMigration.remaining'
# For self-compiled installations:
cd /home/git/gitlab
sudo -u git -H bundle exec rails runner -e production 'puts Gitlab::BackgroundMigration.remaining'
```
It is safe to re-execute the following commands, especially if you have 1000+ pending jobs which would likely overflow your runtime memory.
::Tabs
:::TabTitle Linux package (Omnibus)
```shell
# Start the rails console
sudo gitlab-rails c
# Execute the following in the rails console
scheduled_queue = Sidekiq::ScheduledSet.new
pending_job_classes = scheduled_queue.select { |job| job["class"] == "BackgroundMigrationWorker" }.map { |job| job["args"].first }.uniq
pending_job_classes.each { |job_class| Gitlab::BackgroundMigration.steal(job_class) }
```
:::TabTitle Self-compiled (source)
```shell
# Start the rails console
sudo -u git -H bundle exec rails RAILS_ENV=production
# Execute the following in the rails console
scheduled_queue = Sidekiq::ScheduledSet.new
pending_job_classes = scheduled_queue.select { |job| job["class"] == "BackgroundMigrationWorker" }.map { |job| job["args"].first }.uniq
pending_job_classes.each { |job_class| Gitlab::BackgroundMigration.steal(job_class) }
```
::EndTabs
## Background migrations stuck in 'pending' state
For background migrations stuck in pending, run the following check. If
it returns non-zero and the count does not decrease over time, follow the rest
of the steps in this section.
```shell
# For Linux package installations:
sudo gitlab-rails runner -e production 'puts Gitlab::Database::BackgroundMigrationJob.pending.count'
# For self-compiled installations:
cd /home/git/gitlab
sudo -u git -H bundle exec rails runner -e production 'puts Gitlab::Database::BackgroundMigrationJob.pending.count'
```
It is safe to re-attempt these migrations to clear them out from a pending status:
::Tabs
:::TabTitle Linux package (Omnibus)
```shell
# Start the rails console
sudo gitlab-rails c
# Execute the following in the rails console
Gitlab::Database::BackgroundMigrationJob.pending.find_each do |job|
puts "Running pending job '#{job.class_name}' with arguments #{job.arguments}"
result = Gitlab::BackgroundMigration.perform(job.class_name, job.arguments)
puts "Result: #{result}"
end
```
:::TabTitle Self-compiled (source)
```shell
# Start the rails console
sudo -u git -H bundle exec rails RAILS_ENV=production
# Execute the following in the rails console
Gitlab::Database::BackgroundMigrationJob.pending.find_each do |job|
puts "Running pending job '#{job.class_name}' with arguments #{job.arguments}"
result = Gitlab::BackgroundMigration.perform(job.class_name, job.arguments)
puts "Result: #{result}"
end
```
::EndTabs
## What do you do if your advanced search migrations are stuck?
In GitLab 15.0, an advanced search migration named `DeleteOrphanedCommit` can be permanently stuck
in a pending state across upgrades. This issue
[is corrected in GitLab 15.1](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/89539).
If you are a self-managed customer who uses GitLab 15.0 with advanced search, you will experience performance degradation.
To clean up the migration, upgrade to 15.1 or later.
For other advanced search migrations stuck in pending, see [how to retry a halted migration](../integration/advanced_search/elasticsearch.md#retry-a-halted-migration).
If you upgrade GitLab before all pending advanced search migrations are completed, any pending migrations
that have been removed in the new version cannot be executed or retried.
In this case, you must
[re-create your index from scratch](../integration/advanced_search/elasticsearch_troubleshooting.md#last-resort-to-recreate-an-index).
## What do you do for the error `Elasticsearch version not compatible`
Confirm that your version of Elasticsearch or OpenSearch is [compatible with your version of GitLab](../integration/advanced_search/elasticsearch.md#version-requirements).
|
