Upgrading GitLab

Upgrading GitLab is a relatively straightforward process, but the complexity can increase based on the installation method you have used, how old your GitLab version is, if you’re upgrading to a major version, and so on.

Make sure to read the whole page as it contains information related to every upgrade method.

note
Upgrade GitLab to the latest available patch release, for example 13.8.8 rather than 13.8.0. This includes versions you must stop at on the upgrade path as there may be fixes for issues relating to the upgrade process.

The maintenance policy documentation has additional information about upgrading, including:

  • How to interpret GitLab product versioning.
  • Recommendations on the what release to run.
  • How we use patch and security patch releases.
  • When we backport code changes.

Upgrade based on installation method

Depending on the installation method and your GitLab version, there are multiple official ways to update GitLab:

Linux packages (Omnibus GitLab)

The package upgrade guide contains the steps needed to update a package installed by official GitLab repositories.

There are also instructions when you want to update to a specific version.

Installation from source

In the past we used separate documents for the upgrading instructions, but we have since switched to using a single document. The old upgrading guidelines can still be found in the Git repository:

Installation using Docker

GitLab provides official Docker images for both Community and Enterprise editions, and they are based on the Omnibus package. See how to install GitLab using Docker.

Installation using Helm

GitLab can be deployed into a Kubernetes cluster using Helm. Instructions on how to update a cloud-native deployment are in a separate document.

Use the version mapping from the chart version to GitLab version to determine the upgrade path.

Plan your upgrade

See the guide to plan your GitLab upgrade.

Checking for background migrations before upgrading

Certain releases may require different migrations to be finished before you update to the newer version.

Batched migrations are a migration type available in GitLab 14.0 and later. Background migrations and batched migrations are not the same, so you should check that both are complete before updating.

Decrease the time required to complete these migrations by increasing the number of Sidekiq workers that can process jobs in the background_migration queue.

Background migrations

Pending migrations

For Omnibus installations:

sudo gitlab-rails runner -e production 'puts Gitlab::BackgroundMigration.remaining'
sudo gitlab-rails runner -e production 'puts Gitlab::Database::BackgroundMigration::BatchedMigration.queued.count'

For installations from source:

cd /home/git/gitlab
sudo -u git -H bundle exec rails runner -e production 'puts Gitlab::BackgroundMigration.remaining'
sudo -u git -H bundle exec rails runner -e production 'puts Gitlab::Database::BackgroundMigration::BatchedMigration.queued.count'

Failed migrations

For Omnibus installations:

For GitLab 14.0-14.9:

sudo gitlab-rails runner -e production 'Gitlab::Database::BackgroundMigration::BatchedMigration.failed.count'

For GitLab 14.10 and later:

sudo gitlab-rails runner -e production 'Gitlab::Database::BackgroundMigration::BatchedMigration.with_status(:failed).count'

For installations from source:

For GitLab 14.0-14.9:

cd /home/git/gitlab
sudo -u git -H bundle exec rails runner -e production 'Gitlab::Database::BackgroundMigration::BatchedMigration.failed.count'

For GitLab 14.10 and later:

cd /home/git/gitlab
sudo -u git -H bundle exec rails runner -e production 'Gitlab::Database::BackgroundMigration::BatchedMigration.with_status(:failed).count'

Batched background migrations

GitLab 14.0 introduced batched background migrations.

Some installations may need to run GitLab 14.0 for at least a day to complete the database changes introduced by that upgrade.

Check the status of batched background migrations

To check the status of batched background migrations:

  1. On the top bar, select Menu > Admin.
  2. On the left sidebar, select Monitoring > Background Migrations.

    queued batched background migrations table

All migrations must have a Finished status before you upgrade GitLab.

The status of batched background migrations can also be queried directly in the database.

  1. Log into a psql prompt according to the directions for your instance’s installation method (for example, sudo gitlab-psql for Omnibus installations).
  2. Run the following query in the psql session to see details on incomplete batched background migrations:

    select job_class_name, table_name, column_name, job_arguments from batched_background_migrations where status <> 3;
    

If the migrations are not finished and you try to update to a later version, GitLab prompts you with an error:

Expected batched background migration for the given configuration to be marked as 'finished', but it is 'active':

If you get this error, check the batched background migration options to complete the upgrade.

What do I do if my background migrations are stuck?

caution
The following operations can disrupt your GitLab performance. They run a number of Sidekiq jobs that perform various database or file updates.

Background migrations remain in the Sidekiq queue

Run the following check. If it returns non-zero and the count does not decrease over time, follow the rest of the steps in this section.

# For Omnibus installations:
sudo gitlab-rails runner -e production 'puts Gitlab::BackgroundMigration.remaining'

# For installations from source:
cd /home/git/gitlab
sudo -u git -H bundle exec rails runner -e production 'puts Gitlab::BackgroundMigration.remaining'

It is safe to re-execute the following commands, especially if you have 1000+ pending jobs which would likely overflow your runtime memory.

For Omnibus installations

# Start the rails console
sudo gitlab-rails c

# Execute the following in the rails console
scheduled_queue = Sidekiq::ScheduledSet.new
pending_job_classes = scheduled_queue.select { |job| job["class"] == "BackgroundMigrationWorker" }.map { |job| job["args"].first }.uniq
pending_job_classes.each { |job_class| Gitlab::BackgroundMigration.steal(job_class) }

For installations from source

# Start the rails console
sudo -u git -H bundle exec rails RAILS_ENV=production

# Execute the following in the rails console
scheduled_queue = Sidekiq::ScheduledSet.new
pending_job_classes = scheduled_queue.select { |job| job["class"] == "BackgroundMigrationWorker" }.map { |job| job["args"].first }.uniq
pending_job_classes.each { |job_class| Gitlab::BackgroundMigration.steal(job_class) }

Background migrations stuck in ‘pending’ state

GitLab 13.6 introduced an issue where a background migration named BackfillJiraTrackerDeploymentType2 can be permanently stuck in a pending state across upgrades. To clean up this stuck migration, see the 13.6.0 version-specific instructions.

GitLab 14.2 introduced an issue where a background migration named BackfillDraftStatusOnMergeRequests can be permanently stuck in a pending state across upgrades when the instance lacks records that match the migration’s target. To clean up