• Configure daily backups
  • Configure backup of PostgreSQL data
  • Configure backup of object storage data
  • Configure backup of Git repositories
  • Configure backup of configuration files
• Restore a backup
  • Prerequisites
  • Restore object storage data
  • Restore PostgreSQL data
  • Restore Git repositories

Back up and restore large reference architectures

This document describes how to:

• Configure daily backups
• Take a backup now (planned)
• Restore a backup

This document is intended for environments using:

• Linux package (Omnibus) and cloud-native hybrid reference architectures 60 RPS / 3,000 users and up
• Amazon RDS for PostgreSQL data
• Amazon S3 for object storage
• Object storage to store everything possible, including blobs and container registry

Configure daily backups

Configure backup of PostgreSQL data

The backup command uses pg_dump, which is not appropriate for databases over 100 GB. You must choose a PostgreSQL solution which has native, robust backup capabilities.

Configure backup of object storage data

Object storage, (not NFS) is recommended for storing GitLab data, including blobs and Container registry.

Configure backup of Git repositories

If using Cloud native hybrid, provision a VM running the GitLab Linux package:

1. Spin up a VM with 8 vCPU and 7.2 GB memory. This node will be used to back up Git repositories. Adding support for Gitaly server-side backups to backup-utility is proposed in issue 438393, which would remove the need to provision a VM.
2. Configure the node as another GitLab Rails node as defined in your reference architecture. As with other GitLab Rails nodes, this node must have access to your main PostgreSQL database, Redis, object storage, and Gitaly Cluster. Find your reference architecture and see the Configure GitLab Rails section for an example of how to set the server up. You might need to translate some Helm chart values to the equivalent ones for Linux package installations. Note that a Praefect node cannot be used to back up Git data. It must be a GitLab Rails node.

3. Ensure the GitLab application isn’t running on this node by disabling most services:

   1. Edit /etc/gitlab/gitlab.rb to ensure the following services are disabled. roles(['application_role']) disables Redis, PostgreSQL, and Consul, and is the basis of the reference architecture Rails node definition.

      roles(['application_role'])
      gitlab_workhorse['enable'] = false
      puma['enable'] = false
      sidekiq['enable'] = false
      gitlab_kas['enable'] = false
      gitaly['enable'] = false
      prometheus_monitoring['enable'] = false
      

   2. Reconfigure GitLab:

      sudo gitlab-ctl reconfigure
      

   3. The only service that should be left is logrotate. To verify that logrotate is the only remaining service, run:

      gitlab-ctl status
      

   Issue 6823 proposes to add a role in the Linux package that meets these requirements.

To back up the Git repositories:

1. Configure a server-side backup destination in all Gitaly nodes.
2. Add the destination bucket to your backups of object storage data.

3. Take a full backup of your Git data. Use the REPOSITORIES_SERVER_SIDE variable, and skip PostgreSQL data:

   sudo gitlab-backup create REPOSITORIES_SERVER_SIDE=true SKIP=db
   

   This causes Gitaly nodes to upload the Git data and some metadata to remote storage. Blobs such as uploads, artifacts, and LFS do not need to be explicitly skipped, because the gitlab-backup command does not back up object storage by default.

4. Note the backup ID of the backup, which is needed for the next step. For example, if the backup command outputs 2024-02-22 02:17:47 UTC -- Backup 1708568263_2024_02_22_16.9.0-ce is done., then the backup ID is 1708568263_2024_02_22_16.9.0-ce.
5. Check that the full backup created data in the backup bucket.

6. Run the backup command again, this time specifying incremental backup of Git repositories, and a backup ID. Using the example ID from the previous step, the command is:

   sudo gitlab-backup create REPOSITORIES_SERVER_SIDE=true SKIP=db INCREMENTAL=yes PREVIOUS_BACKUP=1708568263_2024_02_22_16.9.0-ce
   

   The value of PREVIOUS_BACKUP is not used by this command, but it is required by the command. There is an issue for removing this unnecessary requirement, see issue 429141.

7. Check that the incremental backup succeeded, and added data to object storage.

8. Configure cron to make daily backups. Edit the crontab for the root user:

   sudo su -
   crontab -e
   

9. There, add the following lines to schedule the backup for everyday of every month at 2 AM. To limit the number of increments needed to restore a backup, a full backup of Git repositories will be taken on the first of each month, and the rest of the days will take an incremental backup.:

   0 2 1 * * /opt/gitlab/bin/gitlab-backup create REPOSITORIES_SERVER_SIDE=true SKIP=db CRON=1
   0 2 2-31 * * /opt/gitlab/bin/gitlab-backup create REPOSITORIES_SERVER_SIDE=true SKIP=db INCREMENTAL=yes PREVIOUS_BACKUP=1708568263_2024_02_22_16.9.0-ce CRON=1
   

Configure backup of configuration files

If your configuration and secrets are defined outside of your deployment and then deployed into it, then the implementation of the backup strategy depends on your specific setup and requirements. As an example, you can store secrets in AWS Secret Manager with replication to multiple regions and configure a script to back up secrets automatically.

If your configuration and secrets are only defined inside your deployment:

1. Storing configuration files describes how to extract configuration and secrets files.
2. These files should be uploaded to a separate, more restrictive, object storage account.

Restore a backup

Restore a backup of a GitLab instance.

Prerequisites

Before restoring a backup:

1. Choose a working destination GitLab instance.
2. Ensure the destination GitLab instance is in a region where your AWS backups are stored.
3. Check that the destination GitLab instance uses exactly the same version and type (CE or EE) of GitLab on which the backup data was created. For example, CE 15.1.4.
4. Restore backed up secrets to the destination GitLab instance.
5. Ensure that the destination GitLab instance has the same repository storages configured. Additional storages are fine.
6. If the backed up GitLab instance had any blobs stored in object storage, ensure that object storage is configured for those kinds of blobs.
7. If the backed up GitLab instance had any blobs stored on the file system, ensure that NFS is configured.

8. To use new secrets or configuration, and to avoid unexpected configuration changes during restore:

   • Linux package installations on all nodes:
     1. Reconfigure the destination GitLab instance.
     2. Restart the destination GitLab instance.

   • Helm chart (Kubernetes) installations:

     1. On all GitLab Linux package nodes, run:

        sudo gitlab-ctl reconfigure
        sudo gitlab-ctl start
        

     2. Make sure you have a running GitLab instance by deploying the charts. Ensure the Toolbox pod is enabled and running by executing the following command:

        kubectl get pods -lrelease=RELEASE_NAME,app=toolbox
        

     3. The Webservice, Sidekiq and Toolbox pods must be restarted. The safest way to restart those pods is to run:

        kubectl delete pods -lapp=sidekiq,release=<helm release name>
        kubectl delete pods -lapp=webservice,release=<helm release name>
        kubectl delete pods -lapp=toolbox,release=<helm release name>
        

9. Confirm the destination GitLab instance still works. For example:

   • Make requests to the health check endpoints.
   • Run GitLab check Rake tasks.

10. Stop GitLab services which connect to the PostgreSQL database.

    • Linux package installations on all nodes running Puma or Sidekiq, run:

      sudo gitlab-ctl stop
      

    • Helm chart (Kubernetes) installations:

      1. Note the current number of replicas for database clients for subsequent restart:

         kubectl get deploy -n <namespace> -lapp=sidekiq,release=<helm release name> -o jsonpath='{.items[].spec.replicas}{"\n"}'
         kubectl get deploy -n <namespace> -lapp=webservice,release=<helm release name> -o jsonpath='{.items[].spec.replicas}{"\n"}'
         kubectl get deploy -n <namespace> -lapp=prometheus,release=<helm release name> -o jsonpath='{.items[].spec.replicas}{"\n"}'
         

      2. Stop the clients of the database to prevent locks interfering with the restore process:

         kubectl scale deploy -lapp=sidekiq,release=<helm release name> -n <namespace> --replicas=0
         kubectl scale deploy -lapp=webservice,release=<helm release name> -n <namespace> --replicas=0
         kubectl scale deploy -lapp=prometheus,release=<helm release name> -n <namespace> --replicas=0
         

Restore object storage data

Restore PostgreSQL data

Restore Git repositories

Select or create a node to restore:

• For Linux package installations, choose a Rails node, which is a node that normally runs Puma or Sidekiq.

• For Helm chart (Kubernetes) installations, if you don’t already have a Git repository backup node, create one now:

  1. Spin up a VM with 8 vCPU and 7.2 GB memory. This node is used to back up and restore Git repositories because a Praefect node cannot be used to back up Git data.
  2. Configure the node as another GitLab Rails node as defined in your reference architecture. As with other GitLab Rails nodes, this node must have access to your main PostgreSQL database as well as to Gitaly Cluster.
  3. Restore backed up secrets to the target GitLab.

To restore Git repositories:

1. Ensure the node has enough attached storage to store both the .tar file of Git repositories, and its extracted data.
2. SSH into the GitLab Rails node.
3. As part of Restore object storage data, you should have restored a bucket containing the GitLab backup .tar file of Git repositories.

4. Download the backup .tar file from its bucket into the backup directory described in the gitlab.rb configuration gitlab_rails['backup_path']. The default is /var/opt/gitlab/backups. The backup file must be owned by the git user.

   sudo cp 11493107454_2018_04_25_10.6.4-ce_gitlab_backup.tar /var/opt/gitlab/backups/
   sudo chown git:git /var/opt/gitlab/backups/11493107454_2018_04_25_10.6.4-ce_gitlab_backup.tar
   

5. Restore the backup, specifying the ID of the backup you wish to restore:

   The restore command requires additional parameters when your installation is using PgBouncer, for either performance reasons or when using it with a Patroni cluster.

   # This command will overwrite the contents of your GitLab database!
   # NOTE: "_gitlab_backup.tar" is omitted from the name
   sudo gitlab-backup restore BACKUP=11493107454_2018_04_25_10.6.4-ce
   

   If there’s a GitLab version mismatch between your backup tar file and the installed version of GitLab, the restore command aborts with an error message. Install the correct GitLab version, and then try again.

6. Restart and check GitLab:

   • Linux package installations:

     1. In all Puma or Sidekiq nodes, run:

        sudo gitlab-ctl restart
        

     2. In one Puma or Sidekiq node, run:

        sudo gitlab-rake gitlab:check SANITIZE=true
        

   • Helm chart (Kubernetes) installations:

     1. Start the stopped deployments, using the number of replicas noted in Prerequisites:

        kubectl scale deploy -lapp=sidekiq,release=<helm release name> -n <namespace> --replicas=<original value>
        kubectl scale deploy -lapp=webservice,release=<helm release name> -n <namespace> --replicas=<original value>
        kubectl scale deploy -lapp=prometheus,release=<helm release name> -n <namespace> --replicas=<original value>
        

     2. In the Toolbox pod, run:

        sudo gitlab-rake gitlab:check SANITIZE=true
        

7. Check that database values can be decrypted especially if /etc/gitlab/gitlab-secrets.json was restored, or if a different server is the target for the restore:

   • For Linux package installations, in a Puma or Sidekiq node, run:

     sudo gitlab-rake gitlab:doctor:secrets
     

   • For Helm chart (Kubernetes) installations, in the Toolbox pod, run:

     sudo gitlab-rake gitlab:doctor:secrets
     

8. For added assurance, you can perform an integrity check on the uploaded files:

   • For Linux package installations, in a Puma or Sidekiq node, run:

     sudo gitlab-rake gitlab:artifacts:check
     sudo gitlab-rake gitlab:lfs:check
     sudo gitlab-rake gitlab:uploads:check
     

   • For Helm chart (Kubernetes) installations, because these commands can take a long time because they iterate over all rows, run the following commands the GitLab Rails node, rather than a Toolbox pod:

     sudo gitlab-rake gitlab:artifacts:check
     sudo gitlab-rake gitlab:lfs:check
     sudo gitlab-rake gitlab:uploads:check
     

   If missing or corrupted files are found, it does not always mean the back up and restore process failed. For example, the files might be missing or corrupted on the source GitLab instance. You might need to cross-reference prior backups. If you are migrating GitLab to a new environment, you can run the same checks on the source GitLab instance to determine whether the integrity check result is preexisting or related to the backup and restore process.

The restoration should be complete.