Toolbox

The Toolbox chart is used to execute periodic housekeeping tasks within the GitLab application. These tasks include backups, database reindexing, Sidekiq maintenance, and Rake tasks.

Configuration

The following configuration settings are the default settings provided by the Toolbox chart:

gitlab:
  ## doc/charts/gitlab/toolbox
  toolbox:
    enabled: true
    replicas: 1
    backups:
      cron:
        enabled: false
        concurrencyPolicy: Replace
        failedJobsHistoryLimit: 1
        schedule: "0 1 * * *"
        successfulJobsHistoryLimit: 3
        suspend: false
        backoffLimit: 6
        safeToEvict: false
        restartPolicy: "OnFailure"
        resources:
          requests:
            cpu: 50m
            memory: 350M
        persistence:
          enabled: false
          accessMode: ReadWriteOnce
          useGenericEphemeralVolume: false
          size: 10Gi
      objectStorage:
        backend: s3
        config: {}
      registry:
        database: {}
    databaseReindex:
      cron:
        enabled: false
        concurrencyPolicy: Replace
        failedJobsHistoryLimit: 1
        schedule: "12 * * * 0,6"
        successfulJobsHistoryLimit: 3
        suspend: false
        backoffLimit: 6
        safeToEvict: false
        restartPolicy: "OnFailure"
        resources:
          requests:
            cpu: 200m
            memory: 400M
    persistence:
      enabled: false
      accessMode: 'ReadWriteOnce'
      size: '10Gi'
    resources:
      requests:
        cpu: '50m'
        memory: '350M'
    securityContext:
      fsGroup: '1000'
      runAsUser: '1000'
      runAsGroup: '1000'
    containerSecurityContext:
      runAsUser: '1000'
    affinity: {}

Configuring backups

Information concerning configuring backups in the backup and restore documentation. Additional information about the technical implementation of how the backups are performed can be found in the backup and restore architecture documentation.]

Registry metadata database credentials

If you use the container registry metadata database, you can configure the Toolbox to receive registry database connection details for backup and restore operations. The connection parameters (host, port, database name, SSL settings) are sourced automatically from the registry chart’s ConfigMap. You only need to supply the database usernames and password secret in the Toolbox values:

gitlab:
  toolbox:
    backups:
      registry:
        database:
          backupUser: registry_backup
          restoreUser: registry_restore
          password:
            secret: gitlab-toolbox-registry-database-password
            backupPasswordKey: backupPassword
            restorePasswordKey: restorePassword

The default secret name is RELEASE-toolbox-registry-database-password, where RELEASE is replaced by the Helm release name (usually gitlab). Create the Kubernetes secret before deploying:

kubectl create secret generic <RELEASE>-toolbox-registry-database-password \
  --from-literal=backupPassword=<backup_password> \
  --from-literal=restorePassword=<restore_password>

The credentials are mounted into the Toolbox pod at /etc/gitlab/registry-db/ and are available in both the long-running Toolbox deployment and the backup CronJob.

Required database permissions

The backup and restore users require different privilege levels on the registry metadata database. When using the bundled PostgreSQL with Linux package (Omnibus), these users and permissions are created automatically. For external PostgreSQL, create the users manually.

The backup user needs read-only access for pg_dump:

-- Create the backup user
CREATE ROLE registry_backup WITH LOGIN PASSWORD '<backup_password>'
  NOINHERIT NOCREATEDB NOSUPERUSER NOREPLICATION;

-- The partitions schema must exist (created by registry migrations)
-- Grant connect and read-only privileges
GRANT CONNECT ON DATABASE registry TO registry_backup;

GRANT USAGE ON SCHEMA public TO registry_backup;
GRANT SELECT ON ALL TABLES IN SCHEMA public TO registry_backup;
GRANT SELECT ON ALL SEQUENCES IN SCHEMA public TO registry_backup;
ALTER DEFAULT PRIVILEGES FOR ROLE registry IN SCHEMA public
  GRANT SELECT ON TABLES TO registry_backup;
ALTER DEFAULT PRIVILEGES FOR ROLE registry IN SCHEMA public
  GRANT SELECT ON SEQUENCES TO registry_backup;

GRANT USAGE ON SCHEMA partitions TO registry_backup;
GRANT SELECT ON ALL TABLES IN SCHEMA partitions TO registry_backup;
GRANT SELECT ON ALL SEQUENCES IN SCHEMA partitions TO registry_backup;
ALTER DEFAULT PRIVILEGES FOR ROLE registry IN SCHEMA partitions
  GRANT SELECT ON TABLES TO registry_backup;
ALTER DEFAULT PRIVILEGES FOR ROLE registry IN SCHEMA partitions
  GRANT SELECT ON SEQUENCES TO registry_backup;

The ALTER DEFAULT PRIVILEGES statements ensure the backup user automatically receives SELECT on any tables or sequences the registry owner (registry) creates in the future.

The restore user needs superuser privileges to recreate the schema, set role to the original object owner, and create triggers:

CREATE ROLE registry_restore WITH LOGIN PASSWORD '<restore_password>'
  SUPERUSER;

Configure periodic database reindexing

Database reindexing can be executed periodically to:

• Create and delete indexes asynchronously.
• Run PostgreSQL constraint validation in the background.
• Reindex PostgreSQL indexes to reduce index bloat.

Reindexing is performed by the gitlab:db:reindex Rake task. The Toolbox chart provides a CronJob to run the Rake task periodically.

Enable this CronJob by setting the value databaseReindex.cron.enabled to true. The job will automatically:

1. Select two indexes with the most index bloat.
2. Reindex those indexes in the background.

Set the schedule for the CronJob using the databaseReindex.cron.schedule value. You should run the reindexing during low traffic periods. For example, database reindexing runs for GitLab.com on Saturday and Sunday.

Persistence configuration

The persistent stores for backups and restorations are configured separately. Please review the following considerations when configuring GitLab for backup and restore operations.

Backups use the backups.cron.persistence.* properties and restorations use the persistence.* properties. Further descriptions concerning the configuration of a persistence store will use just the final property key (e.g. .enabled or .size) and the appropriate prefix will need to be added.

The persistence stores are disabled by default, thus .enabled needs to be set to true for a backup or restoration of any appreciable size. In addition, either .storageClass needs to be specified for a PersistentVolume to be created by Kubernetes or a PersistentVolume needs to be manually created. If .storageClass is specified as -, then the PersistentVolume will be created using the default StorageClass as specified in the Kubernetes cluster.

If the PersistentVolume is created manually, then the volume can be specified using the .volumeName property or by using the selector .matchLables / .matchExpressions properties.

In most cases the default value of .accessMode will provide adequate controls for only Toolbox accessing the PersistentVolumes. Please consult the documentation for the CSI driver installed in the Kubernetes cluster to ensure that the setting is correct.

Backup considerations

A backup operation needs an amount of disk space to hold the individual components that are being backed up before they are written to the backup object store. The amount of disk space depends on the following factors:

• Number of projects and the amount of data stored under each project
• Size of the PostgresSQL database (issues, MRs, etc.)
• Size of each object store backend

Once the rough size has been determined, the backups.cron.persistence.size property can be set so that backups can commence.

Restore considerations

During the restoration of a backup, the backup needs to be extracted to disk before the files are replaced on the running instance. The size of this restoration disk space is controlled by the persistence.size property. Be mindful that as the size of the GitLab installation grows the size of the restoration disk space also needs to grow accordingly. In most cases the size of the restoration disk space should be the same size as the backup disk space.

Toolbox included tools

The Toolbox container contains useful GitLab tools such as Rails console, Rake tasks, etc. These commands allow one to check the status of the database migrations, execute Rake tasks for administrative tasks, interact with the Rails console:

# locate the Toolbox pod
kubectl get pods -lapp=toolbox

# Launch a shell inside the pod
kubectl exec -it <Toolbox pod name> -- bash

# open Rails console
gitlab-rails console -e production

# execute a Rake task
gitlab-rake gitlab:env:info

affinity

For more information, see affinity.