Manage the container registry metadata database

Tier: Free, Premium, Ultimate Offering: Self-managed
History

The metadata database enables many new registry features, including online garbage collection, and increases the efficiency of many registry operations. This page contains information on how to create the database.

Metadata database feature support

You can migrate existing registries to the metadata database, and use online garbage collection.

Some database-enabled features are only enabled for GitLab.com and automatic database provisioning for the registry database is not available. Review the feature support section in the administration documentation for the status of features related to the container registry database.

Create the database

Follow the steps below to manually create the database and role.

note
These instructions assume you are using the bundled PostgreSQL server. If you are using your own server, there will be some variation in how you connect.
  1. Create the secret with the database password:

    kubectl create secret generic RELEASE_NAME-registry-database-password --from-literal=password=randomstring
    
  2. Log into your database instance:

    kubectl exec -it $(kubectl get pods -l app.kubernetes.io/name=postgresql -o custom-columns=NAME:.metadata.name --no-headers) -- bash
    
    PGPASSWORD=${POSTGRES_POSTGRES_PASSWORD} psql -U postgres -d template1
    
  3. Create the database user:

    CREATE ROLE registry WITH LOGIN;
    
  4. Set the database user password.

    1. Fetch the password:

      kubectl get secret RELEASE_NAME-registry-database-password -o jsonpath="{.data.password}" | base64 --decode
      
    2. Set the password in the psql prompt:

      \password registry
      
  5. Create the database:

    CREATE DATABASE registry WITH OWNER registry;
    
  6. Safely exit from the PostgreSQL command line and then from the container using exit:

    template1=# exit
    ...@gitlab-postgresql-0/$ exit
    

Enable the metadata database for Helm charts installations

Prerequisites:

  • GitLab 17.3 or later.
  • PostgreSQL database version 12 or later, accessible from the registry pods.
  • Access to the Kubernetes cluster and the Helm deployment locally.
  • SSH access to the registry pods.

Follow the instructions that match your situation:

  • New installation or enabling the container registry for the first time.
  • Migrate existing container images to the metadata database:
note
For a list of import times for various test and user registries, see this table in issue 423459. Your registry deployment is unique, and your import times might be longer than those reported in the issue.

Before you start

Read the before you start section of the Registry administration guide.

New installations

To enable the database:

  1. Create the database and Kubernetes secret.
  2. Get the current Helm values for your release and save them to a file. For example, for a release named gitlab and a file named values.yml:

    helm get values gitlab > values.yml
    
  3. Add the following lines to your values.yml file:

    registry:
      enabled: true
      database:
        enabled: true
        name: registry  # must match the database name you created above
        user: registry  # must match the database username you created above
        password:
          secret: gitlab-registry-database-password # must match the secret name
          key: password  # must match the secret key to read the password from
        sslmode: verify-full
        ssl:
          secret: gitlab-registry-postgresql-ssl # you will need to create this secret manually
          clientKey: client-key.pem
          clientCertificate: client-cert.pem
          serverCA: server-ca.pem
        migrations:
          enabled: true  # this option will execute the schema migration as part of the registry deployment
    
  4. Optional. You can verify the schema migrations have been applied properly. You can either:
    • Review the log output of the migrations job, for example:

      kubectl logs jobs/gitlab-registry-migrations-1
      ...
      OK: applied 154 migrations in 13.752s
      
    • Or, connect to the Postgres database and query the schema_migrations table:

      SELECT * FROM schema_migrations;
      

      Ensure the applied_at column timestamp is filled for all rows.

The registry is ready to use the metadata database!

Existing registries

You can migrate your existing container registry data in one step or three steps. A few factors affect the duration of the migration:

  • The size of your existing registry data.
  • The specifications of your PostgresSQL instance.
  • The number of registry pods running in your cluster.
  • Network latency between the registry, PostgresSQL and your configured Object Storage.
note
Work to automate the migration process is being tracked in issue 5293.

Requirements

You must complete the following steps before attempting the one-step or three-step migration:

  1. Create the database and Kubernetes secret.
  2. Get the current Helm values for your release and save them into a file. For example, for a release named gitlab and a file named values.yml:

    helm get values gitlab > values.yml
    

One-step migration

When doing a one-step migration, be aware that:

  • The registry must remain in read-only mode during the migration.
  • If the Pod where the migration is being executed is terminated, you have to completely restart the process. The work to improve this process is tracked in issue 5293.

To migrate existing container registry to the metadata database in one step:

  1. Follow the steps described in the requirements section.
  2. Find the registry: section in the values.yml file and add the database section. Set:
    • database.configure to true.
    • database.enabled to false.
    • maintenance.readonly.enabled to true.
    • migrations.enabled to true.
    registry:
      enabled: true
      maintenance:
        readonly:
          enabled: true  # must remain set to true while the migration is executed
      database:
        configure: true  # must be true for the migration step
        enabled: false  # must be false!
        name: registry  # must match the database name you created above
        user: registry  # must match the database username you created above
        password:
          secret: gitlab-registry-database-password  # must match the secret name
          key: password  # must match the secret key to read the password from
        sslmode: verify-full  # SSL connection mode. See https://www.postgresql.org/docs/current/libpq-ssl.html#LIBPQ-SSL-PROTECTION for more options.
        ssl:
          secret: gitlab-registry-postgresql-ssl  # you will need to create this secret manually
          clientKey: client-key.pem
          clientCertificate: client-cert.pem
          serverCA: server-ca.pem
        migrations:
          enabled: true  # this option will execute the schema migration as part of the registry deployment
    
  3. Upgrade your Helm installation to apply changes in your deployment:

    helm upgrade gitlab gitlab/gitlab -f values.yml
    
  4. Connect to one of the registry pods via SSH, for example for a pod named gitlab-registry-5ddcd9f486-bvb57:

    kubectl exec -ti gitlab-registry-5ddcd9f486-bvb57 bash
    
  5. Change to the home directory and then run the following command:

    cd ~
    /usr/bin/registry database import /etc/docker/registry/config.yml
    
  6. If the command completed successfully, all images are now fully imported. You can now enable the database and turn off read-only mode in the configuration:

    registry:
      enabled: true
      maintenance:
        readonly:
          enabled: false
      database:
        configure: true  # once database.enabled is set to true, this option can be removed
        enabled: true
        name: registry
        user: registry
        password:
          secret: gitlab-registry-database-password
          key: password
        migrations:
          enabled: true
    
  7. Upgrade your Helm installation to apply changes in your deployment:

    helm upgrade gitlab gitlab/gitlab -f values.yml
    

You can now use the metadata database for all operations!

Three-step migration

You can migrate existing container registry data to the metadata database in three separate steps, which is recommended if:

  • The registry contains a large amount of data.
  • You need to minimize downtime during the migration.

To migrate in three steps, you must:

  1. Pre-import repositories
  2. Import all repository data
  3. Import common blobs
note
Users have reported step one import completed at rates of 2 to 4 TB per hour. At the slower speed, registries with over 100TB of data could take longer than 48 hours.
Step 1. Pre-import repositories

For larger instances, this process can take hours or even days to complete, depending on the size of your registry. You can still use the registry during this process.

caution
It is not yet possible to restart the migration, so it’s important to let the migration run to completion. If you must halt the operation, you have to restart this step.
  1. Follow the steps described in the requirements section.
  2. Find the registry: section in the values.yml file and add the database section. Set:
    • database.configure to true.
    • database.enabled to false.
    • migrations.enabled to true.
    registry:
      enabled: true
      database:
        configure: true
        enabled: false  # must be false!
        name: registry  # must match the database name you created above
        user: registry  # must match the database username you created above
        password:
          secret: gitlab-registry-database-password  # must match the secret name
          key: password  # must match the secret key to read the password from
        sslmode: verify-full  # SSL connection mode. See https://www.postgresql.org/docs/current/libpq-ssl.html#LIBPQ-SSL-PROTECTION for more options.
        ssl:
          secret: gitlab-registry-postgresql-ssl  # you will need to create this secret manually
          clientKey: client-key.pem
          clientCertificate: client-cert.pem
          serverCA: server-ca.pem
        migrations:
          enabled: true  # this option will execute the schema migration as part of the registry deployment
    
  3. Save the file and upgrade your Helm installation to apply changes in your deployment:

    helm upgrade gitlab gitlab/gitlab -f values.yml
    
  4. Connect to one of the registry pods with SSH. For example, for a pod named gitlab-registry-5ddcd9f486-bvb57:

    kubectl exec -ti gitlab-registry-5ddcd9f486-bvb57 bash
    
  5. Change to the home directory and then run the following command:

    cd ~
    /usr/bin/registry database import --step-one /etc/docker/registry/config.yml
    

The first step is complete when the registry import complete displays.

note
You should try to schedule the following step as soon as possible to reduce the amount of downtime required. Ideally, less than one week after step one completes. Any new data written to the registry before the next step causes that step to take more time.
Step 2. Import all repository data

This step requires the registry to be set in read-only mode. Allow enough time for downtime during this process.

  1. Set the registry to read-only mode in your values.yml file:

    registry:
      enabled: true
      maintenance:
        readonly:
          enabled: true   # must be true!
      database:
        configure: true
        enabled: false  # must be false!
        name: registry  # must match the database name you created above
        user: registry  # must match the database username you created above
        password:
          secret: gitlab-registry-database-password  # must match the secret name
          key: password  # must match the secret key to read the password from
        sslmode: verify-full  # SSL connection mode. See https://www.postgresql.org/docs/current/libpq-ssl.html#LIBPQ-SSL-PROTECTION for more options.
        ssl:
          secret: gitlab-registry-postgresql-ssl  # you will need to create this secret manually
          clientKey: client-key.pem
          clientCertificate: client-cert.pem
          serverCA: server-ca.pem
        migrations:
          enabled: true  # this option will execute the schema migration as part of the registry deployment
    
  2. Save the file and upgrade your Helm installation to apply changes in your deployment:

    helm upgrade gitlab gitlab/gitlab -f values.yml
    
  3. Connect to one of the registry pods with SSH. For example, for a pod named gitlab-registry-5ddcd9f486-bvb57:

    kubectl exec -ti gitlab-registry-5ddcd9f486-bvb57 bash
    
  4. Change to the home directory and then run the following command:

    cd ~
    /usr/bin/registry database import --step-two /etc/docker/registry/config.yml
    
  5. If the command completed successfully, all images are now fully imported. You can now enable the database and turn off read-only mode in the configuration:

    registry:
      enabled: true
      maintenance:        # this section can be removed
        readonly:
          enabled: false
      database:
        configure: true  # once database.enabled is set to true, this option can be removed
        enabled: true   # must be true!
        name: registry  # must match the database name you created above
        user: registry  # must match the database username you created above
        password:
          secret: gitlab-registry-database-password  # must match the secret name
          key: password  # must match the secret key to read the password from
        sslmode: verify-full  # SSL connection mode. See https://www.postgresql.org/docs/current/libpq-ssl.html#LIBPQ-SSL-PROTECTION for more options.
        ssl:
          secret: gitlab-registry-postgresql-ssl  # you will need to create this secret manually
          clientKey: client-key.pem
          clientCertificate: client-cert.pem
          serverCA: server-ca.pem
        migrations:
          enabled: true  # this option will execute the schema migration as part of the registry deployment
    
  6. Save the file and upgrade your Helm installation to apply changes in your deployment:

    helm upgrade gitlab gitlab/gitlab -f values.yml
    

You can now use the metadata database for all operations!

Step 3. Import common blobs

The registry is now fully using the database for its metadata, but it does not yet have access to any potentially unused layer blobs.

To complete the process, run the final step of the migration:

cd ~
/usr/bin/registry database import --step-three /etc/docker/registry/config.yml

After the command completes successfully, the registry is now fully migrated to the database!