Container registry metadata database

Tier: Free, Premium, Ultimate
Offering: GitLab Self-Managed

The metadata database provides many new registry features, including online garbage collection, and increases the efficiency of many registry operations.

If you have existing registries, you can migrate to the metadata database.

Some database-enabled features are only enabled for GitLab.com and automatic database provisioning for the registry database is not available. Review the feature support section in the administration documentation for the status of features related to the container registry database.

Create an external metadata database

In production, you should create an external metadata database.

Prerequisites:

Set up an external PostgreSQL server.

After you set up the external PostgreSQL server:

Create a secret for the metadata database password:

kubectl create secret generic RELEASE_NAME-registry-database-password --from-literal=password=<your_registry_password>

Use the following SQL commands to create the user and the database:

-- Create the registry user
CREATE USER registry WITH PASSWORD '<your_registry_password>';

-- Create the registry database
CREATE DATABASE registry OWNER registry;

For cloud-managed services, grant additional roles as needed:

GRANT rds_superuser TO registry;

GRANT azure_pg_admin TO registry;

GRANT cloudsqlsuperuser TO registry;

Create a built-in metadata database

You can use the built-in cloud native metadata database for trial purposes only. You should not use it in production.

Create the database automatically

Prerequisites:

Helm chart 9.3 or later.

New installations that set postgresql.install=true when installing the GitLab chart, automatically create the registry database, username, and shared secret RELEASE-registry-database-password.

This automatic provisioning:

Creates a dedicated registry database.
Sets up a registry user with appropriate permissions.
Generates a Kubernetes secret named RELEASE-registry-database-password containing the database password.
Configures the necessary database schema and permissions.

With automatic database creation, you can skip the manual database creation steps and immediately enable the metadata database.

Create the database manually

To manually create the metadata database using the built-in PostgreSQL server:

Create the secret with the database password:

kubectl create secret generic RELEASE_NAME-registry-database-password --from-literal=password=<your_registry_password>

Log into your database instance:

kubectl exec -it $(kubectl get pods -l app.kubernetes.io/name=postgresql -o custom-columns=NAME:.metadata.name --no-headers) -- bash

PGPASSWORD=${POSTGRES_POSTGRES_PASSWORD} psql -U postgres -d template1

Create the database user:
```
CREATE ROLE registry WITH LOGIN;
```

Set the database user password.

Fetch the password:

kubectl get secret RELEASE_NAME-registry-database-password -o jsonpath="{.data.password}" | base64 --decode

Set the password in the psql prompt:
```
\password registry
```

Create the database:

CREATE DATABASE registry WITH OWNER registry;

Safely exit from the PostgreSQL command line and then from the container using exit:
```
template1=# exit
...@gitlab-postgresql-0/$ exit
```

Enable the metadata database

After you’ve created the database, enable it. Additional steps are required when migrating an existing container registry.

Prerequisites

Prerequisites:

GitLab 17.3 or later.
A deployment of the required version of PostgreSQL, accessible from the registry pods.
Access to the Kubernetes cluster and the Helm deployment locally.
SSH access to the registry pods.

Also read the before you start section of the Registry administration guide.

For a list of import times for various test and user registries, see this table in issue 423459. Your registry deployment is unique, and your import times might be longer than those reported in the issue.

Enable for new registries

To enable the database for a new container registry:

Get the current Helm values for your release and save them to a file. For example, for a release named gitlab and a file named values.yml:
```
helm get values gitlab > values.yml
```

Add the following lines to your values.yml file:

registry:
  enabled: true
  database:
    enabled: true
    name: registry  # must match the database name you created above
    user: registry  # must match the database username you created above
    password:
      secret: gitlab-registry-database-password # must match the secret name
      key: password  # must match the secret key to read the password from
    sslmode: verify-full
    # these settings are inherited from `global.psql.ssl`
    ssl:
      secret: gitlab-registry-postgresql-ssl # you will need to create this secret manually
      clientKey: client-key.pem
      clientCertificate: client-cert.pem
      serverCA: server-ca.pem
    migrations:
      enabled: true  # this option will execute the schema migration as part of the registry deployment

Optional. Verify the schema migrations have been applied properly. You can either:
- Review the log output of the migrations job, for example:
```
kubectl logs jobs/gitlab-registry-migrations-1
...
OK: applied 154 migrations in 13.752s
```
- Or, connect to the Postgres database and query the schema_migrations table:
```
SELECT * FROM schema_migrations;
```
  Ensure the applied_at column timestamp is filled for all rows.

The registry is ready to use the metadata database!

Enable for and import existing registries

You can import your existing container registry data in one step or three steps. A few factors affect the duration of the migration:

The size of your existing registry data.
The specifications of your PostgresSQL instance.
The number of registry pods running in your cluster.
Network latency between the registry, PostgresSQL and your configured Object Storage.

Work to automate the import process is being tracked in issue 5293.

Before attempting the one-step or three-step import, get the current Helm values for your release and save them into a file. For example, for a release named gitlab and a file named values.yml:

helm get values gitlab > values.yml

Import in one step

When doing a one-step import, be aware that:

The registry must remain in read-only mode during the import.
If the Pod where the import is being executed is terminated, you have to completely restart the process. The work to improve this process is tracked in issue 5293.

To import existing container registry to the metadata database in one step:

Find the registry: section in the values.yml file and add the database section. Set:

database.configure to true.
database.enabled to false.
maintenance.readonly.enabled to true.
migrations.enabled to true.

registry:
  enabled: true
  maintenance:
    readonly:
      enabled: true  # must remain set to true while the migration is executed
  database:
    configure: true  # must be true for the migration step
    enabled: false  # must be false!
    name: registry  # must match the database name you created above
    user: registry  # must match the database username you created above
    password:
      secret: gitlab-registry-database-password  # must match the secret name
      key: password  # must match the secret key to read the password from
    sslmode: verify-full  # SSL connection mode. See https://www.postgresql.org/docs/current/libpq-ssl.html#LIBPQ-SSL-PROTECTION for more options.
    ssl:
      secret: gitlab-registry-postgresql-ssl  # you will need to create this secret manually
      clientKey: client-key.pem
      clientCertificate: client-cert.pem
      serverCA: server-ca.pem
    migrations:
      enabled: true  # this option will execute the schema migration as part of the registry deployment

Upgrade your Helm installation to apply changes in your deployment:
```
helm upgrade gitlab gitlab/gitlab -f values.yml
```
Connect to one of the registry pods via SSH, for example for a pod named gitlab-registry-5ddcd9f486-bvb57:
```
kubectl exec -ti gitlab-registry-5ddcd9f486-bvb57 bash
```

Change to the home directory and then run the following command:

cd ~
/usr/bin/registry database import /etc/docker/registry/config.yml

If the command completed successfully, all images are now fully imported. You can now enable the database and turn off read-only mode in the configuration:

registry:
  enabled: true
  maintenance:
    readonly:
      enabled: false
  database:
    configure: true  # once database.enabled is set to true, this option can be removed
    enabled: true
    name: registry
    user: registry
    password:
      secret: gitlab-registry-database-password
      key: password
    migrations:
      enabled: true

Upgrade your Helm installation to apply changes in your deployment:
```
helm upgrade gitlab gitlab/gitlab -f values.yml
```

You can now use the metadata database for all operations!

Import in three steps

You can import existing container registry data to the metadata database in three separate steps, which is recommended if:

The registry contains a large amount of data.
You need to minimize downtime during the migration.

To import in three steps, you must:

Pre-import repositories
Import all repository data
Import common blobs

Users have reported step one import completed at rates of 2 to 4 TB per hour. At the slower speed, registries with over 100 TB of data could take longer than 48 hours.

Step 1. Pre-import repositories

For larger instances, this process can take hours or even days to complete, depending on the size of your registry. You can still use the registry during this process.

It is not yet possible to restart the import, so it’s important to let the import run to completion. If you must halt the operation, you have to restart this step.

Find the registry: section in the values.yml file and add the database section. Set:

database.configure to true.
database.enabled to false.
migrations.enabled to true.

registry:
  enabled: true
  database:
    configure: true
    enabled: false  # must be false!
    name: registry  # must match the database name you created above
    user: registry  # must match the database username you created above
    password:
      secret: gitlab-registry-database-password  # must match the secret name
      key: password  # must match the secret key to read the password from
    sslmode: verify-full  # SSL connection mode. See https://www.postgresql.org/docs/current/libpq-ssl.html#LIBPQ-SSL-PROTECTION for more options.
    ssl:
      secret: gitlab-registry-postgresql-ssl  # you will need to create this secret manually
      clientKey: client-key.pem
      clientCertificate: client-cert.pem
      serverCA: server-ca.pem
    migrations:
      enabled: true  # this option will execute the schema migration as part of the registry deployment

Save the file and upgrade your Helm installation to apply changes in your deployment:
```
helm upgrade gitlab gitlab/gitlab -f values.yml
```
Connect to one of the registry pods with SSH. For example, for a pod named gitlab-registry-5ddcd9f486-bvb57:
```
kubectl exec -ti gitlab-registry-5ddcd9f486-bvb57 bash
```

Change to the home directory and then run the following command:

cd ~
/usr/bin/registry database import --step-one /etc/docker/registry/config.yml

The first step is complete when the registry import complete displays.

You should try to schedule the following step as soon as possible to reduce the amount of downtime required. Ideally, less than one week after step one completes. Any new data written to the registry before the next step causes that step to take more time.

Step 2. Import all repository data

This step requires the registry to be set in read-only mode. Allow enough time for downtime during this process.

Set the registry to read-only mode in your values.yml file:

registry:
  enabled: true
  maintenance:
    readonly:
      enabled: true   # must be true!
  database:
    configure: true
    enabled: false  # must be false!
    name: registry  # must match the database name you created above
    user: registry  # must match the database username you created above
    password:
      secret: gitlab-registry-database-password  # must match the secret name
      key: password  # must match the secret key to read the password from
    sslmode: verify-full  # SSL connection mode. See https://www.postgresql.org/docs/current/libpq-ssl.html#LIBPQ-SSL-PROTECTION for more options.
    ssl:
      secret: gitlab-registry-postgresql-ssl  # you will need to create this secret manually
      clientKey: client-key.pem
      clientCertificate: client-cert.pem
      serverCA: server-ca.pem
    migrations:
      enabled: true  # this option will execute the schema migration as part of the registry deployment

Save the file and upgrade your Helm installation to apply changes in your deployment:
```
helm upgrade gitlab gitlab/gitlab -f values.yml
```
Connect to one of the registry pods with SSH. For example, for a pod named gitlab-registry-5ddcd9f486-bvb57:
```
kubectl exec -ti gitlab-registry-5ddcd9f486-bvb57 bash
```

Change to the home directory and then run the following command:

cd ~
/usr/bin/registry database import --step-two /etc/docker/registry/config.yml

If the command completed successfully, all images are now fully imported. You can now enable the database and turn off read-only mode in the configuration:

registry:
  enabled: true
  maintenance:        # this section can be removed
    readonly:
      enabled: false
  database:
    configure: true  # once database.enabled is set to true, this option can be removed
    enabled: true   # must be true!
    name: registry  # must match the database name you created above
    user: registry  # must match the database username you created above
    password:
      secret: gitlab-registry-database-password  # must match the secret name
      key: password  # must match the secret key to read the password from
    sslmode: verify-full  # SSL connection mode. See https://www.postgresql.org/docs/current/libpq-ssl.html#LIBPQ-SSL-PROTECTION for more options.
    ssl:
      secret: gitlab-registry-postgresql-ssl  # you will need to create this secret manually
      clientKey: client-key.pem
      clientCertificate: client-cert.pem
      serverCA: server-ca.pem
    migrations:
      enabled: true  # this option will execute the schema migration as part of the registry deployment

Save the file and upgrade your Helm installation to apply changes in your deployment:
```
helm upgrade gitlab gitlab/gitlab -f values.yml
```

You can now use the metadata database for all operations!

Step 3. Import common blobs

The registry is now fully using the database for its metadata, but it does not yet have access to any potentially unused layer blobs.

To complete the process, run the final step of the migration:

cd ~
/usr/bin/registry database import --step-three /etc/docker/registry/config.yml

After the command completes successfully, the registry is now fully migrated to the database!

Database migrations

The container registry supports two types of migrations:

Regular schema migrations: Changes to the database structure that must run before deploying new application code. These should be fast to avoid deployment delays.
Post-deployment migrations: Changes to the database structure that can run while the application is running. Used for longer operations like creating indexes on large tables, avoiding startup delays and extended upgrade downtime.

Apply database migrations

By default, the registry chart applies both regular schema and post-deployment migrations automatically if database.migrations.enabled is set to true.

To reduce downtime during upgrades, you can skip post-deployment migrations and apply them manually after the application starts:

Set the SKIP_POST_DEPLOYMENT_MIGRATIONS environment variable to true using ExtraEnv for the registry deployment:
```
registry:
  extraEnv:
    SKIP_POST_DEPLOYMENT_MIGRATIONS: true
```
After upgrading, connect to a registry pod.

Apply pending post-deployment migrations:

registry database migrate up /etc/docker/registry/config.yml

The migrate up command offers some extra flags that can be used to control how the migrations are applied. Run registry database migrate up --help for details.

Troubleshooting

Error: `panic: interface conversion: interface {} is nil, not bool`

When importing existing registries, you might see this error:

panic: interface conversion: interface {} is nil, not bool

This is a known issue that is fixed in registry version v4.15.2-gitlab and in GitLab 17.9 and later.

To work around this issue, upgrade your registry version:

In your values.yml file, set the registry image tag:

registry:
  image:
    tag: v4.15.2-gitlab

Upgrade your Helm installation:
```
helm upgrade gitlab -f values.yml
```

Alternatively, you can manually update the registry configuration:

In /etc/docker/registry/config.yml, set parallelwalk to false for your storage provider. For example, with S3:
```
storage:
  s3:
    parallelwalk: false
```

Container registry metadata database

Create an external metadata database

Create a built-in metadata database

Create the database automatically

Create the database manually

Enable the metadata database

Prerequisites

Enable for new registries

Enable for and import existing registries

Import in one step

Import in three steps

Step 1. Pre-import repositories

Step 2. Import all repository data

Step 3. Import common blobs

Database migrations

Apply database migrations

Troubleshooting

Error: panic: interface conversion: interface {} is nil, not bool

Error: `panic: interface conversion: interface {} is nil, not bool`