Application secrets

GitLab must be able to access various secrets such as access tokens and other credentials to function. These secrets are encrypted and stored at rest and may be found in different data stores depending on use. Use this guide to understand how different kinds of secrets are stored and managed.

Application secrets and operational secrets

Broadly speaking, there are two classes of secrets:

  1. Application secrets. The GitLab application uses these to implement a particular feature or function. An example would be access tokens or private keys to create cryptographic signatures. We store these secrets in the database in encrypted columns. See Secure Coding Guidelines: At rest.
  2. Operational secrets. Used to read and store other secrets or bootstrap the application. For this reason, they cannot be stored in the database. These secrets are stored as Rails credentials in the config/secrets.yml file, directly for source installation, or through an installer like Omnibus or Helm (where actual secrets can be stored in an external secrets container like Kubernetes secrets or Vault).

Application secrets

Application secrets should be stored in postgres using ActiveRecord::Encryption:

class MyModel < ApplicationRecord
  encrypts :my_secret
end

Until recently, we used attr_encrypted instead of ActiveRecord::Encryption. We are in the process of migrating all columns to use the new Rails-native encryption framework (see epic 15420).

Despite there being precedent, application secrets should not be stored as an ApplicationSetting. This can lead to the entire application malfunctioning if this secret fails to decode. To reduce coupling to other features, isolate secrets into dedicated tables.

In some cases, it can be undesirable to store secrets in the database. For example, if the secret is needed to bootstrap the Rails application, it may have to access the database in an initializer, which can lead to initialization races as the database connection itself may not yet be ready. In this case, store the secret as an operational secret instead.

Operational secrets

We maintain a number of operational secrets in config/secrets.yml, primarily to manage other secrets. Historically, GitLab used this approach for all secrets, including application secrets, but has meanwhile moved most of these into postgres. The only exception is openid_connect_signing_key since it needs to be accessed from a Rails initializer before the database may be ready.

Secret entries

EntryDescription
secret_key_baseThe base key to be used for generating a various secrets
otp_key_baseThe base key for One Time Passwords, described in User management
db_key_baseThe base key to encrypt the data for attr_encrypted columns
openid_connect_signing_keyThe signing key for OpenID Connect
encrypted_settings_key_baseThe base key to encrypt settings files with
active_record_encryption_primary_keyThe base key to non-deterministically-encrypt data for ActiveRecord::Encryption encrypted columns
active_record_encryption_deterministic_keyThe base key to deterministically-encrypt data for ActiveRecord::Encryption encrypted columns
active_record_encryption_key_derivation_saltThe derivation salt to encrypt data for ActiveRecord::Encryption encrypted columns

Where the secrets are stored

Installation typeLocation
Linux package/etc/gitlab/gitlab-secrets.json
Cloud Native GitLab ChartsKubernetes Secrets
Self-compiled<path-to-gitlab-rails>/config/secrets.yml (Automatically generated by config/initializers/01_secret_token.rb)

Warning: Before you add a new secret to application secrets

Add support to Omnibus GitLab and the Cloud Native GitLab charts

Before you add a new secret to config/initializers/01_secret_token.rb, make sure you also update Omnibus GitLab and the Cloud Native GitLab charts, or the update will fail. Both installation methods are responsible for writing the config/secrets.yml file. If if they don’t know about a secret, Rails attempts to write to the file, and fails because it doesn’t have write access.

Examples

Populate the secrets in live environments

Additionally, in case you need the secret to have the same value on all nodes (which is usually the case), you need to make sure it’s configured for all live environments (GitLab.com, staging, pre) prior to changing this file.

Document the new secrets

  1. Add the new secrets to this documentation file.

  2. Mention the new secrets in the next release upgrade notes. For instance, for the 17.8 release, the notes would go in data/release_posts/17_8/17-8-upgrade.yml and contain something like the following:

    ---
upgrades:
  - reporter: <your username>  # item author username
    description: |
      In Gitlab 17.8, three new secrets have been added to support the upcoming encryption framework:
      - `active_record_encryption_primary_key`
      - `active_record_encryption_deterministic_key`
      - `active_record_encryption_key_derivation_salt`

      **If you have a multi-node configuration, you should ensure these secrets are the same on all nodes.** Otherwise, the application will automatically generate the missing secrets.

      If you use the [GitLab helm chart](https://docs.gitlab.com/charts/) and disabled the [shared-secrets chart](https://docs.gitlab.com/charts/charts/shared-secrets/), you will need to [manually  create these secrets](https://docs.gitlab.com/charts/installation/secrets.html#gitlab-rails-secret).

  3. Mention the new secrets in the next Cloud Native GitLab charts upgrade notes. For instance, for 8.8, you should document the new secrets in https://docs.gitlab.com/charts/releases/8_0.html.

Further iteration

We may either deprecate or remove this automatic secret generation performed by config/initializers/01_secret_token.rb in the future. See issue #222690 for more information.