Use HashiCorp Vault secrets in GitLab CI/CD
- Tier: Premium, Ultimate
- Offering: GitLab.com, GitLab Self-Managed, GitLab Dedicated
You can use HashiCorp Vault secrets in GitLab CI/CD. Use ID tokens to authenticate with HashiCorp Vault.
You must configure your Vault server before you can use Vault secrets in a CI/CD job. The Authenticating and reading secrets with HashiCorp Vault tutorial has more details about configuring Vault and authenticating with ID tokens.
In the following examples, replace vault.example.com
with the URL of your Vault server,
and gitlab.example.com
with the URL of your GitLab instance.
Configure your Vault server
To configure your Vault server:
Enable the authentication method by running these commands. They provide your Vault server the OIDC Discovery URL for your GitLab instance, so Vault can fetch the public signing key and verify the JSON Web Token (JWT) when authenticating:
$ vault auth enable jwt $ vault write auth/jwt/config \ oidc_discovery_url="https://gitlab.example.com" \ bound_issuer="gitlab.example.com"
Configure policies on your Vault server to grant or forbid access to certain paths and operations. This example grants read access to the set of secrets required by your production environment:
vault policy write myproject-production - <<EOF # Read-only permission on 'ops/data/production/*' path path "ops/data/production/*" { capabilities = [ "read" ] } EOF
Configure roles on your Vault server, restricting roles to a project or namespace.
Create the following CI/CD variables to provide details about your Vault server:
VAULT_SERVER_URL
: The URL of your Vault server, such ashttps://vault.example.com:8200
.VAULT_AUTH_ROLE
: Optional. The role to use when attempting to authenticate. If no role is specified, Vault uses the default role specified when the authentication method was configured.VAULT_AUTH_PATH
: Optional. The path where the authentication method is mounted, default isjwt
.VAULT_NAMESPACE
: Optional. The Vault Enterprise namespace to use for reading secrets and authentication. With:- Vault, the
root
("/
") namespace is used when no namespace is specified. - Vault Open source, the setting is ignored.
- HashiCorp Cloud Platform (HCP) Vault, a namespace
is required. HCP Vault uses the
admin
namespace as the root namespace by default. For example,VAULT_NAMESPACE=admin
.
- Vault, the
Configure server roles
When a CI/CD job attempts to authenticate, it specifies a role. You can use roles to group different policies together. If authentication is successful, these policies are attached to the resulting Vault token.
Bound claims are predefined values that are matched to the JWT claims. With bounded claims, you can restrict access to specific GitLab users, specific projects, or even jobs running for specific Git references. You can have as many bounded claims you need, but they must all match for authentication to be successful.
Combining bounded claims with GitLab features like user roles and protected branches, you can tailor these rules to fit your specific use case. In this example, authentication is allowed only for jobs running for protected tags with names matching the pattern used for production releases:
$ vault write auth/jwt/role/myproject-production - <<EOF
{
"role_type": "jwt",
"policies": ["myproject-production"],
"token_explicit_max_ttl": 60,
"user_claim": "user_email",
"bound_audiences": "https://vault.example.com",
"bound_claims_type": "glob",
"bound_claims": {
"project_id": "42",
"ref_protected": "true",
"ref_type": "tag",
"ref": "auto-deploy-*"
}
}
EOF
Always restrict your roles to a project or namespace by using one of the provided
claims like project_id
or namespace_id
. Without these restrictions, any JWT
generated by this GitLab instance may be allowed to authenticate using this role.
For a full list of ID token JWT claims, review the Use HashiCorp Vault secrets in GitLab CI/CD tutorial.
You can also specify some attributes for the resulting Vault tokens, such as time-to-live, IP address range, and number of uses. The full list of options is available in Vault’s documentation on creating roles for the JSON web token method.
Use Vault secrets in a CI/CD job
When a job has at least one ID token defined, the secrets
keyword automatically uses that token to authenticate with Vault.
After configuring your Vault server, use the secrets:vault
keyword to use the secrets stored in Vault:
job_using_vault:
id_tokens:
VAULT_ID_TOKEN:
aud: https://vault.example.com
secrets:
DATABASE_PASSWORD:
vault: production/db/password@ops
token: $VAULT_ID_TOKEN
In this example:
production/db
is the path to the secret.password
is the field.ops
is the path where the secrets engine is mounted.production/db/password@ops
translates to a path ofops/data/production/db
.- Authentication is with
$VAULT_ID_TOKEN
.
After GitLab fetches the secret from Vault, the value is saved in a temporary file.
The path to this file is stored in a CI/CD variable named DATABASE_PASSWORD
,
similar to variables of type file
.
To overwrite the default behavior, set the file
option explicitly:
secrets:
id_tokens:
VAULT_ID_TOKEN:
aud: https://vault.example.com
DATABASE_PASSWORD:
vault: production/db/password@ops
file: false
token: $VAULT_ID_TOKEN
In this example, the secret value is put directly in the DATABASE_PASSWORD
variable
instead of pointing to a file that holds it.
Secrets engines
GitLab Runner supports different secrets engines with the secrets:engine:name
keyword:
Secrets engine | secrets:engine:name value | Runner version | Details |
---|---|---|---|
KV secrets engine - version 2 | kv-v2 | 13.4 | kv-v2 is the default engine GitLab Runner uses when no engine type is explicitly specified. |
KV secrets engine - version 1 | kv-v1 or generic | 13.4 | Support for the generic keyword introduced in GitLab 15.11. |
AWS secrets engine | generic | 16.11 | |
HashiCorp Vault Artifactory Secrets Plugin | generic | 16.11 | This secrets backend talks to JFrog Artifactory server (5.0.0 or later) and dynamically provisions access tokens with specified scopes. |
Use a different secrets engine
The kv-v2
secrets engine is used by default. To use a different engine, add an engine
section
under vault
in the configuration.
For example, to set the secret engine and path for Artifactory:
job_using_vault:
id_tokens:
VAULT_ID_TOKEN:
aud: https://vault.example.com
secrets:
JFROG_TOKEN:
vault:
engine:
name: generic
path: artifactory
path: production/jfrog
field: access_token
file: false
In this example, the secret value is obtained from artifactory/production/jfrog
with a field of access_token
.
Troubleshooting
Self-signed certificate error: certificate signed by unknown authority
When the Vault server is using a self-signed certificate, you see the following error in the job logs:
ERROR: Job failed (system failure): resolving secrets: initializing Vault service: preparing authenticated client: checking Vault server health: Get https://vault.example.com:8000/v1/sys/health?drsecondarycode=299&performancestandbycode=299&sealedcode=299&standbycode=299&uninitcode=299: x509: certificate signed by unknown authority
You have two options to solve this error:
- Add the self-signed certificate to the GitLab Runner server’s CA store. If you deployed GitLab Runner using the Helm chart, you have to create your own GitLab Runner image.
- Use the
VAULT_CACERT
environment variable to configure GitLab Runner to trust the certificate:- If you are using systemd to manage GitLab Runner, see how to add an environment variable for GitLab Runner.
- If you deployed GitLab Runner using the Helm chart:
Provide a custom certificate for accessing GitLab, and make sure to add the certificate for the Vault server instead of the certificate for GitLab. If your GitLab instance is also using a self-signed certificate, you should be able to add both in the same
Secret
.Add the following lines in your
values.yaml
file:## Replace both the <SECRET_NAME> and the <VAULT_CERTIFICATE> ## with the actual values you used to create the secret certsSecretName: <SECRET_NAME> envVars: - name: VAULT_CACERT value: "/home/gitlab-runner/.gitlab-runner/certs/<VAULT_CERTIFICATE>"
If you are running vault server in development mode locally with GitLab Development Kit (GDK), you might also get this error. You can manually ask the system to trust the self signed certificate of Vault server. This sample tutorial explains how to do this on macOS.
resolving secrets: secret not found: MY_SECRET
error
When GitLab is unable to find the secret in the vault, you might receive this error:
ERROR: Job failed (system failure): resolving secrets: secret not found: MY_SECRET
Check that the vault
value is correctly configured in the CI/CD job.
You can use the kv
command with the Vault CLI
to check if the secret is retrievable to help determine the syntax for the vault
value in your CI/CD configuration. For example, to retrieve the secret:
$ vault kv get -field=password -namespace=admin -mount=ops "production/db"
this-is-a-password