- Supported options for self-signed certificates targeting the GitLab server
- Trusting TLS certificates for Docker and Kubernetes executors
Introduced in GitLab Runner 0.7.0
GitLab Runner provides two options to configure certificates to be used to verify TLS peers:
For connections to the GitLab server: the certificate file can be specified as detailed in the Supported options for self-signed certificates targeting the GitLab server section.
This solves the
x509: certificate signed by unknown authorityproblem when registering a runner.
For existing Runners, the same error can be seen in Runner logs when trying to check the jobs:
Couldn't execute POST against https://hostname.tld/api/v4/jobs/request: Post https://hostname.tld/api/v4/jobs/request: x509: certificate signed by unknown authority
A more generic approach which also covers other scenarios such as user scripts, connecting to a cache server or an external Git LFS store: a certificate can be specified and installed on the container as detailed in the Trusting TLS certificates for Docker and Kubernetes executors section.
An example job log error concerning a Git LFS operation that is missing a certificate:
LFS: Get https://object.hostname.tld/lfs-dev/c8/95/a34909dce385b85cee1a943788044859d685e66c002dbf7b28e10abeef20?X-Amz-Expires=600&X-Amz-Date=20201006T043010Z&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=svcgitlabstoragedev%2F20201006%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-SignedHeaders=host&X-Amz-Signature=012211eb0ff0e374086e8c2d37556f2d8ca4cc948763e90896f8f5774a100b55: x509: certificate signed by unknown authority
This section refers to the situation where only the GitLab server requires a custom certificate. If other hosts also require a custom certificate authority (CA), please see the next section.
GitLab Runner supports the following options:
Default - Read the system certificate: GitLab Runner reads the system certificate store and verifies the GitLab server against the certificate authorities (CA) stored in the system. Note that reading from the system certificate store is not supported in Windows.
Specify a custom certificate file: GitLab Runner exposes the
tls-ca-fileoption during registration (
gitlab-runner register --tls-ca-file=/path), and in
[[runners]]section. This allows you to specify a custom certificate file. This file will be read every time the runner tries to access the GitLab server.
Read a PEM certificate: GitLab Runner reads the PEM certificate (DER format is not supported) from a predefined file:
/etc/gitlab-runner/certs/hostname.crton *nix systems when GitLab Runner is executed as root.
If your server address is
https://my.gitlab.server.com:8443/, create the certificate file at:
/etc/gitlab-runner/certs/my.gitlab.server.com.crt. To verify that the file is correctly installed, you can use a tool like
openssl. For example:
echo | openssl s_client -CAfile /etc/gitlab-runner/certs/gitlab-hostname.tld.crt -connect gitlab-hostname.tld:443
~/.gitlab-runner/certs/hostname.crton *nix systems when GitLab Runner is executed as non-root.
./certs/hostname.crton other systems. If running GitLab Runner as a Windows service, this will not work. Specify a custom certificate file instead.
If your GitLab server certificate is signed by your CA, use your CA certificate (not your GitLab server signed certificate). You might need to add the intermediates to the chain as well. For example, if you have a primary, intermediate, and root certificate, you can put all of them into one file:
-----BEGIN CERTIFICATE----- (Your primary SSL certificate: your_domain_name.crt) -----END CERTIFICATE----- -----BEGIN CERTIFICATE----- (Your intermediate certificate) -----END CERTIFICATE----- -----BEGIN CERTIFICATE----- (Your root certificate) -----END CERTIFICATE-----
- If you are updating the certificate for an existing Runner, restart it.
- As a temporary and insecure workaround, to skip the verification of certificates,
variables:section of your
.gitlab-ci.ymlfile, set the CI variable
- If you are using GitLab Runner Helm chart, you will need to configure certificates according to the doc Providing a custom certificate for accessing GitLab.
The runner injects missing certificates to build the CA chain in build containers.
git clone and
artifacts to work with servers that do not use publicly
This approach is secure, but makes the runner a single point of trust.
There are two contexts that need to be taken into account when we consider registering a certificate on a container:
- The user image, which is used to run the user script. In this scenario, the user must take ownership regarding how to install a certificate, since this is highly dependent on the image itself, and the runner has no way of knowing how to install a certificate in each possible scenario.
- The Runner helper image, which is used for standard operations such as fetching sources,
uploading artifacts, etc. In this scenario, the user only needs to make a certificate file
available at a specific location (e.g.
/etc/gitlab-runner/certs/ca.crt), and the Docker container will automatically install it for the user.
If your build script needs to communicate with peers through TLS and needs to rely on
a self-signed certificate or custom Certificate Authority, you will need to perform the
certificate installation in the build job, as the Docker container running the user scripts
doesn’t have the certificate files installed by default. This might be required to use
a custom cache host, perform a secondary
git clone, or fetch a file through a tool like
To install the certificate:
Map the necessary files as a Docker volume so that the Docker container that will run the scripts can see them. Do this by adding a volume inside the respective key inside the
config.tomlfile, for example:
[[runners]] name = "docker" url = "https://example.com/" token = "TOKEN" executor = "docker" [runners.docker] image = "ubuntu:latest" # Add path to your ca.crt file in the volumes list volumes = ["/cache", "/path/to-ca-cert-dir/ca.crt:/etc/gitlab-runner/certs/ca.crt:ro"]
Linux-only: Use the mapped file (e.g
ca.crt) in a
- Copies it to
/usr/local/share/ca-certificates/ca.crtinside the Docker container.
Installs it by running
update-ca-certificates --fresh. For example (commands vary based on the distribution you’re using):
[[runners]] name = "docker" url = "https://example.com/" token = "TOKEN" executor = "docker" # Copy and install CA certificate before each job pre_build_script = """ apt-get update -y > /dev/null apt-get install -y ca-certificates > /dev/null cp /etc/gitlab-runner/certs/ca.crt /usr/local/share/ca-certificates/ca.crt update-ca-certificates --fresh > /dev/null """
[[runners]] name = "docker" url = "https://example.com/" token = "TOKEN" executor = "docker" # Copy and install CA certificate before each job pre_build_script = """ apk update >/dev/null apk add ca-certificates > /dev/null rm -rf /var/cache/apk/* cp /etc/gitlab-runner/certs/ca.crt /usr/local/share/ca-certificates/ca.crt update-ca-certificates --fresh > /dev/null """
- Copies it to
Introduced in GitLab 13.3.
You can map a certificate file to
/etc/gitlab-runner/certs/ca.crt on Linux,
C:\GitLab-Runner\certs\ca.crt on Windows.
The Runner helper image installs this user-defined
ca.crt file at start-up, and uses it
when performing operations like cloning and uploading artifacts, for example.
[[runners]] name = "docker" url = "https://example.com/" token = "TOKEN" executor = "docker" [runners.docker] image = "mcr.microsoft.com/windows/servercore:2004" # Add directory holding your ca.crt file in the volumes list volumes = ["c:\\cache", "c:\\path\\to-ca-cert-dir:C:\\GitLab-Runner\\certs:ro"]
Due to a known issue in the Kubernetes executor’s
handling of the helper image’s
ENTRYPOINT, the mapped certificate file isn’t automatically installed
to the system certificate store.
Refer to the general SSL troubleshooting documentation.
In addition, you can use the
tlsctl tool to debug GitLab certificates from the runner’s end.