• Passing properties to GitLab Runner Operator
• Operator properties
• Cache properties
  • S3 cache
  • GCS cache
  • Azure cache
• Configure a proxy environment
• Customize config.toml with a configuration template
• Configure a custom TLS cert
• Configure the CPU and memory size of runner pods
• Configure job concurrency per runner based on cluster resources
• Troubleshooting
  • Root vs non-root
    • Overriding the HOME environment variable
    • Watch out for SCC
    • Run As anyuid SCC
    • Configure SETFCAP
  • Using FIPS Compliant GitLab Runner
    • Register GitLab Runner by using a self-signed certificate
    • Register GitLab Runner with an external URL that points to an IP address

Configuring GitLab Runner on OpenShift

This document explains how to configure GitLab Runner on OpenShift.

Passing properties to GitLab Runner Operator

When creating a Runner, you can configure it by setting properties in its spec. For example, you can specify the GitLab URL it will be registered in, or the name of the secret that contains the registration token:

apiVersion: apps.gitlab.com/v1beta2
kind: Runner
metadata:
 name: dev
spec:
 gitlabUrl: https://gitlab.example.com
 token: gitlab-runner-secret # Name of the secret containing the Runner token

Read about all the available properties in Operator properties.

Operator properties

This is a list of the supported properties that can be passed to the Operator.

Some properties are only available with more recent versions of the Operator.

Cache properties

S3 cache

GCS cache

Azure cache

Configure a proxy environment

To create a proxy environment:

1. Edit the custom-env.yaml file. For example:

   apiVersion: v1
   data:
     HTTP_PROXY: example.com
   kind: ConfigMap
   metadata:
     name: custom-env
   

2. Update OpenShift to apply the changes.

   oc apply -f custom-env.yaml
   

3. Update your gitlab-runner.yml file.

   apiVersion: apps.gitlab.com/v1beta2
   kind: Runner
   metadata:
     name: dev
   spec:
     gitlabUrl: https://gitlab.example.com
     token: gitlab-runner-secret # Name of the secret containing the Runner token
     env: custom-env
   

If the proxy can’t reach the Kubernetes API, you might see an error in your CI/CD job:

ERROR: Job failed (system failure): prepare environment: setting up credentials: Post https://172.21.0.1:443/api/v1/namespaces/<KUBERNETES_NAMESPACE>/secrets: net/http: TLS handshake timeout. Check https://docs.gitlab.com/runner/shells/index.html#shell-profile-loading for more information

To resolve this error, add the IP address of the Kubernetes API to NO_PROXY configuration in the custom-env.yaml file:

   apiVersion: v1
   data:
     NO_PROXY: 172.21.0.1
     HTTP_PROXY: example.com
   kind: ConfigMap
   metadata:
     name: custom-env

You can verify the IP address of the Kubernetes API by running:

oc get services --namespace default --field-selector='metadata.name=kubernetes' | grep -v NAME | awk '{print $3}'

Customize config.toml with a configuration template

You can customize the runner’s config.toml file by using the configuration template.

1. Create a custom config template file. For example, let’s instruct our runner to mount an EmptyDir volume. Create the custom-config.toml file:

   [[runners]]
     [runners.kubernetes]
       [runners.kubernetes.volumes]
         [[runners.kubernetes.volumes.empty_dir]]
           name = "empty-dir"
           mount_path = "/path/to/empty_dir"
           medium = "Memory"
   

2. Create a ConfigMap named custom-config-toml from our custom-config.toml file:

    oc create configmap custom-config-toml --from-file config.toml=custom-config.toml
   

3. Set the config property of the Runner:

    apiVersion: apps.gitlab.com/v1beta2
    kind: Runner
    metadata:
      name: dev
    spec:
      gitlabUrl: https://gitlab.example.com
      token: gitlab-runner-secret
      config: custom-config-toml
   

Configure a custom TLS cert

1. To set a custom TLS cert, create a secret with key tls.crt. In this example, the file is named custom-tls-ca-secret.yaml:

    apiVersion: v1
    kind: Secret
    metadata:
        name: custom-tls-ca
    type: Opaque
    stringData:
        tls.crt: |
            -----BEGIN CERTIFICATE-----
            MIIEczCCA1ugAwIBAgIBADANBgkqhkiG9w0BAQQFAD..AkGA1UEBhMCR0Ix
            .....
            7vQMfXdGsRrXNGRGnX+vWDZ3/zWI0joDtCkNnqEpVn..HoX
            -----END CERTIFICATE-----
   

2. Create the secret:

   oc apply -f custom-tls-ca-secret.yaml
   

3. Set the ca key in the runner.yaml to the same name as the name of our secret:

    apiVersion: apps.gitlab.com/v1beta2
    kind: Runner
    metadata:
      name: dev
    spec:
      gitlabUrl: https://gitlab.example.com
      token: gitlab-runner-secret
      ca: custom-tls-ca
   

Configure the CPU and memory size of runner pods

To set CPU limits and memory limits in a custom config.toml file, follow the instructions in this topic.

Configure job concurrency per runner based on cluster resources

Set the concurrent property of the Runner resource:

   apiVersion: apps.gitlab.com/v1beta2
   kind: Runner
   metadata:
     name: dev
   spec:
     gitlabUrl: https://gitlab.example.com
     token: gitlab-runner-secret
     concurrent: 2

Job concurrency is dictated by the requirements of the project.

1. Start by trying to determine the compute and memory resources required to execute a CI job.
2. Calculate how many times that job would be able to execute given the resources in the cluster.

If you set too large a concurrency value, the Kubernetes executor will process the jobs as soon as it can. However, the Kubernetes cluster’s scheduler capacity determines when the job is scheduled.

Troubleshooting

Root vs non-root

The GitLab Runner Operator and the GitLab Runner pod run as non-root users. As a result, the build image used in the job would need to run as a non-root user to be able to complete successfully. This is to ensure that jobs can run successfully with the least permission. However, for this to work, the build image used for the CI jobs also needs to be built to run as non-root and should not write to a restricted filesystem. Keep in mind that most container filesystems on an OpenShift cluster will be read-only, except for mounted volumes, /var/tmp, /tmp and other volumes mounted on the root filesystem as tmpfs.

Overriding the HOME environment variable

If creating a custom build image or overriding env variables, ensure that the HOME environment variables is not set to / which would be read-only. Especially if your jobs would need to write files to the home directory. You could create a directory under /home for example /home/ci and set ENV HOME=/home/ci in your Dockerfile.

For the runner pods it’s expected that HOME would be set to /home/gitlab-runner. If this variable is changed, the new location must have the proper permissions. These guidelines are also documented in the Red Hat Container Platform Docs > Creating Images > Support arbitrary user ids.

Watch out for SCC

By default, when installed in a new OpenShift project, the GitLab Runner Operator will run as non-root. There are exceptions, when all the service accounts in a project are granted anyuid access, such as the default project. In that case, the user of the image will be root. This can be easily checked by running the whoami inside any container shell, e.g. a job. Read more about SCC in Red Hat Container Platform Docs > Managing security context constraints.

Run As anyuid SCC

Though discouraged, in the event that is it absolutely necessary for a CI job to run as the root user or to write to the root filesystem, you will need to set the anyuid SCC on the GitLab Runner service account, gitlab-runner-sa, which is used by the GitLab Runner container.

In OpenShift 4.3.8 and earlier:

oc adm policy add-scc-to-user anyuid -z gitlab-runner-sa -n <runner_namespace>

# Check that the anyiud SCC is set:
oc get scc anyuid -o yaml

In OpenShift 4.3.8 and later:

oc create -f - <<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: scc-anyuid
  namespace: <runner_namespace>
rules:
- apiGroups:
  - security.openshift.io
  resourceNames:
  - anyuid
  resources:
  - securitycontextconstraints
  verbs:
  - use
EOF

oc create -f - <<EOF
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: sa-to-scc-anyuid
  namespace: <runner_namespace>
subjects:
  - kind: ServiceAccount
    name: gitlab-runner-sa
roleRef:
  kind: Role
  name: scc-anyuid
  apiGroup: rbac.authorization.k8s.io
EOF

Configure SETFCAP

If you use Red Hat OpenShift Container Platform (RHOCP) 4.11 or later, you may get the following error message:

error reading allowed ID mappings:error reading subuid mappings for user

Some jobs (for example, buildah) need the SETFCAP capability granted to run correctly. To fix this issue:

1. Add the SETFCAP capability to the SCC that GitLab Runner is using (replace the gitlab-scc with the SCC assigned to your GitLab Runner pod):

    oc patch scc gitlab-scc --type merge -p '{"allowedCapabilities":["SETFCAP"]}'
   

2. Update your config.toml and add the SETFCAP capability under the kubernetes section:

    [[runners]]
      [runners.kubernetes]
      [runners.kubernetes.pod_security_context]
        [runners.kubernetes.build_container_security_context]
        [runners.kubernetes.build_container_security_context.capabilities]
          add = ["SETFCAP"]
   

3. Create a configmap using this config.toml in the namespace where GitLab Runner is deployed:

    oc create configmap custom-config-toml --from-file config.toml=config.toml
   

4. Modify the runner you want to fix, adding the config: parameter to point to the recently created configmap (replace my-runner with the correct runner pod name)

    oc patch runner my-runner --type merge -p '{"spec": {"config": "custom-config-toml"}}'
   

For more inforamtion, see the Red Hat documentation.

Using FIPS Compliant GitLab Runner

To use a FIPS compliant GitLab Runner Helper, change the helper image as follows:

apiVersion: apps.gitlab.com/v1beta2
kind: Runner
metadata:
 name: dev
spec:
 gitlabUrl: https://gitlab.example.com
 token: gitlab-runner-secret
 helperImage: gitlab/gitlab-runner-helper:ubi-fips
 concurrent: 2

Register GitLab Runner by using a self-signed certificate

When you use a self-signed certificate with your GitLab self-managed installation, you must create a secret that contains the CA certificate used to sign your private certificates.

The name of the secret is then provided as the CA in the Runner spec section:

KIND:     Runner
VERSION:  apps.gitlab.com/v1beta2

FIELD:    ca <string>

DESCRIPTION:
     Name of tls secret containing the custom certificate authority (CA)
     certificates

The secret can be created using the following command:

oc create secret generic mySecret --from-file=tls.crt=myCert.pem -o yaml

Register GitLab Runner with an external URL that points to an IP address

If the runner cannot match the self-signed certificate with the hostname, you might get an error message. This can happen when the GitLab self-managed instance is configured to be accessed from an IP address instead of a hostname (where ###.##.##.## is the IP address of the GitLab server):

[31;1mERROR: Registering runner... failed               [0;m  [31;1mrunner[0;m=A5abcdEF [31;1mstatus[0;m=couldn't execute POST against https://###.##.##.##/api/v4/runners:
Post https://###.##.##.##/api/v4/runners: x509: cannot validate certificate for ###.##.##.## because it doesn't contain any IP SANs
[31;1mPANIC: Failed to register the runner. You may be having network problems.[0;m

To fix this issue:

1. On the GitLab self-managed server, modify the openssl to add the IP address to the subjectAltName parameter:

   # vim /etc/pki/tls/openssl.cnf
   
   [ v3_ca ]
   subjectAltName=IP:169.57.64.36 <---- Add this line. 169.57.64.36 is your GitLab server IP.
   

2. Then re-generate a self-signed CA with the commands below:

   # cd /etc/gitlab/ssl
   # openssl req -x509 -nodes -days 3650 -newkey rsa:4096 -keyout /etc/gitlab/ssl/169.57.64.36.key -out /etc/gitlab/ssl/169.57.64.36.crt
   # openssl dhparam -out /etc/gitlab/ssl/dhparam.pem 4096
   # gitlab-ctl restart
   

3. Use this new certificate to generate a new secret.