• Workflow
• Connecting to the Kubernetes API
• Kubernetes executor interaction diagram
• The keywords
  • CPU requests and limits
  • Memory requests and limits
  • Storage requests and limits
  • Other keywords
  • Configuring executor Service Account
  • Overwriting Kubernetes Namespace
  • Overwriting Kubernetes Default Service Account
  • Setting Bearer Token to be Used When Making Kubernetes API calls
  • Overwriting pod annotations
  • Overwriting Container Resources
• Define keywords in the configuration TOML
• Using volumes
  • Host Path volumes
  • PVC volumes
  • ConfigMap volumes
  • Secret volumes
  • Empty Dir volumes
• Using Security Context
• Using services
• Adding extra host aliases
• Using Affinity
  • Node Affinity
• Pod’s DNS Config
  • KubernetesDNSConfigOption
• Capabilities configuration
  • Default list of dropped capabilities
  • Example configuration
• Using Docker in your builds
  • Exposing /var/run/docker.sock
  • Using docker:dind
  • Not supplying Git
  • Resource separation
  • Using kaniko
• Job execution

The Kubernetes executor

GitLab Runner can use Kubernetes to run builds on a Kubernetes cluster. This is possible with the use of the Kubernetes executor.

The Kubernetes executor, when used with GitLab CI, connects to the Kubernetes API in the cluster creating a Pod for each GitLab CI Job. This Pod is made up of, at the very least, a build container, a helper container, and an additional container for each service defined in the .gitlab-ci.yml or config.toml files. The names for these containers are as follows:

• The build container is build
• The helper container is helper
• The services containers are svc-X where X is [0-9]+

Note that when services and containers are running in the same Kubernetes pod, they are all sharing the same localhost address. The following restrictions are then applicable:

• Since GitLab Runner 12.8 and Kubernetes 1.7, the services are accessible via their DNS names. If you are using an older version you will have to use localhost.
• You cannot use several services using the same port (e.g., you cannot have two mysql services at the same time).

Workflow

The Kubernetes executor divides the build into multiple steps:

1. Prepare: Create the Pod against the Kubernetes Cluster. This creates the containers required for the build and services to run.
2. Pre-build: Clone, restore cache and download artifacts from previous stages. This is run on a special container as part of the Pod.
3. Build: User build.
4. Post-build: Create cache, upload artifacts to GitLab. This also uses the special container as part of the Pod.

Connecting to the Kubernetes API

The following options are provided, which allow you to connect to the Kubernetes API:

• host: Optional Kubernetes apiserver host URL (auto-discovery attempted if not specified)
• cert_file: Optional Kubernetes apiserver user auth certificate
• key_file: Optional Kubernetes apiserver user auth private key
• ca_file: Optional Kubernetes apiserver ca certificate

The user account provided must have permission to create, list and attach to Pods in the specified namespace in order to function.

If you are running GitLab Runner within the Kubernetes cluster you can omit all of the above fields to have GitLab Runner auto-discover the Kubernetes API. This is the recommended approach.

If you are running it externally to the Cluster then you will need to set each of these keywords and make sure that GitLab Runner has access to the Kubernetes API on the cluster.

Kubernetes executor interaction diagram

The diagram below depicts the interaction with a GitLab Runner hosted on a Kubernetes cluster and the Kubernetes API. The Kubernetes API is the mechanism that is used by GitLab Runner on Kubernetes to create pods on the cluster. The interaction depicted in this diagram is valid on any Kubernetes cluster, whether that’s a turnkey solution hosted on the major public cloud providers or a self-managed Kubernetes installation.

The keywords

The following keywords help to define the behavior of GitLab Runner within Kubernetes.

CPU requests and limits

Memory requests and limits

Storage requests and limits

Other keywords

Configuring executor Service Account

You can set the KUBERNETES_SERVICE_ACCOUNT environment variable or use --kubernetes-service-account flag.

Overwriting Kubernetes Namespace

Additionally, Kubernetes namespace can be overwritten on .gitlab-ci.yml file, by using the variable KUBERNETES_NAMESPACE_OVERWRITE.

This approach allows you to create a new isolated namespace dedicated for CI purposes, and deploy a custom set of Pods. The Pods spawned by the runner will take place on the overwritten namespace, for simple and straight forward access between containers during the CI stages.

variables:
  KUBERNETES_NAMESPACE_OVERWRITE: ci-${CI_COMMIT_REF_SLUG}

Furthermore, to ensure only designated namespaces will be used during CI runs, set the configuration namespace_overwrite_allowed with an appropriate regular expression. When left empty the overwrite behavior is disabled.

Overwriting Kubernetes Default Service Account

Additionally, the Kubernetes service account can be overwritten in the .gitlab-ci.yml file by using the variable KUBERNETES_SERVICE_ACCOUNT_OVERWRITE.

This approach allows you to specify a service account that is attached to the namespace, which is useful when dealing with complex RBAC configurations.

variables:
  KUBERNETES_SERVICE_ACCOUNT_OVERWRITE: ci-service-account

To ensure only designated service accounts will be used during CI runs, set the configuration service_account_overwrite_allowed or set the environment variable KUBERNETES_SERVICE_ACCOUNT_OVERWRITE_ALLOWED with an appropriate regular expression. When left empty the overwrite behavior is disabled.

Setting Bearer Token to be Used When Making Kubernetes API calls

In conjunction with setting the namespace and service account as mentioned above, you may set the bearer token used when making API calls to create the build pods. This will allow project owners to use project secret variables to specify a bearer token. When specifying the bearer token, you must set the Host configuration keyword.

variables:
  KUBERNETES_BEARER_TOKEN: thebearertokenfromanothernamespace

Overwriting pod annotations

Additionally, Kubernetes pod annotations can be overwritten on the .gitlab-ci.yml file, by using KUBERNETES_POD_ANNOTATIONS_* for variables and key=value for the value. The pod annotations will be overwritten to the key=value. Multiple annotations can be applied. For example:

variables:
  KUBERNETES_POD_ANNOTATIONS_1: "Key1=Val1"
  KUBERNETES_POD_ANNOTATIONS_2: "Key2=Val2"
  KUBERNETES_POD_ANNOTATIONS_3: "Key3=Val3"

Overwriting Container Resources

Additionally, Kubernetes CPU and memory allocations for requests and limits for the build, helper and service containers can be overwritten on the .gitlab-ci.yml file with the following variables:

 variables:
   KUBERNETES_CPU_REQUEST: 3
   KUBERNETES_CPU_LIMIT: 5
   KUBERNETES_MEMORY_REQUEST: 2Gi
   KUBERNETES_MEMORY_LIMIT: 4Gi
   KUBERNETES_EPHEMERAL_STORAGE_REQUEST: 512Mi
   KUBERNETES_EPHEMERAL_STORAGE_LIMIT: 1Gi

   KUBERNETES_HELPER_CPU_REQUEST: 3
   KUBERNETES_HELPER_CPU_LIMIT: 5
   KUBERNETES_HELPER_MEMORY_REQUEST: 2Gi
   KUBERNETES_HELPER_MEMORY_LIMIT: 4Gi
   KUBERNETES_HELPER_EPHEMERAL_STORAGE_REQUEST: 512Mi
   KUBERNETES_HELPER_EPHEMERAL_STORAGE_LIMIT: 1Gi

   KUBERNETES_SERVICE_CPU_REQUEST: 3
   KUBERNETES_SERVICE_CPU_LIMIT: 5
   KUBERNETES_SERVICE_MEMORY_REQUEST: 2Gi
   KUBERNETES_SERVICE_MEMORY_LIMIT: 4Gi
   KUBERNETES_SERVICE_EPHEMERAL_STORAGE_REQUEST: 512Mi
   KUBERNETES_SERVICE_EPHEMERAL_STORAGE_LIMIT: 1Gi

The values for these variables are restricted to what the max overwrite for that resource has been set to.

Define keywords in the configuration TOML

Each of the keywords can be defined in the config.toml file.

Here is an example config.toml:

concurrent = 4

[[runners]]
  name = "Kubernetes Runner"
  url = "https://gitlab.com/ci"
  token = "......"
  executor = "kubernetes"
  [runners.kubernetes]
    host = "https://45.67.34.123:4892"
    cert_file = "/etc/ssl/kubernetes/api.crt"
    key_file = "/etc/ssl/kubernetes/api.key"
    ca_file = "/etc/ssl/kubernetes/ca.crt"
    namespace = "gitlab"
    namespace_overwrite_allowed = "ci-.*"
    bearer_token_overwrite_allowed = true
    privileged = true
    cpu_limit = "1"
    memory_limit = "1Gi"
    service_cpu_limit = "1"
    service_memory_limit = "1Gi"
    helper_cpu_limit = "500m"
    helper_memory_limit = "100Mi"
    poll_interval = 5
    poll_timeout = 3600
    dns_policy = "cluster-first"
    [runners.kubernetes.node_selector]
      gitlab = "true"
    [runners.kubernetes.node_tolerations]
      "node-role.kubernetes.io/master" = "NoSchedule"
      "custom.toleration=value" = "NoSchedule"
      "empty.value=" = "PreferNoSchedule"
      "onlyKey" = ""

Using volumes

As described earlier, volumes can be mounted in the build container. At this time hostPath, PVC, configMap, and secret volume types are supported. Users can configure any number of volumes for each of mentioned types.

Here is an example configuration:

concurrent = 4

[[runners]]
  # usual configuration
  executor = "kubernetes"
  [runners.kubernetes]
    [[runners.kubernetes.volumes.host_path]]
      name = "hostpath-1"
      mount_path = "/path/to/mount/point"
      read_only = true
      host_path = "/path/on/host"
    [[runners.kubernetes.volumes.host_path]]
      name = "hostpath-2"
      mount_path = "/path/to/mount/point_2"
      read_only = true
    [[runners.kubernetes.volumes.pvc]]
      name = "pvc-1"
      mount_path = "/path/to/mount/point1"
    [[runners.kubernetes.volumes.config_map]]
      name = "config-map-1"
      mount_path = "/path/to/directory"
      [runners.kubernetes.volumes.config_map.items]
        "key_1" = "relative/path/to/key_1_file"
        "key_2" = "key_2"
    [[runners.kubernetes.volumes.secret]]
      name = "secrets"
      mount_path = "/path/to/directory1"
      read_only = true
      [runners.kubernetes.volumes.secret.items]
        "secret_1" = "relative/path/to/secret_1_file"
    [[runners.kubernetes.volumes.empty_dir]]
      name = "empty-dir"
      mount_path = "/path/to/empty_dir"
      medium = "Memory"

Host Path volumes

HostPath volume configuration instructs Kubernetes to mount a specified host path inside of the container. The volume can be configured with following options:

PVC volumes

PVC volume configuration instructs Kubernetes to use a PersistentVolumeClaim that is defined in Kubernetes cluster and mount it inside of the container. The volume can be configured with following options:

ConfigMap volumes

ConfigMap volume configuration instructs Kubernetes to use a configMap that is defined in Kubernetes cluster and mount it inside of the container.

When using configMap volume, each key from selected configMap will be changed into a file stored inside of the selected mount path. By default all keys are present, configMap’s key is used as file’s name and value is stored as file’s content. The default behavior can be changed with items option.

items option is defining a mapping between key that should be used and path (relative to volume’s mount path) where configMap’s value should be saved. When using items option only selected keys will be added to the volumes and all other will be skipped.

Notice: If a non-existing key will be used then job will fail on Pod creation stage.

Secret volumes

Secret volume configuration instructs Kubernetes to use a secret that is defined in Kubernetes cluster and mount it inside of the container.

When using secret volume each key from selected secret will be changed into a file stored inside of the selected mount path. By default all keys are present, secret’s key is used as file’s name and value is stored as file’s content. The default behavior can be changed with items option.

items option is defining a mapping between key that should be used and path (relative to volume’s mount path) where secret’s value should be saved. When using items option only selected keys will be added to the volumes and all other will be skipped.

Notice: If a non-existing key will be used then job will fail on Pod creation stage.

Empty Dir volumes

emptyDir volume configuration instructs Kubernetes to mount an empty directory inside of the container.

Using Security Context

Pod security context configuration instructs executor to set a pod security policy on the build pod.

Assigning a security context to pods provides security to your Kubernetes cluster. For this to work you’ll need to provide a helper image that conforms to the policy you set here.

Read more about the helper image. Example of building your own helper image:

ARG tag
FROM gitlab/gitlab-runner-helper:${tag}
RUN addgroup -g 59417 -S nonroot && \
    adduser -u 59417 -S nonroot -G nonroot
WORKDIR /home/nonroot
USER 59417:59417

This example creates a user and group called nonroot and sets the image to run as that user.

Example of setting pod security context in your config.toml:

concurrent = %(concurrent)s
check_interval = 30
  [[runners]]
    name = "myRunner"
    url = "gitlab.example.com"
    executor = "kubernetes"
    [runners.kubernetes]
      helper_image = "gitlab-registy.example.com/helper:latest"
      [runners.kubernetes.pod_security_context]
        run_as_non_root = true
        run_as_user = 59417
        run_as_group = 59417
        fs_group = 59417

Using services

Define a list of services.

concurrent = 1
check_interval = 30
  [[runners]]
    name = "myRunner"
    url = "gitlab.example.com"
    executor = "kubernetes"
    [runners.kubernetes]
      helper_image = "gitlab-registy.example.com/helper:latest"
      [[runners.kubernetes.services]]
        name = "postgres:12-alpine"
        alias = "db1"
      [[runners.kubernetes.services]]
        name = "registry.example.com/svc1"
        alias = "svc1"
        entrypoint = ["entrypoint.sh"]
        command = ["executable","param1","param2"]

Adding extra host aliases

This feature is available in Kubernetes 1.7+.

Host aliases configuration instructs Kubernetes to add entries to /etc/hosts file inside of the container. The host aliases can be configured with the following options:

Here is an example configuration:

concurrent = 4

[[runners]]
  # usual configuration
  executor = "kubernetes"
  [runners.kubernetes]
    [[runners.kubernetes.host_aliases]]
      ip = "127.0.0.1"
      hostnames = ["web1", "web2"]
    [[runners.kubernetes.host_aliases]]
      ip = "192.168.1.1"
      hostnames = ["web14", "web15"]

Using Affinity

Node Affinity

Define a list of node affinities to be added to a pod specification at build time.

concurrent = 1
  [[runners]]
    name = "myRunner"
    url = "gitlab.example.com"
    executor = "kubernetes"
    [runners.kubernetes]
      [runners.kubernetes.affinity]
        [runners.kubernetes.affinity.node_affinity]
          [[runners.kubernetes.affinity.node_affinity.preferred_during_scheduling_ignored_during_execution]]
            weight = 100
            [runners.kubernetes.affinity.node_affinity.preferred_during_scheduling_ignored_during_execution.preference]
              [[runners.kubernetes.affinity.node_affinity.preferred_during_scheduling_ignored_during_execution.preference.match_expressions]]
               key = "cpu_speed"
               operator = "In"
               values = ["fast"]
              [[runners.kubernetes.affinity.node_affinity.preferred_during_scheduling_ignored_during_execution.preference.match_expressions]]
               key = "mem_speed"
               operator = "In"
               values = ["fast"]
          [[runners.kubernetes.affinity.node_affinity.preferred_during_scheduling_ignored_during_execution]]
            weight = 50
            [runners.kubernetes.affinity.node_affinity.preferred_during_scheduling_ignored_during_execution.preference]
              [[runners.kubernetes.affinity.node_affinity.preferred_during_scheduling_ignored_during_execution.preference.match_expressions]]
                key = "core_count"
                operator = "In"
                values = ["high", "32"]
              [[runners.kubernetes.affinity.node_affinity.preferred_during_scheduling_ignored_during_execution.preference.match_fields]]
               key = "cpu_type"
               operator = "In"
               values = ["arm64"]
        [runners.kubernetes.affinity.node_affinity.required_during_scheduling_ignored_during_execution]
          [[runners.kubernetes.affinity.node_affinity.required_during_scheduling_ignored_during_execution.node_selector_terms]]
            [[runners.kubernetes.affinity.node_affinity.required_during_scheduling_ignored_during_execution.node_selector_terms.match_expressions]]
              key = "kubernetes.io/e2e-az-name"
              operator = "In"
              values = [
                "e2e-az1",
                "e2e-az2"
              ]

Pod’s DNS Config

Pod’s DNS Config gives users more control on the DNS settings of the created build Pods.

concurrent = 1
check_interval = 30
[[runners]]
  name = "myRunner"
  url = "https://gitlab.example.com"
  token = "__REDACTED__"
  executor = "kubernetes"
  [runners.kubernetes]
    image = "alpine:latest"
    [dns_config]
      nameservers = [
        "1.2.3.4",
      ]
      searches = [
        "ns1.svc.cluster-domain.example",
        "my.dns.search.suffix",
      ]

      [[dns_config.options]]
        name = "ndots"
        value = "2"

      [[dns_config.options]]
        name = "edns0"

KubernetesDNSConfigOption

Capabilities configuration

Kubernetes allows to configure different Linux capabilities that should be added or dropped from a container.

GitLab Runner supports configuration of capabilities with the cap_add and cap_drop keywords in the [runners.kubernetes] section of the configuration file. By default, GitLab Runner provides a list of capabilities that should be dropped.

Because the default list of dropped capabilities can intersect with user-defined capabilities, the following rules are applied to determine the final list:

1. User-defined cap_drop has priority over user-defined cap_add. If you define the same capability in both settings, only the one from cap_drop is passed to the container.

2. User-defined cap_add has priority over the default list of dropped capabilities. If you want to add the capability that is dropped by default, explicitly add it to cap_add.

The final list of capabilities is added to all containers in the job’s pod.

A few notes:

• Remove the CAP_ prefix from capability identifiers passed to the container configuration. For example, if you want to add or drop the CAP_SYS_TIME capability, in the configuration file, set a SYS_TIME string for cap_add or cap_drop.

• The owner of the Kubernetes cluster can define a PodSecurityPolicy, where specific capabilities are allowed, restricted, or added by default. These rules take precedence over any user-defined configuration.

  Container runtimes can also define a default list of capabilities that you can add, like those seen in Docker or containerd.

Default list of dropped capabilities

GitLab Runner tries to drop these capabilities by default. If any of them are required for the job to be executed properly, you should explicitly add the capability with the cap_add setting:

• NET_RAW

Example configuration

concurrent = 1
check_interval = 30
[[runners]]
  name = "myRunner"
  url = "gitlab.example.com"
  executor = "kubernetes"
  [runners.kubernetes]
    # ...
    cap_add = ["SYS_TIME", "IPC_LOCK"]
    cap_drop = ["SYS_ADMIN"]
    # ...

Using Docker in your builds

There are a couple of caveats when using Docker in your builds while running on a Kubernetes cluster. Most of these issues are already discussed in the Using Docker Build section of the GitLab CI documentation but it is worthwhile to revisit them here as you might run into some slightly different things when running this on your cluster.

Exposing /var/run/docker.sock

Exposing your host’s /var/run/docker.sock into your build container, using the runners.kubernetes.volumes.host_path option, brings the same risks with it as always. That node’s containers are accessible from the build container and depending if you are running builds in the same cluster as your production containers it might not be wise to do that.

Using docker:dind

Running the docker:dind also known as the docker-in-docker image is also possible but sadly needs the containers to be run in privileged mode. If you’re willing to take that risk other problems will arise that might not seem as straight forward at first glance. Because the Docker daemon is started as a service usually in your .gitlab-ci.yaml it will be run as a separate container in your Pod. Basically containers in Pods only share volumes assigned to them and an IP address by which they can reach each other using localhost. /var/run/docker.sock is not shared by the docker:dind container and the docker binary tries to use it by default.

To overwrite this and make the client use TCP to contact the Docker daemon, in the other container, be sure to include the environment variables of the build container:

• DOCKER_HOST=tcp://localhost:2375 for no TLS connection.
• DOCKER_HOST=tcp://localhost:2376 for TLS connection.

Make sure to configure those properly. As of Docker 19.03, TLS is enabled by default but it requires mapping certificates to your client. You can enable non-TLS connection for DIND or mount certificates as described in Use Docker In Docker Workflow with Docker executor

Not supplying Git

Do not try to use an image that doesn’t supply Git and add the GIT_STRATEGY=none environment variable for a job that you think doesn’t need to do a fetch or clone. Because Pods are ephemeral and do not keep state of previously run jobs your checked out code will not exist in both the build and the Docker service container. Errors you might run into are things like could not find git binary and the Docker service complaining that it cannot follow some symlinks into your build context because of the missing code.

Resource separation

In both the docker:dind and /var/run/docker.sock cases the Docker daemon has access to the underlying kernel of the host machine. This means that any limits that had been set in the Pod will not work when building Docker images. The Docker daemon will report the full capacity of the node regardless of the limits imposed on the Docker build containers spawned by Kubernetes.

One way to help minimize the exposure of the host’s kernel to any build container when running in privileged mode or by exposing /var/run/docker.sock is to use the node_selector option to set one or more labels that have to match a node before any containers are deployed to it. For example build containers may only run on nodes that are labeled with role=ci while running all other production services on other nodes. Further separation of build containers can be achieved using node taints. This will disallow other pods from scheduling on the same nodes as the build pods without extra configuration for the other pods.

Using kaniko

Another approach for building Docker images inside a Kubernetes cluster is using kaniko. kaniko:

• Allows you to build images without privileged access.
• Works without the Docker daemon.

For more information, see Building images with kaniko and GitLab CI/CD.

Job execution

At the moment we are using kube exec to run the scripts, which relies on having a stable network connection between the Runner and the pod for the duration of the command. This leads to problems like Job marked as success midway. If you are experiencing this problem turn off the feature flag FF_USE_LEGACY_KUBERNETES_EXECUTION_STRATEGY to use kube attach for script execution, which is more stable.

We are rolling this out slowly and have plans to enable the kube attach behavior by default in future release, please follow #10341 for updates.