• Limitations
• Configuration
• Prerequisite software for running a Job
• Stages
  • Services
  • Config
    • job_env usage
  • Prepare
  • Run
  • Cleanup
• Terminating and killing executables
• Error handling
  • Build Failure
  • System Failure
• Job response

The Custom executor

GitLab Runner provides the Custom executor for environments that it doesn’t support natively, for example, Podman or Libvirt.

This gives you the control to create your own executor by configuring GitLab Runner to use some executable to provision, run, and clean up your environment.

The scripts you configure for the custom executor are called Drivers. For example, you could create a Podman driver, an LXD driver or a Libvirt driver.

Limitations

Below are some current limitations when using the Custom executor:

• No Interactive Web Terminal support.

Configuration

There are a few configuration keys that you can choose from. Some of them are optional.

Below is an example of configuration for the Custom executor using all available configuration keys:

[[runners]]
  name = "custom"
  url = "https://gitlab.com"
  token = "TOKEN"
  executor = "custom"
  builds_dir = "/builds"
  cache_dir = "/cache"
  [runners.custom]
    config_exec = "/path/to/config.sh"
    config_args = [ "SomeArg" ]
    config_exec_timeout = 200

    prepare_exec = "/path/to/script.sh"
    prepare_args = [ "SomeArg" ]
    prepare_exec_timeout = 200

    run_exec = "/path/to/binary"
    run_args = [ "SomeArg" ]

    cleanup_exec = "/path/to/executable"
    cleanup_args = [ "SomeArg" ]
    cleanup_exec_timeout = 200

    graceful_kill_timeout = 200
    force_kill_timeout = 200

For field definitions and which ones are required, see [runners.custom] section configuration.

In addition both builds_dir and cache_dir inside of the [[runners]] are required fields.

Prerequisite software for running a Job

The user must set up the environment, including the following that must be present in the PATH:

• Git: Used to clone the repositories.
• Git LFS: Pulls any LFS objects that might be in the repository.
• GitLab Runner: Used to download/update artifacts and cache.

Stages

The Custom executor provides the stages for you to configure some details of the job, prepare and clean up the environment and run the job script within it. Each stage is responsible for specific things and has different things to keep in mind.

Each stage executed by the Custom executor is executed at the time a builtin GitLab Runner executor would execute them.

For each step that will be executed, specific environment variables are exposed to the executable, which can be used to get information about the specific Job that is running. All stages will have the following environment variables available to them:

• Standard CI/CD environment variables, including predefined variables.
• All environment variables provided by the Custom executor Runner host system.
• All services and their available settings. Exposed in JSON format as CUSTOM_ENV_CI_JOB_SERVICES.

Both CI/CD environment variables and predefined variables are prefixed with CUSTOM_ENV_ to prevent conflicts with system environment variables. For example, CI_BUILDS_DIR will be available as CUSTOM_ENV_CI_BUILDS_DIR.

The stages run in the following sequence:

1. config_exec
2. prepare_exec
3. run_exec
4. cleanup_exec

Services

Services are exposed as a JSON array as CUSTOM_ENV_CI_JOB_SERVICES.

Example:

custom:
  script:
    - echo $CUSTOM_ENV_CI_JOB_SERVICES
  services:
    - redis:latest
    - name: my-postgres:9.4
      alias: pg
      entrypoint: ["path", "to", "entrypoint"]
      command: ["path", "to", "cmd"]

The example above will set CUSTOM_ENV_CI_JOB_SERVICES environment variable with the following value:

[{"name":"redis:latest","alias":"","entrypoint":null,"command":null},{"name":"my-postgres:9.4","alias":"pg","entrypoint":["path","to","entrypoint"],"command":["path","to","cmd"]}]

Config

The Config stage is executed by config_exec.

Sometimes you might want to set some settings during execution time. For example settings a build directory depending on the project ID. config_exec reads from STDOUT and expects a valid JSON string with specific keys.

For example:

#!/usr/bin/env bash

cat << EOS
{
  "builds_dir": "/builds/${CUSTOM_ENV_CI_CONCURRENT_PROJECT_ID}/${CUSTOM_ENV_CI_PROJECT_PATH_SLUG}",
  "cache_dir": "/cache/${CUSTOM_ENV_CI_CONCURRENT_PROJECT_ID}/${CUSTOM_ENV_CI_PROJECT_PATH_SLUG}",
  "builds_dir_is_shared": true,
  "hostname": "custom-hostname",
  "driver": {
    "name": "test driver",
    "version": "v0.0.1"
  },
  "job_env" : {
    "CUSTOM_ENVIRONMENT": "example"
  }
}
EOS

Any additional keys inside of the JSON string will be ignored. If it’s not a valid JSON string the stage will fail and be retried two more times.

The STDERR of the executable will print to the job log.

The user can set config_exec_timeout if they want to set a deadline for how long GitLab Runner should wait to return the JSON string before terminating the process.

If any of the config_exec_args are defined, these will be added in order to the executable defined in config_exec. For example we have the config.toml content below:

...
[runners.custom]
  ...
  config_exec = "/path/to/config"
  config_args = [ "Arg1", "Arg2" ]
  ...

GitLab Runner would execute it as /path/to/config Arg1 Arg2.

job_env usage

The main purpose of job_env configuration is to pass variables to the context of custom executor driver calls for subsequent stages of the job execution.

Let’s consider an example driver, where connection with the job execution environment requires preparing some credentials and that this operation is very expensive. Let’s say we need to connect to our local credentials provider to get a temporary SSH username and password that the custom executor can next use to connect with the job execution environment.

With Custom Executor execution flow, where each job execution stage: prepare, multiple run calls and cleanup are separate executions of the driver, the context is separate for each of them. For our credentials resolving example, connection to the credentials provider needs to be done each time.

If this operation is expensive, we might want to do it once for a whole job execution, and then re-use the credentials for all job execution stages. This is where the job_env can help. With this you can connect with the provider once, during the config_exec call and then pass the received credentials with the job_env. They will be next added to the list of variables that the custom executor calls for prepare_exec, run_exec and cleanup_exec are receiving. With this, the driver instead of connecting to the credentials provider each time may just read the variables and use the credentials that are present.

The important thing to understand is that the variables are not automaticaly available for the job itself. It fully depends on how the Custom Executor Driver is implemented and in many cases it will be not present there.

If you’re considering the job_env setting so you can pass a set of variables to every job executed by a particular runner, then look at the environment setting from [[runners]].

If the variables are dynamic and it’s expected that their values will change between different jobs, then you should make sure that your driver is implemented in a way that the variables passed by job_env will be added to the job execution call.

Prepare

The Prepare stage is executed by prepare_exec.

At this point, GitLab Runner knows everything about the job (where and how it’s going to run). The only thing left is for the environment to be set up so the job can run. GitLab Runner will execute the executable that is specified in prepare_exec.

This is responsible for setting up the environment (for example, creating the virtual machine or container, services or anything else). After this is done, we expect that the environment is ready to run the job.

This stage is executed only once, in a job execution.

The user can set prepare_exec_timeout if they want to set a deadline for how long GitLab Runner should wait to prepare the environment before terminating the process.

The STDOUT and STDERR returned from this executable will print to the job log.

If any of the prepare_exec_args are defined, these will be added in order to the executable defined in prepare_exec. For example we have the config.toml content below:

...
[runners.custom]
  ...
  prepare_exec = "/path/to/bin"
  prepare_args = [ "Arg1", "Arg2" ]
  ...

GitLab Runner would execute it as /path/to/bin Arg1 Arg2.

Run

The Run stage is executed by run_exec.

The STDOUT and STDERR returned from this executable will print to the job log.

Unlike the other stages, the run_exec stage is executed multiple times, since it’s split into sub stages listed below in sequential order:

1. prepare_script
2. get_sources
3. restore_cache
4. download_artifacts
5. step_*
6. build_script
7. step_*
8. after_script
9. archive_cache OR archive_cache_on_failure
10. upload_artifacts_on_success OR upload_artifacts_on_failure
11. cleanup_file_variables