Internal Executor Interface
Interfaces
GitLab Runner uses a concept of what we name executors to define a way of how a job may be
executed.
While the current philosophy behind GitLab CI/CD job execution is that everything is a shell script, this script may be executed in a different ways, for example:
- in a shell directly on a host where GitLab Runner is working,
- in a shell on an external host available through SSH,
- in a shell in a virtual machine managed by VirtualBox or Parallels,
- in a shell in a container managed by Docker,
and few others. There is also the Custom Executor, which allows the user to interact with a very simple externally exposed interface to implement their own way of job execution.
All of these executors are orchestrated internally by GitLab Runner process. And for that Runner is using a set of Go interfaces that need to be implemented by the executor to work.
Te two main interfaces (part of the common package) that manage an executor’s lifetime
and job execution are:
ExecutorExecutorProvider
type Executor interface {
// Shell returns data about the shell and scripts this executor is bound to.
Shell() *ShellScriptInfo
// Prepare prepares the environment for build execution. e.g. connects to SSH, creates containers.
Prepare(options ExecutorPrepareOptions) error
// Run executes a command on the prepared environment.
Run(cmd ExecutorCommand) error
// Finish marks the build execution as finished.
Finish(err error)
// Cleanup cleans any resources left by build execution.
Cleanup()
// GetCurrentStage returns current stage of build execution.
GetCurrentStage() ExecutorStage
// SetCurrentStage sets the current stage of build execution.
SetCurrentStage(stage ExecutorStage)
}
type ExecutorProvider interface {
// CanCreate returns whether the executor provider has the necessary data to create an executor.
CanCreate() bool
// Create creates a new executor. No resource allocation happens.
Create() Executor
// Acquire acquires the necessary resources for the executor to run, e.g. finds a virtual machine.
Acquire(config *RunnerConfig) (ExecutorData, error)
// Release releases any resources locked by Acquire.
Release(config *RunnerConfig, data ExecutorData)
// GetFeatures returns metadata about the features the executor supports, e.g. variables, services, shell.
GetFeatures(features *FeaturesInfo) error
// GetConfigInfo extracts metadata about the config the executor is using, e.g. GPUs.
GetConfigInfo(input *RunnerConfig, output *ConfigInfo)
// GetDefaultShell returns the name of the default shell for the executor.
GetDefaultShell() string
}
All the existing executors are also extending the executors.AbstractExecutor struct (named
AbstractExecutor further in this document), which implements a small, common set of features.
While there is no protection in code that would ensure usage of AbstractExecutor (until the
new code implements the interfaces - it will work), it’s expected that the new executors will
extend it - to ensure consistent behavior of some features across executors.
For convenience there is also the executors.DefaultExecutorProvider that implements the
ExecutorProvider interface and is suitable for most cases. However, each executor may
decide to implement its provider independently (which in fact is currently done only by the
Docker Machine executor).
What’s important, because both Executor and ExecutorProvider are interfaces, the implementation
allows to “stack” different structs. The usage of this possibility will be shown with one of the
examples.s
Executor interface
The Executor interface is responsible for the job execution management.
The described methods are managing preparation of the job environment (Prepare()), job script
executions (Run() and Finish(); note: subsequent job steps are executed with a separate Run() calls)
and job environment cleanup (Finish()).
It also provides integration for internal Prometheus metrics exporter to label some relevant metrics
with information about the current executor usage stage (GetCurrentStage(), SetCurrentStage()).
The Shell() method is currently used in one place, and it’s fully implemented in the mentioned
AbstractExecutor struct. Given the existing implementation and evolution of different executors
over time, it seems that this method should be pulled off the executor interface and handled in
some different way. Hopefully - in a way that will enforce usage of AbstractExecutor.
Usage of the interface, in very simplification, goes as follows:
- The instance of an executor was provided and assigned to a received job,
-
Shell()is called to get the configuration of a shell. It’s used to prepare all the scripts that will be executed for the job. -
Prepare()is called to prepare the job environment (for example creating a Kubernetes Pod, a set of Docker containers or a VirtualBox VM). It’s also a place for the specific executor implementation to handle its own preparation. Through the usage ofAbstractExecutorall the executors will also get access to some common features like for example job trace object. -
Run()is called several times, each time containing details about the script for a job execution step to be executed with the executor. -
Finish()is called after execution of all job stages is done and when job is being marked as finished. Some executors may take a usage of this moment. Most of them defers toAbstractExecutor. -
Cleanup()is called to cleanup the job environment. It’s the opposite ofPrepare().
Additionally SetCurrentStage() is called internally by the executors (however most of them
defer to AbstractExecutor) to mark on what executor usage stage the system
is now within this executor instance. And GetCurrentStage() is called externally in random
moments by the metrics collector. The value is then used to summarize information about
different jobs and label some of the metrics.
ExecutorProvider interface
The ExecutorProvider interface is responsible for preparation of the executor itself. It builds
an abstraction around the Executor concept. With this abstraction, what the user configures with
the config.toml executor setting is in fact the executor provider. And then for every job executed
by the runner a new, independent instance of the executor is prepared. The maintenance of the executor
is done by the ExecutorProvider.
The described methods are managing creation of the executor instance (CanCreate(), Create()),
reservation of provider’s resources for a potential job (Acquire(), Release()). There is also
support for gathering some information that should be reported to GitLab when requesting jobs
(GetFeatures(), GetConfigInfo()). And finally a method that gives information about the shell
that should be used with the provided executor (GetDefaultShell()).
Usage of the interface, in very simplification, goes as follows:
-
CanCreate(),GetFeatures()andGetDefaultShell()are executed at the provider registration to validate that the provider is able to work in general. - Before requesting a new job for the specific
[[runners]]worker theAcquire()is called to check and do a reservation of provider resources for the job. This is a place where the provider may control its capacity and return information about some preallocated resources. -
GetFeatures()is called several times to ensure that information about features supported by Runner can be sent back with different API requests to GitLab. On of the calls is made when preparing the initial request for a job. - Same goes for the
GetConfigInfo()which is called only once, when preparing the initial request for a job. It allows to sent some information about used configuration to GitLab. - Same goes for the
GetDefaultShall()which is called also onl once, when preparing the initial request for a job. It allows to sent information about used shell to GitLab. - If the job was received, it’s preparation is started and at some moment
Create()is called to create a new instance of the executor. - When the job execution is fully done,
Release()is called. This is a place where the provider may handle releasing resources that were previously reserved for the job.
List of features that can be reported to GitLab can be found in the FeaturesInfo struct in
common/network.go.
DefaultExecutorProvider
As DefaultExecutorProvider is currently one of two existing implementations of ExecutorProvider
interface and is used by most of the executors, let’s describe how it’s built.
type DefaultExecutorProvider struct {
Creator func() common.Executor
FeaturesUpdater func(features *common.FeaturesInfo)
ConfigUpdater func(input *common.RunnerConfig, output *common.ConfigInfo)
DefaultShellName string
}
The Creator is the most important part. It’s a function that returns a new instance
of the given Executor interface implementation. It is being implemented by each
of the executors. It’s required to be implemented.
The interface’s CanCreate() method will fail if the Creator is left empty. Call to
provider’s Create() is proxied to the Creator function.
FeaturesUpdater and ConfigUpdater are functions that allow to request the feature
and config information. All executors are using these functions to expose information
about supported features or config details. The FeaturesUpdater is optional and every
executor have to report which features from the list are supported. ConfiguUpdater is
optional and can be skipped. DefaultShellName must be set by every executor.
Provider’s GetFeatures(), GetConfigInfo(), GetDefaultShell() calls will use the
defined updaters and the shell name to expose needed data to the caller.
Acquire() and Release() are a NOOP. DefaultExecutorProvider doesn’t use the
concept of resources management and simply creates a new instance of the executor
for every call.
Usage examples
Shell
Shell executor is the simplest executor that GitLab Runner provides. It executes the job script in a simple shell process, created directly on the host where GitLab Runner is running itself. There is no virtualization, no containers, no network communication here.
ExecutorProvider
Shell executor uses the DefaultExecutorProvider. It reports usage of very limited number
of features (two in all cases, two more if the platform is not windows). It doesn’t
expose any configuration details.
The shell depends on what’s the default value for the platform where the Runner is operating. It’s configured as a login shell.
Executor
Prepare() doesn’t have anything specific. As the shell executor executes everything directly
in the system where Runner process exists, it just makes sure that the builds and cache
paths are usable. After that it defers to AbstractExecutor steps of preparation.
Run() uses the provided script details to construct os/exec.Cmd call. Shell executor
ensures that STDIN/STDOUT/STDERR are passed properly between the script execution shell
process started by that call and the job trace object. It also detects the exit code
of the command and reports it back as expected by the interface.
There is no custom implementation of Finish() nor Cleanup(). The executor defers to the common
steps in AbstractExecutor.
Docker
Docker executor is probably the most powerful and mature of GitLab Runner executors. It supports
most of the features available in .gitlab-ci.yml. It allows to run every job in an environment
separated from other jobs. All jobs are however executed on one host and the capacity of the runner
is limited by that host’s available resources.
Docker executor comes with a special variant - the SSH one. To make this documentation easier to understand (as the executor descriptions are just examples to help understand how the executor interface works) we will describe just the “normal” variant of Docker executor.
There is also the windows variant of the executor. We will not include its details in this
description as well.
In Docker executor the jobs are executed in Docker containers. Each job gets a set of connected
containers sharing at least one volume with the working directory. The main container is created
from the image specified by the user. It needs to expose a shell where Runner will execute the
script. Additionally, Runner will create what we call predefined container from the helper
image provided by Runner. This container will be used to execute scripts handling common tasks
like updating the Git and Git LFS sources, operating with cache and operating with artifacts.
Depending on the job configuration Runner may create more containers for the defined services.
These will be linked by the networking to the main container, so that the job script can utilise
network available services exposed by them.
ExecutorProvider
Docker executor also uses the DefaultExecutorProvider. It reports usage of few more executor-related
features, and additionally it reports some configuration details.
The shell is hardcoded and differs between the platforms. In case of the most popular linux variant
of Docker executor, it’s configured as a non-login shell.
Executor
Prepare() is highly utilised in this executor. During that step Runner will prepare different
internal tools (like volumes manager or network manager) and set up the basic configuration that
will be next used by the containers for job execution. It’s also the step when all the images
defined for the job are pulled. Creation of volumes, device binding and service containers
also happens during that step.
After Prepare() is done the environment should be fully ready to start creating predefined/job step
execution containers, connecting them to the whole stack and execute scripts in them.
Run() creates predefined or job step containers, attaches to them and executes the script
in a shell that should be running as the main process of the container. It proxies the STDOUT and
STDERR of the containers to the job trace object. It also uses the Docker Engine API to detect
the script execution exit code.
Finish() doesn’t have any custom behavior here, and it just defers to the AbstractExecutor.
Cleanup() is the opposite of Prepare(), so it removes all the defined resources like containers,
volumes (that were not configured as persistent), job specific network (if used).
Docker Machine (autoscaling capabilities)
Docker Machine executor is in fact an autoscaling provider built on top of the regular Docker executor.
It takes advantage of the interface concept and encapsulates the Docker executor in itself. Responsibility
of Docker Machine executor is mostly focused on the ExecutorProvider interface. With that it manages
a pool of VMs with Docker Engine running on them. Management is done by using the Docker Machine tool
by running an os/exec.Cmd calls to it.
Management of the VMs may be done in “on-demand” or “autoscaled in background” modes. Chosen mode
depends on configuration provided by the user. In the first mode the VMs will be created for each
received job, until the limit of jobs is reached. In the second mode it will maintain a configurable
set of Idle s that await for jobs. Jobs are then requested only when there is at least one
Idle VM. When one Idle VM is taken for a received job, another is created to replace it. When the
VM is returned to the pool (if configured to do so) and the number of Idle exceeds the defined limit,
the provider starts to remove some of them. This loop that tries to maintain the desired number of Idle
VMs and desired total number of managed VMs works all the time in the background.
Docker Machine executor is currently implemented in a way that it allows execution of only one job at once on a single VM.
For the execution of jobs Docker Machine executor uses the Docker executor and fact, that one can
configure access credentials of the Docker Engine API. With that the Docker Machine provider
manages the VMs, chooses a VM for a job and instantiates the Docker executor, automatically configuring
it to use the credentials and API endpoint of the VM. With that jobs are executed like with the normal
Docker executor (supporting all the different features available for it in .gitlab-ci.yml syntax), but
doest that on an external host, independent for each job.
ExecutorProvider
Docker Machine executor brings its own implementation of the ExecutorProvider interface!
However, as it internally uses the Docker executor, it also instantiates the Docker executor
provider (which itself is the specific configuration of DefaultExecutorProvider) and either
proxies some calls to it directly or calls it internally for its own purpose.
CanCreate() is proxied directly to Docker executor. Same goes for GetFeatures(), GetConfigInfo()
and GetDefaultShell().
Create() is very simple as its returns the machineExecutor (implementation of the Executor interface)
with access to itself, so that steps like Prepare() or Cleanup() can use it to maintain the autoscaled
VMs (more about that will be described bellow).
This provider is also the one that finally takes the usage of Acquire() and Release() methods
of the Executor Provider interface.
Behavior of Acquire() depends on the configured mode.
In the “on-demand” mode it’s used as a place to kick one of the old machines cleanup calls. It
doesn’t do any real acquiring and even logs that with IdleCount is set to 0 so the machine will
be created on demand in job context (this is not user facing and available in the Runner process logs).
With that the provider will try to create the VM in context of the job. If there is anything that will
cause a failure: exceeding the defined limits in autoscaling configuration, wrong autoscaling configuration,
cloud provider errors, Docker Engine availability problems - it will cause a failure of the job.
In the “autoscaled in background” mode, it will check if there is any Idle VM that is available.
If it is, it will reserve it and allow Runner to send a request for new job. If job is received,
it will get information about the acquired VM. If there is no available Idle VMs, then the call to
Acuire() will wail, which will prevent Runner from sending a request for a job (and in Runner logs
will be logged as the no free machines that can process builds warning).
Release() behaves in the same in both modes. It will check if the VM that was used for the job
is applicable for removal and will trigger a remove in that case. In other cases, it will signal the
internal autoscaling coordination mechanism that the VM was released and it’s back in the Idle pool,
so that it can be used again.
Executor
The Executor interface implementation is also a mix of a code specific for Docker Machine executor
and encapsulation of Docker executor. Docker Machine executor injects all the work needed
to maintain, chose and use the VMs and to configure the dedicated Docker executor instance, and
then it depends on this executor to handle the rest.
Shell() call defers to Docker executor, which itself defers to AbstractExecutor (as all the executors do).
Prepare() prepares the VM to use. Depending on the configured mode it may mean using the preallocated
VM or creating it on-demand. In the “on-demand” mode this is the place where eventual failure caused
by VM creation may fail the job. Having the VM details it updates the configuration of Docker executor
by pointing the host and credentials to access Docker Engine and instantiates the Docker executor provider.
Finally, it calls Docker Executor’s Prepare() to handle all the job environment preparation as
it was described in the previous example.
Run() and Finish() have no specific behavior. They simply proxy the call to the internal Docker executor.
GetCurrentStage() and SetCurrentStage() are also proxies to the Docker executor, which itself defers
to the AbstractExecutor implementation.
Finally, the Cleanup() call does two things. First, it internally calls Docker executor’s Cleanup() method
to clean the job environment on the VM as it was described in the previous example. Then it calls
providers Release() to signal that the job is done and that the VM can be released.