• Overview
• System requirements
• Supported cloud providers
• Runner configuration
  • Runner global options
  • [[runners]] options
  • [runners.machine] options
  • [runners.cache] options
  • Additional configuration information
• Limit the number of VMs created by the Docker Machine executor
• Autoscaling algorithm and parameters
• How concurrent, limit and IdleCount generate the upper limit of running machines
• The IdleScaleFactor strategy
• Autoscaling periods configuration
• Off Peak time mode configuration (Deprecated)
• Distributed runners caching
• Distributed container registry mirroring
• A complete example of config.toml

Docker Machine Executor autoscale configuration

Autoscale provides the ability to use resources in a more elastic and dynamic way.

GitLab Runner can autoscale, so that your infrastructure contains only as many build instances as are necessary at any time. If you configure GitLab Runner to only use autoscale, the system on which GitLab Runner is installed acts as a bastion for all the machines it creates. This machine is referred to as a “Runner Manager.”

Overview

When this feature is enabled and configured properly, jobs are executed on machines created on demand. Those machines, after the job is finished, can wait to run the next jobs or can be removed after the configured IdleTime. In case of many cloud providers this helps to utilize the cost of already used instances.

Below, you can see a real life example of the GitLab Runner autoscale feature, tested on GitLab.com for the GitLab Community Edition project:

Each machine on the chart is an independent cloud instance, running jobs inside of Docker containers.

System requirements

Before configuring autoscale, you must:

• Prepare your own environment.
• Optionally use a forked version of Docker machine supplied by GitLab, which has some additional fixes.

Supported cloud providers

The autoscale mechanism is based on Docker Machine. All supported virtualization and cloud provider parameters are available at the GitLab-managed fork of Docker Machine.

Runner configuration

This section describes the significant autoscale parameters. For more configurations details read the advanced configuration.

Runner global options

[[runners]] options

[runners.machine] options

Configuration parameters details can be found in GitLab Runner - Advanced Configuration - The [runners.machine] section.

[runners.cache] options

Configuration parameters details can be found in GitLab Runner - Advanced Configuration - The [runners.cache] section

Additional configuration information

There is also a special mode, when you set IdleCount = 0. In this mode, machines are always created on-demand before each job (if there is no available machine in Idle state). After the job is finished, the autoscaling algorithm works the same as it is described below. The machine is waiting for the next jobs, and if no one is executed, after the IdleTime period, the machine is removed. If there are no jobs, there are no machines in Idle state.

If the IdleCount is set to a value greater than 0, then idle VMs are created in the background. The runner acquires an existing idle VM before asking for a new job.

• If the job is assigned to the runner, then that job is sent to the previously acquired VM.
• If the job is not assigned to the runner, then the lock on the idle VM is released and the VM is returned back to the pool.

Limit the number of VMs created by the Docker Machine executor

To limit the number of virtual machines (VMs) created by the Docker Machine executor, use the limit parameter in the [[runners]] section of the config.toml file.

The concurrent parameter does not limit the number of VMs.

As detailed here, one process can be configured to manage multiple runner workers.

This example illustrates the values set in the config.toml file for one runner process:

concurrent = 100

[[runners]]
name = "first"
executor = "shell"
limit = 40
(...)

[[runners]]
name = "second"
executor = "docker+machine"
limit = 30
(...)

[[runners]]
name = "third"
executor = "ssh"
limit = 10

[[runners]]
name = "fourth"
executor = "virtualbox"
limit = 20
(...)

With this configuration:

• One runner process can create four different runner workers using different execution environments.
• The concurrent value is set to 100, so this one runner will execute a maximum of 100 concurrent GitLab CI/CD jobs.
• Only the second runner worker is configured to use the Docker Machine executor and therefore can automatically create VMs.
• The limit setting of 30 means that the second runner worker can execute a maximum of 30 CI/CD jobs on autoscaled VMs at any point in time.
• While concurrent defines the global concurrency limit across multiple [[runners]] workers, limit defines the maximum concurrency for a single [[runners]] worker.

In this example, the runner process handles:

• Across all [[runners]] workers, up to 100 concurrent jobs.
• For the first worker, no more than 40 jobs, which are executed with the shell executor.
• For the second worker, no more than 30 jobs, which are executed with the docker+machine executor. Additionally, Runner will maintain VMs based on the autoscaling configuration in [runners.machine], but no more than 30 VMs in all states (idle, in-use, in-creation, in-removal).
• For the third worker, no more than 10 jobs, executed with the ssh executor.
• For the fourth worker, no more than 20 jobs, executed with the virtualbox executor.

In this second example, there are two [[runners]] workers configured to use the docker+machine executor. With this configuration, each runner worker manages a separate pool of VMs that are constrained by the value of the limit parameter.

concurrent = 100

[[runners]]
name = "first"
executor = "docker+machine"
limit = 80
(...)

[[runners]]
name = "second"
executor = "docker+machine"
limit = 50
(...)

In this example:

• The runner processes no more than 100 jobs (the value of concurrent).
• The runner process executes jobs in two [[runners]] workers, each of which uses the docker+machine executor.
• The first runner can create a maximum of 80 VMs. Therefore this runner can execute a maximum of 80 jobs at any point in time.
• The second runner can create a maximum of 50 VMs. Therefore this runner can execute a maximum of 50 jobs at any point in time.

Autoscaling algorithm and parameters

The autoscaling algorithm is based on these parameters:

• IdleCount
• IdleCountMin
• IdleScaleFactor
• IdleTime
• MaxGrowthRate
• limit

We say that each machine that does not run a job is in Idle state. When GitLab Runner is in autoscale mode, it monitors all machines and ensures that there is always an IdleCount of machines in Idle state.

If there is an insufficient number of Idle machines, GitLab Runner starts provisioning new machines, subject to the MaxGrowthRate limit. Requests for machines above the MaxGrowthRate value are put on hold until the number of machines being created falls below MaxGrowthRate.

At the same time, GitLab Runner is checking the duration of the Idle state of each machine. If the time exceeds the IdleTime value, the machine is automatically removed.

Example: Let’s suppose, that we have configured GitLab Runner with the following autoscale parameters:

[[runners]]
  limit = 10
  # (...)
  executor = "docker+machine"
  [runners.machine]
    MaxGrowthRate = 1
    IdleCount = 2
    IdleTime = 1800
    # (...)

At the beginning, when no jobs are queued, GitLab Runner starts two machines (IdleCount = 2), and sets them in Idle state. Notice that we have also set IdleTime to 30 minutes (IdleTime = 1800).

Now, let’s assume that 5 jobs are queued in GitLab CI. The first 2 jobs are sent to the Idle machines of which we have two. GitLab Runner now notices that the number of Idle is less than IdleCount (0 < 2), so it starts new machines. These machines are provisioned sequentially, to prevent exceeding the MaxGrowthRate.

The remaining 3 jobs are assigned to the first machine that is ready. As an optimization, this can be a machine that was busy, but has now completed its job, or it can be a newly provisioned machine. For the sake of this example, let us assume that provisioning is fast, and the provisioning of new machines completed before any of the earlier jobs completed.

We now have 1 Idle machine, so GitLab Runner starts another 1 new machine to satisfy IdleCount. Because there are no new jobs in queue, those two machines stay in Idle state and GitLab Runner is satisfied.

This is what happened: We had 2 machines, waiting in Idle state for new jobs. After the 5 jobs where queued, new machines were created, so in total we had 7 machines. Five of them were running jobs, and 2 were in Idle state, waiting for the next jobs.

The algorithm still works the same way; GitLab Runner creates a new Idle machine for each machine used for the job execution until IdleCount is satisfied. Those machines are created up to the number defined by limit parameter. If GitLab Runner notices that there is a limit number of total created machines, it stops autoscaling, and new jobs must wait in the job queue until machines start returning to Idle state.

In the above example we always have two idle machines. The IdleTime applies only when we are over the IdleCount. Then we try to reduce the number of machines to IdleCount.

Scaling down: After the job is finished, the machine is set to Idle state and is waiting for the next jobs to be executed. Let’s suppose that we have no new jobs in the queue. After the time designated by IdleTime passes, the Idle machines are removed. In our example, after 30 minutes, all machines are removed (each machine after 30 minutes from when last job execution ended) and GitLab Runner starts to keep an IdleCount of Idle machines running, just like at the beginning of the example.

So, to sum up:

1. We start GitLab Runner
2. GitLab Runner creates 2 idle machines
3. GitLab Runner picks one job
4. GitLab Runner creates one more machine to fulfill the strong requirement of always having the two idle machines
5. Job finishes, we have 3 idle machines
6. When one of the three idle machines goes over IdleTime from the time when last time it picked the job it is removed
7. GitLab Runner always has at least 2 idle machines waiting for fast picking of the jobs

Below you can see a comparison chart of jobs statuses and machines statuses in time:

How concurrent, limit and IdleCount generate the upper limit of running machines

A magic equation doesn’t exist to tell you what to set limit or concurrent to. Act according to your needs. Having IdleCount of Idle machines is a speedup feature. You don’t need to wait 10s/20s/30s for the instance to be cre