- Introduction
- Prerequisites
- Prepare the Runner Manager instance
- Registering the GitLab Runner
- Configuring the runner
- Cutting down costs with Amazon EC2 Spot instances
- Conclusion
Autoscaling GitLab Runner on AWS EC2
One of the biggest advantages of GitLab Runner is its ability to automatically spin up and down VMs to make sure your builds get processed immediately. It’s a great feature, and if used correctly, it can be extremely useful in situations where you don’t use your runners 24/7 and want to have a cost-effective and scalable solution.
Introduction
In this tutorial, we’ll explore how to properly configure GitLab Runner in AWS. The instance in AWS will serve as a Runner Manager that spawns new Docker instances on demand. The runners on these instances are automatically created. They use the parameters covered in this guide and do not require manual configuration after creation.
In addition, we’ll make use of Amazon’s EC2 Spot instances which will greatly reduce the costs of the GitLab Runner instances while still using quite powerful autoscaling machines.
Prerequisites
A familiarity with Amazon Web Services (AWS) is required as this is where most of the configuration will take place.
We suggest a quick read through Docker machine amazonec2
driver
documentation to familiarize
yourself with the parameters we will set later in this article.
Your GitLab Runner is going to need to talk to your GitLab instance over the network, and that is something you need think about when configuring any AWS security groups or when setting up your DNS configuration.
For example, you can keep the EC2 resources segmented away from public traffic in a different VPC to better strengthen your network security. Your environment is likely different, so consider what works best for your situation.
AWS security groups
Docker Machine will attempt to use a
default security group
with rules for port 2376
and SSH 22
, which is required for communication with the Docker
daemon. Instead of relying on Docker, you can create a security group with the
rules you need and provide that in the GitLab Runner options as we will
see below. This way, you can customize it to your
liking ahead of time based on your networking environment.
You have to make sure that ports 2376
and 22
are accessible by the Runner Manager instance.
AWS credentials
You’ll need an AWS Access Key tied to a user with permission to scale (EC2) and update the cache (via S3). Create a new user with policies for EC2 (AmazonEC2FullAccess) and S3 (AmazonS3FullAccess). To be more secure, you can disable console login for that user. Keep the tab open or copy paste the security credentials in an editor as we’ll use them later during the GitLab Runner configuration.
Prepare the Runner Manager instance
The first step is to install GitLab Runner in an EC2 instance that will serve as the Runner Manager that spawns new machines. Choose a distribution that both Docker and GitLab Runner support, like Ubuntu, Debian, CentOS, or RHEL.
This doesn’t have to be a powerful machine because a Runner Manager instance doesn’t run jobs itself. For your initial configuration, you can start with a smaller instance. This machine is a dedicated host because we need it always up and running. Therefore, it is the only host with an ongoing baseline cost.
Install the prerequisites:
- Log in to your server
- Install GitLab Runner from the official GitLab repository
- Install Docker
- Install Docker Machine from the GitLab fork (Docker has deprecated Docker Machine)
Now that the Runner is installed, it’s time to register it.
Registering the GitLab Runner
Before configuring the GitLab Runner, you need to first register it, so that it connects with your GitLab instance:
- Obtain a runner token
- Register the runner
- When asked the executor type, enter
docker+machine
You can now move on to the most important part, configuring the GitLab Runner.
Configuring the runner
Now that the runner is registered, you need to edit its configuration file and add the required options for the AWS machine driver.
Let’s first break it down to pieces.
The global section
In the global section, you can define the limit of the jobs that can be run
concurrently across all runners (concurrent
). This heavily depends on your
needs, like how many users GitLab Runner will accommodate, how much time your
builds take, etc. You can start with something low like 10
, and increase or
decrease its value going forward.
The check_interval
option defines how often the runner should check GitLab
for new jobs, in seconds.
Example:
concurrent = 10
check_interval = 0
Other options are also available.
The runners
section
From the [[runners]]
section, the most important part is the executor
which
must be set to docker+machine
. Most of those settings are taken care of when
you register the runner for the first time.
limit
sets the maximum number of machines (running and idle) that this runner
will spawn. For more information, check the relationship between limit
, concurrent
and IdleCount
.
Example:
[[runners]]
name = "gitlab-aws-autoscaler"
url = "<URL of your GitLab instance>"
token = "<Runner's token>"
executor = "docker+machine"
limit = 20
Other options
under [[runners]]
are also available.
The runners.docker
section
In the [runners.docker]
section you can define the default Docker image to
be used by the child runners if it’s not defined in .gitlab-ci.yml
.
By using privileged = true
, all runners will be able to run
Docker in Docker
which is useful if you plan to build your own Docker images via GitLab CI/CD.
Next, we use disable_cache = true
to disable the Docker executor’s inner
cache mechanism since we will use the distributed cache mode as described
in the following section.
Example:
[runners.docker]
image = "alpine"
privileged = true
disable_cache = true
Other options
under [runners.docker]
are also available.
The runners.cache
section
To speed up your jobs, GitLab Runner provides a cache mechanism where selected directories and/or files are saved and shared between subsequent jobs. While not required for this setup, it is recommended to use the distributed cache mechanism that GitLab Runner provides. Since new instances will be created on demand, it is essential to have a common place where the cache is stored.
In the following example, we use Amazon S3:
[runners.cache]
Type = "s3"
Shared = true
[runners.cache.s3]
ServerAddress = "s3.amazonaws.com"
AccessKey = "<your AWS Access Key ID>"
SecretKey = "<your AWS Secret Access Key>"
BucketName = "<the bucket where your cache should be kept>"
BucketLocation = "us-east-1"
Here’s some more information to further explore the cache mechanism:
- Reference for
runners.cache
- Reference for
runners.cache.s3
- Deploying and using a cache server for GitLab Runner
- How cache works
The runners.machine
section
This is the most important part of the configuration and it’s the one that tells GitLab Runner how and when to spawn new or remove old Docker Machine instances.
We will focus on the AWS machine options, for the rest of the settings read about the:
- Autoscaling algorithm and the parameters it’s based on - depends on the needs of your organization
- Autoscaling periods - useful when there are regular time periods in your organization when no work is done, for example weekends
Here’s an example of the runners.machine
section:
[runners.machine]
IdleCount = 1
IdleTime = 1800
MaxBuilds = 10
MachineDriver = "amazonec2"
MachineName = "gitlab-docker-machine-%s"
MachineOptions = [
"amazonec2-access-key=XXXX",
"amazonec2-secret-key=XXXX",
"amazonec2-region=us-central-1",
"amazonec2-vpc-id=vpc-xxxxx",
"amazonec2-subnet-id=subnet-xxxxx",
"amazonec2-zone=x",
"amazonec2-use-private-address=true",
"amazonec2-tags=runner-manager-name,gitlab-aws-autoscaler,gitlab,true,gitlab-runner-autoscale,true",
"amazonec2-security-group=xxxxx",
"amazonec2-instance-type=m4.2xlarge",
]
[[runners.machine.autoscaling]]
Periods = ["* * 9-17 * * mon-fri *"]
IdleCount = 50
IdleTime = 3600
Timezone = "UTC"
[[runners.machine.autoscaling]]
Periods = ["* * * * * sat,sun *"]
IdleCount = 5
IdleTime = 60
Timezone = "UTC"
The Docker Machine driver is set to amazonec2
and the machine name has a
standard prefix followed by %s
(required) that is replaced by the ID of the
child runner: gitlab-docker-machine-%s
.
Now, depending on your AWS infrastructure, there are many options you can set up
under MachineOptions
. Below you can see the most common ones.
Machine option | Description |
---|---|
amazonec2-access-key=XXXX |
The AWS access key of the user that has permissions to create EC2 instances, see AWS credentials. |
amazonec2-secret-key=XXXX |
The AWS secret key of the user that has permissions to create EC2 instances, see AWS credentials. |
amazonec2-region=eu-central-1 |
The region to use when launching the instance. You can omit this entirely and the default us-east-1 will be used. |
amazonec2-vpc-id=vpc-xxxxx |
Your VPC ID to launch the instance in. |
amazonec2-subnet-id=subnet-xxxx |
The AWS VPC subnet ID. |
amazonec2-zone=x |
If not specified, the availability zone is a , it needs to be set to the same availability zone as the specified subnet, for example when the zone is eu-west-1b it has to be amazonec2-zone=b
|
amazonec2-use-private-address=true |
Use the private IP address of Docker Machines, but still create a public IP address. Useful to keep the traffic internal and avoid extra costs. |
amazonec2-tags=runner-manager-name,gitlab-aws-autoscaler,gitlab,true,gitlab-runner-autoscale,true |
AWS extra tag key-value pairs, useful to identify the instances on the AWS console. The “Name” tag is set to the machine name by default. We set the “runner-manager-name” to match the runner name set in [[runners]] , so that we can filter all the EC2 instances created by a specific manager setup. |
amazonec2-security-group=xxxx |
AWS VPC security group name, not the security group ID. See AWS security groups. |
amazonec2-instance-type=m4.2xlarge |
The instance type that the child runners will run on. |
amazonec2-ssh-user=xxxx |
The user that will have SSH access to the instance. |
amazonec2-iam-instance-profile=xxxx_runner_machine_inst_profile_name |
The IAM instance pro |