GitLab Documentation

Install and configure GitLab Runner for auto-scaling

The auto scale feature was introduced in GitLab Runner 1.1.0.

For an overview of the auto-scale architecture, take a look at the comprehensive documentation on auto-scaling.

Prepare the environment

In order to use the auto-scale feature, Docker and GitLab Runner must be installed in the same machine:

  1. Login to a new Linux-based machine that will serve as a bastion server and where Docker will spawn new machines from
  2. Install GitLab Runner following the GitLab Runner installation documentation
  3. Install Docker Machine following the Docker Machine installation documentation

Prepare the Docker Registry and Cache Server

To speedup the builds we advise to setup a personal Docker registry server working in proxy mode. A cache server is also recommended.

Install Docker Registry

Note: Read more in Distributed registry mirroring.

  1. Login to a dedicated machine where Docker registry proxy will be running
  2. Make sure that Docker Engine is installed on this machine
  3. Create a new Docker registry:

    docker run -d -p 6000:5000 \
        -e REGISTRY_PROXY_REMOTEURL=https://registry-1.docker.io \
        --restart always \
        --name registry registry:2
    

    You can modify the port number (6000) to expose Docker registry on a different port.

  4. Check the IP address of the server:

    hostname --ip-address
    

    You should preferably choose the private networking IP address. The private networking is usually the fastest solution for internal communication between machines of a single provider (DigitalOcean, AWS, Azure, etc,) Usually the private networking is also not accounted to your monthly bandwidth limit.

  5. Docker registry will be accessible under MY_REGISTRY_IP:6000.

Install the cache server

Note: You can use any other S3-compatible server, including Amazon S3. Read more in Distributed runners caching.

  1. Login to a dedicated machine where the cache server will be running
  2. Make sure that Docker Engine is installed on that machine
  3. Start minio, a simple S3-compatible server written in Go:

    docker run -it --restart always -p 9005:9000 \
            -v /.minio:/root/.minio -v /export:/export \
            --name minio \
            minio/minio:latest server /export
    

    You can modify the port 9005 to expose the cache server on different port.

  4. Check the IP address of the server:

    hostname --ip-address
    
  5. Your cache server will be available at MY_CACHE_IP:9005

  6. Read the Access and Secret Key of minio with: sudo cat /.minio/config.json

  7. Create a bucket that will be used by the Runner: sudo mkdir /export/runner. runner is the name of the bucket in that case. If you choose a different bucket then it will be different

  8. All caches will be stored in the /export directory

Configure GitLab Runner

  1. Register a GitLab Runner, selecting the docker+machine executor (Look into runners documentation to learn how to obtain a token):

    sudo gitlab-ci-multi-runner register
    

    Example output:

    Please enter the gitlab-ci coordinator URL (e.g. https://gitlab.com )
    https://gitlab.com
    Please enter the gitlab-ci token for this runner
    xxx
    Please enter the gitlab-ci description for this runner
    my-autoscale-runner
    INFO[0034] fcf5c619 Registering runner... succeeded
    Please enter the executor: shell, docker, docker-ssh, docker+machine, docker-ssh+machine, ssh?
    docker+machine
    Please enter the Docker image (eg. ruby:2.1):
    ruby:2.1
    INFO[0037] Runner registered successfully. Feel free to start it, but if it's
    running already the config should be automatically reloaded!
    
  2. Edit config.toml. You need to fill in the options for [runners.machine] and [runners.cache] and configure the MachineDriver selecting your provider. Also configure MachineOptions, limit and IdleCount.

    For more information visit the dedicated page covering detailed information about GitLab Runner Autoscaling.

    Example configuration using DigitalOcean:

    concurrent = 20
    
    [[runners]]
    executor = "docker+machine"
    limit = 20
    [runners.machine]
      IdleCount = 5
      MachineDriver = "digitalocean"
      MachineName = "auto-scale-runners-%s.my.domain.com"
      MachineOptions = [
          "digitalocean-image=coreos-stable",
          "digitalocean-ssh-user=core",
          "digitalocean-access-token=MY_DIGITAL_OCEAN_TOKEN",
          "digitalocean-region=nyc2",
          "digitalocean-private-networking",
          "engine-registry-mirror=http://MY_REGISTRY_IP:6000"
        ]
    [runners.cache]
      Type = "s3"
      ServerAddress = "MY_CACHE_IP:9005"
      AccessKey = "ACCESS_KEY"
      SecretKey = "SECRET_KEY"
      BucketName = "runner"
      Insecure = true # Use Insecure only when using with Minio, without the TLS certificate enabled
    
  3. Try to build your project. In a few seconds, if you run docker-machine ls you should see a new machine being created.

Upgrading the Runner

  1. Ensure your operating system isn't configured to automatically restart the runner if it exits (which is the default configuration on some platforms).

  2. Stop the runner:

    killall -SIGQUIT gitlab-runner
    

    Sending the SIGQUIT signal will make the Runner to stop gracefully. It will stop accepting new jobs, and will exit as soon as the current builds are finished.

  3. Wait until the Runner exits. You can check its status with gitlab-runner status or await a graceful shutdown for up to 30 minutes with:

    for i in `seq 1 180`; do # 1800 seconds = 30 minutes
        gitlab-runner status || break
        sleep 10
    done
    
  4. You can now safely upgrade the Runner without interrupting any builds

Manage the Docker Machines

  1. Ensure your operating system isn't configured to automatically restart the runner if it exits (which is the default configuration on some platforms).

  2. Stop the Runner:

    killall -SIGQUIT gitlab-runner
    
  3. Wait until the Runner exits. You can check its status with: gitlab-runner status or await a graceful shutdown for up to 30 minutes with:

    for i in `seq 1 180`; do # 1800 seconds = 30 minutes
        gitlab-runner status || break
        sleep 10
    done
    
  4. You can now manage (upgrade or remove) any Docker Machines with the docker-machine command