• Use a proxy for containers
  • GitLab Dependency Proxy
  • Docker Hub Registry mirror
    • Use a Docker Hub Registry mirror
  • Other open source solutions
• Use a distributed cache
  • Use AWS S3
  • Use MinIO
  • Use Google Cloud Storage
  • Use Azure Blob storage

Speed up job execution

You can improve performance of your jobs by caching your images and dependencies.

Use a proxy for containers

You can speed up the time it takes to download Docker images by using:

• The GitLab Dependency Proxy or
• A mirror of the DockerHub Registry
• Other open source solutions

GitLab Dependency Proxy

To more quickly access container images, you can use the Dependency Proxy to proxy container images.

Docker Hub Registry mirror

You can also speed up the time it takes for your jobs to access container images by mirroring Docker Hub. This results in the Registry as a pull through cache. In addition to speeding up job execution, a mirror can make your infrastructure more resilient to Docker Hub outages and Docker Hub rate limits.

When the Docker daemon is configured to use the mirror it automatically checks for the image on your running instance of the mirror. If it’s not available, it pulls the image from the public Docker registry and stores it locally before handing it back to you.

The next request for the same image pulls from your local registry.

More detail on how it works can be found here.

Use a Docker Hub Registry mirror

To create a Docker Hub Registry mirror:

1. Log in to a dedicated machine where the proxy container registry will run.
2. Make sure that Docker Engine is installed on that machine.

3. Create a new container registry:

   docker run -d -p 6000:5000 \
       -e REGISTRY_PROXY_REMOTEURL=https://registry-1.docker.io \
       --restart always \
       --name registry registry:2
   

   You can modify the port number (6000) to expose the registry on a different port. This will start the server with http. If you want to turn on TLS (https) follow the official documentation.

4. Check the IP address of the server:

   hostname --ip-address
   

   You should choose the private network IP address. The private network is usually the fastest solution for internal communication between machines on a single provider, like DigitalOcean, AWS, or Azure. Usually, data transferred on a private network is not applied against your monthly bandwidth limit.

The Docker Hub registry is accessible under MY_REGISTRY_IP:6000.

You can now configure config.toml to use the new registry server.

Other open source solutions

• rpardini/docker-registry-proxy can proxy most container registries locally, including the GitLab Container Registry.

Use a distributed cache

You can speed up the time it takes to download language dependencies by using a distributed cache.

To specify a distributed cache, you set up the cache server and then configure runner to use that cache server.

If you are using autoscaling, learn more about the distributed runners cache feature.

The following cache servers are supported:

• AWS S3
• MinIO or other S3-compatible cache server
• Google Cloud Storage
• Azure Blob storage

Learn more about GitLab CI/CD cache dependencies and best practices.

Use AWS S3

To use AWS S3 as a distributed cache, edit runner’s config.toml file to point to the S3 location and provide credentials for connecting. Make sure the runner has a network path to the S3 endpoint.

If you use a private subnet with a NAT gateway, to save cost on data transfers you can enable an S3 VPC endpoint.

Use MinIO

Instead of using AWS S3, you can create your own cache storage.

1. Log in to a dedicated machine where the cache server will run.
2. Make sure that Docker Engine is installed on that machine.

3. Start MinIO, a simple S3-compatible server written in Go:

   docker run -d --restart always -p 9005:9000 \
           -v /.minio:/root/.minio -v /export:/export \
           -e "MINIO_ROOT_USER=<minio_root_username>" \
           -e "MINIO_ROOT_PASSWORD=<minio_root_password>" \
           --name minio \
           minio/minio:latest server /export
   

   You can modify the port 9005 to expose the cache server on a different port.

4. Check the IP address of the server:

   hostname --ip-address
   

5. Your cache server will be available at MY_CACHE_IP:9005.

6. Create a bucket that will be used by the runner:

   sudo mkdir /export/runner
   

   runner is the name of the bucket in that case. If you choose a different bucket, then it will be different. All caches will be stored in the /export directory.

7. Use the MINIO_ROOT_USER and MINIO_ROOT_PASSWORD values (from above) as your Access and Secret Keys when configuring your runner.

You can now configure config.toml to use the new cache server.

Use Google Cloud Storage

To use Google Cloud Platform as a distributed cache, edit runner’s config.toml file to point to the GCP location and provide credentials for connecting. Make sure the runner has a network path to the GCS endpoint.

Use Azure Blob storage

To use Azure Blob storage as a distributed cache, edit runner’s config.toml file to point to the Azure location and provide credentials for connecting. Make sure the runner has a network path to the Azure endpoint.