Troubleshooting the GitLab container registry

You must sign in to GitLab with administrator rights to troubleshoot most issues with the GitLab container registry.

You can find additional troubleshooting information in the GitLab container registry administration documentation.

Migrating OCI container images to GitLab container registry

Migrating container images to the GitLab registry is not supported, but epic proposes to change this behavior.

You can use third-party tools to migrate container images. For example, skopeo, can copy container images between various storage mechanisms. You can use skopeo to copy from container registries, container storage backends, local directories, and local OCI-layout directories to the GitLab container registry.

Docker connection error

A Docker connection error can occur when there are special characters in either the group, project, or branch name. Special characters include:

  • A leading underscore.
  • A trailing hyphen or dash.

To resolve this error, you can change the group path, the project path or the branch name.

You may get a 404 Not Found or Unknown Manifest error message if you use Docker Engine 17.11 or earlier. Current versions of Docker Engine use the v2 API.

The images in your GitLab container registry must use the Docker v2 API. For information on how to update version 1 images to version 2, see the Docker documentation.

Blob unknown to registry error when pushing a manifest list

When pushing a Docker manifest list to the GitLab container registry, you may receive the error manifest blob unknown: blob unknown to registry. This error is likely caused by having multiple images with different architectures spread out over several repositories instead of the same repository.

For example, you may have two images, each representing an architecture:

  • The amd64 platform.
  • The arm64v8 platform.

To build a multi-arch image with these images, you must push them to the same repository as the multi-arch image.

To address the Blob unknown to registry error, include the architecture in the tag name of individual images. For example, use mygroup/myapp:1.0.0-amd64 and mygroup/myapp:1.0.0-arm64v8. You can then tag the manifest list with mygroup/myapp:1.0.0.

Unable to change project path or transfer a project

If you try to change a project path or transfer a project to a new namespace, you may receive one of the following errors:

  • Project cannot be transferred because tags are present in its container registry.
  • Namespace cannot be moved because at least one project has tags in the container registry.

This error occurs when the project has images in the container registry. You must delete or move these images before you change the path or transfer the project.

The following procedure uses these sample project names:

  • For the current project: gitlab.example.com/org/build/sample_project/cr:v2.9.1.
  • For the new project: gitlab.example.com/new_org/build/new_sample_project/cr:v2.9.1.

  1. Download the Docker images on your computer: 

    docker login gitlab.example.com
docker pull gitlab.example.com/org/build/sample_project/cr:v2.9.1
    note
    Use either a personal access token or a deploy token to authenticate your user account.

  2. Rename the images to match the new project name: 

    docker tag gitlab.example.com/org/build/sample_project/cr:v2.9.1 gitlab.example.com/new_org/build/new_sample_project/cr:v2.9.1
  3. Delete the images in the old project by using the UI or API. There may be a delay while the images are queued and deleted.

  4. Change the path or transfer the project:

    1. On the left sidebar, select Search or go to and find your project.
    2. Select Settings > General.
    3. Expand the Advanced section.
    4. In the Change path text box, edit the path.
    5. Select Change path.

  5. Restore the images: 

    docker push gitlab.example.com/new_org/build/new_sample_project/cr:v2.9.1

See this issue for details.

unauthorized: authentication required when pushing large images

When pushing large images, you may see an authentication error like the following: 

docker push gitlab.example.com/myproject/docs:latest
The push refers to a repository [gitlab.example.com/myproject/docs]
630816f32edb: Preparing
530d5553aec8: Preparing
...
4b0bab9ff599: Waiting
d1c800db26c7: Waiting
42755cf4ee95: Waiting
unauthorized: authentication required

This error happens when your authentication token expires before the image push is complete. By default, tokens for the container registry on self-managed GitLab instances expire every five minutes. On GitLab.com, the token expiration time is set to 15 minutes.

If you are using self-managed GitLab, an administrator can increase the token duration.

Failed to pull image messages

You might receive a `Failed to pull image’ error message when a CI/CD job is unable to pull a container image from a project with a limited CI/CD job token scope.

Slow uploads when using kaniko to push large images

When you push large images with kaniko, you might experience uncharacteristically long delays.

This is typically a result of a performance issue with kaniko and HTTP/2. The current workaround is to use HTTP/1.1 when pushing with kaniko.

To use HTTP/1.1, set the GODEBUG environment variable to "http2client=0".

docker login command fails with access forbidden

The container registry returns the GitLab API URL to the Docker client to validate credentials. The Docker client uses basic auth, so the request contains the Authorization header. If the Authorization header is missing in the request to the /jwt/auth endpoint configured in the token_realm for the registry configuration, you receive an access forbidden error message.

For example: 

> docker login gitlab.example.com:4567

Username: user
Password:
Error response from daemon: Get "https://gitlab.company.com:4567/v2/": denied: access forbidden

To avoid this error, ensure the Authorization header is not stripped from the request. For example, a proxy in front of GitLab might be redirecting to the /jwt/auth endpoint.

OCI manifest found, but accept header does not support OCI manifests error

If you are unable to pull an image, the registry logs could have an error similar to: 

manifest unknown: OCI manifest found, but accept header does not support OCI manifests

This error happens when a client does not submit the correct Accept: application/vnd.oci.image.manifest.v1+json header. Make sure your Docker client version is up to date. If you are using a third party tool, make sure it can handle OCI manifests.