- Confirm your GitLab and GitLab Runner versions
- What does
- Where are logs stored when run as a service?
- Configure DNS for a Docker executor runner
- I’m seeing
x509: certificate signed by unknown authority
- I get
Permission Deniedwhen accessing the
- The Docker executor gets timeout when building Java project
- I get 411 when uploading artifacts
warning: You appear to have cloned an empty repository.
zoneinfo.zip: no such file or directoryerror when using
- Why can’t I run more than one instance of GitLab Runner?
Job failed (system failure): preparing environment:
- Helm Chart:
ERROR .. Unauthorized
This section can assist when troubleshooting GitLab Runner.
GitLab aims to guarantee backward compatibility. However, as a first troubleshooting step, you should ensure your version of GitLab Runner is the same as your GitLab version.
coordinator is the GitLab installation from which a job is requested.
In other words, runners are isolated (virtual) machines that pick up jobs
requested by their
- If the GitLab Runner is run as service on Linux/macOS the daemon logs to syslog.
- If GitLab Runner is running as a service on Windows, it creates system event logs. To view them, open the Event Viewer (from the Run menu, type
eventvwr.mscor search for “Event Viewer”). Then go to Windows Logs > Application. The Source for Runner logs is
gitlab-runner. If you are using Windows Server Core, run this PowerShell command to get the last 20 log entries:
get-eventlog Application -Source gitlab-runner -Newest 20 | format-table -wrap -auto.
Is it possible to run GitLab Runner in debug/verbose mode. From a terminal, run:
gitlab-runner --debug run
Debug logging can be enabled in the global section of the
config.toml by setting the
log_level setting to
## Configure the GitLab Runner logging level. Available values are: debug, info, warn, error, fatal, panic ## ref: https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-global-section ## logLevel: debug
When configuring a GitLab Runner with the Docker executor, it is possible to run into a problem where the Runner daemon on the host can access GitLab but the built container cannot. This can happen when DNS is configured in the host but those configurations are not passed to the container.
GitLab service and GitLab Runner exist in two different networks that are bridged in two ways (for example, over the Internet and through a VPN). If the routing mechanism that the Runner uses to find the GitLab service queries DNS, the container’s DNS configuration doesn’t know to use the DNS service over the VPN and may default to the one provided over the Internet. This configuration would result in the following message:
Created fresh repository. ++ echo 'Created fresh repository.' ++ git -c 'http.userAgent=gitlab-runner 13.9.0 linux/amd64' fetch origin +da39a3ee5e6b4b0d3255bfef95601890afd80709:refs/pipelines/435345 +refs/heads/master:refs/remotes/origin/master --depth 50 --prune --quiet fatal: Authentication failed for 'https://gitlab.example.com/group/example-project.git/'
In this case, the authentication failure is caused by a service in between the Internet and the GitLab service. This service uses separate credentials, which the runner could circumvent if they used the DNS service over the VPN.
You can tell Docker which DNS server to use by using the
dns configuration in the
[runners.docker] section of the Runner’s
dns = ["192.168.xxx.xxx","192.168.xxx.xxx"]
If you want to use Docker executor,
and you are connecting to Docker Engine installed on server.
You can see the
Permission Denied error.
The most likely cause is that your system uses SELinux (enabled by default on CentOS, Fedora and RHEL).
Check your SELinux policy on your system for possible denials.
This most likely happens, because of the broken AUFS storage driver: Java process hangs on inside container. The best solution is to change the storage driver to either OverlayFS (faster) or DeviceMapper (slower).
This happens due to fact that GitLab Runner uses
Transfer-Encoding: chunked which is broken on early version of NGINX (https://serverfault.com/questions/164220/is-there-a-way-to-avoid-nginx-411-content-length-required-errors).
Upgrade your NGINX to newer version. For more information see this issue: https://gitlab.com/gitlab-org/gitlab-runner/-/issues/1031
git clone using HTTP(s) (with GitLab Runner or manually for
tests) and you see the following output:
$ git clone https://git.example.com/user/repo.git Cloning into 'repo'... warning: You appear to have cloned an empty repository.
Make sure, that the configuration of the HTTP Proxy in your GitLab server installation is done properly. Especially if you are using some HTTP Proxy with its own configuration, make sure that GitLab requests are proxied to the GitLab Workhorse socket, not to the GitLab Unicorn socket.
Git protocol via HTTP(S) is resolved by the GitLab Workhorse, so this is the main entrypoint of GitLab.
If you are using Omnibus GitLab, but don’t want to use the bundled NGINX server, please read using a non-bundled web-server.
In the GitLab Recipes repository there are web-server configuration examples for Apache and NGINX.
If you are using GitLab installed from source, please also read the above documentation and examples, and make sure that all HTTP(S) traffic is going through the GitLab Workhorse.
It’s possible to configure the timezone in which
are described. This feature should work on most Unix systems out of the box. However on some
Unix systems, and probably on most non-Unix systems (including Windows, for which we’re providing
GitLab Runner binaries), when used, the runner will crash at start with an error similar to:
Failed to load config Invalid OffPeakPeriods value: open /usr/local/go/lib/time/zoneinfo.zip: no such file or directory
The error is caused by the
time package in Go. Go uses the IANA Time Zone database to load
the configuration of the specified timezone. On most Unix systems, this database is already present on
one of well-known paths (
time package looks for the Time Zone database in all those three paths. If it doesn’t find any
of them, but the machine has a configured Go development environment, then it will fallback to
If none of those paths are present (for example on a production Windows host) the above error is thrown.
In case your system has support for the IANA Time Zone database, but it’s not available by default, you can try to install it. For Linux systems it can be done for example by:
# on Debian/Ubuntu based systems sudo apt-get install tzdata # on RPM based systems sudo yum install tzdata # on Linux Alpine sudo apk add -U tzdata
If your system doesn’t provide this database in a native way, then you can make
working by following the steps below:
zoneinfo.zip. Starting with version v9.1.0 you can download the file from a tagged path. In that case you should replace
latestwith the tag name (e.g.,
v9.1.0) in the
Store this file in a well known directory. We’re suggesting to use the same directory where the
config.tomlfile is present. So for example, if you’re hosting Runner on Windows machine and your configuration file is stored at
C:\gitlab-runner\config.toml, then save the
ZONEINFOenvironment variable containing a full path to the
zoneinfo.zipfile. If you are starting the Runner using the
runcommand, then you can do this with:
ZONEINFO=/etc/gitlab-runner/zoneinfo.zip gitlab-runner run <other options ...>
or if using Windows:
C:\gitlab-runner> set ZONEINFO=C:\gitlab-runner\zoneinfo.zip C:\gitlab-runner> gitlab-runner run <other options ...>
If you are starting GitLab Runner as a system service then you will need to update/override the service configuration in a way that is provided by your service manager software (unix systems) or by adding the
ZONEINFOvariable to the list of environment variables available for the GitLab Runner user through System Settings (Windows).
You can, but not sharing the same
Running multiple instances of GitLab Runner using the same configuration file can cause
unexpected and hard-to-debug behavior. In
GitLab Runner 12.2,
only a single instance of GitLab Runner can use a specific
config.toml file at
This error is often due to your shell loading your profile, and one of the scripts is causing the failure.
Example of dotfiles that are known to cause failure:
SELinux can also be the culprit of this error. You can confirm this by looking at the SELinux audit log:
sealert -a /var/log/audit/audit.log
Before uninstalling or upgrading runners deployed with Helm, pause them in GitLab and wait for any jobs to complete.
If you remove a runner pod with
helm uninstall or
while a job is running,
Unauthorized errors like the following
may occur when the job completes:
ERROR: Error cleaning up pod: Unauthorized ERROR: Error cleaning up secrets: Unauthorized ERROR: Job failed (system failure): Unauthorized
This probably occurs because when the runner is removed, the role bindings are removed. The runner pod continues until the job completes, and then the runner tries to delete it. Without the role binding, the runner pod no longer has access.
See this issue for details.