Troubleshooting Omnibus GitLab installation issues

Use this page to learn about common issues users can encounter when installing Omnibus GitLab packages.

Hash Sum mismatch when downloading packages

apt-get install outputs something like:

E: Failed to fetch https://packages.gitlab.com/gitlab/gitlab-ce/ubuntu/pool/trusty/main/g/gitlab-ce/gitlab-ce_8.1.0-ce.0_amd64.deb  Hash Sum mismatch

Run the following to fix this:

sudo rm -rf /var/lib/apt/lists/partial/*
sudo apt-get update
sudo apt-get clean

See Joe Damato’s from Packagecloud comment and his blog article for more context.

Another workaround is to download the package manually by selecting the correct package from the CE packages or EE packages repository:

curl -LJO "https://packages.gitlab.com/gitlab/gitlab-ce/packages/ubuntu/trusty/gitlab-ce_8.1.0-ce.0_amd64.deb/download"
dpkg -i gitlab-ce_8.1.0-ce.0_amd64.deb

Installation on openSUSE and SLES platforms warns about unknown key signature

Omnibus GitLab packages are signed with GPG keys in addition to the package repositories providing signed metadata. This ensures authenticity and integrity of the packages that are distributed to the users. However, the package manager used in openSUSE and SLES operating systems may sometime raise false warnings with these signatures, similar to

File 'repomd.xml' from repository 'gitlab_gitlab-ce' is signed with an unknown key '14219A96E15E78F4'. Continue? [yes/no] (no):
File 'repomd.xml' from repository 'gitlab_gitlab-ce' is signed with an unknown key '14219A96E15E78F4'. Continue? [yes/no] (no): yes

This is a known bug with zypper where zypper ignores the gpgkey keyword in the repository configuration file. With later versions of Packagecloud, there may be improvements regarding this, but currently users have to manually agree to package installation.

So, in openSUSE or SLES systems, if such a warning is displayed, it is safe to continue installation.

apt/yum complains about GPG signatures

You already have GitLab repositories configured, and ran apt-get update, apt-get install or yum install, and saw errors like the following:

The following signatures couldn’t be verified because the public key is not available: NO_PUBKEY 3F01618A51312F3F

or

https://packages.gitlab.com/gitlab/gitlab-ee/el/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml signature could not be verified for gitlab-ee

This is because on April 2020, GitLab changed the GPG keys used to sign metadata of the apt and yum repositories available through the Packagecloud instance. If you see this error, it generally means you do not have the public keys currently used to sign repository metadata in your keyring. To fix this error, follow the steps to fetch the new key.

Reconfigure shows an error: NoMethodError - undefined method '[]=' for nil:NilClass

You ran sudo gitlab-ctl reconfigure or package upgrade triggered the reconfigure which produced error similar to:

================================================================================
Recipe Compile Error in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/default.rb
================================================================================

NoMethodError
-------------
undefined method `[]=' for nil:NilClass

Cookbook Trace:
---------------
  /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/config.rb:21:in `from_file'
  /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/default.rb:26:in `from_file'

Relevant File Content:

This error is thrown when /etc/gitlab/gitlab.rb configuration file contains configuration that is invalid or unsupported. Double check that there are no typos or that the configuration file does not contain obsolete configuration.

You can check the latest available configuration by using sudo gitlab-ctl diff-config (Command available starting with GitLab 8.17) or check the latest gitlab.rb.template.

GitLab is unreachable in my browser

Try specifying an external_url in /etc/gitlab/gitlab.rb. Also check your firewall settings; port 80 (HTTP) or 443 (HTTPS) might be closed on your GitLab server.

Note that specifying the external_url for GitLab, or any other bundled service (Registry and Mattermost) doesn’t follow the key=value format that other parts of gitlab.rb follow. Make sure that you have them set in the following format:

external_url "https://gitlab.example.com"
registry_external_url "https://registry.example.com"
mattermost_external_url "https://mattermost.example.com"
note
Don’t add the equal sign (=) between external_url and the value.

Emails are not being delivered

To test email delivery you can create a new GitLab account for an email that is not used in your GitLab instance yet.

If necessary, you can modify the ‘From’ field of the emails sent by GitLab with the following setting in /etc/gitlab/gitlab.rb:

gitlab_rails['gitlab_email_from'] = 'gitlab@example.com'

Run sudo gitlab-ctl reconfigure for the change to take effect.

TCP ports for GitLab services are already taken

By default, Puma listens at TCP address 127.0.0.1:8080. NGINX listens on port 80 (HTTP) and/or 443 (HTTPS) on all interfaces.

The ports for Redis, PostgreSQL and Puma can be overridden in /etc/gitlab/gitlab.rb as follows:

redis['port'] = 1234
postgresql['port'] = 2345
puma['port'] = 3456

For NGINX port changes please see settings/nginx.md.

Git user does not have SSH access

SELinux-enabled systems

On SELinux-enabled systems the Git user’s .ssh directory or its contents can get their security context messed up. You can fix this by running sudo gitlab-ctl reconfigure, which sets the gitlab_shell_t security context on /var/opt/gitlab/.ssh.

In GitLab 10.0 this behavior was improved by setting the context permanently using semanage. The runtime dependency policycoreutils-python has been added to the RPM package for RHEL based operating systems in order to ensure the semanage command is available.

Diagnose and resolve SELinux issues

Omnibus GitLab detects default path changes in /etc/gitlab/gitlab.rb and should apply the correct file contexts. For installations using custom data path configuration, the administrator may have to manually resolve SELinux issues.

Data paths may be altered via gitlab.rb, however, a common scenario forces the use of symlink paths. Administrators should be cautious, because symlink paths are not supported for all scenarios, such as Gitaly data paths.

For example, if /data/gitlab replaced /var/opt/gitlab as the base data directory, the following fixes the security context:

sudo semanage fcontext -a -t gitlab_shell_t /data/gitlab/.ssh/
sudo semanage fcontext -a -t gitlab_shell_t /data/gitlab/.ssh/authorized_keys
sudo restorecon -Rv /data/gitlab/
sudo semanage fcontext -a -t gitlab_shell_t /data/gitlab/gitlab-shell/config.yml
sudo restorecon -Rv /data/gitlab/gitlab-shell/
sudo semanage fcontext -a -t gitlab_shell_t /data/gitlab/gitlab-rails/etc/gitlab_shell_secret
sudo restorecon -Rv /data/gitlab/gitlab-rails/
sudo semanage fcontext --list | grep /data/gitlab/

After the policies are applied, you can verify the SSH access is working by getting the welcome message:

ssh -T git@gitlab-hostname

All systems

The Git user is created, by default, with a locked password, shown by '!' in /etc/shadow. Unless “UsePam yes” is enabled, the OpenSSH daemon prevents the Git user from authenticating even with SSH keys. An alternative secure solution is to unlock the password by replacing '!' with '*' in /etc/shadow. The Git user is still unable to change the password because it runs in a restricted shell and the passwd command for non-superusers requires entering the current password prior to a new password. The user cannot enter a password that matches '*', which means the account continues to not have a password.

Keep in mind that the Git user must have access to the system so please review your security settings at /etc/security/access.conf and make sure the Git user is not blocked.

PostgreSQL error FATAL: could not create shared memory segment: Cannot allocate memory

The packaged PostgreSQL instance tries to allocate 25% of total memory as shared memory. On some Linux (virtual) servers, there is less shared memory available, which prevents PostgreSQL from starting. In