Fast lookup of SSH keys

When the number of users grows, SSH operations become slow because OpenSSH performs a linear search through the authorized_keys file to authenticate users. This process requires significant time and disk I/O, which delays users attempting to push or pull to a repository. If users add or remove keys frequently, the operating system may not cache the authorized_keys file, which causes repeated disk reads.

Instead of using the authorized_keys file, you can configure GitLab Shell to look up SSH keys. It is faster because the lookup is indexed in the GitLab database.

Fast lookup is required for Geo

Unlike Cloud Native GitLab, by default Linux package installations manage an authorized_keys file that is located in the git user’s home directory. For most installations, this file is located under /var/opt/gitlab/.ssh/authorized_keys, but you can use the following command to locate the authorized_keys on your system:

getent passwd git | cut -d: -f6 | awk '{print $1"/.ssh/authorized_keys"}'

The authorized_keys file contains all the public SSH keys for users allowed to access GitLab. However, to maintain a single source of truth, Geo must be configured to perform SSH fingerprint lookups with database lookup.

When you set up Geo, you must follow the steps below for both the primary and secondary nodes. Do not select Write to authorized keys file on the primary node, because it is reflected automatically on the secondary if database replication is working.

Set up fast lookup

GitLab Shell provides a way to authorize SSH users with a fast, indexed lookup to the GitLab database. GitLab Shell uses the fingerprint of the SSH key to check whether the user is authorized to access GitLab.

Fast lookup can be enabled with the following SSH servers:

• gitlab-sshd
• OpenSSH

You can run both services simultaneously by using separate ports for each service.

With gitlab-sshd

To set up gitlab-sshd, see the gitlab-sshd documentation. After gitlab-sshd is enabled, GitLab Shell and gitlab-sshd are configured to use fast lookup automatically.

With OpenSSH

Prerequisites:

• OpenSSH 6.9 or later is required because AuthorizedKeysCommand must accept a fingerprint. To check your version, run sshd -V.

To set up fast lookup with OpenSSH:

1. Add the following to your sshd_config file:

   Match User git    # Apply the AuthorizedKeysCommands to the git user only
     AuthorizedKeysCommand /opt/gitlab/embedded/service/gitlab-shell/bin/gitlab-shell-authorized-keys-check git %u %k
     AuthorizedKeysCommandUser git
   Match all    # End match, settings apply to all users again

   This file is usually located in:

   • Linux package installations: /etc/ssh/sshd_config
   • Docker installations: /assets/sshd_config
   • Self-compiled installations: If you followed the instructions for installing GitLab Shell from source, the command should be located at /home/git/gitlab-shell/bin/gitlab-shell-authorized-keys-check. Consider creating a wrapper script somewhere else, as this command must be owned by root, and not be writable by a group or others. Also consider changing the ownership of this command as needed, but this might require temporary ownership changes during gitlab-shell upgrades.

2. Reload OpenSSH:

   # Debian or Ubuntu installations
   sudo service ssh reload
   
   # CentOS installations
   sudo service sshd reload

3. Confirm that SSH is working:

   1. Comment out your user’s key in the authorized_keys file. To do this, start the line with #.

   2. From your local machine, attempt to pull a repository or run:

      ssh -T git@gitlab.example.com

      A successful pull or welcome message means that GitLab found the key in the database, as the key is not present in the file.

If there are lookup failures, the authorized_keys file is still scanned. Git SSH performance might still be slow for many users, as long as the large file exists.

To resolve this, you can disable writes to the authorized_keys file:

1. Confirm SSH works. This step is important because otherwise the file quickly becomes out-of-date.

2. Disable writes to the authorized_keys file:

   1. On the left sidebar, at the bottom, select Admin.
   2. Select Settings > Network.
   3. Expand Performance optimization.
   4. Clear the Use authorized_keys file to authenticate SSH keys checkbox.
   5. Select Save changes.

3. Verify the change:

   1. Remove your SSH key in the UI.
   2. Add a new key.
   3. Try to pull a repository.

4. Back up and delete your authorized_keys file. The current users’ keys are already present in the database, so there is no need for migration or for users to re-add their keys.

How to go back to using the authorized_keys file

This overview is brief. Refer to the above instructions for more context.

1. Rebuild the authorized_keys file.
2. Enable writes to the authorized_keys file.
   1. On the left sidebar, at the bottom, select Admin.
   2. On the left sidebar, select Settings > Network.
   3. Expand Performance optimization.
   4. Select the Use authorized_keys file to authenticate SSH keys checkbox.
3. Remove the AuthorizedKeysCommand lines from /etc/ssh/sshd_config or from /assets/sshd_config if you are using Docker from a Linux package installation.
4. Reload sshd: sudo service sshd reload.

SELinux support

GitLab supports authorized_keys database lookups with SELinux.

Because the SELinux policy is static, GitLab doesn’t support changing internal web server ports. Administrators would have to create a special .te file for the environment, as it isn’t generated dynamically.

Additional documentation

Additional technical documentation for gitlab-sshd may be found in the GitLab Shell documentation.

Troubleshooting

SSH traffic slow or high CPU load

If your SSH traffic is slow or causing high CPU load:

• Check the size of /var/log/btmp.
• Ensure it is rotated on a regular basis, or after reaching a certain size.

If this file is very large, GitLab SSH fast lookup can cause the bottleneck to be hit more frequently, thus decreasing performance even further. Consider disabling UsePAM in your sshd_config to avoid reading /var/log/btmp altogether.

Running strace and lsof on a running sshd: git process returns debugging information. To get an strace on an in-progress Git over SSH connection for IP x.x.x.x, run:

sudo strace -s 10000 -p $(sudo netstat -tp | grep x.x.x.x | egrep 'ssh.*: git' | sed -e 's/.*ESTABLISHED *//' -e 's#/.*##')

Or get an lsof for a running Git over SSH process:

sudo lsof -p $(sudo netstat -tp | egrep 'ssh.*: git' | head -1 | sed -e 's/.*ESTABLISHED *//' -e 's#/.*##')