Configure Gitaly

The Gitaly service itself is configured by using a TOML configuration file.

To change Gitaly settings:

For Omnibus GitLab

  1. Edit /etc/gitlab/gitlab.rb and add or change the Gitaly settings.
  2. Save the file and reconfigure GitLab.

For installations from source

  1. Edit /home/git/gitaly/config.toml and add or change the Gitaly settings.
  2. Save the file and restart GitLab.

The following configuration options are also available:

About the Gitaly token

The token referred to throughout the Gitaly documentation is just an arbitrary password selected by the administrator. It is unrelated to tokens created for the GitLab API or other similar web API tokens.

Run Gitaly on its own server

By default, Gitaly is run on the same server as Gitaly clients and is configured as above. Single-server installations are best served by this default configuration used by:

However, Gitaly can be deployed to its own server, which can benefit GitLab installations that span multiple machines.

note
When configured to run on their own servers, Gitaly servers must be upgraded before Gitaly clients in your cluster.

The process for setting up Gitaly on its own server is:

  1. Install Gitaly.
  2. Configure authentication.
  3. Configure Gitaly servers.
  4. Configure Gitaly clients.
  5. Disable Gitaly where not required (optional).

When running Gitaly on its own server, note the following regarding GitLab versions:

  • From GitLab 11.4, Gitaly was able to serve all Git requests without requiring a shared NFS mount for Git repository data, except for the Elasticsearch indexer.
  • From GitLab 11.8, the Elasticsearch indexer also uses Gitaly for data access. NFS can still be leveraged for redundancy on block-level Git data, but should be mounted only on the Gitaly servers.
  • From GitLab 11.8 to 12.2, it is possible to use Elasticsearch in a Gitaly setup that doesn’t use NFS. To use Elasticsearch in these versions, the repository indexer must be enabled in your GitLab configuration.
  • In GitLab 12.3 and later, the new indexer is the default and no configuration is required.

Network architecture

The following list depicts the network architecture of Gitaly:

  • GitLab Rails shards repositories into repository storages.
  • /config/gitlab.yml contains a map from storage names to (Gitaly address, Gitaly token) pairs.
  • The storage name -> (Gitaly address, Gitaly token) map in /config/gitlab.yml is the single source of truth for the Gitaly network topology.
  • A (Gitaly address, Gitaly token) corresponds to a Gitaly server.
  • A Gitaly server hosts one or more storages.
  • A Gitaly client can use one or more Gitaly servers.
  • Gitaly addresses must be specified in such a way that they resolve correctly for all Gitaly clients.
  • Gitaly clients are:
    • Puma.
    • Sidekiq.
    • GitLab Workhorse.
    • GitLab Shell.
    • Elasticsearch indexer.
    • Gitaly itself.
  • A Gitaly server must be able to make RPC calls to itself by using its own (Gitaly address, Gitaly token) pair as specified in /config/gitlab.yml.
  • Authentication is done through a static token which is shared among the Gitaly and GitLab Rails nodes.

The following digraph illustrates communication between Gitaly servers and GitLab Rails showing the default ports for HTTP and HTTPs communication.

Gitaly network architecture diagram

caution
Gitaly servers must not be exposed to the public internet as Gitaly’s network traffic is unencrypted by default. The use of firewall is highly recommended to restrict access to the Gitaly server. Another option is to use TLS.

In the following sections, we describe how to configure two Gitaly servers with secret token abc123secret:

  • gitaly1.internal.
  • gitaly2.internal.

We assume your GitLab installation has three repository storages:

  • default.
  • storage1.
  • storage2.

You can use as few as one server with one repository storage if desired.

Install Gitaly

Install Gitaly on each Gitaly server using either Omnibus GitLab or install it from source:

  • For Omnibus GitLab, download and install the Omnibus GitLab package you want but do not provide the EXTERNAL_URL= value.
  • To install from source, follow the steps at Install Gitaly.

Configure authentication

Gitaly and GitLab use two shared secrets for authentication:

  • One to authenticate gRPC requests to Gitaly.
  • A second for authentication callbacks from GitLab Shell to the GitLab internal API.

For Omnibus GitLab

To configure the Gitaly token:

  1. On the Gitaly clients, edit /etc/gitlab/gitlab.rb:

    gitlab_rails['gitaly_token'] = 'abc123secret'
    
  2. Save the file and reconfigure GitLab.
  3. On the Gitaly server, edit /etc/gitlab/gitlab.rb:

    gitaly['auth_token'] = 'abc123secret'
    
  4. Reconfigure GitLab.

There are two ways to configure the GitLab Shell token.

Method 1:

  1. Copy /etc/gitlab/gitlab-secrets.json from the Gitaly client to same path on the Gitaly servers (and any other Gitaly clients).
  2. Reconfigure GitLab on Gitaly servers.

Method 2:

  1. On the Gitaly clients, edit /etc/gitlab/gitlab.rb:

    gitlab_shell['secret_token'] = 'shellsecret'
    
  2. Save the file and reconfigure GitLab.
  3. On the Gitaly servers, edit /etc/gitlab/gitlab.rb:

    gitlab_shell['secret_token'] = 'shellsecret'
    
  4. Reconfigure GitLab.

For installations from source

  1. Copy /home/git/gitlab/.gitlab_shell_secret from the Gitaly client to the same path on the Gitaly servers (and any other Gitaly clients).
  2. On the Gitaly clients, edit /home/git/gitlab/config/gitlab.yml:

    gitlab:
      gitaly:
        token: 'abc123secret'
    
  3. Save the file and restart GitLab.
  4. On the Gitaly servers, edit /home/git/gitaly/config.toml:

    [auth]
    token = 'abc123secret'
    
  5. Save the file and restart GitLab.

Configure Gitaly servers

On the Gitaly servers, you must configure storage paths and enable the network listener. The Gitaly server must be able to read, write, and set permissions on the configured path.

If you want to reduce the risk of downtime when you enable authentication, you can temporarily disable enforcement. For more information, see the documentation on configuring Gitaly authentication.

For Omnibus GitLab

  1. Edit /etc/gitlab/gitlab.rb:
   # Avoid running unnecessary services on the Gitaly server
   postgresql['enable'] = false
   redis['enable'] = false
   nginx['enable'] = false
   puma['enable'] = false
   sidekiq['enable'] = false
   gitlab_workhorse['enable'] = false
   grafana['enable'] = false
   gitlab_exporter['enable'] = false
   gitlab_kas['enable'] = false

   # If you run a separate monitoring node you can disable these services
   prometheus['enable'] = false
   alertmanager['enable'] = false

   # If you don't run a separate monitoring node you can
   # enable Prometheus access & disable these extra services.
   # This makes Prometheus listen on all interfaces. You must use firewalls to restrict access to this address/port.
   # prometheus['listen_address'] = '0.0.0.0:9090'
   # prometheus['monitor_kubernetes'] = false

   # If you don't want to run monitoring services uncomment the following (not recommended)
   # node_exporter['enable'] = false

   # Prevent database connections during 'gitlab-ctl reconfigure'
   gitlab_rails['auto_migrate'] = false

   # Configure the gitlab-shell API callback URL. Without this, `git push` will
   # fail. This can be your 'front door' GitLab URL or an internal load
   # balancer.
   # Don't forget to copy `/etc/gitlab/gitlab-secrets.json` from Gitaly client to Gitaly server.
   gitlab_rails['internal_api_url'] = 'https://gitlab.example.com'

   # Make Gitaly accept connections on all network interfaces. You must use
   # firewalls to restrict access to this address/port.
   # Comment out following line if you only want to support TLS connections
   gitaly['listen_addr'] = "0.0.0.0:8075"

   # Authentication token to ensure only authorized servers can communicate with
   # Gitaly server
   gitaly['auth_token'] = 'AUTH_TOKEN'
  1. Append the following to /etc/gitlab/gitlab.rb for each respective Gitaly server:

    On gitaly1.internal:

    git_data_dirs({
      'default' => {
        'path' => '/var/opt/gitlab/git-data'
      },
      'storage1' => {
        'path' => '/mnt/gitlab/git-data'
      },
    })
    

    On gitaly2.internal:

    git_data_dirs({
      'storage2' => {
        'path' => '/srv/gitlab/git-data'
      },
    })
    
  2. Save the file and reconfigure GitLab.
  3. Run sudo /opt/gitlab/embedded/bin/gitaly-hooks check /var/opt/gitlab/gitaly/config.toml to confirm that Gitaly can perform callbacks to the GitLab internal API.

For installations from source

  1. Edit /home/git/gitaly/config.toml:

    listen_addr = '0.0.0.0:8075'
    
    runtime_dir = '/var/opt/gitlab/gitaly'
    
    [logging]
    format = 'json'
    level = 'info'
    dir = '/var/log/gitaly'
    

    For GitLab 14.9 and earlier, set internal_socket_dir = '/var/opt/gitlab/gitaly' instead of runtime_dir.

  2. Append the following to