Configuring a Monitoring node for Scaling and High Availability

Introduced in GitLab 12.0.

You can configure a Prometheus node to monitor GitLab.

Standalone Monitoring node using GitLab Omnibus

The GitLab Omnibus package can be used to configure a standalone Monitoring node running Prometheus and Grafana. The monitoring node is not highly available. See Scaling and High Availability for an overview of GitLab scaling and high availability options.

The steps below are the minimum necessary to configure a Monitoring node running Prometheus and Grafana with Omnibus:

  1. SSH into the Monitoring node.
  2. Download/install the Omnibus GitLab package you want using steps 1 and 2 from the GitLab downloads page.
    • Do not complete any other steps on the download page.
  3. Make sure to collect CONSUL_SERVER_NODES, which are the IP addresses or DNS records of the Consul server nodes, for the next step. Note they are presented as Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z

  4. Edit /etc/gitlab/gitlab.rb and add the contents:

    external_url 'http://gitlab.example.com'
    
    # Enable Prometheus
    prometheus['enable'] = true
    prometheus['listen_address'] = '0.0.0.0:9090'
    prometheus['monitor_kubernetes'] = false
    
    # Enable Grafana
    grafana['enable'] = true
    grafana['admin_password'] = 'toomanysecrets'
    
    # Enable service discovery for Prometheus
    consul['enable'] = true
    consul['monitoring_service_discovery'] =  true
    
    # Replace placeholders
    # Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z
    # with the addresses of the Consul server nodes
    consul['configuration'] = {
       retry_join: %w(Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z),
    }
    
    # Disable all other services
    gitlab_rails['auto_migrate'] = false
    alertmanager['enable'] = false
    gitaly['enable'] = false
    gitlab_monitor['enable'] = false
    gitlab_workhorse['enable'] = false
    nginx['enable'] = true
    postgres_exporter['enable'] = false
    postgresql['enable'] = false
    redis['enable'] = false
    redis_exporter['enable'] = false
    sidekiq['enable'] = false
    unicorn['enable'] = false
    node_exporter['enable'] = false
    
  5. Run sudo gitlab-ctl reconfigure to compile the configuration.

Migrating to Service Discovery

Once monitoring using Service Discovery is enabled with consul['monitoring_service_discovery'] = true, ensure that prometheus['scrape_configs'] is not set in /etc/gitlab/gitlab.rb. Setting both consul['monitoring_service_discovery'] = true and prometheus['scrape_configs'] in /etc/gitlab/gitlab.rb will result in errors.