- Overview
- Configuring Prometheus
- Viewing performance metrics
- Prometheus as a Grafana data source
- GitLab metrics
- Bundled software metrics
- Configuring Prometheus to monitor Kubernetes
- Troubleshooting
Monitoring GitLab with Prometheus
Prometheus is a powerful time-series monitoring service, providing a flexible platform for monitoring GitLab and other software products.
GitLab provides out-of-the-box monitoring with Prometheus, providing easy access to high quality time-series monitoring of GitLab services.
Prometheus and the various exporters listed in this page are bundled in the Omnibus GitLab package. Check each exporter’s documentation for the timeline they got added. For installations from source you must install them yourself. Over subsequent releases additional GitLab metrics are captured.
Prometheus services are on by default.
Prometheus and its exporters don’t authenticate users, and are available to anyone who can access them.
Overview
Prometheus works by periodically connecting to data sources and collecting their performance metrics through the various exporters. To view and work with the monitoring data, you can either connect directly to Prometheus or use a dashboard tool like Grafana.
Configuring Prometheus
For installations from source, you must install and configure it yourself.
Prometheus and its exporters are on by default.
Prometheus runs as the gitlab-prometheus
user and listen on
http://localhost:9090
. By default, Prometheus is only accessible from the GitLab server itself.
Each exporter is automatically set up as a
monitoring target for Prometheus, unless individually disabled.
To disable Prometheus and all of its exporters, as well as any added in the future:
- Edit
/etc/gitlab/gitlab.rb
-
Add or find and uncomment the following line, making sure it’s set to
false
:prometheus_monitoring['enable'] = false
- Save the file and reconfigure GitLab for the changes to take effect.
Changing the port and address Prometheus listens on
To access Prometheus from outside the GitLab server, change the address/port that Prometheus listens on:
- Edit
/etc/gitlab/gitlab.rb
-
Add or find and uncomment the following line:
prometheus['listen_address'] = 'localhost:9090'
Replace
localhost:9090
with the address or port you want Prometheus to listen on. If you would like to allow access to Prometheus to hosts other thanlocalhost
, leave out the host, or use0.0.0.0
to allow public access:prometheus['listen_address'] = ':9090' # or prometheus['listen_address'] = '0.0.0.0:9090'
- Save the file and reconfigure GitLab for the changes to take effect
Adding custom scrape configurations
You can configure additional scrape targets for the Omnibus GitLab-bundled
Prometheus by editing prometheus['scrape_configs']
in /etc/gitlab/gitlab.rb
using the Prometheus scrape target configuration
syntax.
Here is an example configuration to scrape http://1.1.1.1:8060/probe?param_a=test¶m_b=additional_test
:
prometheus['scrape_configs'] = [
{
'job_name': 'custom-scrape',
'metrics_path': '/probe',
'params' => {
'param_a' => ['test'],
'param_b' => ['additional_test']
},
'static_configs' => [
'targets' => ['1.1.1.1:8060'],
],
},
]
Standalone Prometheus using Omnibus GitLab
The Omnibus GitLab package can be used to configure a standalone Monitoring node running Prometheus and Grafana.
The steps below are the minimum necessary to configure a Monitoring node running Prometheus and Grafana with Omnibus GitLab:
- SSH into the Monitoring node.
- Install the Omnibus GitLab package you want using steps 1 and 2 from the GitLab downloads page, but do not follow the remaining steps.
- Make sure to collect the IP addresses or DNS records of the Consul server nodes, for the next step.
-
Edit
/etc/gitlab/gitlab.rb
and add the contents:roles ['monitoring_role'] external_url 'http://gitlab.example.com' # Prometheus prometheus['listen_address'] = '0.0.0.0:9090' prometheus['monitor_kubernetes'] = false # Grafana grafana['enable'] = true grafana['admin_password'] = 'toomanysecrets' grafana['disable_login_form'] = false # Enable service discovery for Prometheus consul['enable'] = true consul['monitoring_service_discovery'] = true consul['configuration'] = { retry_join: %w(10.0.0.1 10.0.0.2 10.0.0.3), # The addresses can be IPs or FQDNs } # Nginx - For Grafana access nginx['enable'] = true
- Run
sudo gitlab-ctl reconfigure
to compile the configuration.
The next step is to tell all the other nodes where the monitoring node is:
-
Edit
/etc/gitlab/gitlab.rb
, and add, or find and uncomment the following line:gitlab_rails['prometheus_address'] = '10.0.0.1:9090'
Where
10.0.0.1:9090
is the IP address and port of the Prometheus node. -
Save the file and reconfigure GitLab for the changes to take effect.
After monitoring using Service Discovery is enabled with consul['monitoring_service_discovery'] = true
,
ensure that prometheus['scrape_configs']
is not set in /etc/gitlab/gitlab.rb
. Setting both
consul['monitoring_service_discovery'] = true
and prometheus['scrape_configs']
in /etc/gitlab/gitlab.rb
results in errors.
Using an external Prometheus server
A few configuration changes are required to allow GitLab to be monitored by an external Prometheus server. External servers are recommended for GitLab deployments with multiple nodes.
To use an external Prometheus server:
- Edit
/etc/gitlab/gitlab.rb
. -
Disable the bundled Prometheus:
prometheus['enable'] = false
-
Set each bundled service’s exporter to listen on a network address, for example:
node_exporter['listen_address'] = '0.0.0.0:9100' gitlab_workhorse['prometheus_listen_addr'] = "0.0.0.0:9229" # Rails nodes gitlab_exporter['listen_address'] = '0.0.0.0' gitlab_exporter['listen_port'] = '9168' # Sidekiq nodes sidekiq['listen_address'] = '0.0.0.0' # Redis nodes redis_exporter['listen_address'] = '0.0.0.0:9121' # PostgreSQL nodes postgres_exporter['listen_address'] = '0.0.0.0:9187' # Gitaly nodes gitaly['prometheus_listen_addr'] = "0.0.0.0:9236"
- Install and set up a dedicated Prometheus instance, if necessary, using the official installation instructions.
-
Add the Prometheus server IP address to the monitoring IP allowlist. For example:
gitlab_rails['monitoring_whitelist'] = ['127.0.0.0/8', '192.168.0.1']
-
On all GitLab Rails(Puma, Sidekiq) servers, set the Prometheus server IP address and listen port. For example:
gitlab_rails['prometheus_address'] = '192.168.0.1:9090'
-
To scrape NGINX metrics, you must also configure NGINX to allow the Prometheus server IP. For example:
nginx['status']['options'] = { "server_tokens" => "off", "access_log" => "off", "allow" => "192.168.0.1", "deny" => "all", }
- Reconfigure GitLab to apply the changes.
- Edit the Prometheus server’s configuration file.
-
Add each node’s exporters to the Prometheus server’s scrape target configuration. For example, a sample snippet using
static_configs
:scrape_configs: - job_name: nginx static_configs: - targets: - 1.1.1.1:8060 - job_name: redis static_configs: - targets: - 1.1.1.1:9121 - job_name: postgres static_configs: - targets: - 1.1.1.1:9187 - job_name: node static_configs: - targets: - 1.1.1.1:9100 - job_name: gitlab-workhorse static_configs: - targets: - 1.1.1.1:9229 - job_name: gitlab-rails metrics_path: "/-/metrics" scheme: https static_configs: - targets: - 1.1.1.1 - job_name: gitlab-sidekiq static_configs: - targets: - 1.1.1.1:8082 - job_name: gitlab_exporter_database metrics_path