GitLab Prometheus metrics

Note: Available since Omnibus GitLab 9.3. For installations from source you’ll have to configure it yourself.

To enable the GitLab Prometheus metrics:

  1. Log into GitLab as an administrator, and go to the Admin area.
  2. Navigate to GitLab’s Settings > Metrics and profiling.
  3. Find the Metrics - Prometheus section, and click Enable Prometheus Metrics.
  4. Restart GitLab for the changes to take effect.

Collecting the metrics

GitLab monitors its own internal service metrics, and makes them available at the /-/metrics endpoint. Unlike other Prometheus exporters, in order to access it, the client IP needs to be included in a whitelist.

For Omnibus and Chart installations, these metrics are automatically enabled and collected as of GitLab 9.4. For source installations or earlier versions, these metrics will need to be enabled manually and collected by a Prometheus server.

Metrics available

The following metrics are available:

MetricTypeSinceDescriptionLabels
gitlab_banzai_cached_render_real_duration_secondsHistogram9.4Duration of rendering Markdown into HTML when cached output existscontroller, action
gitlab_banzai_cacheless_render_real_duration_secondsHistogram9.4Duration of rendering Markdown into HTML when cached outupt does not existcontroller, action
gitlab_cache_misses_totalCounter10.2Cache read misscontroller, action
gitlab_cache_operation_duration_secondsHistogram10.2Cache access time 
gitlab_cache_operations_totalCounter12.2Cache operations by controller/actioncontroller, action, operation
gitlab_database_transaction_secondsHistogram12.1Time spent in database transactions, in seconds 
gitlab_method_call_duration_secondsHistogram10.2Method calls real durationcontroller, action, module, method
gitlab_rails_queue_duration_secondsHistogram9.4Measures latency between GitLab Workhorse forwarding a request to Rails 
gitlab_sql_duration_secondsHistogram10.2SQL execution time, excluding SCHEMA operations and BEGIN / COMMIT 
gitlab_transaction_allocated_memory_bytesHistogram10.2Allocated memory for all transactions (gitlab_transaction_* metrics) 
gitlab_transaction_cache_<key>_count_totalCounter10.2Counter for total Rails cache calls (per key) 
gitlab_transaction_cache_<key>_duration_totalCounter10.2Counter for total time (seconds) spent in Rails cache calls (per key) 
gitlab_transaction_cache_count_totalCounter10.2Counter for total Rails cache calls (aggregate) 
gitlab_transaction_cache_duration_totalCounter10.2Counter for total time (seconds) spent in Rails cache calls (aggregate) 
gitlab_transaction_cache_read_hit_count_totalCounter10.2Counter for cache hits for Rails cache callscontroller, action
gitlab_transaction_cache_read_miss_count_totalCounter10.2Counter for cache misses for Rails cache callscontroller, action
gitlab_transaction_duration_secondsHistogram10.2Duration for all transactions (gitlab_transaction_* metrics)controller, action
gitlab_transaction_event_build_found_totalCounter9.4Counter for build found for API /jobs/request 
gitlab_transaction_event_build_invalid_totalCounter9.4Counter for build invalid due to concurrency conflict for API /jobs/request 
gitlab_transaction_event_build_not_found_cached_totalCounter9.4Counter for cached response of build not found for API /jobs/request 
gitlab_transaction_event_build_not_found_totalCounter9.4Counter for build not found for API /jobs/request 
gitlab_transaction_event_change_default_branch_totalCounter9.4Counter when default branch is changed for any repository 
gitlab_transaction_event_create_repository_totalCounter9.4Counter when any repository is created 
gitlab_transaction_event_etag_caching_cache_hit_totalCounter9.4Counter for etag cache hit.endpoint
gitlab_transaction_event_etag_caching_header_missing_totalCounter9.4Counter for etag cache miss - header missingendpoint
gitlab_transaction_event_etag_caching_key_not_found_totalCounter9.4Counter for etag cache miss - key not foundendpoint
gitlab_transaction_event_etag_caching_middleware_used_totalCounter9.4Counter for etag middleware accessedendpoint
gitlab_transaction_event_etag_caching_resource_changed_totalCounter9.4Counter for etag cache miss - resource changedendpoint
gitlab_transaction_event_fork_repository_totalCounter9.4Counter for repository forks (RepositoryForkWorker). Only incremented when source repository exists 
gitlab_transaction_event_import_repository_totalCounter9.4Counter for repository imports (RepositoryImportWorker) 
gitlab_transaction_event_push_branch_totalCounter9.4Counter for all branch pushes 
gitlab_transaction_event_push_commit_totalCounter9.4Counter for commitsbranch
gitlab_transaction_event_push_tag_totalCounter9.4Counter for tag pushes 
gitlab_transaction_event_rails_exception_totalCounter9.4Counter for number of rails exceptions 
gitlab_transaction_event_receive_email_totalCounter9.4Counter for recieved emailshandler
gitlab_transaction_event_remote_mirrors_failed_totalCounter10.8Counter for failed remote mirrors 
gitlab_transaction_event_remote_mirrors_finished_totalCounter10.8Counter for finished remote mirrors 
gitlab_transaction_event_remote_mirrors_running_totalCounter10.8Counter for running remote mirrors 
gitlab_transaction_event_remove_branch_totalCounter9.4Counter when a branch is removed for any repository 
gitlab_transaction_event_remove_repository_totalCounter9.4Counter when a repository is removed 
gitlab_transaction_event_remove_tag_totalCounter9.4Counter when a tag is remove for any repository 
gitlab_transaction_event_sidekiq_exception_totalCounter9.4Counter of Sidekiq exceptions 
gitlab_transaction_event_stuck_import_jobs_totalCounter9.4Count of stuck import jobsprojects_without_jid_count, projects_with_jid_count
gitlab_transaction_event_update_build_totalCounter9.4Counter for update build for API /jobs/request/:id 
gitlab_transaction_new_redis_connections_totalCounter9.4Counter for new Redis connections 
gitlab_transaction_queue_duration_totalCounter9.4Duration jobs were enqueued before processing 
gitlab_transaction_rails_queue_duration_totalCounter9.4Measures latency between GitLab Workhorse forwarding a request to Railscontroller, action
gitlab_transaction_view_duration_totalCounter9.4Duration for viewscontroller, action, view
gitlab_view_rendering_duration_secondsHistogram10.2Duration for views (histogram)controller, action, view
http_requests_totalCounter9.4Rack request countmethod
http_request_duration_secondsHistogram9.4HTTP response time from rack middlewaremethod, status
pipelines_created_totalCounter9.4Counter of pipelines created 
rack_uncaught_errors_totalCounter9.4Rack connections handling uncaught errors count 
user_session_logins_totalCounter9.4Counter of how many users have logged in 
upload_file_does_not_existCounter10.7 in EE, 11.5 in CENumber of times an upload record could not find its file 
failed_login_captcha_totalGauge11.0Counter of failed CAPTCHA attempts during login 
successful_login_captcha_totalGauge11.0Counter of successful CAPTCHA attempts during login 

Metrics controlled by a feature flag

The following metrics can be controlled by feature flags:

MetricFeature Flag
gitlab_method_call_duration_secondsprometheus_metrics_method_instrumentation
gitlab_transaction_allocated_memory_bytesprometheus_metrics_transaction_allocated_memory
gitlab_view_rendering_duration_secondsprometheus_metrics_view_instrumentation

Sidekiq Metrics available for Geo

Sidekiq jobs may also gather metrics, and these metrics can be accessed if the Sidekiq exporter is enabled (e.g. via the monitoring.sidekiq_exporter configuration option in gitlab.yml.

MetricTypeSinceDescriptionLabels
geo_db_replication_lag_secondsGauge10.2Database replication lag (seconds)url
geo_repositoriesGauge10.2Total number of repositories available on primaryurl
geo_repositories_syncedGauge10.2Number of repositories synced on secondaryurl
geo_repositories_failedGauge10.2Number of repositories failed to sync on secondaryurl
geo_lfs_objectsGauge10.2Total number of LFS objects available on primaryurl
geo_lfs_objects_syncedGauge10.2Number of LFS objects synced on secondaryurl
geo_lfs_objects_failedGauge10.2Number of LFS objects failed to sync on secondaryurl
geo_attachmentsGauge10.2Total number of file attachments available on primaryurl
geo_attachments_syncedGauge10.2Number of attachments synced on secondaryurl
geo_attachments_failedGauge10.2Number of attachments failed to sync on secondaryurl
geo_last_event_idGauge10.2Database ID of the latest event log entry on the primaryurl
geo_last_event_timestampGauge10.2UNIX timestamp of the latest event log entry on the primaryurl
geo_cursor_last_event_idGauge10.2Last database ID of the event log processed by the secondaryurl
geo_cursor_last_event_timestampGauge10.2Last UNIX timestamp of the event log processed by the secondaryurl
geo_status_failed_totalCounter10.2Number of times retrieving the status from the Geo Node failedurl
geo_last_successful_status_check_timestampGauge10.2Last timestamp when the status was successfully updatedurl
geo_lfs_objects_synced_missing_on_primaryGauge10.7Number of LFS objects marked as synced due to the file missing on the primaryurl
geo_job_artifacts_synced_missing_on_primaryGauge10.7Number of job artifacts marked as synced due to the file missing on the primaryurl
geo_attachments_synced_missing_on_primaryGauge10.7Number of attachments marked as synced due to the file missing on the primaryurl
geo_repositories_checksummed_countGauge10.7Number of repositories checksummed on primaryurl
geo_repositories_checksum_failed_countGauge10.7Number of repositories failed to calculate the checksum on primaryurl
geo_wikis_checksummed_countGauge10.7Number of wikis checksummed on primaryurl
geo_wikis_checksum_failed_countGauge10.7Number of wikis failed to calculate the checksum on primaryurl
geo_repositories_verified_countGauge10.7Number of repositories verified on secondaryurl
geo_repositories_verification_failed_countGauge10.7Number of repositories failed to verify on secondaryurl
geo_repositories_checksum_mismatch_countGauge10.7Number of repositories that checksum mismatch on secondaryurl
geo_wikis_verified_countGauge10.7Number of wikis verified on secondaryurl
geo_wikis_verification_failed_countGauge10.7Number of wikis failed to verify on secondaryurl
geo_wikis_checksum_mismatch_countGauge10.7Number of wikis that checksum mismatch on secondaryurl
geo_repositories_checked_countGauge11.1Number of repositories that have been checked via git fsckurl
geo_repositories_checked_failed_countGauge11.1Number of repositories that have a failure from git fsckurl
geo_repositories_retrying_verification_countGauge11.2Number of repositories verification failures that Geo is actively trying to correct on secondaryurl
geo_wikis_retrying_verification_countGauge11.2Number of wikis verification failures that Geo is actively trying to correct on secondaryurl

Database load balancing metrics

The following metrics are available:

MetricTypeSinceDescription
db_load_balancing_hostsGauge12.3Current number of load balancing hosts

Ruby metrics

Some basic Ruby runtime metrics are available:

MetricTypeSinceDescription
ruby_gc_duration_secondsCounter11.1Time spent by Ruby in GC
ruby_gc_stat_...Gauge11.1Various metrics from GC.stat
ruby_file_descriptorsGauge11.1File descriptors per process
ruby_memory_bytesGauge11.1Memory usage by process
ruby_sampler_duration_secondsCounter11.1Time spent collecting stats
ruby_process_cpu_seconds_totalGauge12.0Total amount of CPU time per process
ruby_process_max_fdsGauge12.0Maximum number of open file descriptors per process
ruby_process_resident_memory_bytesGauge12.0Memory usage by process, measured in bytes
ruby_process_start_time_secondsGauge12.0UNIX timestamp of process start time

Unicorn Metrics

Unicorn specific metrics, when Unicorn is used.

MetricTypeSinceDescription
unicorn_active_connectionsGauge11.0The number of active Unicorn connections (workers)
unicorn_queued_connectionsGauge11.0The number of queued Unicorn connections
unicorn_workersGauge12.0The number of Unicorn workers

Puma Metrics (EXPERIMENTAL)

When Puma is used instead of Unicorn, the following metrics are available:

MetricTypeSinceDescription
puma_workersGauge12.0Total number of workers
puma_running_workersGauge12.0Number of booted workers
puma_stale_workersGauge12.0Number of old workers
puma_runningGauge12.0Number of running threads
puma_queued_connectionsGauge12.0Number of connections in that worker’s “todo” set waiting for a worker thread
puma_active_connectionsGauge12.0Number of threads processing a request
puma_pool_capacityGauge12.0Number of requests the worker is capable of taking right now
puma_max_threadsGauge12.0Maximum number of worker threads
puma_idle_threadsGauge12.0Number of spawned threads which are not processing a request
puma_killer_terminations_totalGauge12.0Number of workers terminated by PumaWorkerKiller

Metrics shared directory

GitLab’s Prometheus client requires a directory to store metrics data shared between multi-process services. Those files are shared among all instances running under Unicorn server. The directory needs to be accessible to all running Unicorn’s processes otherwise metrics will not function correctly.

For best performance its advisable that this directory will be located in tmpfs.

Its location is configured using environment variable prometheus_multiproc_dir.

If GitLab is installed using Omnibus and tmpfs is available then metrics directory will be automatically configured.

← Back to the main Prometheus page