GitLab Prometheus metrics

To enable the GitLab Prometheus metrics:

  1. Log in to GitLab as a user with administrator access.
  2. On the top bar, select Main menu > Admin.
  3. On the left sidebar, select Settings > Metrics and profiling.
  4. Find the Metrics - Prometheus section, and select Add link to Prometheus.
  5. Restart GitLab for the changes to take effect.

For installations from source you must configure it yourself.

Collecting the metrics

GitLab monitors its own internal service metrics, and makes them available at the /-/metrics endpoint. Unlike other Prometheus exporters, to access the metrics, the client IP address must be explicitly allowed.

These metrics are enabled and collected for Omnibus GitLab and Chart installations. For source installations, these metrics must be enabled manually and collected by a Prometheus server.

For enabling and viewing metrics from Sidekiq nodes, see Sidekiq metrics.

Metrics available

The following metrics are available:

MetricTypeSinceDescriptionLabels
gitlab_banzai_cached_render_real_duration_secondsHistogram9.4Duration of rendering Markdown into HTML when cached output exists controller, action
gitlab_banzai_cacheless_render_real_duration_secondsHistogram9.4Duration of rendering Markdown into HTML when cached output does not exist controller, action
gitlab_cache_misses_totalCounter10.2Cache read miss controller, action
gitlab_cache_operation_duration_secondsHistogram10.2Cache access time 
gitlab_cache_operations_totalCounter12.2Cache operations by controller or action controller, action, operation
gitlab_ci_pipeline_builder_scoped_variables_durationHistogram14.5Time in seconds it takes to create the scoped variables for a CI/CD job 
gitlab_ci_pipeline_creation_duration_secondsHistogram13.0Time in seconds it takes to create a CI/CD pipeline 
gitlab_ci_pipeline_size_buildsHistogram13.1Total number of builds within a pipeline grouped by a pipeline sourcesource
gitlab_ci_runner_authentication_success_totalCounter15.2Total number of times that runner authentication has succeededtype
gitlab_ci_runner_authentication_failure_totalCounter15.2Total number of times that runner authentication has failed 
job_waiter_started_totalCounter12.9Number of batches of jobs started where a web request is waiting for the jobs to completeworker
job_waiter_timeouts_totalCounter12.9Number of batches of jobs that timed out where a web request is waiting for the jobs to completeworker
gitlab_ci_active_jobsHistogram14.2Count of active jobs when pipeline is created 
gitlab_database_transaction_secondsHistogram12.1Time spent in database transactions, in seconds 
gitlab_method_call_duration_secondsHistogram10.2Method calls real duration controller, action, module, method
gitlab_page_out_of_boundsCounter12.8Counter for the PageLimiter pagination limit being hit controller, action, bot
gitlab_rails_boot_time_secondsGauge14.8Time elapsed for Rails primary process to finish startup 
gitlab_rails_queue_duration_secondsHistogram9.4Measures latency between GitLab Workhorse forwarding a request to Rails 
gitlab_sql_duration_secondsHistogram10.2SQL execution time, excluding SCHEMA operations and BEGIN / COMMIT  
gitlab_sql_<role>_duration_secondsHistogram13.10SQL execution time, excluding SCHEMA operations and BEGIN / COMMIT, grouped by database roles (primary/replica) 
gitlab_ruby_threads_max_expected_threadsGauge13.3Maximum number of threads expected to be running and performing application work 
gitlab_ruby_threads_running_threadsGauge13.3Number of running Ruby threads by name 
gitlab_transaction_cache_<key>_count_totalCounter10.2Counter for total Rails cache calls (per key) 
gitlab_transaction_cache_<key>_duration_totalCounter10.2Counter for total time (seconds) spent in Rails cache calls (per key) 
gitlab_transaction_cache_count_totalCounter10.2Counter for total Rails cache calls (aggregate) 
gitlab_transaction_cache_duration_totalCounter10.2Counter for total time (seconds) spent in Rails cache calls (aggregate) 
gitlab_transaction_cache_read_hit_count_totalCounter10.2Counter for cache hits for Rails cache calls controller, action
gitlab_transaction_cache_read_miss_count_totalCounter10.2Counter for cache misses for Rails cache calls controller, action
gitlab_transaction_duration_secondsHistogram10.2Duration for successful requests (gitlab_transaction_* metrics) controller, action
gitlab_transaction_event_build_found_totalCounter9.4Counter for build found for API /jobs/request 
gitlab_transaction_event_build_invalid_totalCounter9.4Counter for build invalid due to concurrency conflict for API /jobs/request 
gitlab_transaction_event_build_not_found_cached_totalCounter9.4Counter for cached response of build not found for API /jobs/request 
gitlab_transaction_event_build_not_found_totalCounter9.4Counter for build not found for API /jobs/request 
gitlab_transaction_event_change_default_branch_totalCounter9.4Counter when default branch is changed for any repository 
gitlab_transaction_event_create_repository_totalCounter9.4Counter when any repository is created 
gitlab_transaction_event_etag_caching_cache_hit_totalCounter9.4Counter for ETag cache hit.endpoint
gitlab_transaction_event_etag_caching_header_missing_totalCounter9.4Counter for ETag cache miss - header missingendpoint
gitlab_transaction_event_etag_caching_key_not_found_totalCounter9.4Counter for ETag cache miss - key not foundendpoint
gitlab_transaction_event_etag_caching_middleware_used_totalCounter9.4Counter for ETag middleware accessedendpoint
gitlab_transaction_event_etag_caching_resource_changed_totalCounter9.4Counter for ETag cache miss - resource changedendpoint
gitlab_transaction_event_fork_repository_totalCounter9.4Counter for repository forks (RepositoryForkWorker). Only incremented when source repository exists 
gitlab_transaction_event_import_repository_totalCounter9.4Counter for repository imports (RepositoryImportWorker) 
gitlab_transaction_event_patch_hard_limit_bytes_hit_totalCounter13.9Counter for diff patch size limit hits 
gitlab_transaction_event_push_branch_totalCounter9.4Counter for all branch pushes 
gitlab_transaction_event_rails_exception_totalCounter9.4Counter for number of rails exceptions 
gitlab_transaction_event_receive_email_totalCounter9.4Counter for received emailshandler
gitlab_transaction_event_remote_mirrors_failed_totalCounter10.8Counter for failed remote mirrors 
gitlab_transaction_event_remote_mirrors_finished_totalCounter10.8Counter for finished remote mirrors 
gitlab_transaction_event_remote_mirrors_running_totalCounter10.8Counter for running remote mirrors 
gitlab_transaction_event_remove_branch_totalCounter9.4Counter when a branch is removed for any repository 
gitlab_transaction_event_remove_repository_totalCounter9.4Counter when a repository is removed 
gitlab_transaction_event_remove_tag_totalCounter9.4Counter when a tag is remove for any repository 
gitlab_transaction_event_sidekiq_exception_totalCounter9.4Counter of Sidekiq exceptions 
gitlab_transaction_event_stuck_import_jobs_totalCounter9.4Count of stuck import jobs projects_without_jid_count, projects_with_jid_count
gitlab_transaction_event_update_build_totalCounter9.4Counter for update build for API /jobs/request/:id  
gitlab_transaction_new_redis_connections_totalCounter9.4Counter for new Redis connections 
gitlab_transaction_rails_queue_duration_totalCounter9.4Measures latency between GitLab Workhorse forwarding a request to Rails controller, action
gitlab_transaction_view_duration_totalCounter9.4Duration for views controller, action, view
gitlab_view_rendering_duration_secondsHistogram10.2Duration for views (histogram) controller, action, view
http_requests_totalCounter9.4Rack request count method, status
http_request_duration_secondsHistogram9.4HTTP response time from rack middleware for successful requestsmethod
gitlab_transaction_db_count_totalCounter13.1Counter for total number of SQL calls controller, action
gitlab_transaction_db_<role>_count_totalCounter13.10Counter for total number of SQL calls, grouped by database roles (primary/replica) controller, action
gitlab_transaction_db_write_count_totalCounter13.1Counter for total number of write SQL calls controller, action
gitlab_transaction_db_cached_count_totalCounter13.1Counter for total number of cached SQL calls controller, action
gitlab_transaction_db_<role>_cached_count_totalCounter13.1Counter for total number of cached SQL calls, grouped by database roles (primary/replica) controller, action
gitlab_transaction_db_<role>_wal_count_totalCounter14.0Counter for total number of WAL (write ahead log location) queries, grouped by database roles (primary/replica) controller, action
gitlab_transaction_db_<role>_wal_cached_count_totalCounter14.1Counter for total number of cached WAL (write ahead log location) queries, grouped by database roles (primary/replica) controller, action
http_elasticsearch_requests_duration_seconds Histogram13.1Elasticsearch requests duration during web transactions controller, action
http_elasticsearch_requests_total Counter13.1Elasticsearch requests count during web transactions controller, action
pipelines_created_totalCounter9.4Counter of pipelines created 
rack_uncaught_errors_totalCounter9.4Rack connections handling uncaught errors count 
user_session_logins_totalCounter9.4Counter of how many users have logged in since GitLab was started or restarted 
upload_file_does_not_existCounter10.7Number of times an upload record could not find its file. 
failed_login_captcha_totalGauge11.0Counter of failed CAPTCHA attempts during login 
successful_login_captcha_totalGauge11.0Counter of successful CAPTCHA attempts during login 
auto_devops_pipelines_completed_totalCounter12.7Counter of completed Auto DevOps pipelines, labeled by status 
artifact_report_<report_type>_builds_completed_totalCounter15.3Counter of completed CI Builds with report-type artifacts, grouped by report type and labeled by status 
gitlab_metrics_dashboard_processing_time_msSummary12.10Metrics dashboard processing time in millisecondsservice, stages
action_cable_active_connectionsGauge13.4Number of ActionCable WS clients currently connectedserver_mode
action_cable_broadcasts_totalCounter13.10The number of ActionCable broadcasts emittedserver_mode
action_cable_pool_min_sizeGauge13.4Minimum number of worker threads in ActionCable thread poolserver_mode
action_cable_pool_max_sizeGauge13.4Maximum number of worker threads in ActionCable thread poolserver_mode
action_cable_pool_current_sizeGauge13.4Current number of worker threads in ActionCable thread poolserver_mode
action_cable_pool_largest_sizeGauge13.4Largest number of worker threads observed so far in ActionCable thread poolserver_mode
action_cable_pool_pending_tasksGauge13.4Number of tasks waiting to be executed in ActionCable thread poolserver_mode
action_cable_pool_tasks_totalGauge13.4Total number of tasks executed in ActionCable thread poolserver_mode
gitlab_ci_trace_operations_totalCounter13.4Total amount of different operations on a build traceoperation
gitlab_ci_trace_bytes_totalCounter13.4Total amount of build trace bytes transferred 
action_cable_single_client_transmissions_totalCounter13.10The number of ActionCable messages transmitted to any client in any channelserver_mode
action_cable_subscription_confirmations_totalCounter13.10The number of ActionCable subscriptions from clients confirmedserver_mode
action_cable_subscription_rejections_totalCounter13.10The number of ActionCable subscriptions from clients rejectedserver_mode
action_cable_transmitted_bytesHistogram14.1Message size, in bytes, transmitted over action cable operation, channel
gitlab_issuable_fast_count_by_state_totalCounter13.5Total number of row count operations on issue/merge request list pages 
gitlab_issuable_fast_count_by_state_failures_totalCounter13.5Number of soft-failed row count operations on issue/merge request list pages 
gitlab_ci_trace_finalize_duration_secondsHistogram13.6Duration of build trace chunks migration to object storage 
gitlab_external_http_totalCounter13.8Total number of HTTP calls to external systems controller, action
gitlab_external_http_duration_secondsCounter13.8Duration in seconds spent on each HTTP call to external systems 
gitlab_external_http_exception_totalCounter13.8Total number of exceptions raised when making external HTTP calls 
ci_report_parser_duration_secondsHistogram13.9Time to parse CI/CD report artifactsparser
pipeline_graph_link_calculation_duration_secondsHistogram13.9Total time spent calculating links, in seconds 
pipeline_graph_links_totalHistogram13.9Number of links per graph 
pipeline_graph_links_per_job_ratioHistogram13.9Ratio of links to job per graph 
gitlab_ci_pipeline_security_orchestration_policy_processing_duration_secondsHistogram13.12Time in seconds it takes to process Security Policies in CI/CD pipeline 
gitlab_spamcheck_request_duration_secondsHistogram13.12The duration for requests between Rails and the anti-spam engine 
service_desk_thank_you_emailCounter14.0Total number of email responses to new Service Desk emails 
service_desk_new_note_emailCounter14.0Total number of email notifications on new Service Desk comment 
email_receiver_errorCounter14.1Total number of errors when processing incoming emails 
gitlab_snowplow_events_totalCounter14.1Total number of GitLab Snowplow product intelligence events emitted 
gitlab_snowplow_failed_events_totalCounter14.1Total number of GitLab Snowplow product intelligence events emission failures 
gitlab_snowplow_successful_events_totalCounter14.1Total number of GitLab Snowplow product intelligence events emission successes 
gitlab_ci_build_trace_errors_totalCounter14.4Total amount of different error types on a build traceerror_reason
gitlab_presentable_object_cacheless_render_real_duration_secondsHistogram15.3Duration of real time spent caching and representing specific web request objects controller, action
cached_object_operations_totalCounter15.3Total number of objects cached for specific web requests controller, action

Metrics controlled by a feature flag

The following metrics can be controlled by feature flags:

MetricFeature Flag
gitlab_method_call_duration_secondsprometheus_metrics_method_instrumentation
gitlab_view_rendering_duration_secondsprometheus_metrics_view_instrumentation

Praefect metrics

You can configure Praefect to report metrics. For information on available metrics, see the relevant documentation.

Sidekiq metrics

Sidekiq jobs may also gather metrics, and these metrics can be accessed if the Sidekiq exporter is enabled: for example, using the monitoring.sidekiq_exporter configuration option in gitlab.yml. These metrics are served from the /metrics path on the configured port.

MetricTypeSinceDescriptionLabels  
sidekiq_jobs_cpu_secondsHistogram12.4Seconds of CPU time to run Sidekiq job queue, boundary, external_dependencies, feature_category, job_status, urgency   
sidekiq_jobs_completion_secondsHistogram12.2Seconds to complete Sidekiq job queue, boundary, external_dependencies, feature_category, job_status, urgency   
sidekiq_jobs_db_secondsHistogram12.9Seconds of DB time to run Sidekiq job queue, boundary, external_dependencies, feature_category, job_status, urgency   
sidekiq_jobs_gitaly_secondsHistogram12.9Seconds of Gitaly time to run Sidekiq job queue, boundary, external_dependencies, feature_category, job_status, urgency   
sidekiq_redis_requests_duration_secondsHistogram13.1Duration in seconds that a Sidekiq job spent querying a Redis server queue, boundary, external_dependencies, feature_category, job_status, urgency   
sidekiq_elasticsearch_requests_duration_secondsHistogram13.1Duration in seconds that a Sidekiq job spent in requests to an Elasticsearch server queue, boundary, external_dependencies, feature_category, job_status, urgency   
sidekiq_jobs_queue_duration_secondsHistogram12.5Duration in seconds that a Sidekiq job was queued before being executed queue, boundary, external_dependencies, feature_category, urgency   
sidekiq_jobs_failed_totalCounter12.2Sidekiq jobs failed queue, boundary, external_dependencies, feature_category, urgency   
sidekiq_jobs_retried_totalCounter12.2Sidekiq jobs retried queue, boundary, external_dependencies, feature_category, urgency   
sidekiq_jobs_interrupted_totalCounter15.2Sidekiq jobs interrupted queue, boundary, external_dependencies, feature_category, urgency   
sidekiq_jobs_dead_totalCounter13.7Sidekiq dead jobs (jobs that have run out of retries) queue, boundary, external_dependencies, feature_category, urgency   
sidekiq_redis_requests_totalCounter13.1Redis requests during a Sidekiq job execution queue, boundary, external_dependencies, feature_category, job_status, urgency   
sidekiq_elasticsearch_requests_totalCounter13.1Elasticsearch requests during a Sidekiq job execution queue, boundary, external_dependencies, feature_category, job_status, urgency   
sidekiq_running_jobsGauge12.2Number of Sidekiq jobs running queue, boundary, external_dependencies, feature_category, urgency   
sidekiq_concurrencyGauge12.5Maximum number of Sidekiq jobs   
sidekiq_mem_total_bytesGauge15.3Number of bytes allocated for both objects consuming an object slot and objects that required a malloc’   
geo_db_replication_lag_secondsGauge10.2Database replication lag (seconds)url  
geo_repositoriesGauge10.2Total number of repositories available on primaryurl  
geo_repositories_syncedGauge10.2Number of repositories synced on secondaryurl  
geo_repositories_failedGauge10.2Number of repositories failed to sync on secondaryurl  
geo_lfs_objectsGauge10.2Number of LFS objects on primaryurl  
geo_lfs_objects_checksummedGauge14.6Number of LFS objects checksummed successfully on primaryurl  
geo_lfs_objects_checksum_failedGauge14.6Number of LFS objects failed to calculate the checksum on primaryurl  
geo_lfs_objects_checksum_totalGauge14.6Number of LFS objects tried to checksum on primaryurl  
geo_lfs_objects_syncedGauge10.2Number of syncable LFS objects synced on secondaryurl  
geo_lfs_objects_failedGauge10.2Number of syncable LFS objects failed to sync on secondaryurl  
geo_lfs_objects_registryGauge14.6Number of LFS objects in the registryurl  
geo_lfs_objects_verifiedGauge14.6Number of LFS objects verified on secondaryurl  
geo_lfs_objects_verification_failedGauge14.6Number of LFS objects’ verifications failed on secondaryurl  
geo_lfs_objects_verification_totalGauge14.6Number of LFS objects’ verifications tried on secondaryurlLFS objects failed to sync on secondaryurl
geo_attachmentsGauge10.2Total number of file attachments available on primaryurl  
geo_attachments_syncedGauge10.2Number of attachments synced on secondaryurl  
geo_attachments_failedGauge10.2Number of attachments failed to sync on secondaryurl  
geo_last_event_idGauge10.2Database ID of the latest event log entry on the primaryurl  
geo_last_event_timestampGauge10.2UNIX timestamp of the latest event log entry on the primaryurl  
geo_cursor_last_event_idGauge10.2Last database ID of the event log processed by the secondaryurl  
geo_cursor_last_event_timestampGauge10.2Last UNIX timestamp of the event log processed by the secondaryurl  
geo_status_failed_totalCounter10.2Number of times retrieving the status from the Geo Node failedurl  
geo_last_successful_status_check_timestampGauge10.2Last timestamp when the status was successfully updatedurl  
geo_job_artifacts_synced_missing_on_primaryGauge10.7Number of job artifacts marked as synced due to the file missing on the primaryurl  
geo_repositories_checksummedGauge10.7Number of repositories checksummed on primaryurl  
geo_repositories_checksum_failedGauge10.7Number of repositories failed to calculate the checksum on primaryurl  
geo_wikis_checksummedGauge10.7Number of wikis checksummed on primaryurl  
geo_wikis_checksum_failedGauge10.7Number of wikis failed to calculate the checksum on primaryurl  
geo_repositories_verifiedGauge10.7Number of repositories verified on secondaryurl  
geo_repositories_verification_failedGauge10.7Number of repositories failed to verify on secondaryurl  
geo_repositories_checksum_mismatchGauge10.7Number of repositories that checksum mismatch on secondaryurl  
geo_wikis_verifiedGauge10.7Number of wikis verified on secondaryurl  
geo_wikis_verification_failedGauge10.7Number of wikis failed to verify on secondaryurl  
geo_wikis_checksum_mismatchGauge10.7Number of wikis that checksum mismatch on secondaryurl  
geo_repositories_checkedGauge11.1Number of repositories that have been checked via git fsck url  
geo_repositories_checked_failedGauge11.1Number of repositories that have a failure from git fsck url  
geo_repositories_retrying_verificationGauge11.2Number of repositories verification failures that Geo is actively trying to correct on secondaryurl  
geo_wikis_retrying_verificationGauge11.2Number of wikis verification failures that Geo is actively trying to correct on secondaryurl  
geo_package_filesGauge13.0Number of package files on primaryurl  
geo_package_files_checksummedGauge13.0Number of package files checksummed on primaryurl  
geo_package_files_checksum_failedGauge13.0Number of package files failed to calculate the checksum on primaryurl  
geo_package_files_syncedGauge13.3Number of syncable package files synced on secondaryurl  
geo_package_files_failedGauge13.3Number of syncable package files failed to sync on secondaryurl  
geo_package_files_registryGauge13.3Number of package files in the registryurl  
geo_terraform_state_versionsGauge13.5Number of terraform state versions on primaryurl  
geo_terraform_state_versions_checksummedGauge13.5Number of terraform state versions checksummed successfully on primaryurl  
geo_terraform_state_versions_checksum_failedGauge13.5Number of terraform state versions failed to calculate the checksum on primaryurl  
geo_terraform_state_versions_checksum_totalGauge13.12Number of terraform state versions tried to checksum on primaryurl  
geo_terraform_state_versions_syncedGauge13.5Number of syncable terraform state versions synced on secondaryurl  
geo_terraform_state_versions_failedGauge13.5Number of syncable terraform state versions failed to sync on secondaryurl  
geo_terraform_state_versions_registryGauge13.5Number of terraform state versions in the registryurl  
geo_terraform_state_versions_verifiedGauge13.12Number of terraform state versions verified on secondaryurl  
geo_terraform_state_versions_verification_failedGauge13.12Number of terraform state versions verifications failed on secondaryurl  
geo_terraform_state_versions_verification_totalGauge13.12Number of terraform state versions verifications tried on secondaryurl  
global_search_bulk_cron_queue_sizeGauge12.10Number of database records waiting to be synchronized to Elasticsearch   
global_search_awaiting_indexing_queue_sizeGauge13.2Number of database updates waiting to be synchronized to Elasticsearch while indexing is paused   
geo_merge_request_diffsGauge13.4Number of merge request diffs on primaryurl  
geo_merge_request_diffs_checksum_totalGauge13.12Number of merge request diffs tried to checksum on primaryurl  
geo_merge_request_diffs_checksummedGauge13.4Number of merge request diffs successfully checksummed on primaryurl  
geo_merge_request_diffs_checksum_failedGauge13.4Number of merge request diffs failed to calculate the checksum on primaryurl  
geo_merge_request_diffs_syncedGauge13.4Number of syncable merge request diffs synced on secondaryurl  
geo_merge_request_diffs_failedGauge13.4Number of syncable merge request diffs failed to sync on secondaryurl  
geo_merge_request_diffs_registryGauge13.4Number of merge request diffs in the registryurl  
geo_merge_request_diffs_verification_totalGauge13.12Number of merge request diffs verifications tried on secondaryurl  
geo_merge_request_diffs_verifiedGauge13.12Number of merge request diffs verified on secondaryurl  
geo_merge_request_diffs_verification_failedGauge13.12Number of merge request diffs verifications failed on secondaryurl  
geo_snippet_repositoriesGauge13.4Number of snippets on primaryurl  
geo_snippet_repositories_checksummedGauge13.4Number of snippets checksummed on primaryurl  
geo_snippet_repositories_checksum_failedGauge13.4Number of snippets failed to calculate the checksum on primaryurl  
geo_snippet_repositories_syncedGauge13.4Number of syncable snippets synced on secondaryurl  
geo_snippet_repositories_failedGauge13.4Number of syncable snippets failed on secondaryurl  
geo_snippet_repositories_registryGauge13.4Number of syncable snippets in the registryurl  
geo_group_wiki_repositoriesGauge13.10Number of group wikis on primaryurl  
geo_group_wiki_repositories_checksummedGauge13.10Number of group wikis checksummed on primaryurl  
geo_group_wiki_repositories_checksum_failedGauge13.10Number of group wikis failed to calculate the checksum on primaryurl  
geo_group_wiki_repositories_syncedGauge13.10Number of syncable group wikis synced on secondaryurl  
geo_group_wiki_repositories_failedGauge13.10Number of syncable group wikis failed on secondaryurl  
geo_group_wiki_repositories_registryGauge13.10Number of syncable group wikis in the registryurl  
geo_pages_deploymentsGauge14.3Number of pages deployments on primaryurl  
geo_pages_deployments_checksum_totalGauge14.6Number of pages deployments tried to checksum on primaryurl  
geo_pages_deployments_checksummedGauge14.6Number of pages deployments successfully checksummed on primaryurl  
geo_pages_deployments_checksum_failedGauge14.6Number of pages deployments failed to calculate the checksum on primaryurl  
geo_pages_deployments_syncedGauge14.3Number of syncable pages deployments synced on secondaryurl  
geo_pages_deployments_failedGauge14.3Number of syncable pages deployments failed to sync on secondaryurl  
geo_pages_deployments_registryGauge14.3Number of pages deployments in the registryurl  
geo_pages_deployments_verification_totalGauge14.6Number of pages deployments verifications tried on secondaryurl  
geo_pages_deployments_verifiedGauge14.6Number of pages deployments verified on secondaryurl  
geo_pages_deployments_verification_failedGauge14.6Number of pages deployments verifications failed on secondaryurl  
geo_job_artifactsGauge14.8Number of job artifacts on primaryurl  
geo_job_artifacts_checksum_totalGauge14.8Number of job artifacts tried to checksum on primaryurl  
geo_job_artifacts_checksummedGauge14.8Number of job artifacts successfully checksummed on primaryurl  
geo_job_artifacts_checksum_failedGauge14.8Number of job artifacts failed to calculate the checksum on primaryurl  
geo_job_artifacts_syncedGauge14.8Number of syncable job artifacts synced on secondaryurl  
geo_job_artifacts_failedGauge14.8Number of syncable job artifacts failed to sync on secondaryurl  
geo_job_artifacts_registryGauge14.8Number of job artifacts in the registryurl  
geo_job_artifacts_verification_totalGauge14.8Number of job artifacts verifications tried on secondaryurl  
geo_job_artifacts_verifiedGauge14.8Number of job artifacts verified on secondaryurl  
geo_job_artifacts_verification_failedGauge14.8Number of job artifacts verifications failed on secondaryurl  
limited_capacity_worker_running_jobsGauge13.5Number of running jobsworker  
limited_capacity_worker_max_running_jobsGauge13.5Maximum number of running jobsworker  
limited_capacity_worker_remaining_work_countGauge13.5Number of jobs waiting to be enqueuedworker  
destroyed_job_artifacts_count_totalCounter13.6Number of destroyed expired job artifacts   
destroyed_pipeline_artifacts_count_totalCounter13.8Number of destroyed expired pipeline artifacts   
gitlab_optimistic_locking_retriesHistogram13.10Number of retry attempts to execute optimistic retry lock   
geo_uploadsGauge14.1Number of uploads on primaryurl  
geo_uploads_syncedGauge14.1Number of uploads synced on secondaryurl  
geo_uploads_failedGauge14.1Number of syncable uploads failed to sync on secondaryurl  
geo_uploads_registryGauge14.1Number of uploads in the registryurl  
geo_uploads_checksum_totalGauge14.6Number of uploads tried to checksum on primaryurl  
geo_uploads_checksummedGauge14.6Number of uploads successfully checksummed on primaryurl  
geo_uploads_checksum_failedGauge14.6Number of uploads failed to calculate the checksum on primaryurl  
geo_uploads_verification_totalGauge14.6Number of uploads verifications tried on secondaryurl  
geo_uploads_verifiedGauge14.6Number of uploads verified on secondaryurl  
geo_uploads_verification_failedGauge14.6Number of uploads verifications failed on secondaryurl  
gitlab_sli:rails_request_apdex:totalCounter14.4The number of request-apdex measurements, more information the development documentation endpoint_id, feature_category, request_urgency   
gitlab_sli:rails_request_apdex:success_totalCounter14.4The number of successful requests that met the target duration for their urgency. Divide by gitlab_sli:rails_requests_apdex:total to get a success ratio endpoint_id, feature_category, request_urgency   
geo_ci_secure_filesGauge15.3Number of secure files on primaryurl  
geo_ci_secure_files_checksum_totalGauge15.3Number of secure files tried to checksum on primaryurl  
geo_ci_secure_files_checksummedGauge15.3Number of secure files successfully checksummed on primaryurl  
geo_ci_secure_files_checksum_failedGauge15.3Number of secure files failed to calculate the checksum on primaryurl  
geo_ci_secure_files_syncedGauge15.3Number of syncable secure files synced on secondaryurl  
geo_ci_secure_files_failedGauge15.3Number of syncable secure files failed to sync on secondaryurl  
geo_ci_secure_files_registryGauge15.3Number of secure files in the registryurl  
geo_ci_secure_files_verification_totalGauge15.3Number of secure files verifications tried on secondaryurl  
geo_ci_secure_files_verifiedGauge15.3Number of secure files verified on secondaryurl  
geo_ci_secure_files_verification_failedGauge15.3Number of secure files verifications failed on secondaryurl  

Database load balancing metrics

The following metrics are available:

MetricTypeSinceDescriptionLabels
db_load_balancing_hostsGauge12.3Current number of load balancing hosts 
sidekiq_load_balancing_countCounter13.11Sidekiq jobs using load balancing with data consistency set to :sticky or :delayed queue, boundary, external_dependencies, feature_category, job_status, urgency, data_consistency, load_balancing_strategy
gitlab_transaction_caught_up_replica_pick_count_totalCounter14.1Number of search attempts for caught up replicaresult

Database partitioning metrics

The following metrics are available:

MetricTypeSinceDescription
db_partitions_presentGauge13.4Number of database partitions present
db_partitions_missingGauge13.4Number of database partitions currently expected, but not present

Connection pool metrics

These metrics record the status of the database connection pools, and the metrics all have these labels:

  • class - the Ruby class being recorded.
    • ActiveRecord::Base is the main database connection.
    • Geo::TrackingBase is the connection to the Geo tracking database, if enabled.
  • host - the host name used to connect to the database.
  • port - the port used to connect to the database.
MetricTypeSinceDescription
gitlab_database_connection_pool_sizeGauge13.0Total connection pool capacity
gitlab_database_connection_pool_connectionsGauge13.0Current connections in the pool
gitlab_database_connection_pool_busyGauge13.0Connections in use where the owner is still alive
gitlab_database_connection_pool_deadGauge13.0Connections in use where the owner is not alive
gitlab_database_connection_pool_idleGauge13.0Connections not in use
gitlab_database_connection_pool_waitingGauge13.0Threads currently waiting on this queue

Ruby metrics

Some basic Ruby runtime metrics are available:

MetricTypeSinceDescription
ruby_gc_duration_secondsCounter11.1Time spent by Ruby in GC
ruby_gc_stat_...Gauge11.1Various metrics from GC.stat
ruby_gc_stat_ext_heap_fragmentationGauge15.2Degree of Ruby heap fragmentation as live objects versus eden slots (range 0 to 1)
ruby_file_descriptorsGauge11.1File descriptors per process
ruby_sampler_duration_secondsCounter11.1Time spent collecting stats
ruby_process_cpu_seconds_totalGauge12.0Total amount of CPU time per process
ruby_process_max_fdsGauge12.0Maximum number of open file descriptors per process
ruby_process_resident_memory_bytesGauge12.0Memory usage by process (RSS/Resident Set Size)
ruby_process_unique_memory_bytesGauge13.0Memory usage by process (USS/Unique Set Size)
ruby_process_proportional_memory_bytesGauge13.0Memory usage by process (PSS/Proportional Set Size)
ruby_process_start_time_secondsGauge12.0UNIX timestamp of process start time

Puma Metrics

MetricTypeSinceDescription
puma_workersGauge12.0Total number of workers
puma_running_workersGauge12.0Number of booted workers
puma_stale_workersGauge12.0Number of old workers
puma_runningGauge12.0Number of running threads
puma_queued_connectionsGauge12.0Number of connections in that worker’s “to do” set waiting for a worker thread
puma_active_connectionsGauge12.0Number of threads processing a request
puma_pool_capacityGauge12.0Number of requests the worker is capable of taking right now
puma_max_threadsGauge12.0Maximum number of worker threads
puma_idle_threadsGauge12.0Number of spawned threads which are not processing a request
puma_killer_terminations_totalGauge12.0Number of workers terminated by PumaWorkerKiller

Redis metrics

These client metrics are meant to complement Redis server metrics. These metrics are broken down per Redis instance. These metrics all have a storage label which indicates the Redis instance (cache, shared_state, and so on).

MetricTypeSinceDescription
gitlab_redis_client_exceptions_totalCounter13.2Number of Redis client exceptions, broken down by exception class
gitlab_redis_client_requests_totalCounter13.2Number of Redis client requests
gitlab_redis_client_requests_duration_secondsHistogram13.2Redis request latency, excluding blocking commands

Metrics shared directory

The GitLab Prometheus client requires a directory to store metrics data shared between multi-process services. Those files are shared among all instances running under Puma server. The directory must be accessible to all running Puma’s processes, or metrics can’t function correctly.

This directory’s location is configured using environment variable prometheus_multiproc_dir. For best performance, create this directory in tmpfs.

If GitLab is installed using Omnibus GitLab and tmpfs is available, then GitLab configures the metrics directory for you.