Troubleshooting Service Ping

Set up and test Service Ping locally

To set up Service Ping locally, you must:

  1. Set up local repositories.
  2. Test local setup.
  3. Optional. Test Prometheus-based Service Ping.

Set up local repositories

  1. Clone and start GitLab.
  2. Clone and start Versions Application. Make sure you run docker-compose up to start a PostgreSQL and Redis instance.
  3. Point GitLab to the Versions Application endpoint instead of the default endpoint:
    1. Open service_ping/submit_service.rb locally and modify STAGING_BASE_URL.
    2. Set it to the local Versions Application URL: http://localhost:3000.

Test local setup

  1. Using the gitlab Rails console, manually trigger Service Ping: 

    GitlabServicePingWorker.new.perform('triggered_from_cron' => false)

  2. Use the versions Rails console to check the Service Ping was successfully received, parsed, and stored in the Versions database: 

    UsageData.last

Test Prometheus-based Service Ping

If the data submitted includes metrics queried from Prometheus you want to inspect and verify, you must:

  • Ensure that a Prometheus server is running locally.
  • Ensure the respective GitLab components are exporting metrics to the Prometheus server.

If you do not need to test data coming from Prometheus, no further action is necessary. Service Ping should degrade gracefully in the absence of a running Prometheus server.

Three kinds of components may export data to Prometheus, and are included in Service Ping:

  • node_exporter: Exports node metrics from the host machine.
  • gitlab-exporter: Exports process metrics from various GitLab components.
  • Other various GitLab services, such as Sidekiq and the Rails server, which export their own metrics.

Test with an Omnibus container

This is the recommended approach to test Prometheus-based Service Ping.

To verify your change, build a new Omnibus image from your code branch using CI/CD, download the image, and run a local container instance:

  1. From your merge request, select the qa stage, then trigger the e2e:package-and-test job. This job triggers an Omnibus build in a downstream pipeline of the omnibus-gitlab-mirror project.
  2. In the downstream pipeline, wait for the gitlab-docker job to finish.
  3. Open the job logs and locate the full container name including the version. It takes the following form: registry.gitlab.com/gitlab-org/build/omnibus-gitlab-mirror/gitlab-ee:<VERSION>.
  4. On your local machine, make sure you are signed in to the GitLab Docker registry. You can find the instructions for this in Authenticate to the GitLab container registry.
  5. Once signed in, download the new image by using docker pull registry.gitlab.com/gitlab-org/build/omnibus-gitlab-mirror/gitlab-ee:<VERSION>
  6. For more information about working with and running Omnibus GitLab containers in Docker, refer to GitLab Docker images documentation.

Test with GitLab development toolkits

This is the less recommended approach, because it comes with a number of difficulties when emulating a real GitLab deployment.

The GDK is not set up to run a Prometheus server or node_exporter alongside other GitLab components. If you would like to do so, Monitoring the GDK with Prometheus is a good start.

The GCK has limited support for testing Prometheus based Service Ping. By default, it comes with a fully configured Prometheus service that is set up to scrape a number of components. However, it has the following limitations:

  • It does not run a gitlab-exporter instance, so several process_* metrics from services such as Gitaly may be missing.
  • While it runs a node_exporter, docker-compose services emulate hosts, meaning that it usually reports itself as not associated with any of the other running services. That is not how node metrics are reported in a production setup, where node_exporter always runs as a process alongside other GitLab components on any given node. For Service Ping, none of the node data would therefore appear to be associated to any of the services running, because they all appear to be running on different hosts. To alleviate this problem, the node_exporter in GCK was arbitrarily “assigned” to the web service, meaning only for this service node_* metrics appears in Service Ping.

Generate Service Ping

Generate or get the cached Service Ping in rails console

Use the following method in the rails console. 

Gitlab::Usage::ServicePingReport.for(output: :all_metrics_values, cached: true)

Generate a fresh new Service Ping

Use the following method in the rails console.

This also refreshes the cached Service Ping displayed in the Admin Area. 

Gitlab::Usage::ServicePingReport.for(output: :all_metrics_values)

Generate and print

Generates Service Ping data in JSON format. 

gitlab-rake gitlab:usage_data:generate

Generates Service Ping data in YAML format: 

gitlab-rake gitlab:usage_data:dump_sql_in_yaml

Generate and send Service Ping

Prints the metrics saved in conversational_development_index_metrics. 

gitlab-rake gitlab:usage_data:generate_and_send