Service Ping Guide

Version history
  • Introduced in GitLab Ultimate 11.2, more statistics.
  • In GitLab 14.1, renamed from Usage Ping to Service Ping. In 14.0 and earlier, use the Usage Ping documentation for the Rails commands appropriate to your version.

Service Ping is a GitLab process that collects and sends a weekly payload to GitLab. The payload provides important high-level data that helps our product, support, and sales teams understand how GitLab is used. The data helps to:

  • Compare counts month over month (or week over week) to get a rough sense for how an instance uses different product features.
  • Collect other facts that help us classify and understand GitLab installations.
  • Calculate our stage monthly active users (SMAU), which helps to measure the success of our stages and features.

Service Ping information is not anonymous. It’s linked to the instance’s hostname, but does not contain project names, usernames, or any other specific data.

Sending a Service Ping payload is optional and you can disable it on any self-managed instance. When Service Ping is enabled, GitLab gathers data from the other instances and can show your instance’s usage statistics to your users.

Service Ping terminology

We use the following terminology to describe the Service Ping components:

  • Service Ping: the process that collects and generates a JSON payload.
  • Service Data: the contents of the Service Ping JSON payload. This includes metrics.
  • Metrics: primarily made up of row counts for different tables in an instance’s database. Each metric has a corresponding metric definition in a YAML file.
  • MAU: monthly active users.
  • WAU: weekly active users.

Limitations

  • Service Ping does not track frontend events things like page views, link clicks, or user sessions.
  • Service Ping focuses only on aggregated backend events.

Because of these limitations we recommend you:

  • Instrument your products with Snowplow for more detailed analytics on GitLab.com.
  • Use Service Ping to track aggregated backend events on self-managed instances.

Service Ping request flow

The following example shows a basic request/response flow between a GitLab instance, the Versions Application, the License Application, Salesforce, the GitLab S3 Bucket, the GitLab Snowflake Data Warehouse, and Sisense:

sequenceDiagram participant GitLab Instance participant Versions Application participant Licenses Application participant Salesforce participant S3 Bucket participant Snowflake DW participant Sisense Dashboards GitLab Instance->>Versions Application: Send Service Ping loop Process usage data Versions Application->>Versions Application: Parse usage data Versions Application->>Versions Application: Write to database Versions Application->>Versions Application: Update license ping time end loop Process data for Salesforce Versions Application-xLicenses Application: Request Zuora subscription id Licenses Application-xVersions Application: Zuora subscription id Versions Application-xSalesforce: Request Zuora account id by Zuora subscription id Salesforce-xVersions Application: Zuora account id Versions Application-xSalesforce: Usage data for the Zuora account end Versions Application->>S3 Bucket: Export Versions database S3 Bucket->>Snowflake DW: Import data Snowflake DW->>Snowflake DW: Transform data using dbt Snowflake DW->>Sisense Dashboards: Data available for querying Versions Application->>GitLab Instance: DevOps Score (Conversational Development Index)

How Service Ping works

  1. The Service Ping cron job is set in Sidekiq to run weekly.
  2. When the cron job runs, it calls Gitlab::Usage::ServicePingReport.for(output: :all_metrics_values).
  3. Gitlab::Usage::ServicePingReport.for(output: :all_metrics_values) cascades down to ~400+ other counter method calls.
  4. The response of all methods calls are merged together into a single JSON payload.
  5. The JSON payload is then posted to the Versions application If a firewall exception is needed, the required URL depends on several things. If the hostname is version.gitlab.com, the protocol is TCP, and the port number is 443, the required URL is https://version.gitlab.com/.
  6. In case of an error, it will be reported to the Version application along with following pieces of information:

    • uuid - GitLab instance unique identifier
    • hostname - GitLab instance hostname
    • version - GitLab instance current versions
    • elapsed - Amount of time which passed since Service Ping report process started and moment of error occurrence
    • message - Error message
     
     {
       "uuid"=>"02333324-1cd7-4c3b-a45b-a4993f05fb1d",
       "hostname"=>"127.0.0.1",
       "version"=>"14.7.0-pre",
       "elapsed"=>0.006946,
       "message"=>'PG::UndefinedColumn: ERROR:  column \"non_existent_attribute\" does not exist\nLINE 1: SELECT COUNT(non_existent_attribute) FROM \"issues\" /*applica...'
     }
     
     
  7. Finally, the timing metadata information that is used for diagnostic purposes is submitted to the Versions application. It consists of a list of metric identifiers and the time it took to calculate the metrics:

    Introduced in GitLab 15.0 [with a flag(../../user/feature_flags.md), enabled by default.

On self-managed GitLab, by default this feature is available. To hide the feature, ask an administrator to disable the feature flag named measure_service_ping_metric_collection. On GitLab.com, this feature is available.
    {"metadata"=>
      {"metrics"=>
        [{"name"=>"version", "time_elapsed"=>1.1811964213848114e-05},
         {"name"=>"installation_type", "time_elapsed"=>0.00017242692410945892},
         {"name"=>"license_billable_users", "time_elapsed"=>0.009520471096038818},
         ....
         {"name"=>"counts.clusters_platforms_eks",
          "time_elapsed"=>0.05638605775311589},
         {"name"=>"counts.clusters_platforms_gke",
          "time_elapsed"=>0.40995341585949063},
         {"name"=>"counts.clusters_platforms_user",
          "time_elapsed"=>0.06410990096628666},
         {"name"=>"counts.clusters_management_project",
          "time_elapsed"=>0.24020783510059118},
         {"name"=>"counts.clusters_integrations_elastic_stack",
          "time_elapsed"=>0.03484998410567641}
        ]
      }
    }
    ```

### On a Geo secondary site

We also collect metrics specific to [Geo](../../administration/geo/index.md) secondary sites to send with Service Ping.

1. The [Geo secondary service ping cron job](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/workers/geo/secondary_usage_data_cron_worker.rb) is set in Sidekiq to run weekly.
1. When the cron job runs, it calls [`SecondaryUsageData.update_metrics!`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/models/geo/secondary_usage_data.rb#L33). This collects the relevant metrics from Prometheus and stores the data in the Geo secondary tracking database for transmission to the primary site during a [Geo node status update](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/models/geo_node_status.rb#L105).
1. Geo node status data is sent with the JSON payload in the process described above. The following is an example of the payload where each object in the array represents a Geo node:

   ```json
   [
     {
       "repository_verification_enabled"=>true,
       "repositories_replication_enabled"=>true,
       "repositories_synced_count"=>24,
       "repositories_failed_count"=>0,
       "git_fetch_event_count_weekly"=>nil,
       "git_push_event_count_weekly"=>nil,
       ... other geo node status fields
     }
   ]

Enable or disable service ping metadata reporting

Service Ping timing metadata reporting is under development but ready for production use. It is deployed behind a feature flag that is enabled by default. GitLab administrators with access to the GitLab Rails console can opt to disable it.

To enable it:

Feature.enable(:measure_service_ping_metric_collection)

To disable it:

Feature.disable(:measure_service_ping_metric_collection)

Implementing Service Ping

See the implement Service Ping guide.

Example Service Ping payload

The following is example content of the Service Ping payload.

{
  "uuid": "0000000-0000-0000-0000-000000000000",
  "hostname": "example.com",
  "version": "12.10.0-pre",
  "installation_type": "omnibus-gitlab",
  "active_user_count": 999,
  "recorded_at": "2020-04-17T07:43:54.162+00:00",
  "edition": "EEU",
  "license_md5": "00000000000000000000000000000000",
  "license_id": null,
  "historical_max_users": 999,
  "licensee": {
    "Name": "ABC, Inc.",
    "Email": "email@example.com",
    "Company": "ABC, Inc."
  },
  "license_user_count": 999,
  "license_starts_at": "2020-01-01",
  "license_expires_at": "2021-01-01",
  "license_plan": "ultimate",
  "license_add_ons": {
  },
  "license_trial": false,
  "counts": {
    "assignee_lists": 999,
    "boards": 999,
    "ci_builds": 999,
    ...
  },
  "container_registry_enabled": true,
  "dependency_proxy_enabled": false,
  "gitlab_shared_runners_enabled": true,
  "gravatar_enabled": true,
  "influxdb_metrics_enabled": true,
  "ldap_enabled": false,
  "mattermost_enabled": false,
  "omniauth_enabled": true,
  "prometheus_enabled": false,
  "prometheus_metrics_enabled": false,
  "reply_by_email_enabled": "incoming+%{key}@incoming.gitlab.com",
  "signup_enabled": true,
  "web_ide_clientside_preview_enabled": true,
  "projects_with_expiration_policy_disabled": 999,
  "projects_with_expiration_policy_enabled": 999,
  ...
  "elasticsearch_enabled": true,
  "license_trial_ends_on": null,
  "geo_enabled": false,
  "git": {
    "version":