GitLab Observability development guidelines

GitLab Observability development setup

There are several options for developing and debugging GitLab Observability:

Run GDK and connect to the staging instance of GitLab Observability Backend

This method takes advantage of our Cloud Connected Observability Backend. Your GitLab instance will require a valid Cloud License and will be treated as a self-managed instance, connected to a multi-tenant GitLab-hosted instance of the GitLab Observability Backend. See this design document for more details on how this works.

How to enable:

  1. Add a GitLab Ultimate Self-Managed subscription to your GDK instance.

    1. Sign in to the staging Customers Portal by selecting the Continue with GitLab.com account button. If you do not have an existing account, you are prompted to create one.
    2. If you do not have an existing cloud activation code, click the Staging Self-Managed Ultimate Subscription link on the new subscription purchase links page.
    3. Select enough seats to cover the number of users in your GDK instance (200 should be plenty).
    4. Purchase the subscription using a test credit card.

    After this step is complete, you will have an activation code for a GitLab Ultimate Self-Managed subscription.

  2. Set environment variables to point customers-dot to staging, and the Observability URL to staging. For GDK, this can be done in <gdk-root>/env.runit: 

    export GITLAB_SIMULATE_SAAS=0
export GITLAB_LICENSE_MODE=test
export CUSTOMER_PORTAL_URL=https://customers.staging.gitlab.com
export OVERRIDE_OBSERVABILITY_QUERY_URL=https://observe.staging.gitlab.com
export OVERRIDE_OBSERVABILITY_INGEST_URL=https://observe.staging.gitlab.com

    On a non-GDK/GCK instance, you can set the variables using gitlab_rails['env'] in the gitlab.rb file: 

    gitlab_rails['env'] = {
'GITLAB_LICENSE_MODE' => 'test',
'CUSTOMER_PORTAL_URL' => 'https://customers.staging.gitlab.com',
'OVERRIDE_OBSERVABILITY_QUERY_URL' => 'https://observe.staging.gitlab.com',
'OVERRIDE_OBSERVABILITY_INGEST_URL' => 'https://observe.staging.gitlab.com'
}
  3. Enable the feature flag for GitLab Observability features:
    1. Start a rails console session:
    2. Run Feature.enable(:observability_features);
  4. Restart your instance (e.g. gdk restart).
  5. Follow the instructions to activate your new license.
  6. Test out the GitLab Observability feature by navigating to a project and selecting Tracing, Metrics, or Logs from the Monitor section of the navigation menu.
  7. If you are seeing 404 errors you might need to manually refresh your license data.

Use the purpose built devvm

Visit devvm and follow the README instructions for setup and developing against it.

Use the OpenTelemetry Demo app to send data to a project

The OpenTelemetry Demo app is a great way to run several Docker containers (representing a distributed system), and to send the logs, metrics, and traces to your local GDK instance.

You can reference the instructions for running the demo app here.

OpenTelemetry Demo app Quickstart

  1. Clone the Demo repository: 

    git clone https://github.com/open-telemetry/opentelemetry-demo.git

  2. Change to the demo folder: 

    cd opentelemetry-demo/
  3. Create a project in your local GDK instance. Take note of the project ID.
  4. In the newly created project, create a project access token with Developer role and API scope. Save the token for use in the next step.

  5. With an editor, edit the configuration in src/otelcollector/otelcol-config-extras.yml. Add the following YAML, replacing:

    • $GDK_HOST with the host and $GDK_PORT with the port number of your GitLab instance.
    • $PROJECT_ID with the project ID and $TOKEN with the token created in the previous steps. 
    exporters:
   otlphttp/gitlab:
      endpoint: http://$GDK_HOST:$GDK_PORT/api/v4/projects/$PROJECT_ID/observability/
      headers:
         "private-token": "$TOKEN"

service:
   pipelines:
      traces:
         exporters: [spanmetrics, otlphttp/gitlab]
      metrics:
         exporters: [otlphttp/gitlab]
      logs:
         exporters: [otlphttp/gitlab]
note
For GDK and Docker to communicate you may need to set up a loopback interface.

  1. Save the configuration and start the demo app: 

    docker compose up --force-recreate --remove-orphans --detach
  2. Visit the UI to generate data.
  3. Verify Telemetry by exploring logs, metrics, and traces under the Monitor menu in your GitLab project.

Run GDK with mocked Observability data

Apply the following patch to override Observability API calls with local mocks: 

git apply < <(curl --silent "https://gitlab.com/gitlab-org/opstrace/opstrace/-/snippets/3747939/raw/main/mock.patch")