Testing AI features

This document highlights AI-specific testing considerations that complement GitLab standard testing guidelines. It focuses on the challenges AI features bring to testing, such as non-deterministic responses from third-party providers. Examples are included for each testing level.

AI-powered features depend on system components outside the GitLab monolith, such as the AI Gateway and IDE extensions. In addition to these guidelines, consult any testing guidelines documented in each component project.

Unit testing

Follow standard unit testing guidelines. For AI features, always mock third-party AI provider calls to ensure fast, reliable tests.

Unit test examples

• GitLab: ee/spec/lib/code_suggestions/tasks/code_completion_spec.rb
• VS Code extension: code_suggestions/code_suggestions.test.ts

Integration tests

Use integration tests to verify request construction and response handling for AI providers. Mock AI provider responses to ensure predictable, fast tests that handle various responses, errors, and status codes.

Integration test examples

• GitLab: ee/spec/requests/api/code_suggestions_spec.rb
• VS Code extension: main/test/integration/chat.test.js

Frontend feature tests

Use frontend feature tests to validate AI features from an end-user perspective. Mock AI providers to maintain speed and reliability. Focus on happy paths with selective negative path testing for high-risk scenarios.

Frontend feature test example

• GitLab Duo Chat: ee/spec/features/duo_chat_spec.rb

DAP feature tests in core feature pages

To load DAP components in a feature page by default, include the shared context include_context 'with duo features enabled and agentic chat available for group on SaaS' in a feature spec.

Duo agentic chat feature specs

Agentic chat feature-spec coverage is consolidated in ee/spec/features/duo_chat/agentic_chat_smoke_spec.rb. We strongly suggest to prefer the MSW integration suite at ee/spec/frontend/integration/ai_duo_panel/duo_agentic_chat/ over adding a new feature spec to test Agentic Chat.

Implement a feature spec when you want to test an interaction between agentic chat and other features. For example, the agentic_chat_smoke_spec.rb test suite exercises opening agentic chat using Call-to-Action buttons in the Work Item, Pipeline, and Merge Request pages.

Run DAP feature tests when making a change in AI Gateway

These feature tests also run when we make a change to the AI Gateway repository, to verify that an MR does not accidentally break DAP features, for example:

1. A developer opens an MR in AI Gateway project.
2. A pipeline runs for the MR, which triggers downstream pipeline in GitLab project against aigw/test-branch test branch. This branch points to the same SHA as master.
3. If a pipeline fails, the developer should investigate if the proposed change doesn’t accidentally introduce regressions.

Run a feature spec locally with your DWS/AIGW change

1. Run gdk start to start services including DWS.
2. Open terminal at <gdk-root>/gitlab and use one of the following options:
   • Run export TEST_AI_GATEWAY_REPO_REF=<your-remote-feature-branch> and delete <gitlab-rails-root>/tmp/tests/gitlab-ai-gateway/ cache dir, OR
   • Run export TEST_DUO_WORKFLOW_SERVICE_ENABLED="false" && export TEST_DUO_WORKFLOW_SERVICE_PORT=<your-local-dws-port>. This allows the feature tests to request to your local DWS instance. Make sure the following configuration is set to your local DWS and it’s running:
     • Set true to AIGW_MOCK_MODEL_RESPONSES
     • Set true to AIGW_USE_AGENTIC_MOCK
3. Run a feature spec, for example bundle exec rspec ee/spec/features/duo_chat/agentic_chat_smoke_spec.rb.

See logs of a test case

DAP consists of multiple services and API calls. To debug a test case failure, you may need to examine service logs to identify the root cause. Here are the couple of pointers:

• GitLab-Rails REST API … log/api_json.log

• GitLab-Rails GraphQL API … log/graphql_json.log

• GitLab-Workhorse … log/workhorse-test.log

• DWS … log/duo-workflow-service.log. CI collects this file as a job artifact, and on a failing example tagged :duo_workflow_service, the last 60 lines print automatically in the RSpec output.

• You can also examine the state of VueJS app by having JS console log output:

  ruby

  it 'runs a test' do
    ...
  
    # This prints the browser logs. Combine with `console.log()` in JavaScript.
    browser_logs.each do |log|
      puts "#{log.level}: #{log.message}"
    end
  
    ...
  end

End-to-End testing

Use end-to-end tests sparingly to verify AI features work with real provider responses. Key considerations:

• Keep tests minimal due to slower execution and potential provider outages.
• Account for non-deterministic AI responses in test design. For example, use deterministic assertions on controlled elements like chatbot names, not AI-generated content.

E2E test examples

• GitLab: specs/features/ee/browser_ui/3_create/web_ide/code_suggestions_in_web_ide_spec.rb
• JetBrains: test/kotlin/com/gitlab/plugin/e2eTest/tests/CodeSuggestionTest.kt

Live environment testing

• GitLab.com: We run minimal E2E tests continuously against staging and production environments. For example, Code Suggestions smoke tests.
• GitLab Self-Managed: We use the gitlab-qa orchestrator with AI Gateway scenarios to test AI features on GitLab Self-Managed instances.

Duo Agent Platform foundational flows

The Duo Agent Platform foundational-flow smoke tests are orchestrated end-to-end tests that drive a flow through a real CI pipeline. The gitlab-qa orchestrator runs both tests as the duo-agent-platform job in the e2e:test-on-omnibus-ee child pipeline.

The developer/v1 test creates a workflow through POST /ai/duo_workflows/workflows with start_workflow: true, then asserts that the duo_workflow source pipeline succeeds and the workflow reaches finished. The code_review/v1 test creates a merge request, then assigns the GitLabDuo review bot as its reviewer, the way a review is requested in the product. It then asserts that the duo_workflow source pipeline succeeds, the flow reaches finished, and the review bot posts its review summary on the merge request.

The tests span two repositories. The GitLab repository holds the specs and the flow provisioning helpers. The gitlab-qa orchestrator holds the infrastructure that boots the Duo Workflow Service and routes the GitLab instance to it:

• Component::DuoWorkflowService boots the Duo Workflow Service from the same model-gateway image as the AI Gateway, switched to gRPC and agentic-mock mode, and captures the container logs to the job artifacts on teardown.
• Test::Integration::AiGatewayBase wires the Duo Workflow Service into the scenario and passes the GITLAB_DUO_WORKFLOW_SERVICE_URL and GITLAB_DUO_WORKFLOW_SECURE values into the omnibus Rails environment.
• Test::Integration::DuoAgentPlatform is a dedicated scenario, a subclass of the AI Gateway scenario, so the flows run in their own duo-agent-platform omnibus job and the Duo Workflow Service container stays out of the ai-gateway job.

The agentic-mock mode returns deterministic responses driven by directives, instead of calling a real model, the same approach the Duo Chat and Code Suggestions tests use for the AI Gateway. Where those directives go depends on the flow. The developer/v1 goal is free-form, so its test puts directives in the flow goal. A code_review/v1 goal is validated to be a merge request IID or a merge request URL, so anything else returns 400. The Code Review test instead puts its directives in custom instructions, committed to the project’s default branch as .gitlab/duo/mr-review-instructions.yaml, because that is the only field the flow copies into the model prompt unescaped.

Getting the escaping wrong makes the mock silently return placeholder text, so keep these rules in mind:

1. The mock parses the entire prompt as XML, so any unescaped <, >, or & anywhere in the merge request diff or title makes every scripted response degrade to a placeholder. Keep the seeded diff plain ASCII.
2. The mock reads only the text of the <tool_calls> element, so the JSON payload inside it must have its <, >, and & XML-escaped.
3. Each step of a flow builds its own model instance, so the script replays from the beginning in every step of the flow that calls a model. A tool call that a given step’s toolset does not include fails that step, which then advances anyway.

Gitlab::Duo::CodeReview::Modes::Dap refuses Duo Agent Platform routing for a user holding a Duo Enterprise seat unless the root namespace has recorded consent for :code_review_flow_dap_routing. Without that consent the reviewer assignment still succeeds, but falls back to the non-agentic review service, so no flow starts at all. enable_on_group! therefore sends create_code_review_flow_consent: true whenever code_review/v1 is the flow being enabled. The API ignores that parameter when the namespace has no Duo Enterprise add-on, where consent is not required.

The following files in the GitLab repository make up the tests:

A foundational flow runs after four setup steps have completed in foundational_flow.rb:

1. assign_duo_seat! assigns a Duo seat to the acting user.
2. enable_on_group! enables the flow on the top-level group, which cascades to create an item consumer and service account.
3. enable_remote_flows_on_project! enables remote flows on the project.
4. wait_for_flow_consumer! waits until the consumer, its active service account, and the service-account project membership all resolve. The cascade in step 2 can lag on a cold instance, so this helper re-enables the flow until provisioning settles.

To add another foundational flow:

1. Pass the flow reference to the provisioning helpers from the spec.
2. Extend enable_on_group! if the flow gates on namespace consent or on a specific add-on.
3. Work out where the flow’s prompt accepts mock directives, because the goal is not always available for that.
4. If the spec creates the flow through POST /ai/duo_workflows/workflows, register the flow reference in the FOUNDATIONAL_FLOWS registry in qa/qa/ee/resource/ai/duo_workflow.rb, with a default_goal only if the flow’s goal is static. A flow whose goal names a resource, as Code Review does, needs goal, and source_branch if the flow checks out a branch, supplied by the spec. A flow that GitLab triggers itself is discovered with DuoWorkflow.find_latest_in_project instead, and needs no registry entry.

Custom catalog flows do not use the foundational flows list, so you must create the catalog item and its consumer directly rather than call enable_on_group!.

Exploratory testing

Perform exploratory testing before significant milestones to uncover bugs outside expected workflows and UX issues. This is especially important for AI features as they progress through experiment, beta, and GA phases.

Dogfooding

We dogfood everything. This is especially important for AI features given the rapidly changing nature of the field. See the dogfooding process for details.