AI features based on 3rd-party integrations

GitLab Duo features are powered by AI models and integrations. This document provides an overview of developing with AI features in GitLab.

For detailed instructions on setting up GitLab Duo licensing in your development environment, see GitLab Duo licensing for local development.

Instructions for setting up GitLab Duo features in the local development environment

Here is a list of all of the main steps to go through from a fresh, GDK-less computer to fully working ai-development ready.

1. Setup GDK. You should follow the local network binding documentation as well.
2. Configure license
3. Setup your Google Cloud Platform account and the CLI
4. Get your Anthropic license key by making an access request like this
5. If you want to use Duo Chat, Code Suggestions, and other non-Duo Agent Platform features, install and configure AI gateway
6. If you want to use Duo Agent Platform or Agentic Chat locally, setup Duo Workflow Service
7. Run the Duo setup Rake task

More details on each step can be found down below for help and troubleshooting.

Setup GDK

After following the installation documentation, executing gdk status should show all services as up and running. Login as the root user and make sure there are no errors.

Configure licenses

See GitLab Duo licensing for local development.

Important Before tackling any other step, validate that you have a working License. Navigate to admin/subscriptions and make sure you see only one license and that is has the online label.

If there is any issue, check out the troubleshooting documentation.

Install AI gateway

This step includes getting your Google Cloud account setup, getting your Anthropic key and then setting up AI Gateway. Follow these instructions to install the AI gateway with GDK.

Why: This ensures that your instance or group has the correct licenses, settings, and feature flags to test Duo features locally. AI gateway is what routes request between GitLab Rails and the LLM. The script should take care of most of the setup required. Once it has been run, make sure to check in your GDK database that the ai gateway URL is correct. Run:

Ai::Setting.first.ai_gateway_url

This should return a URL that points to your local and uses the right port: http://0.0.0.0:5052.

If the value points to a non-local URL, you can manually execute an update to your GDK database:

Ai::Setting.first.update!(ai_gateway_url: "http://0.0.0.0:5052")

Now in your gdk directory, you can cd into the gitlab-ai-gateway directory and run poetry sync. This should install all project dependency. If this resolves without error, try now to run the test of the project with make test. If there are errors, check the results as it can help debug potential issues with your configuration.

Finally, run gdk tail gitlab-ai-gateway from the GitLab project directory. There should be no errors in the log.

Setup Duo Workflow Service

After following the steps in the setup, run gdk status. You should see the duo-workflow-service running. You can run gdk tail duo-workflow-service in case there might be errors.

Run gitlab:duo:setup script

Note: this task is idempotent and skips reseeding if the gitlab-duo group already exists. To force reseeding from this task, set GITLAB_DUO_RESEED=1. For details on the seeds used, see Development seed files.

This ensures that your instance or group has the correct licenses, settings, and feature flags to test Duo features locally. Below are several options. If you are unsure, use option 1.

1. GitLab.com (SaaS) mode

   GITLAB_SIMULATE_SAAS=1 bundle exec 'rake gitlab:duo:setup'

   This:

   • Creates a test group called gitlab-duo, which contains a project called test
   • Applies an Ultimate license to the group
   • Sets up Duo Enterprise seats for the group
   • Enables all feature flags for the group
   • Updates group settings to enable all available GitLab Duo features

   Alternatively, if you want to add GitLab Duo Pro licenses for the group instead (which only enables a subset of features), you can run:

   GITLAB_SIMULATE_SAAS=1 bundle exec 'rake gitlab:duo:setup[duo_pro]'

2. GitLab Self-Managed / Dedicated mode

   GITLAB_SIMULATE_SAAS=0 bundle exec 'rake gitlab:duo:setup'

   This:

   • Creates a test group called gitlab-duo, which contains a project called test
   • Applies an Ultimate license to the instance
   • Sets up Duo Enterprise seats for the instance
   • Enables all feature flags for the instance
   • Updates instance settings to enable all available GitLab Duo features

   Alternatively, if you want to add GitLab Duo Pro add-on for the instance instead (which only enables a subset of features), you can run:

   GITLAB_SIMULATE_SAAS=0 bundle exec 'rake gitlab:duo:setup[duo_pro]'

After the script finishes without error, now go to gitlab-duo/test and validate that you can see Duo Chat. Send a question to Chat and make sure there are no errors. If there are, the two most common problems in development are A1003 and A9999.

A9999 is a catchall error. The biggest offender is not having the ai gateway URL setup properly as described in Install AI gateway. If not, make sure to check the tests are passing in the gitlab-ai-gateway repository with make test and that gdk tail gitlab-ai-gateway returns no error.

A1003 is more around permissions, either an invalid/missing Anthropic token or a misconfiguration of gcloud.

Tips for local development

1. When responses are taking too long to appear in the user interface, consider restarting Sidekiq by running gdk restart rails-background-jobs. If that doesn’t work, try gdk kill and then gdk start.
2. Alternatively, bypass Sidekiq entirely and run the service synchronously. This can help with debugging errors as GraphQL errors are now available in the network inspector instead of the Sidekiq logs. To do that, temporarily alter the perform_for method in Llm::CompletionWorker class by changing perform_async to perform_inline.
3. When testing model selection, add export FETCH_MODEL_SELECTION_DATA_FROM_LOCAL=1 to your env.runit file, so that your GDK fetches model information from your local AI Gateway rather than cloud-connected AIGW.

Feature development (Abstraction Layer)

Feature flags

Apply the following feature flags to any AI feature work:

• A general flag (ai_global_switch) that applies to all other AI features. It’s enabled by default.
• A flag specific to that feature. The feature flag name must be different than the licensed feature name.

See the feature flag tracker epic for the list of all feature flags and how to use them.

Push feature flags to AI gateway

You can push feature flags to AI gateway. This is helpful to gradually rollout user-facing changes even if the feature resides in AI gateway. See the following example:

# Push a feature flag state to AI gateway.
Gitlab::AiGateway.push_feature_flag(:new_prompt_template, user)

Later, you can use the feature flag state in AI gateway in the following way:

from ai_gateway.feature_flags import is_feature_enabled

# Check if the feature flag "new_prompt_template" is enabled.
if is_feature_enabled('new_prompt_template'):
  # Build a prompt from the new prompt template
else:
  # Build a prompt from the old prompt template

IMPORTANT: At the cleaning up step, remove the feature flag in AI gateway repository before removing the flag in GitLab-Rails repository. If you clean up the flag in GitLab-Rails repository at first, the feature flag in AI gateway will be disabled immediately as it’s the default state, hence you might encounter a surprising behavior.

IMPORTANT: Cleaning up the feature flag in AI gateway will immediately distribute the change to all GitLab instances, including GitLab.com, GitLab Self-Managed, and GitLab Dedicated.

Technical details:

• When push_feature_flag runs on an enabled feature flag, the name of the flag is cached in the current context, which is later attached to the x-gitlab-enabled-feature-flags HTTP header when GitLab-Sidekiq/Rails sends requests to AI gateway.

• When frontend clients (for example, VS Code Extension or LSP) request a User JWT (UJWT) for direct AI gateway communication, GitLab returns:

  • Public headers (including x-gitlab-enabled-feature-flags).
  • The generated UJWT (1-hour expiration).

Frontend clients must regenerate UJWT upon expiration. Backend changes such as feature flag updates through ChatOps render the header values to become stale. These header values are refreshed at the next UJWT generation.

Similarly, we also have push_frontend_feature_flag to push feature flags to frontend.

GraphQL API

To connect to the AI provider API using the Abstraction Layer, use an extendable GraphQL API called aiAction. The input accepts key/value pairs, where the key is the action that needs to be performed. We only allow one AI action per mutation request.

Example of a mutation:

mutation {
  aiAction(input: {summarizeComments: {resourceId: "gid://gitlab/Issue/52"}}) {
    clientMutationId
  }
}

As an example, assume we want to build an “explain code” action. To do this, we extend the input with a new key, explainCode. The mutation would look like this:

mutation {
  aiAction(
    input: {
      explainCode: { resourceId: "gid://gitlab/MergeRequest/52", code: "foo() { console.log() }" }
    }
  ) {
    clientMutationId
  }
}

The GraphQL API then uses the Anthropic Client to send the response.

How to receive a response

The API requests to AI providers are handled in a background job. We therefore do not keep the request alive and the Frontend needs to match the request to the response from the subscription.

To match a response on the aiCompletionResponse subscription, you can provide a clientSubscriptionId to the aiAction mutation.

• The clientSubscriptionId should be unique per feature and within a page to not interfere with other AI features. We recommend to use a UUID.
• Only when the clientSubscriptionId is provided as part of the aiAction mutation, it will be used for broadcasting the aiCompletionResponse.
• If the clientSubscriptionId is not provided, only userId and resourceId are used for the aiCompletionResponse.

As an example mutation for summarizing comments, we provide a randomId as part of the mutation:

mutation {
  aiAction(
    input: {
      summarizeComments: { resourceId: "gid://gitlab/Issue/52" }
      clientSubscriptionId: "randomId"
    }
  ) {
    clientMutationId
  }
}

In our component, we then listen on the aiCompletionResponse using the userId, resourceId and clientSubscriptionId ("randomId"):

subscription aiCompletionResponse(
  $userId: UserID
  $resourceId: AiModelID
  $clientSubscriptionId: String
) {
  aiCompletionResponse(
    userId: $userId
    resourceId: $resourceId
    clientSubscriptionId: $clientSubscriptionId
  ) {
    content
    errors
  }
}

The subscription for Chat behaves differently.

To not have many concurrent subscriptions, you should also only subscribe to the subscription once the mutation is sent by using skip().

Clarifying different ID parameters

When working with the aiAction mutation, several ID parameters are used for routing requests and responses correctly. Here’s what each parameter does:

• user_id (required)

  • Purpose: Identifies and authenticates the requesting user
  • Used for: Permission checks, request attribution, and response routing
  • Example: gid://gitlab/User/123
  • Note: This ID is automatically included by the GraphQL API framework

• client_subscription_id (recommended for streaming or multiple features)

  • Client-generated UUID for tracking specific request/response pairs
  • Required when using streaming responses or when multiple AI features share the same page
  • Example: "9f5dedb3-c58d-46e3-8197-73d653c71e69"
  • Can be omitted for simple, isolated requests with no streaming

• resource_id (contextual - required for some features)

  • Purpose: References a specific GitLab entity (project, issue, MR) that provides context for the AI operation
  • Used for: Permission verification and contextual information gathering
  • Real example: "gid://gitlab/Issue/164723626"
  • Note: Some features may not require a specific resource

• project_id (contextual - required for some features)

  • Purpose: Identifies the project context for the AI operation
  • Used for: Project-specific permission checks and context
  • Real example: "gid://gitlab/Project/278964"
  • Note: Some features may not require a specific project

Current abstraction layer flow

The following graph uses VertexAI as an example. You can use different providers.

flowchart TD
A[GitLab frontend] -->B[AiAction GraphQL mutation]
B --> C[Llm::ExecuteMethodService]
C --> D[One of services, for example: Llm::GenerateSummaryService]
D -->|scheduled| E[AI worker:Llm::CompletionWorker]
E -->F[::Gitlab::Llm::Completions::Factory]
F -->G[#96;::Gitlab::Llm::VertexAi::Completions::...#96; class using #96;::Gitlab::Llm::Templates::...#96; class]
G -->|calling| H[Gitlab::Llm::VertexAi::Client]
H --> |response| I[::Gitlab::Llm::GraphqlSubscriptionResponseService]
I --> J[GraphqlTriggers.ai_completion_response]
J --> K[::GitlabSchema.subscriptions.trigger]

Reuse the existing AI components for multiple models

We thrive optimizing AI components, such as prompt, input/output parser, tools/function-calling, for each LLM, however, diverging the components for each model could increase the maintenance overhead. Hence, it’s generally advised to reuse the existing components for multiple models as long as it doesn’t degrade a feature quality. Here are the rules of thumbs:

1. Iterate on the existing prompt template for multiple models. Do NOT introduce a new one unless it causes a quality degradation for a particular model.
2. Iterate on the existing input/output parsers and tools/functions-calling for multiple models. Do NOT introduce a new one unless it causes a quality degradation for a particular model.
3. If a quality degradation is detected for a particular model, the shared component should be diverged for the particular model.

An example of this case is that we can apply Claude specific CoT optimization to the other models such as Mixtral as long as it doesn’t cause a quality degradation.

Monitoring

• Error ratio and response latency apdex for each Ai action can be found on Sidekiq Service dashboard under SLI Detail: llm_completion.
• Spent tokens, usage of each Ai feature and other statistics can be found on periscope dashboard.
• AI gateway logs.
• AI gateway metrics.
• Feature usage dashboard via proxy.

Security

Refer to the secure coding guidelines for Artificial Intelligence (AI) features.

Help

• Here’s how to reach us!
• View guidelines for working with GitLab Duo Chat.