Configure GitLab to access GitLab Duo Self-Hosted

Prerequisites:

• Upgrade GitLab to version 17.9 or later.
• You must be an administrator.

To configure your GitLab instance to access the available self-hosted models in your infrastructure:

1. Confirm that a fully self-hosted configuration is appropriate for your use case.
2. Configure your GitLab instance to access the AI Gateway.
3. In GitLab 18.4 and later, configure your GitLab instance to access the GitLab Duo Agent Platform service.
4. Configure the self-hosted model.
5. Configure the GitLab Duo features to use your self-hosted model.

Configure access to the local AI Gateway

To configure access between your GitLab instance and your local AI Gateway:

1. In the upper-right corner, select Admin.
2. Select GitLab Duo.
3. Select Change configuration.
4. Under Local AI Gateway URL, enter your AI Gateway URL.
5. Select Save changes.

Configure timeout for the AI Gateway

Prerequisites:

• You must be an administrator.

Configure the timeout for GitLab requests to the AI Gateway when using self-hosted models. You can use longer timeouts for models that need extended processing times for large context windows or complex queries.

You can configure a timeout between 60 and 600 seconds (10 minutes). If you don’t set the timeout, GitLab uses the default timeout of 60 seconds.

To configure the AI Gateway timeout:

1. In the upper-right corner, select Admin.
2. Select GitLab Duo.
3. Select Change configuration.
4. Under AI Gateway request timeout, enter the timeout value in seconds (between 60 and 600).
5. Select Save changes.

Determine the timeout value

The timeout value depends on your specific deployment and use case.

To determine the timeout value:

• Start with the default timeout of 60 seconds and monitor for timeout errors.
• Monitor your logs for A1000 timeout errors in your logs. If these errors occur frequently, consider increasing the timeout.
• Consider your use case. Larger prompts, complex code generation tasks, or processing large design documents might require longer timeouts.
• Consider your infrastructure. Model performance depends on available GPU resources, network latency between the AI Gateway and model endpoint, and the model’s processing capabilities.
• Increase incrementally. If you experience timeouts, increase the value gradually (for example, by 30-60 seconds) and monitor the results.

For more information on troubleshooting timeout errors, see Error A1000.

Configure access to the GitLab Duo Agent Platform

Prerequisites:

• Self-hosted beta models and features are turned on.

To access the Agent Platform service from your GitLab instance:

1. In the upper-right corner, select Admin.
2. Select GitLab Duo.
3. Select Change configuration.
4. Under Local URL for the GitLab Duo Agent Platform service, enter the URL for the local Agent Platform service.

   • The URL prefix cannot start with http:// or https://.

   • If the URL is not set up with TLS, you must set the DUO_AGENT_PLATFORM_SERVICE_SECURE environment variable in your GitLab instance:

     • For Linux package installations, in gitlab_rails['env'], set 'DUO_AGENT_PLATFORM_SERVICE_SECURE' => false
     • For self-compiled installations, in /etc/default/gitlab set export DUO_AGENT_PLATFORM_SERVICE_SECURE=false
5. Select Save changes.

Add a self-hosted model

You must add a self-hosted model to your GitLab instance to use it with GitLab Duo features.

To add a self-hosted model:

1. In the upper-right corner, select Admin.

2. Select GitLab Duo.

3. Select Configure GitLab Duo Self-Hosted.

   • If Configure GitLab Duo Self-Hosted is not available, synchronize your subscription after purchase:
     1. On the left sidebar, select Subscription.
     2. In Subscription details, to the right of Last sync, select synchronize subscription ( ).

4. Select Add self-hosted model.

5. Complete the fields:

   • Deployment name: Enter a name to uniquely identify the model deployment, for example, Mixtral-8x7B-it-v0.1 on GCP.
   • Model family: Select the model family the deployment belongs to. You can select either a supported or compatible model.
   • Endpoint: Enter the URL where the model is hosted.
   • API key: Optional. Add an API key if you need one to access the model.
   • Model identifier: Enter the model identifier based on your deployment method. The model identifier should match the following format:

     Deployment method	Format	Example
     vLLM	custom_openai/<name of the model served through vLLM>	custom_openai/Mixtral-8x7B-Instruct-v0.1
     Amazon Bedrock	bedrock/<model ID of the model>	bedrock/mistral.mixtral-8x7b-instruct-v0:1
     Azure OpenAI	azure/<model ID of the model>	azure/gpt-35-turbo

6. Select Create self-hosted model.

Set the model identifier for Amazon Bedrock models

To set a model identifier for an Amazon Bedrock model:

1. Set your AWS_REGION. Ensure you have access to models in that region in your AI gateway Docker configuration.

2. Add the region prefix to the model’s inference profile ID for cross-region inferencing.

3. Use the bedrock/ prefix region as the prefix for the model identifier.

   For example, for the Anthropic Claude 3.5 v2 model in the Tokyo region:

   • The AWS_REGION is ap-northeast-1.
   • The cross-region inferencing prefix is apac..
   • The model identifier is bedrock/apac.anthropic.claude-3-5-sonnet-20241022-v2:0.

Some regions are not supported by cross-region inferencing. For these regions, do not specify a region prefix in the model identifier. For example:

• The AWS_REGION is eu-west-2.
• The model identifier is bedrock/anthropic.claude-3-7-sonnet-20250219-v1:0.

Turn on self-hosted beta models and features

To enable self-hosted beta models and features:

1. In the upper-right corner, select Admin.
2. Select GitLab Duo.
3. Select Change configuration.
4. Under Self-hosted beta models and features, select the Use beta models and features in GitLab Duo Self-Hosted checkbox.
5. Select Save changes.

Configure GitLab Duo features to use self-hosted models

View configured features

1. In the upper-right corner, select Admin.
2. Select GitLab Duo.
3. Select Configure GitLab Duo Self-Hosted.
   • If Configure GitLab Duo Self-Hosted is not available, synchronize your subscription after purchase:
     1. On the left sidebar, select Subscription.
     2. In Subscription details, to the right of Last sync, select synchronize subscription ( ).
4. Select the AI-native features tab.

Configure a feature to use a self-hosted model

Configure a GitLab Duo feature and sub-feature to send queries to the self-hosted model:

1. In the upper-right corner, select Admin.

2. Select GitLab Duo.

3. Select Configure GitLab Duo Self-Hosted.

4. Select the AI-native features tab.

5. For the feature and sub-feature you want to configure, from the dropdown list, choose the self-hosted model you want to use.

   For example, for the code generation, you can select Claude-3 on Bedrock deployment (Claude 3).

   

Configure a feature to use a GitLab AI vendor model

You can configure a GitLab Duo feature to use the GitLab AI vendor model, even if you use a self-hosted AI gateway and models.

1. In the upper-right corner, select Admin.
2. Select GitLab Duo.
3. Select Configure GitLab Duo Self-Hosted.
4. Select the AI-native features tab.
5. For the feature and sub-feature you want to configure, from the dropdown list, select GitLab AI vendor model.

Disable GitLab Duo features

GitLab Duo features remain turned on even if you have not chosen a model for a sub-feature.

To disable a GitLab Duo feature or sub-feature:

1. In the upper-right corner, select Admin.
2. Select GitLab Duo.
3. Select Configure GitLab Duo Self-Hosted.
4. Select the AI-native features tab.
5. For the feature or sub-feature you want to disable, from the dropdown list, select Disabled.

Self-host the GitLab documentation

If your GitLab Duo Self-Hosted setup prevents you from accessing the GitLab documentation at docs.gitlab.com, you can self-host the documentation. For more information, see Host the GitLab product documentation.

Related topics

• Supported models
• Compatible models
• GitLab Duo Self-Hosted configuration types