Configure GitLab to access GitLab Duo Self-Hosted
Offering: GitLab Self-Managed
Status: Beta
-
Introduced in GitLab 17.1 with a flag named
ai_custom_model
. Disabled by default. - Enabled on GitLab Self-Managed in GitLab 17.6.
- Changed to require GitLab Duo add-on in GitLab 17.6 and later.
- Feature flag
ai_custom_model
removed in GitLab 17.8
To configure your GitLab instance to access the available self-hosted models in your infrastructure:
- Confirm that a fully self-hosted configuration is appropriate for your use case.
- Configure your GitLab instance.
- Configure the self-hosted model.
- Configure the GitLab Duo features to use your self-hosted model.
Configure your GitLab instance
Prerequisites:
To configure your GitLab instance to access the AI gateway:
-
Where your GitLab instance is installed, update the
/etc/gitlab/gitlab.rb
file:sudo vim /etc/gitlab/gitlab.rb
-
Add and save the following environment variables:
gitlab_rails['env'] = { 'AI_GATEWAY_URL' => '<path_to_your_ai_gateway>:<port>' }
-
Run reconfigure:
sudo gitlab-ctl reconfigure
Configure the self-hosted model
Prerequisites:
- You must be an administrator.
- You must have a Duo Enterprise license add-on.
To configure a self-hosted model:
- On the left sidebar, at the bottom, select Admin.
- Select GitLab Duo.
- In the GitLab Duo section, select Change configuration.
- Under Self-hosted AI models, select Turn on self-hosted models.
- Select Save changes.
- On the left sidebar, select Self-hosted models.
- If the Self-hosted models menu item is not available, synchronize your
subscription after purchase:
- On the left sidebar, select Subscription.
- In Subscription details, to the right of Last sync, select synchronize subscription ().
- If the Self-hosted models menu item is not available, synchronize your
subscription after purchase:
- Select Add self-hosted model.
- Complete the fields:
-
Deployment name: Enter a name to uniquely identify the model deployment, for example,
Mixtral-8x7B-it-v0.1 on GCP
. - Model family: Select the model family the deployment belongs to. Only GitLab-approved models are in this list.
-
Endpoint: Enter the URL where the model is hosted.
- For models hosted through vLLM, you must suffix the URL with
/v1
. The default port is8000
, so the default endpoint URL ishttps://<hostname>:8000/v1
.
- For models hosted through vLLM, you must suffix the URL with
- API key: Optional. Add an API key if you need one to access the model.
-
Model identifier: This is a required field if your deployment method is vLLM, Bedrock or Azure. The value of this field is based on your deployment method, and should match the following structure:
Deployment method Format Example vLLM custom_openai/<name of the model served through vLLM>
custom_openai/Mixtral-8x7B-Instruct-v0.1
Bedrock bedrock/<model ID of the model>
bedrock/mistral.mixtral-8x7b-instruct-v0:1
Azure OpenAI azure/<model ID of the model>
azure/gpt-35-turbo
Others The field is optional
-
Deployment name: Enter a name to uniquely identify the model deployment, for example,
- Select Create self-hosted model.
Configure GitLab Duo features to use self-hosted models
Prerequisites:
- You must be an administrator.
- You must have a Duo Enterprise license add-on.
View configured features
- On the left sidebar, at the bottom, select Admin.
- Select Self-hosted models.
- If the Self-hosted models menu item is not available, synchronize your
subscription after purchase:
- On the left sidebar, select Subscription.
- In Subscription details, to the right of Last sync, select synchronize subscription ().
- If the Self-hosted models menu item is not available, synchronize your
subscription after purchase:
- Select the AI-powered features tab.
Configure the feature to use a self-hosted model
Configure the GitLab Duo feature to send queries to the configured self-hosted model:
- On the left sidebar, at the bottom, select Admin.
- Select Self-hosted models.
- Select the AI-powered features tab.
- For the feature you want to configure, from the dropdown list, choose the self-hosted model you want to use. For example,
Mistral
.
Configure the feature to use GitLab AI Vendor models
Prerequisites:
- You must use an online license.
Configure the GitLab Duo feature to send queries to the Cloud Connector and use the GitLab AI Vendor model:
- On the left sidebar, at the bottom, select Admin.
- Select Self-hosted models.
- Select the AI-powered features tab.
- For the feature you want to configure, from the dropdown list, select GitLab AI Vendor.