Configure GitLab to access self-hosted models
- Tier: Ultimate with GitLab Duo Enterprise - Start a trial
- Offering: Self-managed
- Status: Beta
To configure your GitLab instance to access the available self-hosted models in your infrastructure:
- Use a locally hosted or GitLab.com AI gateway.
- Configure your GitLab instance.
- Configure the self-hosted model.
- Configure the GitLab Duo features to use your self-hosted model.
Configure your GitLab instance
Prerequisites:
To configure your GitLab instance to access the AI gateway:
-
Where your GitLab instance is installed, update the
/etc/gitlab/gitlab.rb
file:sudo vim /etc/gitlab/gitlab.rb
-
Add and save the following environment variables:
gitlab_rails['env'] = { 'AI_GATEWAY_URL' => '<path_to_your_ai_gateway>:<port>' }
-
Run reconfigure:
sudo gitlab-ctl reconfigure
Configure the self-hosted model
Prerequisites:
- You must be an administrator.
To configure a self-hosted model:
-
On the left sidebar, at the bottom, select Admin.
-
Select Self-hosted models.
- If the Self-hosted models menu item is not available, synchronize your
subscription after purchase:
- On the left sidebar, select Subscription.
- In Subscription details, to the right of Last sync, select synchronize subscription ( ).
- If the Self-hosted models menu item is not available, synchronize your
subscription after purchase:
-
Select Add self-hosted model.
-
Complete the fields:
- Deployment name: Enter a name to uniquely identify the model deployment, for example,
Mixtral-8x7B-it-v0.1 on GCP
. - Model family: Select the model family the deployment belongs to. Only GitLab-approved models are in this list.
- Endpoint: Enter the URL where the model is hosted.
- For models hosted through vLLM, it is essential to suffix the URL with
/v1
.
- For models hosted through vLLM, it is essential to suffix the URL with
- API key: Optional. Add an API key if you need one to access the model.
- Model identifier (optional): Optional. The model identifier is based on your deployment method:
Deployment method Format Example vLLM custom_openai/<name of the model served through vLLM>
custom_openai/Mixtral-8x7B-Instruct-v0.1
Bedrock bedrock/<model ID of the model>
bedrock/mistral.mixtral-8x7b-instruct-v0:1
Azure azure/<model ID of the model>
azure/gpt-35-turbo
Others The field is optional
- Deployment name: Enter a name to uniquely identify the model deployment, for example,
-
Select Create self-hosted model.
Configure GitLab Duo features to use self-hosted models
Prerequisites:
- You must be an administrator.
View configured features
- On the left sidebar, at the bottom, select Admin.
- Select Self-hosted models.
- If the Self-hosted models menu item is not available, synchronize your
subscription after purchase:
- On the left sidebar, select Subscription.
- In Subscription details, to the right of Last sync, select synchronize subscription ( ).
- If the Self-hosted models menu item is not available, synchronize your
subscription after purchase:
- Select the AI-powered features tab.
Configure the feature to use a self-hosted model
Configure the GitLab Duo feature to send queries to the configured self-hosted model:
- On the left sidebar, at the bottom, select Admin.
- Select Self-hosted models.
- Select the AI-powered features tab.
- For the feature you want to configure, from the dropdown list, choose the self-hosted model you want to use. For example,
Mistral
.
Docs
Edit this page to fix an error or add an improvement in a merge request.
Create an issue to suggest an improvement to this page.
Product
Create an issue if there's something you don't like about this feature.
Propose functionality by submitting a feature request.
Feature availability and product trials
View pricing to see all GitLab tiers and features, or to upgrade.
Try GitLab for free with access to all features for 30 days.
Get help
If you didn't find what you were looking for, search the docs.
If you want help with something specific and could use community support, post on the GitLab forum.
For problems setting up or using this feature (depending on your GitLab subscription).
Request support