Configure GitLab to access self-hosted models

Tier: For a limited time, Premium and Ultimate. In the future, GitLab Duo Enterprise. Offering: Self-managed Status: Experiment
History
The availability of this feature is controlled by a feature flag. For more information, see the history.
caution
This feature is considered experimental and is not intended for customer usage outside of initial design partners. We expect major changes to this feature.
This page contains information related to upcoming products, features, and functionality. It is important to note that the information presented is for informational purposes only. Please do not rely on this information for purchasing or planning purposes. The development, release, and timing of any products, features, or functionality may be subject to change or delay and remain at the sole discretion of GitLab Inc.

To configure your GitLab instance to access the available self-hosted large language models (LLMs) in your infrastructure:

  1. Configure the self-hosted model.
  2. Configure the GitLab Duo AI-powered features to use your self-hosted models.

Configure the self-hosted model

Prerequisites:

  • You must be an administrator.

To configure a self-hosted model:

  1. On the left sidebar, at the bottom, select Admin.
  2. Select AI-powered features.
    • If the AI-powered features menu item is not available, synchronize your subscription after purchase:
      1. On the left sidebar, select Subscription.
      2. In Subscription details, to the right of Last sync, select synchronize subscription ().
  3. Select Models.
  4. Set your model details by selecting New self-hosted model.
  5. Complete the fields:
    • Enter the model name, for example, Mistral.
    • From the Model dropdown list, select the model. Only GitLab-approved models are listed here.
    • For Endpoint, select the self-hosted model endpoint, for example, the server hosting the model.
    • Optional. For API token, add an API key if you need one to access the model.
  6. Select Create model.

Configure the features to your models

Prerequisites:

  • You must be an administrator.

View the configured features

  1. On the left sidebar, at the bottom, select Admin.
  2. Select AI-powered features.
    • If the AI-powered features menu item is not available, synchronize your subscription after purchase:
      1. On the left sidebar, select Subscription.
      2. In Subscription details, to the right of Last sync, select synchronize subscription ().
  3. Select Features.

Configure the features to use self-hosted models

Use a self-hosted AI Gateway to execute queries to the configured self-hosted model:

  1. For the feature you want to set, select Edit. For example, Code Generation.
  2. Select the model provider for the feature:
    • From the list, select Self-Hosted Model.
    • Choose the self-hosted model you would like to use, for example, Mistral.
  3. Select Save Changes.

Configure the features to use GitLab AI Vendor models

You can set a GitLab Duo feature’s model provider to the GitLab AI Vendor. That feature then uses GitLab hosted models through the Cloud Connector:

  1. For the feature you want to set, select Edit.
  2. In the list of model providers for the feature, select AI Vendor.
  3. Select Save Changes.