Configure GitLab to access GitLab Duo Self-Hosted

Tier: Ultimate with GitLab Duo Enterprise - Start a trial
Offering: GitLab Self-Managed
Status: Beta
History

To configure your GitLab instance to access the available self-hosted models in your infrastructure:

  1. Confirm that a fully self-hosted configuration is appropriate for your use case.
  2. Configure your GitLab instance.
  3. Configure the self-hosted model.
  4. Configure the GitLab Duo features to use your self-hosted model.

Configure your GitLab instance

Prerequisites:

To configure your GitLab instance to access the AI gateway:

  1. Where your GitLab instance is installed, update the /etc/gitlab/gitlab.rb file:

    sudo vim /etc/gitlab/gitlab.rb
    
  2. Add and save the following environment variables:

    gitlab_rails['env'] = {
      'AI_GATEWAY_URL' => '<path_to_your_ai_gateway>:<port>'
    }
    
  3. Run reconfigure:

    sudo gitlab-ctl reconfigure
    

Configure the self-hosted model

Prerequisites:

  • You must be an administrator.
  • You must have a Duo Enterprise license add-on.

To configure a self-hosted model:

  1. On the left sidebar, at the bottom, select Admin.
  2. Select GitLab Duo.
  3. In the GitLab Duo section, select Change configuration.
  4. Under Self-hosted AI models, select Turn on self-hosted models.
  5. Select Save changes.
  6. On the left sidebar, select Self-hosted models.
    • If the Self-hosted models menu item is not available, synchronize your subscription after purchase:
      1. On the left sidebar, select Subscription.
      2. In Subscription details, to the right of Last sync, select synchronize subscription ().
  7. Select Add self-hosted model.
  8. Complete the fields:
    • Deployment name: Enter a name to uniquely identify the model deployment, for example, Mixtral-8x7B-it-v0.1 on GCP.
    • Model family: Select the model family the deployment belongs to. Only GitLab-approved models are in this list.
    • Endpoint: Enter the URL where the model is hosted.
      • For models hosted through vLLM, you must suffix the URL with /v1. The default port is 8000, so the default endpoint URL is https://<hostname>:8000/v1.
    • API key: Optional. Add an API key if you need one to access the model.
    • Model identifier: This is a required field if your deployment method is vLLM, Bedrock or Azure. The value of this field is based on your deployment method, and should match the following structure:

      Deployment method Format Example
      vLLM custom_openai/<name of the model served through vLLM> custom_openai/Mixtral-8x7B-Instruct-v0.1
      Bedrock bedrock/<model ID of the model> bedrock/mistral.mixtral-8x7b-instruct-v0:1
      Azure OpenAI azure/<model ID of the model> azure/gpt-35-turbo
      Others The field is optional  
  9. Select Create self-hosted model.

Configure GitLab Duo features to use self-hosted models

Prerequisites:

  • You must be an administrator.
  • You must have a Duo Enterprise license add-on.

View configured features

  1. On the left sidebar, at the bottom, select Admin.
  2. Select Self-hosted models.
    • If the Self-hosted models menu item is not available, synchronize your subscription after purchase:
      1. On the left sidebar, select Subscription.
      2. In Subscription details, to the right of Last sync, select synchronize subscription ().
  3. Select the AI-powered features tab.

Configure the feature to use a self-hosted model

Configure the GitLab Duo feature to send queries to the configured self-hosted model:

  1. On the left sidebar, at the bottom, select Admin.
  2. Select Self-hosted models.
  3. Select the AI-powered features tab.
  4. For the feature you want to configure, from the dropdown list, choose the self-hosted model you want to use. For example, Mistral.

Configure the feature to use GitLab AI Vendor models

Prerequisites:

Configure the GitLab Duo feature to send queries to the Cloud Connector and use the GitLab AI Vendor model:

  1. On the left sidebar, at the bottom, select Admin.
  2. Select Self-hosted models.
  3. Select the AI-powered features tab.
  4. For the feature you want to configure, from the dropdown list, select GitLab AI Vendor.