Configure LLM platforms

The AI Gateway supports multiple LLM providers through LiteLLM. GitLab has validated and tested the following platforms.

If the platform you want to use is not listed, share your feedback in issue 526144.

Use multiple models and platforms

You can use multiple models and platforms in the same GitLab instance.

For example, you can configure one feature to use Azure OpenAI, and another feature to use AWS Bedrock, or self-hosted models served with vLLM.

This setup gives you flexibility to choose the best model and platform for each use case. Models must be supported and served through a compatible platform.

Self-hosted model deployments

vLLM

vLLM is a high-performance inference server optimized for serving LLMs with memory efficiency. It supports model parallelism and integrates easily with existing workflows.

To install vLLM, see the vLLM Installation Guide. You should install version v0.18.1 or later.

For a prescriptive setup guide for serving GPT OSS 120B with vLLM, see Serve GPT OSS 120B with vLLM.

Configuring the endpoint URL

When configuring the endpoint URL for any OpenAI API compatible platforms (such as vLLM) in GitLab:

• The URL must be suffixed with /v1
• If using the default vLLM configuration, the endpoint URL would be https://<hostname>:8000/v1
• If your server is configured behind a proxy or load balancer, you might not need to specify the port, in which case the URL would be https://<hostname>/v1

Find the model name

After the model has been deployed, to get the model name for the model identifier field in GitLab, query the vLLM server’s /v1/models endpoint:

curl \
  --header "Authorization: Bearer API_KEY" \
  --header "Content-Type: application/json" \
  http://your-vllm-server:8000/v1/models

The model name is the value of the data.id field in the response.

Example response:

{
  "object": "list",
  "data": [
    {
      "id": "Mixtral-8x22B-Instruct-v0.1",
      "object": "model",
      "created": 1739421415,
      "owned_by": "vllm",
      "root": "mistralai/Mixtral-8x22B-Instruct-v0.1",
      // Additional fields removed for readability
    }
  ]
}

In this example, if the model’s id is Mixtral-8x22B-Instruct-v0.1, you would set the model identifier in GitLab as custom_openai/Mixtral-8x22B-Instruct-v0.1.

For more information, see the following documentation:

• vLLM supported models, see the vLLM Supported Models documentation.
• Available options when using vLLM to run a model, see the vLLM documentation on engine arguments.

Mistral-7B-Instruct-v0.2

1. Download the model from HuggingFace:

   shell

   git clone https://<your-hugging-face-username>:<your-hugging-face-token>@huggingface.co/mistralai/Mistral-7B-Instruct-v0.3

2. Run the server:

   shell

   vllm serve <path-to-model>/Mistral-7B-Instruct-v0.3 \
      --served_model_name <choose-a-name-for-the-model>  \
      --tokenizer_mode mistral \
      --tensor_parallel_size <number-of-gpus> \
      --load_format mistral \
      --config_format mistral \
      --tokenizer <path-to-model>/Mistral-7B-Instruct-v0.3

Mixtral-8x7B-Instruct-v0.1

1. Download the model from HuggingFace:

   shell

   git clone https://<your-hugging-face-username>:<your-hugging-face-token>@huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1

2. Rename the token config:

   shell

   cd <path-to-model>/Mixtral-8x7B-Instruct-v0.1
   cp tokenizer.model tokenizer.model.v3

3. Run the model:

   shell

   vllm serve <path-to-model>/Mixtral-8x7B-Instruct-v0.1 \
     --tensor_parallel_size 4 \
     --served_model_name <choose-a-name-for-the-model> \
     --tokenizer_mode mistral \
     --load_format safetensors \
     --tokenizer <path-to-model>/Mixtral-8x7B-Instruct-v0.1

Disable request logging to reduce latency

When running vLLM in production, you can significantly reduce latency by using the --disable-log-requests flag to disable request logging.

Disabling request logging minimizes the overhead introduced by verbose logs, especially under high load, and can help improve performance levels.

vllm serve <path-to-model>/<model-version> \
--served_model_name <choose-a-name-for-the-model>  \
--disable-log-requests

This change has been observed to notably improve response times in internal benchmarks.

Cloud-hosted model deployments

GitLab has validated and tested the following providers. The AI Gateway supports LLM providers that are compatible with LiteLLM.

• AWS Bedrock
• Amazon Bedrock Mantle
• Gemini Enterprise Agent Platform
• Azure OpenAI
• Anthropic
• OpenAI

Configure authentication with AWS Bedrock

You can use several methods to authenticate AWS Bedrock with your AI Gateway.

Prerequisites:

• Models are automatically enabled in Bedrock when first invoked. For more information, see Bedrock model access.
• Have AWS credentials configured with appropriate IAM permissions.

Amazon EKS with Helm Chart (Recommended)

Use IRSA (IAM Roles for Service Accounts) for your AI Gateway pods to authenticate to AWS Bedrock, without storing static credentials.

After you authenticate Amazon EKS with IRSA, the AI Gateway automatically obtains temporary credentials from the IRSA role.

To use IRSA to authenticate Amazon EKS:

1. Create an IAM policy that grants access to Bedrock models. You can scope this to specific models if you require more security:

   json

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Effect": "Allow",
         "Action": [
           "bedrock:InvokeModel",
           "bedrock:InvokeModelWithResponseStream"
         ],
         "Resource": "arn:aws:bedrock:*::foundation-model/*"
       }
     ]
   }

   shell

   aws iam create-policy \
     --policy-name bedrock-ai-gateway-access \
     --policy-document file://bedrock-policy.json \
     --description "Bedrock access for AI Gateway"

2. Optional. For stricter access control, replace the wildcard resource with specific model Amazon Resource Name (ARN). This ensures only approved models can be accessed, even if GitLab configuration changes. For available model ARNs, see Amazon Bedrock model IDs.

   json

   "Resource": [
     "arn:aws:bedrock:us-east-1::foundation-model/anthropic.claude-3-5-sonnet-20241022-v2:0",
     "arn:aws:bedrock:us-east-1::foundation-model/anthropic.claude-3-haiku-20240307-v1:0"
   ]

   Some models might use different ARN formats. For example, newer models might require inference profile ARNs in addition to foundation model ARNs. To check the ARN format for your specific model, see the Amazon Bedrock model IDs.

3. Create an IAM role with a trust policy for your Amazon EKS service account to use. Replace the following values:

   • YOUR_ACCOUNT_ID: Your AWS account ID.
   • REGION: Your Amazon EKS cluster region (for example, us-east-1).
   • YOUR_OIDC_ID: Your Amazon EKS cluster’s OIDC provider ID.
   • NAMESPACE: Kubernetes namespace where AI Gateway is deployed.
   json

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Effect": "Allow",
         "Principal": {
           "Federated": "arn:aws:iam::YOUR_ACCOUNT_ID:oidc-provider/oidc.eks.REGION.amazonaws.com/id/YOUR_OIDC_ID"
         },
         "Action": "sts:AssumeRoleWithWebIdentity",
         "Condition": {
           "StringEquals": {
             "oidc.eks.REGION.amazonaws.com/id/YOUR_OIDC_ID:sub": "system:serviceaccount:NAMESPACE:ai-gateway",
             "oidc.eks.REGION.amazonaws.com/id/YOUR_OIDC_ID:aud": "sts.amazonaws.com"
           }
         }
       }
     ]
   }

   shell

   # Create the role
   aws iam create-role \
     --role-name eks-ai-gateway-bedrock \
     --assume-role-policy-document file://trust-policy.json \
     --description "EKS IRSA role for AI Gateway to access Bedrock"

4. Attach the Bedrock IAM policy to this role.

   shell

   # Attach the role
   aws iam attach-role-policy \
     --role-name eks-ai-gateway-bedrock \
     --policy-arn arn:aws:iam::YOUR_ACCOUNT_ID:policy/bedrock-ai-gateway-access

5. To configure the Helm chart, install the AI Gateway with the IAM role annotation:

   yaml

   serviceAccount:
     create: true
     name: ai-gateway
     annotations:
       eks.amazonaws.com/role-arn: arn:aws:iam::YOUR_ACCOUNT_ID:role/YOUR_ROLE_NAME
   extraEnvironmentVariables:
     - name: AWS_REGION
       value: us-east-1

For more information, see IAM roles for service accounts.

Docker deployments

Configure IAM credentials through environment variables when starting the AI Gateway container:

docker run -d \
  -e AWS_ACCESS_KEY_ID=your-access-key \
  -e AWS_SECRET_ACCESS_KEY=your-secret-key \
  -e AWS_REGION=us-east-1 \
  -p 5052:5052 \
  registry.gitlab.com/gitlab-org/modelops/applied-ml/code-suggestions/ai-assist/model-gateway:self-hosted-vX.Y.Z-ee

The IAM user or role must have a policy similar to the one you would set in Amazon EKS with Helm Chart.

Kubernetes deployments

For Kubernetes clusters other than Amazon EKS, you can use Kubernetes secrets to store AWS credentials:

1. Create a Kubernetes secret:

   shell

   kubectl create secret generic aws-credentials \
     --from-literal=access-key-id=YOUR_ACCESS_KEY_ID \
     --from-literal=secret-access-key=YOUR_SECRET_ACCESS_KEY \
     -n YOUR_NAMESPACE

2. Configure the Helm chart to reference the secret:

   yaml

   extraEnvironmentVariables:
     - name: AWS_ACCESS_KEY_ID
       valueFrom:
         secretKeyRef:
           name: aws-credentials
           key: access-key-id
     - name: AWS_SECRET_ACCESS_KEY
       valueFrom:
         secretKeyRef:
           name: aws-credentials
           key: secret-access-key
     - name: AWS_REGION
       value: us-east-1

Assume an IAM role with AWS Bedrock

By default, the AI Gateway calls Bedrock with the credentials it already has (for example, the IRSA role or environment variables). To call Bedrock with a different role (for example, a role in another AWS account), provide an IAM role for each self-hosted model. The AI Gateway uses its own credentials to assume that role through the AWS Security Token Service (STS) AssumeRole action. The AI Gateway then uses the temporary credentials to call Bedrock.

Prerequisites:

• Administrator access.
• The AI Gateway authenticated with AWS. The AWS identity the AI Gateway runs as can call sts:AssumeRole on the role.

To assume an IAM role when you call Bedrock:

1. In the target AWS account, create an IAM role with the following trust relationship policy. Replace:

   • AI_GATEWAY_PRINCIPAL_ARN with the ARN of the AWS identity the AI Gateway runs as (for example, the arn:aws:iam::AI_GATEWAY_ACCOUNT_ID:role/eks-ai-gateway-bedrock IRSA role).
   • GITLAB_INSTANCE_ID with the ID of your GitLab instance as it appears in the IAM role to assume text box.
   json

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Effect": "Allow",
         "Principal": {
           "AWS": "AI_GATEWAY_PRINCIPAL_ARN"
         },
         "Action": "sts:AssumeRole",
         "Condition": {
           "StringEquals": {
             "sts:ExternalId": "GITLAB_INSTANCE_ID"
           }
         }
       }
     ]
   }

   The AI Gateway sends the instance ID as the STS external ID when it assumes the role. The condition prevents another AI Gateway customer who knows the role ARN from using the AI Gateway’s identity to assume your role (the confused deputy problem). The instance ID is not a secret.

2. To invoke Bedrock models, attach a permission policy to the role. You can scope Resource to specific models:

   json

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Effect": "Allow",
         "Action": [
           "bedrock:InvokeModel",
           "bedrock:InvokeModelWithResponseStream"
         ],
         "Resource": "arn:aws:bedrock:*::foundation-model/*"
       }
     ]
   }

3. If the AI Gateway runs in a different account than the role, also allow the AI Gateway identity to call sts:AssumeRole on the role with an identity-based policy:

   json

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Effect": "Allow",
         "Action": "sts:AssumeRole",
         "Resource": "arn:aws:iam::ROLE_ACCOUNT_ID:role/YOUR_ROLE_NAME"
       }
     ]
   }

4. In the upper-right corner, select Admin.

5. In the left sidebar, select GitLab Duo.

6. Select Configure models for GitLab Duo.

7. For the Amazon Bedrock platform, add or edit a self-hosted model.

8. Optional. In the IAM role to assume text box, enter the ARN of the role (for example, arn:aws:iam::123456789012:role/YOUR_ROLE_NAME). When you leave it empty, the AI Gateway uses its own credentials.

9. Optional. Select Test connection.

10. Select Add self-hosted model.

An IAM role and an API key (bearer token) are mutually exclusive. GitLab does not accept an IAM role for a model that has an API key, and the AI Gateway rejects a request that contains both.

AWS Bedrock API keys

To use AWS Bedrock API keys as an alternative to IAM credentials:

1. Create a Bedrock API key

2. Create a Kubernetes secret with the API key:

   shell

   kubectl create secret generic bedrock-api-key \
     --from-literal=token=YOUR_BEDROCK_API_KEY \
     -n YOUR_NAMESPACE

3. Configure the AI Gateway (add to your values.yaml):

   yaml

   extraEnvironmentVariables:
     - name: AWS_BEARER_TOKEN_BEDROCK
       valueFrom:
         secretKeyRef:
           name: bedrock-api-key
           key: token
     - name: AWS_REGION
       value: us-east-1

Private VPC endpoints

To use a private Bedrock endpoint in a VPC, set the AWS_BEDROCK_RUNTIME_ENDPOINT environment variable.

For Helm deployments:

extraEnvironmentVariables:
  - name: AWS_BEDROCK_RUNTIME_ENDPOINT
    value: https://bedrock-runtime.us-east-1.amazonaws.com

For Docker deployments:

docker run -d \
  -e AWS_BEDROCK_RUNTIME_ENDPOINT=https://bedrock-runtime.us-east-1.amazonaws.com \
  -e AWS_REGION=us-east-1 \
  # ... other configuration

For VPC endpoints, use the format: https://vpce-{vpc-endpoint-id}-{service-name}.{region}.vpce.amazonaws.com

Bedrock guardrails

You can use Amazon Bedrock Guardrails to provide safety and privacy controls for your Bedrock model requests.

To apply these guardrails, set the AIGW_BEDROCK_GUARDRAIL_CONFIG environment variable value to a JSON object with the following fields:

For a Helm deployment, set the environment variable as follows:

extraEnvironmentVariables:
  - name: AIGW_BEDROCK_GUARDRAIL_CONFIG
    value: '{"guardrailIdentifier": "<guardrail_id>", "guardrailVersion": "1", "trace": "disabled"}'

For a Docker deployment:

docker run -d \
  -e AIGW_BEDROCK_GUARDRAIL_CONFIG='{"guardrailIdentifier": "<guardrail_id>", "guardrailVersion": "1", "trace": "disabled"}' \
  # ... other configuration

For more information, see Amazon Bedrock Guardrails.

Configure Amazon Bedrock Mantle

Amazon Bedrock Mantle is an OpenAI API-compatible inference service from AWS. Configure Amazon Bedrock Mantle with the API platform like other OpenAI-compatible endpoints.

The following models are validated and supported on Amazon Bedrock Mantle:

The AI Gateway sends requests for GPT-5 and later models to the Responses API of Amazon Bedrock Mantle. Use the same bedrock_mantle/ prefix for all models.

Anthropic models are not supported on Amazon Bedrock Mantle. Amazon Bedrock Mantle serves these models through the Anthropic Messages API only, and the AI Gateway does not use that API for this platform. To use Anthropic models on AWS, add an Amazon Bedrock model instead.

To configure an Amazon Bedrock Mantle model, add a self-hosted model with the following values:

• For Model family, select the family that matches the model. For OpenAI models, select GPT.
• For Endpoint, enter the regional endpoint in the form https://bedrock-mantle.<region>.api.aws/v1 (for example, https://bedrock-mantle.us-east-1.api.aws/v1).
• For Model identifier, use the bedrock_mantle/ prefix (for example, bedrock_mantle/openai.gpt-oss-120b).
• For API key, enter an Amazon Bedrock Mantle API key. For more information, see AWS Bedrock API keys.

Configure Azure OpenAI

Azure OpenAI serves models through the Chat Completions API and the Responses API. Most models support the Chat Completions API. Some newer models are available only through the Responses API.

To configure an Azure OpenAI model, add a self-hosted model with the following values:

• For Model family, select the family that matches the model. For GPT-5, select GPT.
• For Endpoint, enter the endpoint of your Azure OpenAI resource in the https://<resource-name>.openai.azure.com format. Do not add a path or an API version.
• For Model identifier, use the azure/ prefix followed by the name of your deployment (for example, azure/gpt-4o). This identifier sends requests to the Chat Completions API. To send requests to the Responses API instead, see use the Responses API.
• For API key, enter an API key for your Azure OpenAI resource.

Use the Responses API

For a deployment that is available only through the Responses API, use the azure/responses/ prefix. For Model identifier, enter the prefix followed by the name of your deployment (for example, azure/responses/gpt-5).

The AI Gateway sends requests to the /openai/responses path of your resource. The AI Gateway converts each request to the Responses API format and converts the response back. The conversion supports streamed responses and tool calls. You do not need a proxy to translate between the two APIs.

Configure authentication with Gemini Enterprise Agent Platform

To use models from Gemini Enterprise Agent Platform, you must authenticate your AI Gateway instance. You can use any of the following mechanisms:

• Export the environment variables when starting the Docker container. To do this, set the following environment variables when running the AI Gateway container:

  shell

  GOOGLE_APPLICATION_CREDENTIALS=/path/to/application_default_credentials.json
  VERTEXAI_PROJECT=<gcp-project-id>
  VERTEXAI_LOCATION=global # or any specific location, e.g., "europe-west1"

• Run the AI Gateway container on Google Cloud Run and use the Cloud Run service account for Gemini Enterprise Agent Platform access.

Related topics

• Supported models and hardware requirements documentation.
• Amazon Bedrock supported foundation models
• AWS IAM best practices
• Amazon Bedrock Security
• For configuration information, see the following documentation:
  • Anthropic API overview
  • OpenAI API overview
  • Working with Azure OpenAI models