- Install large language model serving infrastructure
- Configure your GitLab instance
- Install the GitLab AI Gateway
- Upgrade the AI Gateway Docker image
- Alternative installation methods
- Troubleshooting
Set up your self-hosted model infrastructure
-
Introduced in GitLab 17.1 with a flag named
ai_custom_model
. Disabled by default.
By self-hosting the model, AI Gateway, and GitLab instance, there are no calls to external architecture, ensuring maximum levels of security.
To set up your self-hosted model infrastructure:
- Install the large language model (LLM) serving infrastructure.
- Configure your GitLab instance.
- Install the GitLab AI Gateway.
For an installation video guide, see Self-Hosted Models Deployment.
For an installation video guide in French, see Self-Hosted Models Deployment (French Language version).
Install large language model serving infrastructure
Install one of the following GitLab-approved LLM models:
Model | Code completion | Code generation | GitLab Duo Chat |
---|---|---|---|
CodeGemma 2b | Yes | No | No |
CodeGemma 7b-it (Instruction) | No | Yes | No |
CodeGemma 7b-code (Code) | Yes | No | No |
Code-Llama 13b-code | Yes | No | No |
Code-Llama 13b | No | Yes | No |
Codestral 22B (see setup instructions) | Yes | Yes | No |
Mistral 7B | No | Yes | Yes |
Mixtral 8x22B | No | Yes | Yes |
Mixtral 8x7B | No | Yes | Yes |
DeepSeek Coder 33b Instruct | Yes | Yes | No |
DeepSeek Coder 33b Base | Yes | No | No |
Use a serving architecture
To host your models, you should use:
- For non-cloud on-premise deployments, vLLM.
- For cloud deployments, AWS Bedrock as a cloud provider.
Configure your GitLab instance
Prerequisites:
- Upgrade to the latest version of GitLab.
-
The GitLab instance must be able to access the AI Gateway.
-
Where your GitLab instance is installed, update the
/etc/gitlab/gitlab.rb
file.sudo vim /etc/gitlab/gitlab.rb
-
Add and save the following environment variables.
gitlab_rails['env'] = { 'GITLAB_LICENSE_MODE' => 'production', 'CUSTOMER_PORTAL_URL' => 'https://customers.gitlab.com', 'AI_GATEWAY_URL' => '<path_to_their_ai_gateway>' }
-
Run reconfigure:
sudo gitlab-ctl reconfigure
-
-
sudo gitlab-rails console
In the console, enable feature flags:
Feature.enable(:self_hosted_models_beta_ended) Feature.enable(:ai_custom_model)
Exit the Rails console.
Install the GitLab AI Gateway
Install by using Docker
Prerequisites:
- Install a Docker container engine, such as Docker.
- Use a valid hostname accessible within your network. Do not use
localhost
.
The GitLab AI Gateway Docker image contains all necessary code and dependencies in a single container.
Find the AI Gateway release
Find the GitLab official Docker image at:
- AI Gateway Docker image on Container Registry.
- AI Gateway Docker image on DockerHub.
- Release process for self-hosted AI Gateway.
Use the image tag that corresponds to your GitLab version. For example, if the
GitLab version is v17.5.0
, use self-hosted-v17.5.0-ee
tag.
Set up the volumes location
Create a directory where the logs will reside on the Docker host. It can be under
your user’s home directory (for example ~/gitlab-agw
), or in a directory like
/srv/gitlab-agw
. To create that directory, run:
sudo mkdir -p /srv/gitlab-agw
If you’re running Docker with a user other than root
, ensure appropriate
permissions have been granted to that directory.
Start a container from the image
For Docker images with version self-hosted-17.4.0-ee
and later, run the following:
docker run -e AIGW_GITLAB_URL=<your_gitlab_instance> \
-e AIGW_GITLAB_API_URL=https://<your_gitlab_domain>/api/v4/ \
<image>
From the container host, accessing http://localhost:5052/docs
should open the AI Gateway API documentation.
Install by using the AI Gateway Helm chart
Prerequisites:
- You must have a:
- Domain you own, that you can add a DNS record to.
- Kubernetes cluster.
- Working installation of
kubectl
. - Working installation of Helm, version v3.11.0 or later.
For more information, see Test the GitLab chart on GKE or EKS.
Add the AI Gateway Helm repository
Add the AI Gateway Helm repository to Helm’s configuration:
helm repo add ai-gateway \
https://gitlab.com/api/v4/projects/gitlab-org%2fcharts%2fai-gateway-helm-chart/packages/helm/devel
Install the AI Gateway
-
Create the
ai-gateway
namespace:kubectl create namespace ai-gateway
- Generate the certificate for the domain where you plan to expose the AI Gateway.
-
Create the TLS secret in the previously created namespace:
kubectl -n ai-gateway create secret tls ai-gateway-tls --cert="<path_to_cert>" --key="<path_to_cert_key>"
-
For the AI Gateway to access the API, it must know where the GitLab instance is located. To do this, set the
gitlab.url
andgitlab.apiUrl
together with theingress.hosts
andingress.tls
values as follows:helm repo add ai-gateway \ https://gitlab.com/api/v4/projects/gitlab-org%2fcharts%2fai-gateway-helm-chart/packages/helm/devel helm repo update helm upgrade --install ai-gateway \ ai-gateway/ai-gateway \ --version 0.1.1 \ --namespace=ai-gateway \ --set="image.tag=<ai-gateway-image>" \ --set="gitlab.url=https://<your_gitlab_domain>" \ --set="gitlab.apiUrl=https://<your_gitlab_domain>/api/v4/" \ --set "ingress.enabled=true" \ --set "ingress.hosts[0].host=<your_gateway_domain>" \ --set "ingress.hosts[0].paths[0].path=/" \ --set "ingress.hosts[0].paths[0].pathType=ImplementationSpecific" \ --set "ingress.tls[0].secretName=ai-gateway-tls" \ --set "ingress.tls[0].hosts[0]=<your_gateway_domain>" \ --set="ingress.className=nginx" \ --timeout=300s --wait --wait-for-jobs
This step can take will take a few seconds in order for all resources to be allocated and the AI Gateway to start.
Wait for your pods to get up and running:
kubectl wait pod \
--all \
--for=condition=Ready \
--namespace=ai-gateway \
--timeout=300s
When your pods are up and running, you can set up your IP ingresses and DNS records.
Configure the GitLab instance
Configure the GitLab instance.
With those steps completed, your Helm chart installation is complete.
Upgrade the AI Gateway Docker image
To upgrade the AI Gateway, download the newest Docker image tag.
-
Stop the running container:
sudo docker stop gitlab-aigw
-
Remove the existing container:
sudo docker rm gitlab-aigw
-
Pull and run the new image.
-
Ensure that the environment variables are all set correctly.
Alternative installation methods
For information on alternative ways to install the AI Gateway, see issue 463773.
Troubleshooting
First, run the debugging scripts to verify your self-hosted model setup.
For more information on other actions to take, see the troubleshooting documentation.