Supported self-hosted models and hardware requirements

Tier: Ultimate with GitLab Duo Enterprise - Start a trial
Offering: Self-managed
Status: Beta

History

The following table shows the supported models along with their specific features and hardware requirements to help you select the model that best fits your infrastructure needs for optimal performance.

Approved LLMs

Install one of the following GitLab-approved LLM models:

Model family	Model	Code completion	Code generation	GitLab Duo Chat
Mistral Codestral	Codestral 22B	Yes	Yes	No
Mistral	Mistral 7B	Yes	No	No
Mistral	Mistral 7B-it	Yes	Yes	Yes
Mistral	Mixtral 8x7B-it	Yes	Yes	Yes
Mistral	Mixtral 8x22B-it	Yes	Yes	Yes
Claude 3	Claude 3.5 Sonnet	Yes	Yes	Yes
GPT	GPT-3.5-Turbo	Yes	Yes	No
GPT	GPT-4 Turbo	Yes	Yes	No
GPT	GPT-4o	Yes	Yes	No
GPT	GPT-4o-mini	Yes	Yes	No

The following models are under evaluation, and support is limited:

Model family	Model	Code completion	Code generation	GitLab Duo Chat
CodeGemma	CodeGemma 2b	Yes	No	No
CodeGemma	CodeGemma 7b-it	No	Yes	No
CodeGemma	CodeGemma 7b-code	Yes	No	No
Code Llama	Code-Llama 13b-code	Yes	No	No
Code Llama	Code-Llama 13b	No	Yes	No
DeepSeek Coder	DeepSeek Coder 33b Instruct	Yes	Yes	No
DeepSeek Coder	DeepSeek Coder 33b Base	Yes	No	No

Hardware Requirements

For optimal performance, the following hardware specifications are recommended as baselines for hosting these models. Hosting requirements may vary depending model to model, so we recommend checking model vendor documentation as well:

CPU: Minimum 8 cores (16 threads recommended).
RAM: At least 32 GB (64 GB or more recommended for larger models).
GPU:
- Minimum: 2x NVIDIA A100 or equivalent for optimal inference performance.
- Note: For running Mixtral 8x22B and Mixtral 8x22B-it, it is recommended to use 8x NVIDIA A100 GPUs.
Storage: SSD with sufficient space for model weights and data.

Docs

Edit this page to fix an error or add an improvement in a merge request.

Create an issue to suggest an improvement to this page.

Product

Create an issue if there's something you don't like about this feature.

Propose functionality by submitting a feature request.

Feature availability and product trials

View pricing to see all GitLab tiers and features, or to upgrade.

Try GitLab for free with access to all features for 30 days.

Get help

If you didn't find what you were looking for, search the docs.

If you want help with something specific and could use community support, post on the GitLab forum.

For problems setting up or using this feature (depending on your GitLab subscription).

Request support