Reference architectures

The GitLab Reference Architectures have been designed and tested by the GitLab Quality and Support teams to provide recommended deployments at scale.

Available reference architectures

Depending on your workflow, the following recommended reference architectures may need to be adapted accordingly. Your workload is influenced by factors including how active your users are, how much automation you use, mirroring, and repository/change size. Additionally, the displayed memory values are provided by GCP machine types. For different cloud vendors, attempt to select options that best match the provided architecture.

GitLab package (Omnibus)

The following reference architectures, where the GitLab package is used, are available:

Cloud native hybrid

The following Cloud Native Hybrid reference architectures, where select recommended components can be run in Kubernetes, are available:

A GitLab Premium or Ultimate license is required to get assistance from Support with troubleshooting the 2,000 users and higher reference architectures. Read more about our definition of scaled architectures.

Deciding which architecture to use

The Reference Architectures are designed to strike a balance between two important factors–performance and resilience.

While they are designed to make it easier to set up GitLab at scale, it can still be a challenge to know which one will meet your requirements.

As a general guide, the more performant and/or resilient you want your environment to be, the more involved it will be.

This section explains the designs you can choose from. It begins with the least complexity, goes to the most, and ends with a decision tree.

Backups

For environments serving 2,000 or fewer users we generally recommend that an automated backup strategy is used instead of HA.

Backups can provide a good level of RPO / RTO while avoiding the complexities that come with HA.

High Availability (HA)

High Availability ensures every component in the GitLab setup can handle failures through various mechanisms. To achieve this however is involved, and the environments required can be sizable.

For environments serving 3,000 or more users we generally recommend that a HA strategy is used as at this level outages will have a bigger impact against more users. All the architectures in this range have HA built in by design for this reason.

For users who still need to have HA for a lower number of users this can also be achieved with an adjusted 3K architecture as detailed here.

Do you need High Availability (HA)?

As mentioned above, achieving HA does come at a cost. The environment’s required are sizable as each component needs to be multiplied, which comes with additional actual and maintenance costs.

For a lot of our customers with fewer than 3,000 users, we’ve found a backup strategy is sufficient and even preferable. While this does have a slower recovery time, it also means you have a much smaller architecture and less maintenance costs as a result.

In general then, we’d only recommend you employ HA in the following scenarios:

  • When you have 3,000 or more users.
  • When GitLab being down would critically impact your workflow.

Zero Downtime Upgrades

Zero Downtime Upgrades are available for standard Reference Architecture environments with HA (Cloud Native Hybrid is not supported at this time). This allows for an environment to stay up during an upgrade, but the process is more involved as a result and has some limitations as detailed in the documentation.

When going through this process it’s worth noting that there may still be brief moments of downtime when the HA mechanisms tale effect.

In most cases the downtime required for doing an upgrade in general shouldn’t be substantial, so this is only recommended if it’s a key requirement for you.

Cloud Native Hybrid (Kubernetes HA)

As an additional layer of HA resilience you can deploy select components in Kubernetes, known as a Cloud Native Hybrid Reference Architecture.

Note that this is an alternative and more advanced setup compared to a standard Reference Architecture. Running services in Kubernetes is well known to be complex. This setup is only recommended if you have strong working knowledge and experience in Kubernetes.

GitLab Geo (Cross Regional Distribution / Disaster Recovery)

With GitLab Geo you can have both distributed environments in different regions and a full Disaster Recovery (DR) setup in place. With this setup you would have 2 or more separate environments, with one being a primary that gets replicated to the others. In the rare event the primary site went down completely you could fail over to one of the other environments.

This is an advanced and involved setup and should only be undertaken if you have DR as a key requirement. Decisions then on how each environment are configured would also need to be taken, such as if each environment itself would be the full size and / or have HA.

Decision Tree

Below you can find the above guidance in the form of a decision tree. It’s recommended you read through the above guidance in full first before though.

%%{init: { 'theme': 'base' } }%% graph TD L1A(<b>What Reference Architecture should I use?</b>) --> L2A(More than 3000 users?) L2A -->|No| L3A("<a href=#do-you-need-high-availability-ha>Do you need HA?</a><br>(or Zero-Downtime Upgrades)") --> |Yes| L4A><b>Recommendation</b><br><br>3K architecture with HA<br>including supported modifications] L3A -->|No| L4B><b>Recommendation</b><br><br>Architecture closest to user<br>count with Backups] L2A -->|Yes| L3B[Do you have experience with<br/>and want additional resilience<br/>with select components in Kubernetes?] L3B -->|No| L4C><b>Recommendation</b><br><br>Architecture closest to user<br>count with HA] L3B -->|Yes| L4D><b>Recommendation</b><br><br>Cloud Native Hybrid architecture<br>closest to user count] L5A("<a href=#gitlab-geo-cross-regional-distribution-disaster-recovery>Do you need cross regional distribution or disaster recovery?"</a>) --> |Yes| L6A><b>Additional Recommendation</b><br><br> GitLab Geo] L4A -.- L5A L4B -.- L5A L4C -.- L5A L4D -.- L5A classDef default fill:#FCA326 linkStyle default fill:none,stroke:#7759C2
note
The following lists are non-exhaustive. Generally, other cloud providers not listed here likely work with the same specs, but this hasn’t been validated. Additionally, when it comes to other cloud provider services not listed here, it’s advised to be cautious as each implementation can be notably different and should be tested thoroughly before production use.

Through testing and real life usage, the Reference Architectures are validated and supported on the following cloud providers:

Reference ArchitectureGCPAWSAzureBare Metal
Omnibus
Cloud Native Hybrid

Additionally, the following cloud provider services are validated and supported for use as part of the Reference Architectures:

Cloud ServiceGCPAWSBare Metal
Object Storage✅   Cloud Storage ✅   S3 ✅   MinIO
Database✅   Cloud SQL ✅   RDS
Redis✅   ElastiCache

The following specific cloud provider services have been found to have issues in terms of either functionality or performance. As such, they either have caveats that should be considered or are not recommended:

  • Google AlloyDB and Amazon RDS Multi-AZ DB clusters have not been tested and are not recommended. Both solutions are specifically not expected to work with GitLab Geo.
  • Amazon Aurora is incompatible. See 14.4.0 for more details.
  • Azure Blob Storage has been found to have performance limits that can impact production use at certain times. For larger Reference Architectures the service may not be sufficient for production use and an alternative is recommended for use instead.
  • Azure Database for PostgreSQL Server (Single / Flexible) is not recommended for use due to notable performance issues or missing functionality.
note
As a general rule we unfortunately don’t recommend Azure Services at this time. If required, we advise thorough testing is done at your intended scale over a sustained period to validate if the service is suitable.

Validation and test results

The Quality Engineering team does regular smoke and performance tests for the reference architectures to ensure they remain compliant.

Why we perform the tests

The Quality Department has a focus on measuring and improving the performance of GitLab, as well as creating and validating reference architectures that self-managed customers can rely on as performant configurations.

For more information, see our handbook page.

How we perform the tests

Testing occurs against all reference architectures and cloud providers in an automated and ad-hoc fashion. This is done by two tools:

Network latency on the test environments between components on all Cloud Providers were measured at <5ms. Note that this is shared as an observation and not as an implicit recommendation.

We aim to have a “test smart” approach where architectures tested have a good range that can also apply to others. Testing focuses on 10k Omnibus on GCP as the testing has shown this is a good bellwether for the other architectures and cloud providers as well as Cloud Native Hybrids.

The Standard Reference Architectures are designed to be platform-agnostic, with everything being run on VMs via Omnibus GitLab. While testing occurs primarily on GCP, ad-hoc testing has shown that they perform similarly on equivalently specced hardware on other Cloud Providers or if run on premises (bare-metal).

Testing on these reference architectures is performed with the GitLab Performance Tool at specific coded workloads, and the throughputs used for testing are calculated based on sample customer data. Select the reference architecture that matches your scale.

Each endpoint type is tested with the following number of requests per second (RPS) per 1,000 users:

  • API: 20 RPS
  • Web: 2 RPS
  • Git (Pull): 2 RPS
  • Git (Push): 0.4 RPS (rounded to the nearest integer)

How to interpret the results

Testing is done publicly, and all results are shared.

The following table details the testing done against the reference architectures along with the frequency and results. Additional testing is continuously evaluated, and the table is updated accordingly.

Reference
Architecture
GCP (* also proxy for Bare-Metal)AWSAzure
OmnibusCloud Native HybridOmnibusCloud Native HybridOmnibus
1kWeekly
2kWeekly
3kWeeklyWeekly
5kWeekly
10kDailyWeeklyWeeklyWeeklyAd-Hoc
25kWeeklyAd-Hoc
50kWeeklyAd-Hoc (inc Cloud Services)

Cost to run

The following table details the cost to run the different reference architectures across GCP, AWS, and Azure. Bare-metal costs are not included here as it varies widely depending on each customer configuration.

Reference
Architecture
GCPAWSAzure
OmnibusCloud Native HybridOmnibusCloud Native HybridOmnibus
1kCalculated costCalculated costCalculated cost
2kCalculated costCalculated costCalculated cost
3kCalculated costCalculated costCalculated cost
5kCalculated costCalculated costCalculated cost
10kCalculated costCalculated costCalculated cost
25kCalculated costCalculated costCalculate cost
50kCalculated costCalculated costCalculated cost

Deviating from the suggested reference architectures

As a general guideline, the further away you move from the Reference Architectures, the harder it is to get support for it. With any deviation, you’re introducing a layer of complexity that adds challenges to finding out where potential issues might lie.

The reference architectures use the official GitLab Linux packages (Omnibus GitLab) or Helm Charts to install and configure the various components. The components are installed on separate machines (virtualized or bare metal), with machine hardware requirements listed in the “Configuration” column and equivalent VM standard sizes listed in GCP/AWS/Azure columns of each available reference architecture.

Running components on Docker (including Compose) with the same specs should be fine, as Docker is well known in terms of support. However, it is still an additional layer and may still add some support complexities, such as not being able to run strace easily in containers.

Other technologies, like Docker swarm are not officially supported, but can be implemented at your own risk. In that case, GitLab Support is not able to help you.