Provision GitLab Cloud Native Hybrid on AWS EKS

GitLab “Cloud Native Hybrid” is a hybrid of the cloud native technology Kubernetes (EKS) and EC2. While as much of the GitLab application as possible runs in Kubernetes or on AWS services (PaaS), the GitLab service Gitaly must still be run on EC2. Gitaly is a layer designed to overcome limitations of the Git binaries in a horizontally scaled architecture. You can read more here about why Gitaly was built and why the limitations of Git mean that it must currently run on instance compute in Git Characteristics That Make Horizontal Scaling Difficult.

Amazon provides a managed Kubernetes service offering known as Amazon Elastic Kubernetes Service (EKS).

Tested AWS Bill of Materials by reference architecture size

GitLab Cloud Native Hybrid Ref Arch GitLab Baseline Perf Test Results Omnibus on Instances AWS Bill of Materials (BOM) for CNH AWS Build Performance Testing Results for CNH CNH Cost Estimate 3 AZs*
2K Omnibus 2K Baseline 2K Cloud Native Hybrid on EKS GPT Test Results 1 YR Ec2 Compute Savings + 1 YR RDS & ElastiCache RIs
(2 AZ Cost Estimate is in BOM Below)
3K 3k Baseline 3K Cloud Native Hybrid on EKS 3K Full Fixed Scale GPT Test Results

3K Elastic Auto Scale GPT Test Results
1 YR Ec2 Compute Savings + 1 YR RDS & ElastiCache RIs
(2 AZ Cost Estimate is in BOM Below)
5K 5k Baseline 5K Cloud Native Hybrid on EKS 5K Full Fixed Scale GPT Test Results

5K AutoScale from 25% GPT Test Results
1 YR Ec2 Compute Savings + 1 YR RDS & ElastiCache RIs
10K 10k Baseline 10K Cloud Native Hybrid on EKS 10K Full Fixed Scale GPT Test Results

10K Elastic Auto Scale GPT Test Results
10K 1 YR Ec2 Compute Savings + 1 YR RDS & ElastiCache RIs
50K 50k Baseline 50K Cloud Native Hybrid on EKS 50K Full Fixed Scale GPT Test Results

10K Elastic Auto Scale GPT Test Results
50K 1 YR Ec2 Compute Savings + 1 YR RDS & ElastiCache RIs

*Cost calculations for actual implementations are a rough guideline with the following considerations:

  • Actual choices about instance types should be based on GPT testing of your configuration.
  • The first year of actual usage will reveal potential savings due to lower than expected usage, especially for ramping migrations where the full loading takes months, so be careful not to commit to savings plans too early or for too long.
  • The cost estimates assume full scale of the Kubernetes cluster nodes 24 x 7 x 365. Savings due to ‘idling scale-in’ are not considered because they are highly dependent on the usage patterns of the specific implementation.
  • Costs such as GitLab Runners, data egress and storage costs are not included as they are very dependent on the configuration of a specific implementation and on development behaviors (for example, frequency of committing or frequency of builds).
  • These estimates will change over time as GitLab tests and optimizes compute choices.

Available Infrastructure as Code for GitLab Cloud Native Hybrid

The AWS Quick Start for GitLab Cloud Native Hybrid on EKS is developed by AWS, GitLab, and the community that contributes to AWS Quick Starts, whether directly to the GitLab Quick Start or to the underlying Quick Start dependencies GitLab inherits (for example, EKS Quick Start).

note
This automation is in Open Beta. GitLab is working with AWS on resolving the outstanding issues before it is fully released. You can subscribe to this issue to be notified of progress and release announcements: AWS Quick Start for GitLab Cloud Native Hybrid on EKS Status: Beta.

The Beta version deploys Aurora PostgreSQL, but the release version will deploy Amazon RDS PostgreSQL due to known issues with Aurora. All performance testing results will also be redone after this change has been made.

The GitLab Environment Toolkit (GET) is an effort made by GitLab to create a multi-cloud, multi-GitLab (Omnibus + Cloud Native Hybrid) toolkit to provision GitLab. GET is developed by GitLab developers and is open to community contributions. It is helpful to review the GitLab Environment Toolkit (GET) Issues to understand if any of them may affect your provisioning plans.

  AWS Quick Start for GitLab Cloud Native Hybrid on EKS GitLab Environment Toolkit (GET)
Overview and Vision AWS Quick Start GitLab Environment Toolkit
Licensing Open Source (Apache 2.0) GitLab Enterprise Edition license (GitLab Premium tier)
GitLab Support GitLab Beta Support GitLab GA Support
GitLab Reference Architecture Compliant Yes Yes
GitLab Performance Tool (GPT) Tested Yes Yes
Amazon Well Architected Compliant Yes
(via Quick Start program)
Critical portions
reviewed by AWS
Target Cloud Platforms AWS AWS, Google, Azure
IaC Languages CloudFormation (Quick Starts) Terraform, Ansible
Community Contributions and Participation (EcoSystem) GitLab QSG: Getting Started
For QSG Dependencies (for example, EKS): Substantial
Getting Started
Compatible with AWS Meta-Automation Services (via CloudFormation) - AWS Service Catalog (Direct Import)
- ServiceNow via an AWS Service Catalog Connector
- Jira Service Manager via an AWS Service Catalog Connector
- AWS Control Tower (Integration)
- Quick Starts
- AWS SaaS Factory
No
Results in a Ready-to-Use instance Yes Manual Actions or
Supplemental IaC Required
Configuration Features    
Can deploy Omnibus GitLab (non-Kubernetes) No Yes
Results in a self-healing Gitaly Cluster configuration Yes No
Complete Internal Encryption 85%, Targeting 100% Manual
AWS GovCloud Support Yes TBD

Two and Three Zone High Availability

While GitLab Reference Architectures generally encourage three zone redundancy, AWS Quick Starts and AWS Well Architected consider two zone redundancy as AWS Well Architected. Individual implementations should weigh the costs of two and three zone configurations against their own high availability requirements for a final configuration.

Gitaly Cluster uses a consistency voting system to implement strong consistency between synchronized nodes. Regardless of the number of availability zones implemented, there will always need to be a minimum of three Gitaly and three Praefect nodes in the cluster to avoid voting stalemates cause by an even number of nodes.

Streamlined Performance Testing of AWS Quick Start Prepared GitLab Instances

A set of performance testing instructions have been abbreviated for testing a GitLab instance prepared using the AWS Quick Start for GitLab Cloud Native Hybrid on EKS. They assume zero familiarity with GitLab Performance Tool. They can be accessed here: Performance Testing an Instance Prepared using AWS Quick Start for GitLab Cloud Native Hybrid on EKS.

AWS GovCloud Support for AWS Quick Start for GitLab CNH on EKS

The AWS Quick Start for GitLab Cloud Native Hybrid on EKS has been tested with GovCloud and works with the following restrictions and understandings.

  • GovCloud does not have public Route53 hosted zones, so you must set the following parameters:

    CloudFormation Quick Start form field CloudFormation Parameter Setting
    Create Route 53 hosted zone CreatedHostedZone No
    Request AWS Certificate Manager SSL certificate CreateSslCertificate No
  • The Quick Start creates public load balancer IPs, so that you can easily configure your local hosts file to get to the GUI for GitLab when deploying tests. However, you may need to manually alter this if public load balancers are not part of your provisioning plan. We are planning to make non-public load balancers a configuration option issue link: Short Term: Documentation and/or Automation for private GitLab instance with no internet Ingress
  • As of 2021-08-19, AWS GovCloud has Graviton instances for Amazon RDS PostgreSQL available, but does not for ElastiCache Redis.
  • It is challenging to get the Quick Start template to load in GovCloud from the Standard Quick Start URL, so the generic ones are provided here:

AWS PaaS qualified for all GitLab implementations

For both Omnibus GitLab or Cloud Native Hybrid implementations, the following GitLab Service roles can be performed by AWS Services (PaaS). Any PaaS solutions that require preconfigured sizing based on the scale of your instance will also be listed in the per-instance size Bill of Materials lists. Those PaaS that do not require specific sizing, are not repeated in the BOM lists (for example, AWS Certification Manager).

These services have been tested with GitLab.

Some services, such as log aggregation, outbound email are not specified by GitLab, but where provided are noted.

GitLab Services AWS PaaS (Tested) Provided by AWS Cloud
Native Hybrid Quick Start
Tested PaaS Mentioned in Reference Architectures    
PostgreSQL Database Amazon RDS PostgreSQL Yes.
Redis Caching Redis ElastiCache Yes.
Gitaly Cluster (Git Repository Storage)
(Including Praefect and PostgreSQL)
ASG and Instances Yes - ASG and Instances
Note: Gitaly cannot be put into a Kubernetes Cluster.
All GitLab storages besides Git Repository Storage
(Includes Git-LFS which is S3 Compatible)
AWS S3 Yes
     
Tested PaaS for Supplemental Services    
Front End Load Balancing AWS ELB Yes
Internal Load Balancing AWS ELB Yes
Outbound Email Services AWS Simple Email Service (SES) Yes
Certificate Authority and Management