GitLab Architecture Overview

Software delivery

There are two software distributions of GitLab: the open source Community Edition (CE), and the open core Enterprise Edition (EE). GitLab is available under different subscriptions.

New versions of GitLab are released in stable branches and the master branch is for bleeding edge development.

For information, see the GitLab Release Process.

Both EE and CE require some add-on components called GitLab Shell and Gitaly. These components are available from the GitLab Shell and Gitaly repositories respectively. New versions are usually tags but staying on the master branch will give you the latest stable version. New releases are generally around the same time as GitLab CE releases with exception for informal security updates deemed critical.

Components

A typical install of GitLab will be on GNU/Linux. It uses NGINX or Apache as a web front end to proxypass the Unicorn web server. By default, communication between Unicorn and the front end is via a Unix domain socket but forwarding requests via TCP is also supported. The web front end accesses /home/git/gitlab/public bypassing the Unicorn server to serve static pages, uploads (e.g. avatar images or attachments), and precompiled assets. GitLab serves web pages and the GitLab API using the Unicorn web server. It uses Sidekiq as a job queue which, in turn, uses Redis as a non-persistent database backend for job information, meta data, and incoming jobs.

We also support deploying GitLab on Kubernetes using our GitLab Helm chart.

The GitLab web app uses PostgreSQL for persistent database information (e.g. users, permissions, issues, other meta data). GitLab stores the bare Git repositories it serves in /home/git/repositories by default. It also keeps default branch and hook information with the bare repository.

When serving repositories over HTTP/HTTPS GitLab utilizes the GitLab API to resolve authorization and access as well as serving Git objects.

The add-on component GitLab Shell serves repositories over SSH. It manages the SSH keys within /home/git/.ssh/authorized_keys which should not be manually edited. GitLab Shell accesses the bare repositories through Gitaly to serve Git objects and communicates with Redis to submit jobs to Sidekiq for GitLab to process. GitLab Shell queries the GitLab API to determine authorization and access.

Gitaly executes Git operations from GitLab Shell and the GitLab web app, and provides an API to the GitLab web app to get attributes from Git (e.g. title, branches, tags, other meta data), and to get blobs (e.g. diffs, commits, files).

You may also be interested in the production architecture of GitLab.com.

Simplified Component Overview

This is a simplified architecture diagram that can be used to understand GitLab’s architecture.

A complete architecture diagram is available in our component diagram below.

Simplified Component Overview

Component diagram

graph TB HTTP[HTTP/HTTPS] -- TCP 80, 443 --> NGINX[NGINX] SSH -- TCP 22 --> GitLabShell[GitLab Shell] SMTP[SMTP Gateway] Geo[GitLab Geo Node] -- TCP 22, 80, 443 --> NGINX GitLabShell --TCP 8080 -->Unicorn["Unicorn (GitLab Rails)"] GitLabShell --> Praefect GitLabShell --> Redis Unicorn --> PgBouncer[PgBouncer] Unicorn --> Redis Unicorn --> Praefect Sidekiq --> Redis Sidekiq --> PgBouncer Sidekiq --> Praefect GitLabWorkhorse[GitLab Workhorse] --> Unicorn GitLabWorkhorse --> Redis GitLabWorkhorse --> Praefect Praefect --> Gitaly NGINX --> GitLabWorkhorse NGINX -- TCP 8090 --> GitLabPages[GitLab Pages] NGINX --> Grafana[Grafana] Grafana -- TCP 9090 --> Prometheus[Prometheus] Prometheus -- TCP 80, 443 --> Unicorn RedisExporter[Redis Exporter] --> Redis Prometheus -- TCP 9121 --> RedisExporter PostgreSQLExporter[PostgreSQL Exporter] --> PostgreSQL PgBouncerExporter[PgBouncer Exporter] --> PgBouncer Prometheus -- TCP 9187 --> PostgreSQLExporter Prometheus -- TCP 9100 --> NodeExporter[Node Exporter] Prometheus -- TCP 9168 --> GitLabExporter[GitLab Exporter] Prometheus -- TCP 9127 --> PgBouncerExporter GitLabExporter --> PostgreSQL GitLabExporter --> GitLabShell GitLabExporter --> Sidekiq PgBouncer --> Consul PostgreSQL --> Consul PgBouncer --> PostgreSQL NGINX --> Registry Unicorn --> Registry NGINX --> Mattermost Mattermost --- Unicorn Prometheus --> Alertmanager Migrations --> PostgreSQL Runner -- TCP 443 --> NGINX Unicorn -- TCP 9200 --> Elasticsearch Sidekiq -- TCP 9200 --> Elasticsearch Sidekiq -- TCP 80, 443 --> Sentry Unicorn -- TCP 80, 443 --> Sentry Sidekiq -- UDP 6831 --> Jaeger Unicorn -- UDP 6831 --> Jaeger Gitaly -- UDP 6831 --> Jaeger GitLabShell -- UDP 6831 --> Jaeger GitLabWorkhorse -- UDP 6831 --> Jaeger Alertmanager -- TCP 25 --> SMTP Sidekiq -- TCP 25 --> SMTP Unicorn -- TCP 25 --> SMTP Unicorn -- TCP 369 --> LDAP Sidekiq -- TCP 369 --> LDAP Unicorn -- TCP 443 --> ObjectStorage["Object Storage"] Sidekiq -- TCP 443 --> ObjectStorage GitLabWorkhorse -- TCP 443 --> ObjectStorage Registry -- TCP 443 --> ObjectStorage Geo -- TCP 5432 --> PostgreSQL

Component legend

  • ✅ - Installed by default
  • ⚙ - Requires additional configuration, or GitLab Managed Apps
  • ⤓ - Manual installation required
  • ❌ - Not supported or no instructions available
  • N/A - Not applicable

Component statuses are linked to configuration documentation for each component.

Component list

ComponentDescriptionOmnibus GitLabGitLab chartMinikube MinimalGitLab.comSourceGDKCE/EE
NGINXRoutes requests to appropriate components, terminates SSL✅✅⚙✅⤓❌CE & EE
Unicorn (GitLab Rails)Handles requests for the web interface and API✅✅✅✅⚙✅CE & EE
SidekiqBackground jobs processor✅✅✅✅✅✅CE & EE
GitalyGit RPC service for handling all Git calls made by GitLab✅✅✅✅⚙✅CE & EE
PraefectA transparent proxy between any Git client and Gitaly storage nodes.✅❌❌✅⚙✅CE & EE
GitLab WorkhorseSmart reverse proxy, handles large HTTP requests✅✅✅✅⚙✅CE & EE
GitLab ShellHandles git over SSH sessions✅✅✅✅⚙✅CE & EE
GitLab PagesHosts static websites⚙❌❌✅⚙⚙CE & EE
RegistryContainer registry, allows pushing and pulling of images⚙✅✅✅⤓⚙CE & EE
RedisCaching service✅✅✅✅⤓✅CE & EE
PostgreSQLDatabase✅✅✅✅⤓✅CE & EE
PgBouncerDatabase connection pooling, failover⚙❌❌✅❌❌EE Only
ConsulDatabase node discovery, failover⚙❌❌✅❌❌EE Only
GitLab self-monitoring: PrometheusTime-series database, metrics collection, and query service✅✅⚙✅❌❌CE & EE
GitLab self-monitoring: AlertmanagerDeduplicates, groups, and routes alerts from Prometheus⚙✅⚙✅❌❌CE & EE
GitLab self-monitoring: GrafanaMetrics dashboard✅⤓⤓✅❌❌CE & EE
GitLab self-monitoring: SentryTrack errors generated by the GitLab instance⤓❌❌✅⤓⤓CE & EE
GitLab self-monitoring: JaegerView traces generated by the GitLab instance❌❌❌❌⤓⚙CE & EE
Redis ExporterPrometheus endpoint with Redis metrics✅✅✅✅❌❌CE & EE
PostgreSQL ExporterPrometheus endpoint with PostgreSQL metrics✅✅✅✅❌❌CE & EE
PgBouncer ExporterPrometheus endpoint with PgBouncer metrics⚙❌❌✅❌❌CE & EE
GitLab ExporterGenerates a variety of GitLab metrics✅✅✅✅❌❌CE & EE
Node ExporterPrometheus endpoint with system metrics✅N/AN/A✅❌❌CE & EE
MattermostOpen-source Slack alternative⚙⤓⤓⤓❌❌CE & EE
MinIOObject storage service⤓✅✅✅❌⚙CE & EE
RunnerExecutes GitLab CI/CD jobs⤓✅⚙✅⚙⚙CE & EE
Database MigrationsDatabase migrations✅✅✅✅⚙✅CE & EE
Certificate ManagementTLS Settings, Let’s Encrypt✅✅⚙✅⚙⚙CE & EE
GitLab Geo NodeGeographically distributed GitLab nodes⚙❌❌✅❌⚙EE Only
LDAP AuthenticationAuthenticate users against centralized LDAP directory⤓⤓⤓❌⤓⤓CE & EE
Outbound email (SMTP)Send email messages to users⤓⤓⤓✅⤓⤓CE & EE
Inbound email (SMTP)Receive messages to update issues⤓⤓⤓✅⤓⤓CE & EE
ElasticsearchImproved search within GitLab⤓⤓⤓❌⤓⤓EE Only
Sentry integrationError tracking for deployed apps⤓⤓⤓⤓⤓⤓CE & EE
Jaeger integrationDistributed tracing for deployed apps⤓⤓⤓⤓⤓⤓EE Only
GitLab Managed AppsDeploy Helm, Ingress, Cert-Manager, Prometheus, a Runner, JupyterHub, Knative to a cluster⤓⤓⤓⤓⤓⤓CE & EE

Component details

This document is designed to be consumed by systems administrators and GitLab Support Engineers who want to understand more about the internals of GitLab and how they work together.

When deployed, GitLab should be considered the amalgamation of the below processes. When troubleshooting or debugging, be as specific as possible as to which component you are referencing. That should increase clarity and reduce confusion.

Layers

GitLab can be considered to have two layers from a process perspective:

  • Monitoring: Anything from this layer is not required to deliver GitLab the application, but will allow administrators more insight into their infrastructure and what the service as a whole is doing.
  • Core: Any process that is vital for the delivery of GitLab as a platform. If any of these processes halt there will be a GitLab outage. For the Core layer, you can further divide into:
    • Processors: These processes are responsible for actually performing operations and presenting the service.
    • Data: These services store/expose structured data for the GitLab service.

Alertmanager

Alert manager is a tool provided by Prometheus that “handles alerts sent by client applications such as the Prometheus server. It takes care of deduplicating, grouping, and routing them to the correct receiver integration such as email, PagerDuty, or OpsGenie. It also takes care of silencing and inhibition of alerts.” You can read more in issue #45740 about what we will be alerting on.

Certificate management

Consul

Consul is a tool for service discovery and configuration. Consul is distributed, highly available, and extremely scalable.

Database migrations

Elasticsearch

Elasticsearch is a distributed RESTful search engine built for the cloud.

Gitaly

Gitaly is a service designed by GitLab to remove our need for NFS for Git storage in distributed deployments of GitLab (think GitLab.com or High Availability Deployments). As of 11.3.0, this service handles all Git level access in GitLab. You can read more about the project in the project’s readme.

Praefect

Praefect is a transparent proxy between each Git client and the Gitaly coordinating the replication of repository updates to secondary nodes.

GitLab Geo

GitLab Exporter

GitLab Exporter is a process designed in house that allows us to export metrics about GitLab application internals to Prometheus. You can read more in the project’s readme.

GitLab Pages

GitLab Pages is a feature that allows you to publish static websites directly from a repository in GitLab.

You can use it either for personal or business websites, such as portfolios, documentation, manifestos, and business presentations. You can also attribute any license to your content.

GitLab Runner

GitLab Runner runs tests and sends the results to GitLab.

GitLab CI/CD is the open-source continuous integration service included with GitLab that coordinates the testing. The old name of this project was GitLab CI Multi Runner but please use GitLab Runner (without CI) from now on.

GitLab Shell

GitLab Shell is a program designed at GitLab to handle SSH-based git sessions, and modifies the list of authorized keys. GitLab Shell is not a Unix shell nor a replacement for Bash or Zsh.

GitLab Workhorse

GitLab Workhorse is a program designed at GitLab to help alleviate pressure from Unicorn. You can read more about the historical reasons for developing. It’s designed to act as a smart reverse proxy to help speed up GitLab as a whole.

Grafana

Grafana is an open source, feature rich metrics dashboard and graph editor for Graphite, Elasticsearch, OpenTSDB, Prometheus, and InfluxDB.

Jaeger

Jaeger, inspired by Dapper and OpenZipkin, is a distributed tracing system. It can be used for monitoring microservices-based distributed systems.

Logrotate

GitLab is comprised of a large number of services that all log. We started bundling our own logrotate as of 7.4 to make sure we were logging responsibly. This is just a packaged version of the common open source offering.

Mattermost

Mattermost is an open source, private cloud, Slack-alternative from https://mattermost.com.

MinIO

MinIO is an object storage server released under Apache License v2.0. It is compatible with Amazon S3 cloud storage service. It is best suited for storing unstructured data such as photos, videos, log files, backups, and container / VM images. Size of an object can range from a few KBs to a maximum of 5TB.

NGINX

NGINX has an Ingress port for all HTTP requests and routes them to the appropriate sub-systems within GitLab. We are bundling an unmodified version of the popular open source webserver.

Node Exporter

Node Exporter is a Prometheus tool that gives us metrics on the underlying machine (think CPU/Disk/Load). It’s just a packaged version of the common open source offering from the Prometheus project.

PgBouncer

Lightweight connection pooler for PostgreSQL.

PgBouncer Exporter

Prometheus exporter for PgBouncer. Exports metrics at 9127/metrics.

PostgreSQL

GitLab packages the popular Database to provide storage for Application meta data and user information.

PostgreSQL Exporter

postgres_exporter is the community provided Prometheus exporter that will deliver data about PostgreSQL to Prometheus for use in Grafana Dashboards.

Prometheus

Prometheus is a time-series tool that helps GitLab administrators expose metrics about the individual processes used to provide GitLab the service.

Redis

Redis is packaged to provide a place to store:

  • session data
  • temporary cache information
  • background job queues

Redis Exporter

Redis Exporter is designed to give specific metrics about the Redis process to Prometheus so that we can graph these metrics in Grafana.

Registry

The registry is what users use to store their own Docker images. The bundled registry uses NGINX as a load balancer and GitLab as an authentication manager. Whenever a client requests to pull or push an image from the registry, it will return a 401 response along with a header detailing where to get an authentication token, in this case the GitLab instance. The client will then request a pull or push auth token from GitLab and retry the original request to the registry. Learn more about token authentication.

An external registry can also be configured to use GitLab as an auth endpoint.

Sentry

Sentry fundamentally is a service that helps you monitor and fix crashes in realtime. The server is in Python, but it contains a full API for sending events from any language, in any application.

Sidekiq

Sidekiq is a Ruby background job processor that pulls jobs from the Redis queue and processes them. Background jobs allow GitLab to provide a faster request/response cycle by moving work into the background.

Unicorn

Unicorn is a Ruby application server that is used to run the core Rails Application that provides the user facing features in GitLab. Often process output you will see this as bundle or config.ru depending on the GitLab version.

LDAP Authentication

Outbound Email

Inbound Email

GitLab Managed Apps

GitLab provides GitLab Managed Apps, a one-click install for various applications which can be added directly to your configured cluster. These applications are needed for Review Apps and deployments when using Auto DevOps. You can install them after you create a cluster.

GitLab by Request Type

GitLab provides two “interfaces” for end users to access the service:

  • Web HTTP Requests (Viewing the UI/API)
  • Git HTTP/SSH Requests (Pushing/Pulling Git Data)

It’s important to understand the distinction as some processes are used in both and others are exclusive to a specific request type.

GitLab Web HTTP Request Cycle

When making a request to an HTTP Endpoint (think /users/sign_in) the request will take the following path through the GitLab Service:

  • NGINX - Acts as our first line reverse proxy.
  • GitLab Workhorse - This determines if it needs to go to the Rails application or somewhere else to reduce load on Unicorn.
  • Unicorn - Since this is a web request, and it needs to access the application it will go to Unicorn.
  • PostgreSQL/Gitaly/Redis - Depending on the type of request, it may hit these services to store or retrieve data.

GitLab Git Request Cycle

Below we describe the different pathing that HTTP vs. SSH Git requests will take. There is some overlap with the Web Request Cycle but also some differences.

Web Request (80/443)

Git operations over HTTP use the stateless “smart” protocol described in the Git documentation, but responsibility for handling these operations is split across several GitLab components.

Here is a sequence diagram for git fetch. Note that all requests pass through NGINX as well as any other HTTP load balancers, but are not transformed in any way by them. All paths are presented relative to a /namespace/project.git URL.

sequenceDiagram participant Git on client participant NGINX participant Workhorse participant Rails participant Gitaly participant Git on server Note left of Git on client: git fetch
info-refs Git on client->>+Workhorse: GET /info/refs?service=git-upload-pack Workhorse->>+Rails: GET /info/refs?service=git-upload-pack Note right of Rails: Auth check Rails-->>-Workhorse: Gitlab::Workhorse.git_http_ok Workhorse->>+Gitaly: SmartHTTPService.InfoRefsUploadPack request Gitaly->>+Git on server: git upload-pack --stateless-rpc --advertise-refs Git on server-->>-Gitaly: git upload-pack response Gitaly-->>-Workhorse: SmartHTTPService.InfoRefsUploadPack response Workhorse-->>-Git on client: 200 OK Note left of Git on client: git fetch
fetch-pack Git on client->>+Workhorse: POST /git-upload-pack Workhorse->>+Rails: POST /git-upload-pack Note right of Rails: Auth check Rails-->>-Workhorse: Gitlab::Workhorse.git_http_ok Workhorse->>+Gitaly: SmartHTTPService.PostUploadPack request Gitaly->>+Git on server: git upload-pack --stateless-rpc Git on server-->>-Gitaly: git upload-pack response Gitaly-->>-Workhorse: SmartHTTPService.PostUploadPack response Workhorse-->>-Git on client: 200 OK

The sequence is similar for git push, except git-receive-pack is used instead of git-upload-pack.

SSH Request (22)

Git operations over SSH can use the stateful protocol described in the Git documentation, but responsibility for handling them is split across several GitLab components.

No GitLab components speak SSH directly - all SSH connections are made between Git on the client machine and the SSH server, which terminates the connection. To the SSH server, all connections are authenticated as the git user; GitLab users are differentiated by the SSH key presented by the client.

Here is a sequence diagram for git fetch, assuming Fast SSH key lookup is enabled. Note that AuthorizedKeysCommand is an executable provided by GitLab Shell:

sequenceDiagram participant Git on client participant SSH server participant AuthorizedKeysCommand participant GitLab Shell participant Rails participant Gitaly participant Git on server Note left of Git on client: git fetch Git on client->>+SSH server: ssh git fetch-pack request SSH server->>+AuthorizedKeysCommand: gitlab-shell-authorized-keys-check git AAAA... AuthorizedKeysCommand->>+Rails: GET /internal/api/authorized_keys?key=AAAA... Note right of Rails: Lookup key ID Rails-->>-AuthorizedKeysCommand: 200 OK, command="gitlab-shell upload-pack key_id=1" AuthorizedKeysCommand-->>-SSH server: command="gitlab-shell upload-pack key_id=1" SSH server->>+GitLab Shell: gitlab-shell upload-pack key_id=1 GitLab Shell->>+Rails: GET /internal/api/allowed?action=upload_pack&key_id=1 Note right of Rails: Auth check Rails-->>-GitLab Shell: 200 OK, { gitaly: ... } GitLab Shell->>+Gitaly: SSHService.SSHUploadPack request Gitaly->>+Git on server: git upload-pack request Note over Git on client,Git on server: Bidirectional communication between Git client and server Git on server-->>-Gitaly: git upload-pack response Gitaly -->>-GitLab Shell: SSHService.SSHUploadPack response GitLab Shell-->>-SSH server: gitlab-shell upload-pack response SSH server-->>-Git on client: ssh git fetch-pack response

The git push operation is very similar, except git receive-pack is used instead of git upload-pack.

If fast SSH key lookups are not enabled, the SSH server reads from the ~git/.ssh/authorized_keys file to determine what command to run for a given SSH session. This is kept up to date by an AuthorizedKeysWorker in Rails, scheduled to run whenever an SSH key is modified by a user.

SSH certificates may be used instead of keys. In this case, AuthorizedKeysCommand is replaced with an AuthorizedPrincipalsCommand. This extracts a username from the certificate without using the Rails internal API, which is used instead of key_id in the /api/internal/allowed call later.

GitLab Shell also has a few operations that do not involve Gitaly, such as resetting two-factor authentication codes. These are handled in the same way, except there is no round-trip into Gitaly - Rails performs the action as part of the internal API call, and GitLab Shell streams the response back to the user directly.

System Layout

When referring to ~git in the pictures it means the home directory of the Git user which is typically /home/git.

GitLab is primarily installed within the /home/git user home directory as git user. Within the home directory is where the gitlabhq server software resides as well as the repositories (though the repository location is configurable).

The bare repositories are located in /home/git/repositories. GitLab is a ruby on rails application so the particulars of the inner workings can be learned by studying how a ruby on rails application works.

To serve repositories over SSH there’s an add-on application called GitLab Shell which is installed in /home/git/gitlab-shell.

Installation Folder Summary

To summarize here’s the directory structure of the git user home directory.

Processes

ps aux | grep '^git'

GitLab has several components to operate. It requires a persistent database (PostgreSQL) and Redis database, and uses Apache httpd or NGINX to proxypass Unicorn. All these components should run as different system users to GitLab (e.g., postgres, redis and www-data, instead of git).

As the git user it starts Sidekiq and Unicorn (a simple ruby HTTP server running on port 8080 by default). Under the GitLab user there are normally 4 processes: unicorn_rails master (1 process), unicorn_rails worker (2 processes), sidekiq (1 process).

Repository access

Repositories get accessed via HTTP or SSH. HTTP cloning/push/pull utilizes the GitLab API and SSH cloning is handled by GitLab Shell (previously explained).

Troubleshooting

See the README for more information.

Init scripts of the services

The GitLab init script starts and stops Unicorn and Sidekiq:

/etc/init.d/gitlab
Usage: service gitlab {start|stop|restart|reload|status}

Redis (key-value store/non-persistent database):

/etc/init.d/redis
Usage: /etc/init.d/redis {start|stop|status|restart|condrestart|try-restart}

SSH daemon:

/etc/init.d/sshd
Usage: /etc/init.d/sshd {start|stop|restart|reload|force-reload|condrestart|try-restart|status}

Web server (one of the following):

/etc/init.d/httpd
Usage: httpd {start|stop|restart|condrestart|try-restart|force-reload|reload|status|fullstatus|graceful|help|configtest}

$ /etc/init.d/nginx
Usage: nginx {start|stop|restart|reload|force-reload|status|configtest}

Persistent database:

$ /etc/init.d/postgresql
Usage: /etc/init.d/postgresql {start|stop|restart|reload|force-reload|status} [version ..]

Log locations of the services

gitlabhq (includes Unicorn and Sidekiq logs):

  • /home/git/gitlab/log/ contains application.log, production.log, sidekiq.log, unicorn.stdout.log, git_json.log and unicorn.stderr.log normally.

GitLab Shell:

  • /home/git/gitlab-shell/gitlab-shell.log

SSH:

  • /var/log/auth.log auth log (on Ubuntu).
  • /var/log/secure auth log (on RHEL).

NGINX:

  • /var/log/nginx/ contains error and access logs.

Apache httpd:

  • Explanation of Apache logs.
  • /var/log/apache2/ contains error and output logs (on Ubuntu).
  • /var/log/httpd/ contains error and output logs (on RHEL).

Redis:

  • /var/log/redis/redis.log there are also log-rotated logs there.

PostgreSQL:

  • /var/log/postgresql/*

GitLab specific config files

GitLab has configuration files located in /home/git/gitlab/config/*. Commonly referenced config files include:

  • gitlab.yml - GitLab configuration.
  • unicorn.rb - Unicorn web server settings.
  • database.yml - Database connection settings.

GitLab Shell has a configuration file at /home/git/gitlab-shell/config.yml.

Maintenance Tasks

GitLab provides Rake tasks with which you see version information and run a quick check on your configuration to ensure it is configured properly within the application. See maintenance Rake tasks. In a nutshell, do the following:

sudo -i -u git
cd gitlab
bundle exec rake gitlab:env:info RAILS_ENV=production
bundle exec rake gitlab:check RAILS_ENV=production

Note: It is recommended to log into the git user using sudo -i -u git or sudo su - git. While the sudo commands provided by gitlabhq work in Ubuntu they do not always work in RHEL.

GitLab.com

We’ve also detailed our architecture of GitLab.com but this is probably over the top unless you have millions of users.