Using the GitLab-Gitaly chart

The gitaly sub-chart provides a configurable deployment of Gitaly Servers.

Requirements

This chart depends on access to the Workhorse service, either as part of the complete GitLab chart or provided as an external service reachable from the Kubernetes cluster this chart is deployed onto.

Design Choices

The Gitaly container used in this chart also contains the GitLab Shell codebase in order to perform the actions on the Git repositories that have not yet been ported into Gitaly. The Gitaly container includes a copy of the GitLab Shell container within it, and as a result we also need to configure GitLab Shell within this chart.

Configuration

The gitaly chart is configured in two parts: external services, and chart settings.

Gitaly is by default deployed as a component when deploying the GitLab chart. If deploying Gitaly separately, global.gitaly.enabled needs to be set to false and additional configuration will need to be performed as described in the external Gitaly documentation.

Installation command line options

The table below contains all the possible charts configurations that can be supplied to the helm install command using the --set flags.

Chart configuration examples

extraEnv

extraEnv allows you to expose additional environment variables in all containers in the pods.

Below is an example use of extraEnv:

extraEnv:
  SOME_KEY: some_value
  SOME_OTHER_KEY: some_other_value

When the container is started, you can confirm that the environment variables are exposed:

env | grep SOME
SOME_KEY=some_value
SOME_OTHER_KEY=some_other_value

extraEnvFrom

extraEnvFrom allows you to expose additional environment variables from other data sources in all containers in the pods.

Below is an example use of extraEnvFrom:

extraEnvFrom:
  MY_NODE_NAME:
    fieldRef:
      fieldPath: spec.nodeName
  MY_CPU_REQUEST:
    resourceFieldRef:
      containerName: test-container
      resource: requests.cpu
  SECRET_THING:
    secretKeyRef:
      name: special-secret
      key: special_token
      # optional: boolean
  CONFIG_STRING:
    configMapKeyRef:
      name: useful-config
      key: some-string
      # optional: boolean

image.pullSecrets

pullSecrets allows you to authenticate to a private registry to pull images for a pod.

Additional details about private registries and their authentication methods can be found in the Kubernetes documentation.

Below is an example use of pullSecrets

image:
  repository: my.gitaly.repository
  tag: latest
  pullPolicy: Always
  pullSecrets:
  - name: my-secret-name
  - name: my-secondary-secret-name

serviceAccount

This section controls if a ServiceAccount should be created and if the default access token should be mounted in pods.

tolerations

tolerations allow you schedule pods on tainted worker nodes

Below is an example use of tolerations:

tolerations:
- key: "node_label"
  operator: "Equal"
  value: "true"
  effect: "NoSchedule"
- key: "node_label"
  operator: "Equal"
  value: "true"
  effect: "NoExecute"

affinity

For more information, see affinity.

annotations

annotations allows you to add annotations to the Gitaly pods.

Below is an example use of annotations:

annotations:
  kubernetes.io/example-annotation: annotation-value

priorityClassName

priorityClassName allows you to assign a PriorityClass to the Gitaly pods.

Below is an example use of priorityClassName:

priorityClassName: persistence-enabled

git.config

git.config allows you to add configuration to all Git commands spawned by Gitaly. Accepts configuration as documented in git-config(1) in key / value pairs, as shown below.

git:
  config:
    - key: "pack.threads"
      value: 4
    - key: "fsck.missingSpaceBeforeDate"
      value: ignore

cgroups

To prevent exhaustion, Gitaly uses cgroups to assign Git processes to a cgroup based on the repository being operated on. Each cgroup has memory and CPU limits, ensuring system stability and preventing resource saturation.

Please note that the initContainer that runs before Gitaly starts requires to be executed as root. This container will configure the permissions so that Gitaly can manage cgroups. Hence, it will mount a volume on the filesystem to have write access to /sys/fs/cgroup.

Example of Oversubscription

cgroups:
  enabled: true
  # Total limit across all repository cgroups
  memoryBytes: 64424509440 # 60GiB
  cpuShares: 1024
  cpuQuotaUs: 1200000 # 12 cores
  # Per repository limits, 1000 repository cgroups
  repositories:
    count: 1000
    memoryBytes: 32212254720 # 30GiB
    cpuShares: 512
    cpuQuotaUs: 400000 # 4 cores

External Services

This chart should be attached the Workhorse service.

Workhorse

workhorse:
  host: workhorse.example.com
  serviceName: webservice
  port: 8181

Chart settings

The following values are used to configure the Gitaly Pods.

Git Repository Persistence

This chart provisions a PersistentVolumeClaim and mounts a corresponding persistent volume for the Git repository data. You’ll need physical storage available in the Kubernetes cluster for this to work. If you’d rather use emptyDir, disable PersistentVolumeClaim with: persistence.enabled: false.

persistence:
  enabled: true
  storageClass: standard
  accessMode: ReadWriteOnce
  size: 50Gi
  matchLabels: {}
  matchExpressions: []
  subPath: "data"
  annotations: {}

Running Gitaly over TLS

Gitaly supports communicating with other components over TLS. This is controlled by the settings global.gitaly.tls.enabled and global.gitaly.tls.secretName. Follow the steps to run Gitaly over TLS:

1. The Helm chart expects a certificate to be provided for communicating over TLS with Gitaly. This certificate should apply to all the Gitaly nodes that are present. Hence all hostnames of each of these Gitaly nodes should be added as a Subject Alternate Name (SAN) to the certificate.

   To know the hostnames to use, check the file /srv/gitlab/config/gitlab.yml file in the Toolbox pod and check the various gitaly_address fields specified under repositories.storages key within it.

   kubectl exec -it <Toolbox pod> -- grep gitaly_address /srv/gitlab/config/gitlab.yml

1. Create a k8s TLS secret using the certificate created.

   kubectl create secret tls gitaly-server-tls --cert=gitaly.crt --key=gitaly.key

2. Redeploy the Helm chart by passing --set global.gitaly.tls.enabled=true.

Global server hooks

The Gitaly StatefulSet has support for Global server hooks. The hook scripts run on the Gitaly pod, and are therefore limited to the tools available in the Gitaly container.

The hooks are populated using ConfigMaps, and can be used by setting the following values as appropriate:

1. global.gitaly.hooks.preReceive.configmap
2. global.gitaly.hooks.postReceive.configmap
3. global.gitaly.hooks.update.configmap

To populate the ConfigMap, you can point kubectl to a directory of scripts:

kubectl create configmap MAP_NAME --from-file /PATH/TO/SCRIPT/DIR

GPG signing commits created by GitLab

Gitaly has the ability to GPG sign all commits created via the GitLab UI, e.g. the WebIDE, as well as commits created by GitLab, such as merge commits and squashes.

1. Create a k8s secret using your GPG private key.

   kubectl create secret generic gitaly-gpg-signing-key --from-file=signing_key=/path/to/gpg_signing_key.gpg

2. Enable GPG signing in your values.yaml.

   gitlab:
     gitaly:
       gpgSigning:
         enabled: true
         secret: gitaly-gpg-signing-key
         key: signing_key

Server-side backups

The chart supports Gitaly server-side backups. To use them:

1. Create a bucket to store the backups.

2. Configure the object store credentials and the storage URL.

   gitlab:
     gitaly:
       extraEnvFrom:
          # Mount the exisitign object store secret to the expected environment variables.
          AWS_ACCESS_KEY_ID:
            secretKeyRef:
              name: <Rails object store secret>
              key: aws_access_key_id
          AWS_SECRET_ACCESS_KEY:
            secretKeyRef:
              name: <Rails object store secret>
              key: aws_secret_access_key
       backup:
         # This is the connection string for Gitaly server side backups.
         goCloudUrl: <object store connection URL>

   For the expected environment variables and storage URL format for your object storage backend, see the Gitaly documentation.

3. Enable server-side backups with backup-utility.