Using the Praefect chart

The Praefect chart is used to manage a Gitaly cluster inside a GitLab installment deployed with the Helm charts.

Known Limitations

  1. The database has to be manually created.
  2. Migrating from an existing Gitaly setup to Praefect is not supported.

Requirements

This chart consumes the Gitaly chart. Settings from global.gitaly are used to configure the instances created by this chart. Documentation of these settings can be found in Gitaly chart documentation.

Important: global.gitaly.tls is independent of global.praefect.tls. They are configured separately.

By default, this chart will create 3 Gitaly Replicas.

Configuration

The chart is disabled by default. To enable it as part of a chart deploy set global.praefect.enabled=true.

Replicas

The default number of replicas to deploy is 3. This can be changed by setting global.praefect.virtualStorages[].gitalyReplicas with the desired number of replicas. For example:

global:
  praefect:
    enabled: true
    virtualStorages:
    - name: default
      gitalyReplicas: 4
      maxUnavailable: 1

Multiple virtual storages

Multiple virtual storages can be configured (see Gitaly Cluster documentation). For example:

global:
  praefect:
    enabled: true
    virtualStorages:
    - name: default
      gitalyReplicas: 4
      maxUnavailable: 1
    - name: vs2
      gitalyReplicas: 5
      maxUnavailable: 2

This will create two sets of resources for Gitaly. This includes two Gitaly StatefulSets (one per virtual storage).

Administrators can then choose where new repositories are stored.

Persistence

It is possible to provide persistence configuration per virtual storage.

global:
  praefect:
    enabled: true
    virtualStorages:
    - name: default
      gitalyReplicas: 4
      maxUnavailable: 1
      persistence:
        enabled: true
        size: 50Gi
        accessMode: ReadWriteOnce
        storageClass: storageclass1
    - name: vs2
      gitalyReplicas: 5
      maxUnavailable: 2
      persistence:
        enabled: true
        size: 100Gi
        accessMode: ReadWriteOnce
        storageClass: storageclass2

Creating the database

Praefect uses its own database to track its state. This has to be manually created in order for Praefect to be functional.

noteThese instructions assume you are using the bundled PostgreSQL server. If you are using your own server, there will be some variation in how you connect.
  1. Log into your database instance:

    kubectl exec -it $(kubectl get pods -l app=postgresql -o custom-columns=NAME:.metadata.name --no-headers) -- bash
    PGPASSWORD=$(cat $POSTGRES_POSTGRES_PASSWORD_FILE) psql -U postgres -d template1
    
  2. Create the database user:

    template1=# CREATE ROLE praefect WITH LOGIN;
    
  3. Set the database user password.

    By default, the shared-secrets chart will generate a secret for you.

    1. Fetch the password:

      kubectl get secret RELEASE_NAME-praefect-dbsecret -o jsonpath="{.data.secret}" | base64 --decode
      
    2. Set the password in the psql prompt:

      template1=# \password praefect
      Enter new password:
      Enter it again:
      
  4. Create the database:

    CREATE DATABASE praefect WITH OWNER praefect;
    

Running Praefect over TLS

Praefect supports communicating with client and Gitaly nodes over TLS. This is controlled by the settings global.praefect.tls.enabled and global.praefect.tls.secretName. To run Praefect over TLS follow these steps:

  1. The Helm chart expects a certificate to be provided for communicating over TLS with Praefect. This certificate should apply to all the Praefect nodes that are present. Hence all hostnames of each of these nodes should be added as a Subject Alternate Name (SAN) to the certificate or alternatively, you can use wildcards.

    To know the hostnames to use, check the file /srv/gitlab/config/gitlab.yml file in the Task Runner Pod and check the various gitaly_address fields specified under repositories.storages key within it.

    kubectl exec -it <Task Runner Pod> -- grep gitaly_address /srv/gitlab/config/gitlab.yml
    
noteA basic script for generating custom signed certificates for internal Praefect Pods can be found in this repo. Users can use or refer that script to generate certificates with proper SAN attributes.
  1. Create a TLS Secret using the certificate created.

    kubectl create secret tls <secret name> --cert=praefect.crt --key=praefect.key
    
  2. Redeploy the Helm chart by passing the additional arguments --set global.praefect.tls.enabled=true --set global.praefect.tls.secretName=<secret name>

When running Gitaly over TLS, a secret name must be provided for each virtual storage.

global:
  gitaly:
    tls:
      enabled: true
  praefect:
    enabled: true
    tls:
      enabled: true
      secretName: praefect-tls
    virtualStorages:
    - name: default
      gitalyReplicas: 4
      maxUnavailable: 1
      tlsSecretName: default-tls
    - name: vs2
      gitalyReplicas: 5
      maxUnavailable: 2
      tlsSecretName: vs2-tls

Installation command line options

The table below contains all the possible charts configurations that can be supplied to the helm install command using the --set flags.

Parameter Default Description
failover.enabled true Whether Praefect should perform failover on node failure
failover.readonlyAfter false Whether the nodes should be in read-only mode after failover
autoMigrate true Automatically run migrations on startup
electionStrategy sql See election strategy
image.repository registry.gitlab.com/gitlab-org/build/cng/gitaly The default image repository to use. Praefect is bundled as part of the Gitaly image
service.name praefect The name of the service to create
service.type ClusterIP The type of service to create
service.internalPort 8075 The internal port number that the Praefect pod will be listening on
service.externalPort 8075 The port number the Praefect service should expose in the cluster
init.resources    
init.image    
logging.level   Log level
logging.format json Log format
logging.sentryDsn   Sentry DSN URL - Exceptions from Go server
logging.rubySentryDsn   Sentry DSN URL - Exceptions from gitaly-ruby
logging.sentryEnvironment   Sentry environment to be used for logging
metrics.enabled true  
metrics.port 9236  
securityContext.runAsUser 1000  
securityContext.fsGroup 1000