Use TLS between components of the GitLab chart

The GitLab charts can use transport-layer security (TLS) between the various components. This requires you to provide certificates for the services you want to enable, and configure those services to make use of those certificates and the certificate authority (CA) that signed them.

Preparation

Each chart has documentation regarding enabling TLS for that service, and the various settings required to ensure that appropriate configuration.

Generating certificates for internal use

GitLab does not purport to provide high-grade PKI infrastructure, or certificate authorities.

For the purposes of this documentation, we provide a Proof of Concept script below, which makes use of Cloudflare’s CFSSL to produce a self-signed Certificate Authority, and a wildcard certificate that can be used for all services.

This script:

  • Generates a CA key pair.
  • Signs a certificate meant to service all GitLab component service endpoints.
  • Creates two Kubernetes Secret objects:
    • A secret of type kuberetes.io/tls which has the server certificate and key pair.
    • A secret of type Opaque which only contains the public certificate of the CA as ca.crt as need by NGINX Ingress.

Prerequisites:

  • Bash, or compatible shell.
  • cfssl is available to your shell, and within PATH.
  • kubectl is available, and configured to point to your Kubernetes cluster where GitLab will later be installed.
    • Be sure to have created the namespace you wish to have these certificates installed into before operating the script.

You may copy the content of this script to your computer, and make the resulting file executable. We suggest poc-gitlab-internal-tls.sh.

#!/bin/bash
set -e
#############
## make and change into a working directory
pushd $(mktemp -d)

#############
## setup environment
NAMESPACE=${NAMESPACE:-default}
RELEASE=${RELEASE:-gitlab}
## stop if variable is unset beyond this point
set -u
## known expected patterns for SAN
CERT_SANS="*.${NAMESPACE}.svc,${RELEASE}-metrics.${NAMESPACE}.svc,*.${RELEASE}-gitaly.${NAMESPACE}.svc"

#############
## generate default CA config
cfssl print-defaults config > ca-config.json
## generate a CA
echo '{"CN":"'${RELEASE}.${NAMESPACE}.internal.ca'","key":{"algo":"ecdsa","size":256}}' | \
  cfssl gencert -initca - | \
  cfssljson -bare ca -
## generate certificate
echo '{"CN":"'${RELEASE}.${NAMESPACE}.internal'","key":{"algo":"ecdsa","size":256}}' | \
  cfssl gencert -config=ca-config.json -ca=ca.pem -ca-key=ca-key.pem -profile www -hostname="${CERT_SANS}" - |\
  cfssljson -bare ${RELEASE}-services

#############
## load certificates into K8s
kubectl -n ${NAMESPACE} create secret tls ${RELEASE}-internal-tls \
  --cert=${RELEASE}-services.pem \
  --key=${RELEASE}-services-key.pem
kubectl -n ${NAMESPACE} create secret generic ${RELEASE}-internal-tls-ca \
  --from-file=ca.crt=ca.pem

This script does not preserve the CA’s private key. It is a Proof-of-Concept helper, and is not intended for production use.

The script expects two environment variables to be set:

  1. NAMESPACE: The Kubernetes Namespace you will later install GitLab to. This defaults to default, as with kubectl.
  2. RELEASE: The Helm Release name you will later use to install GitLab. This defaults to gitlab.

To operate this script, you may export the two variables, or prepend the script name with their values.

export NAMESPACE=testing
export RELEASE=gitlab

./poc-gitlab-internal-tls.sh

After the script has run, you will find the two secrets created, and the temporary working directory contains all certificates and their keys.

$ pwd
/tmp/tmp.swyMgf9mDs
$ kubectl -n ${NAMESPACE} get secret | grep internal-tls
testing-internal-tls      kubernetes.io/tls                     2      11s
testing-internal-tls-ca   Opaque                                1      10s
$ ls -1
ca-config.json
ca.csr
ca-key.pem
ca.pem
testing-services.csr
testing-services-key.pem
testing-services.pem

Required certificate CN and SANs

The various GitLab components speak to each other over their Service’s DNS names. For TLS certificate verification to pass, each certificate requires a SAN that includes the component’s service name, or a wildcard acceptable to the Kubernetes Service DNS entry.

  • service-name.namespace.svc
  • *.namespace.svc

Failure to ensure these SANs within certificates will result in a non-functional instance, and logs that can be quite cryptic, refering to “connection failure” or “SSL verification failed”.

You can make use of helm template to retrieve a full list of all Service object names, if needed. If your GitLab has been deployed without TLS, you can query Kubernetes for those names:

kubectl -n ${NAMESPACE} get service -lrelease=${RELEASE}

Ingress traffic

By default, network traffic from the Ingress or Gateway API controller to the backend services is expected to be unencrypted. To enable internal TLS for these connections, additional configuration is required depending on your networking solution.

NGINX Ingress

When internal TLS is enabled, the GitLab chart automatically annotates the Ingress objects so that NGINX Ingress initiates TLS connections to the backend services and verifies their certificates against the configured CA. No additional user configuration is required.

If you use a different Ingress implementation, you must add provider-specific annotations or configuration to enable TLS connections between the controller and the backends.

Envoy Gateway

The chart provides BackendTLSPolicy resources to configure Envoy Gateway, or other spec-compliant Gateway API controllers, to initiate TLS connections with the backends.

For details, see the Gateway API documentation.

Configuration

Example configurations can be found in examples/internal-tls.

For the purposes of this documentation, we have provided shared-cert-values.yaml which configures the GitLab components to consume the certificates generated with the script above, in generating certificates for internal use.

Key items to configure:

  1. Global Custom Certificate Authorities.
  2. Per-component TLS for the service listeners. (See each chart’s documentation, under charts/)

This process is greatly simplified by making use of YAML’s native anchor functionality. A truncated snippet of shared-cert-values.yaml shows this:

.internal-ca: &internal-ca gitlab-internal-tls-ca
.internal-tls: &internal-tls gitlab-internal-tls

global:
  certificates:
    customCAs:
    - secret: *internal-ca
  workhorse:
    tls:
      enabled: true
gitlab:
  webservice:
    tls:
      secretName: *internal-tls
    workhorse:
       tls:
          verify: true # default
          secretName: *internal-tls
          caSecretName: *internal-ca

Result

When all components have been configured to provide TLS on their service listeners, all communication between GitLab components will traverse the network with TLS security, including connections from NGINX Ingress to each GitLab component.

NGINX Ingress will terminate any inbound TLS, determine the appropriate services to pass the traffic to, and then form a new TLS connection to the GitLab component. When configured as shown here, it will also verify the certificates served by the GitLab components against the CA.

This can be verified by connecting to the Toolbox pod, and querying the various component Services. One such example, connecting to the Webservice Pod’s primary service port that NGINX Ingress uses:

$ kubectl -n ${NAMESPACE} get pod -lapp=toolbox,release=${RELEASE}
NAME                              READY   STATUS    RESTARTS   AGE
gitlab-toolbox-5c447bfdb4-pfmpc   1/1     Running   0          65m
$ kubectl exec -ti gitlab-toolbox-5c447bfdb4-pfmpc -c toolbox -- \
    curl -Iv "https://gitlab-webservice-default.testing.svc:8181"

The output should be similar to following example:

*   Trying 10.60.0.237:8181...
* Connected to gitlab-webservice-default.testing.svc (10.60.0.237) port 8181 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: CN=gitlab.testing.internal
*  start date: Jul 18 19:15:00 2022 GMT
*  expire date: Jul 18 19:15:00 2023 GMT
*  subjectAltName: host "gitlab-webservice-default.testing.svc" matched cert's "*.testing.svc"
*  issuer: CN=gitlab.testing.internal.ca
*  SSL certificate verify ok.
> HEAD / HTTP/1.1
> Host: gitlab-webservice-default.testing.svc:8181

Troubleshooting

If your GitLab instance appears unreachable from the browser, by rendering an HTTP 503 error, NGINX Ingress is likely having a problem verifying the certificates of the GitLab components.

You may work around this by temporarily setting gitlab.webservice.workhorse.tls.verify to false.

The NGINX Ingress controller can be connected to, and will evidence a message in nginx.conf, regarding problems verifying the certificate(s).

Example content, where the Secret is not reachable:

# Location denied. Reason: "error obtaining certificate: local SSL certificate
  testing/gitlab-internal-tls-ca was not found"
return 503;

Common problems that cause this:

  • CA certificate is not in a key named ca.crt within the Secret.
  • The Secret was not properly supplied, or may not exist within the Namespace.