CI

Cluster tests

Of the cluster tests, merge request pipelines require only the primary k3d cluster test and the e2e_operator suite. The OpenShift test, the MicroShift cluster test, is optional, and you start it by hand. It also runs automatically on default-branch pipelines. It runs in a child pipeline, so chart-version pipelines do not have it, because they are already at the maximum descendant-pipeline depth.

The cluster tests have no GitLab environment and no cleanup job. Each job creates and destroys its own cluster.

Token Management

Read about our IaC managed Project Access Tokens.

k3d cluster tests

The k3d trigger job in .gitlab/ci/review-k3d.gitlab-ci.yml holds a parallel:matrix with one row per tested Kubernetes version. Each row spawns a child pipeline from .gitlab/ci/k3d-version-pipeline.gitlab-ci.yml whose review_k3d job creates a single-use k3d cluster inside the job’s Docker-in-Docker environment. The job deploys the operator and a GitLab custom resource, runs the QA smoke suite against it over a nip.io domain, and destroys the cluster when the job ends. The jobs have no GitLab environment or cleanup job because nothing outlives the job.

These jobs run on the privileged e2e runner fleet, except the arm64 row, which sets its own RUNNER_TAG (see Tested configurations).

The primary version (matrix row with K3D_PRIMARY: "primary") runs automatically on merge request and branch pipelines. The other versions are manual there, and all versions run automatically on nightly (scheduled), stable-branch, and default-branch pipelines.

Chart-version pipelines are already at the maximum descendant-pipeline depth, so they run the same test as the direct review_k3d_chart_version job with the same matrix instead of the trigger job.

Each child pipeline also has a manual qa_k3d_full job, with one parallel worker per full-suite shard. Like review_k3d, each worker is self-contained: it provisions its own k3d cluster, operator, and GitLab instance, then runs its shard of the full suite against it. Trigger it from any merge request or branch pipeline to run the full suite for that Kubernetes version. Nightly pipelines also run it automatically.

Tested configurations

The review jobs cover these combinations of operator scope and Ingress path:

The namespaced operator (CLUSTER_MODE: "false") deploys with nginx-ingress.create=false, so it does not create the cluster-scoped resources (IngressClass, RBAC) that the chart-bundled NGINX controller requires to start. Namespaced rows therefore pair with Gateway API instead. The k3d jobs serve plain HTTP because a single-use cluster has no pre-provisioned wildcard TLS certificate.

The operator never installs Envoy Gateway. It applies no CustomResourceDefinition and does not deploy the chart’s subchart dependencies, so the Gateway API CRDs and the Envoy Gateway controller must already be in the cluster. The v137 row therefore installs them in the job through INSTALL_ENVOY_GATEWAY: "true", which runs task install_envoy_gateway. The global.gatewayApi.installEnvoy: true of the gatewayapi overlay only makes the chart render the GatewayClass and EnvoyProxy resources that the reconciler applies.

review_microshift covers the opposite arrangement. Its microshift overlay sets installEnvoy: false, so the chart renders no GatewayClass and no EnvoyProxy. The Gateway attaches to the ci-stack GatewayClass that the job serves with Envoy Gateway.

TEST_CR_FILES_DIR selects the test CR from config/test. base routes through the chart-bundled NGINX controller and is the default, overlays/gatewayapi switches to the Gateway API, and overlays/microshift attaches it to the ci-stack GatewayClass of the MicroShift cluster and enables GitLab Shell.

Add or remove a Kubernetes version

Edit the parallel:matrix in .gitlab/ci/review-k3d.gitlab-ci.yml. Each row sets:

• K3D_K8S_IMAGE: the rancher/k3s image tag that pins the Kubernetes version.
• K3D_K8S_VERSION_SLUG: a short label, for example v135, used in the child pipeline name.
• CLUSTER_MODE: "true" for the cluster-wide operator, "false" for the namespaced operator.
• Optional. K3D_PRIMARY: "primary" on exactly one row: the version that runs automatically on merge request pipelines.
• Optional. TEST_CR_FILES_DIR and INSTALL_ENVOY_GATEWAY: "true" for the Gateway API/Envoy variant instead of the default NGINX Ingress overlay.
• Optional. RUNNER_TAG to run the row on a runner fleet other than the default e2e, for example saas-linux-large-arm64 for the arm64 row.

MicroShift cluster test

review_microshift tests the operator on OpenShift. The microshift job runs it in a child pipeline from .gitlab/ci/microshift-pipeline.gitlab-ci.yml. The microshift job runs automatically on default-branch pipelines. In other pipelines, start it by hand. It runs a single-use MicroShift cluster as a privileged container in the job’s Docker-in-Docker environment, then runs the same deploy and QA smoke steps as review_k3d. It runs on the e2e runner fleet.

MicroShift serves the SecurityContextConstraints and Route APIs, so the operator takes its OpenShift code paths, and every GitLab pod is admitted through an SCC. GitLab is served over plain HTTP through Gateway API, including Git over SSH. MicroShift gives the Gateway’s LoadBalancer Service the node IP, on which the job publishes ports 80 and 22.

qa_microshift_full, which is manual in the same child pipeline, runs the full suite in 7 self-contained workers, like qa_k3d_full.

.gitlab/ci/scripts/microshift.sh creates the cluster:

• It fails unless SCC admission rejects a pod with a fixed UID outside the namespace range.
• It skips the TopoLVM manifests of the image, which need an LVM volume group, and installs local-path-provisioner instead.
• It removes the OpenShift router, which would hold ports 80 and 443, and installs Envoy Gateway and the ci-stack GatewayClass. Envoy Gateway may use the nonroot-v2 SCC, because it runs as UID 65532.
• It lets the CloudNativePG operator use the nonroot-v2 SCC, because the operator chart pins a UID that restricted-v2 rejects.
• It adds a ClientTrafficPolicy that keeps %2F in paths for the Gateway, which the chart renders only when it configures Envoy Gateway itself.

Update the MicroShift image

MICROSHIFT_IMAGE in .gitlab/ci/scripts/microshift.sh pins a multi-architecture image of ghcr.io/microshift-io/microshift by digest. The MicroShift upstream releases list its tags, for example 4.22.0_202610071017_g0a516b4eb_4.22.0_okd_scos.ec.16. If no image exists for the OpenShift version you want, pin the nearest one and record the choice in ADR 22.

microshift_create applies only the kindnet and kube-proxy manifests of the image. When you update the image, compare /usr/lib/microshift/manifests.d/ in the new image with that list.

End-to-end suite jobs

.gitlab/ci/e2e.gitlab-ci.yml runs the black-box suites in test/e2e, one suite per job, each in a single-use k3d cluster inside its own Docker-in-Docker environment. They reuse .gitlab/ci/scripts/k3d.sh and k3d_collect_debug, and they pull the bridge image build_bridge_image already published rather than building their own.

They are peers of the k3d cluster tests above, not part of them: those deploy a whole GitLab and run gitlab-qa, while these drive custom resources through the API server. e2e_siphon runs on two Kubernetes versions to cover the CEL validation and the schema defaults of both API servers. e2e_upgrades deploys a whole GitLab: it installs a Static version and follows it through Patch and Latest upgrades. It is manual, because it takes most of an hour. e2e_migrate deploys a whole GitLab through the v1beta1 resource and migrates it to a GitLabCore. It runs on scheduled pipelines, and is manual on branch and merge request pipelines, because it rolls the instance out twice.

Both k3d-debug/ and e2e-artifacts/ are uploaded whatever the outcome. The second is written by the harness itself, before it deletes the namespace, which is the one k3d_collect_debug cannot do. See Tests.

QA pipelines

By default, QA pipelines run the smoke suite, a small subset of fast end-to-end functional tests that quickly checks basic functionality. To run the full suite of end-to-end tests instead, trigger the manual full-suite job for the cluster: qa_k3d_full for k3d (see k3d cluster tests), or microshift and then qa_microshift_full in its child pipeline for MicroShift (see MicroShift cluster test).

To debug failures in tests, please follow investigate QA failures guide.

Container image tags

CI pushes the Operator image to the project repository, $CI_REGISTRY_IMAGE (registry.gitlab.com/gitlab-org/cloud-native/gitlab-operator). It pushes the bridge image, built with the bridge Go build tag, to its own repository, $BRIDGE_REGISTRY_IMAGE ($CI_REGISTRY_IMAGE/bridge-staging), so the project repository holds only public images:

Bridge releases

Bridge is released apart from the Operator, with versions of its own. Release tags and stable branches build no Bridge artifact, and no Operator release names one, so the two never write to the same image tag or package. To release Bridge:

1. Open a default branch pipeline of the canonical project whose e2e_operator job passed.
2. Select the name of the manual release_bridge job, set its version input to the Bridge version, such as 0.1.0, and select Run job.

release_bridge then:

1. Refuses a version that is not X.Y.Z, the only form the kubectl bridge migrate preflight compares, and a version whose bridge-staging:<version> image or kubectl-bridge package exists. It also refuses a commit whose bridge-staging:<short-sha> image is missing. It checks all of this before it builds anything.
2. Builds the kubectl bridge plugin for each platform in KUBECTL_BRIDGE_PLATFORMS, Linux and macOS on amd64 and arm64, with the version stamped in. Each platform gets a kubectl-bridge-<os>-<arch>.tar.gz that holds the binary and the license, the archive format krew installs. It also runs scripts/v2alpha1_manifests.sh, which writes v2alpha1-crds.yaml and v2alpha1-rbac.yaml: the v2alpha1 definitions and the manager RBAC that the Helm chart does not ship, the same files task install_v2alpha1_crds applies. It also writes v2alpha1-rbac-namespaced.yaml, the same rules as a Role and RoleBinding, for an Operator installed with watchCluster: false.
3. Uploads the archives, the manifests, and a SHA256SUMS file over all of them to the kubectl-bridge generic package, with the Bridge version as its version.
4. Tags bridge-staging:<short-sha>, the image build_bridge_image pushed for the commit and e2e_operator tested, as bridge-staging:<version>. The image is not rebuilt, and the release signs nothing of its own. The tag comes last, so a version has an image only once its package is complete.

A Bridge release publishes no chart. A released Operator chart, 3.5.0 or later, installs the bridge image with image.name and image.tag set, as Install from a release shows.

If the job fails after the upload started, the package of that version blocks it from running again. Delete the kubectl-bridge package of that version, and run the job again. The job is interruptible: false and in the bridge-release resource group, so a newer pipeline does not cancel it and two releases do not run at once. Its helpers are in .gitlab/ci/scripts/release_bridge.sh.

build_kubectl_bridge builds the plugin without a version and publishes nothing. It runs on the default branch, and in branch and merge request pipelines that change the SPA, cmd/kubectl-bridge, Dockerfile.kubectl-bridge, go.mod, go.sum, the v2alpha1 definitions or RBAC, or the CI configuration, so a broken build shows before a release. bridge_unit_tests compiles the plugin for any other Go change. Both jobs share .kubectl_bridge_build, which runs Dockerfile.kubectl-bridge with buildx, like the container builds, but exports the files instead of pushing an image. The Dockerfile builds the SPA and cross-compiles the plugin with it embedded. To build the same binaries locally:

docker buildx build -f Dockerfile.kubectl-bridge \
  --platform linux/amd64,darwin/arm64 --output type=local,dest=out .

Stable branches skip the bridge image, the plugin build, and the end-to-end suites, which test the bridge image. They still run bridge_unit_tests and fetch the Siphon and Orbit charts that the unit tests of their reconcilers need. To install a Bridge release, see Install from a release.

Container builds

The Operator image is built for every platform in BUILD_PLATFORMS (linux/amd64 and linux/arm64), on a single amd64 runner, with no emulation. .docker_build_job creates a docker-container buildx driver in its own DinD service, because the default docker driver cannot produce an image for a foreign platform.

One Dockerfile builds both images. The image jobs do not compile the manager. build_manager compiles it for every platform in BUILD_PLATFORMS, with the Go build cache, and docker_build_and_push builds with --build-arg MANAGER_SOURCE=prebuilt. The prebuilt stage copies MANAGER_DIR/manager-<os>-<arch> into the image, and BuildKit skips the builder stage. build_bridge_image sets BUILD_TAGS: bridge, so docker_build_and_push takes the manager build_manager compiled with the bridge Go build tag, from .manager/bridge. The public image jobs leave BUILD_TAGS empty and take .manager/default. Release tags compile only default.

Without MANAGER_SOURCE, the builder stage compiles the manager from source, with BUILD_TAGS as its build tags. Both build with -trimpath, so the binary does not depend on where it was built. CI never builds the builder stage. Instead, go_build_cache runs .gitlab/ci/scripts/check_builder_sources.sh, which fails when the manager imports a package from a directory that the stage does not copy.

docker_build_and_push writes the pushed index digest to --metadata-file, and sign reads it back from there. cosign stores a signature against the digest, at <repo>:sha256-<digest>.sig, and cosign verify <repo>:<tag> resolves the tag to its digest first — so one signature covers every tag the build pushed, and the digest never has to be re-resolved from a tag another pipeline could have moved.

Every stage that executes anything is pinned to the build platform with --platform=${BUILDPLATFORM}. The Go stage cross-compiles through GOOS/GOARCH. The certs and runtime stages are deliberately left unpinned so that the published image is the target architecture; they only COPY, so no foreign binary is ever executed.

Adding a RUN to a runtime stage would make that stage need emulation. Build it on a native runner instead, for example with the saas-linux-large-arm64 tag, and join the results with docker buildx imagetools create.

image_certification certifies one architecture per entry in BUILD_PLATFORMS, so the platform list has a single definition in the repository.

build_bundle_image is OLM metadata rather than a runnable workload, and stays single-arch on podman.

Go build cache

The Go jobs cache the Go build cache, GOCACHE, which holds compiled packages. They do not cache modules, because downloading them from the proxy is faster than restoring them. For the reasons and the measurements, see ADR 47.

The .go_cache template sets GOCACHE to .gocache/ and restores the cache of the job’s GO_CACHE_KIND. Every job of a kind, on every branch, shares one cache:

Each kind has one job that pushes its cache, so that jobs do not overwrite one another’s entries. The other jobs set no policy and only pull. go_build_cache compiles what the test jobs compile, with their flags. No job needs it, so the test jobs of a pipeline use the cache an earlier pipeline pushed. When you change how a test job compiles, for example its build tags or its -cover flag, change go_build_cache the same way. Otherwise the job finds nothing in the cache.

Entries built from other sources or with another toolchain are never hit, and Go deletes entries unused for five days. To start every kind from an empty cache, raise GO_CACHE_GENERATION in .gitlab/ci/templates.gitlab-ci.yml.

GitLab keeps protected and unprotected caches apart, and adds -protected or -non_protected to each key. A pipeline uses the protected cache when it runs for a protected branch or tag, or when a user with the Maintainer or Owner role starts it, on any branch. Pipelines that a Developer starts on an unprotected branch use the unprotected cache. For more information, see Caching in GitLab CI/CD.

Neither cache is trusted where CI publishes what it builds. The runner extracts the cache archive into the checkout, so an archive pushed with other cache:paths can replace any file, sources included. On release tags, the default branch, and stable branches, build_manager therefore compiles from an empty cache and only pushes the image cache. The test and lint jobs still pull their caches there, because they publish nothing.

DockerHub rate limits

By default, CI uses images from DockerHub. The shared runners by default use a mirror to avoid hitting DockerHub rate limits. If you use custom runnners, that don’t use caching or mirroring, you should enable the dependency proxy by setting the DOCKERHUB_PREFIX to your proxy, for example DOCKERHUB_PREFIX: ${CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX}, and DEPENDENCY_PROXY_LOGIN="true".

The container build context by default uses the gcr DockerHub mirror. This behavior can be changed by overriding the DOCKER_OPTIONS or DOCKER_MIRROR variables.