CI

Review environments

Of the cluster tests, merge request pipelines require only the primary k3d cluster test and the e2e_operator suite. The OpenShift tests, review_ocp and the MicroShift cluster test, are optional in every pipeline, and you start them by hand. Each runs in a child pipeline. Chart-version pipelines have neither, because they are already at the maximum descendant-pipeline depth.

review_ocp is the only review job with an externally accessible GitLab environment. To run it, start the ocp job, which runs review_ocp and its QA jobs in a child pipeline.

review_ocp deploys to the shared OpenShift CI cluster and is uninstalled automatically after 1 hour. To keep it up longer, pin the environment on the Environments page. When you are done, run stop_review_openshift_ocp in the child pipeline, so that the cluster has enough resources to run review apps for other merge requests.

The k3d cluster tests and the MicroShift cluster test have no environment and no cleanup job. Each job creates and destroys its own cluster.

For more information, see the environments documentation.

Token Management

Read about our IaC managed Project Access Tokens.

OpenShift CI cluster

We manage one OpenShift cluster in Google Cloud that is used for acceptance tests, including the QA suite. review_ocp, the qa_ocp jobs, and validate_olm_release run against it.

The kubeconfig file for connecting to the cluster is stored in the 1Password cloud-native vault. Search for ocp-ci.

The cluster is orchestrated using the openshift-provisioning project. CI access is managed using kube-agents.

k3d cluster tests

The k3d trigger job in .gitlab/ci/review-k3d.gitlab-ci.yml holds a parallel:matrix with one row per tested Kubernetes version. Each row spawns a child pipeline from .gitlab/ci/k3d-version-pipeline.gitlab-ci.yml whose review_k3d job creates a single-use k3d cluster inside the job’s Docker-in-Docker environment. The job deploys the operator and a GitLab custom resource, runs the QA smoke suite against it over a nip.io domain, and destroys the cluster when the job ends. The jobs have no GitLab environment or cleanup job because nothing outlives the job.

These jobs run on the privileged e2e runner fleet, except the arm64 row, which sets its own RUNNER_TAG (see Tested configurations).

The primary version (matrix row with K3D_PRIMARY: "primary") runs automatically on merge request and branch pipelines. The other versions are manual there, and all versions run automatically on nightly (scheduled), stable-branch, and default-branch pipelines.

Chart-version pipelines are already at the maximum descendant-pipeline depth, so they run the same test as the direct review_k3d_chart_version job with the same matrix instead of the trigger job.

Each child pipeline also has a manual qa_k3d_full job, with one parallel worker per full-suite shard. Like review_k3d, each worker is self-contained: it provisions its own k3d cluster, operator, and GitLab instance, then runs its shard of the full suite against it. Trigger it from any merge request or branch pipeline to run the full suite for that Kubernetes version. Nightly pipelines also run it automatically.

Tested configurations

The review jobs cover these combinations of operator scope and Ingress path:

The namespaced operator (CLUSTER_MODE: "false") deploys with nginx-ingress.create=false, so it does not create the cluster-scoped resources (IngressClass, RBAC) that the chart-bundled NGINX controller requires to start. Namespaced rows therefore pair with Gateway API instead. The k3d jobs serve plain HTTP because a single-use cluster has no pre-provisioned wildcard TLS certificate.

The operator never installs Envoy Gateway. It applies no CustomResourceDefinition and does not deploy the chart’s subchart dependencies, so the Gateway API CRDs and the Envoy Gateway controller must already be in the cluster. The v137 row therefore installs them in the job through INSTALL_ENVOY_GATEWAY: "true", which runs task install_envoy_gateway. The global.gatewayApi.installEnvoy: true of the gatewayapi overlay only makes the chart render the GatewayClass and EnvoyProxy resources that the reconciler applies.

review_ocp covers the opposite arrangement. Its gatewayapi-external-ocp overlay sets installEnvoy: false, so the chart renders no GatewayClass and no EnvoyProxy, and the Gateway attaches to the GatewayClass and OpenShift Gateway API controller that the cluster already provides. review_microshift does the same through the microshift overlay, with the ci-stack GatewayClass that the job serves with Envoy Gateway.

TEST_CR_FILES_DIR selects the test CR from config/test. base routes through the chart-bundled NGINX controller and is the default, overlays/gatewayapi switches to the Gateway API, overlays/gatewayapi-external-ocp adds the OpenShift specifics on top of it, and overlays/microshift enables GitLab Shell on top of those for the MicroShift cluster.

Add or remove a Kubernetes version

Edit the parallel:matrix in .gitlab/ci/review-k3d.gitlab-ci.yml. Each row sets:

• K3D_K8S_IMAGE: the rancher/k3s image tag that pins the Kubernetes version.
• K3D_K8S_VERSION_SLUG: a short label, for example v135, used in the child pipeline name.
• CLUSTER_MODE: "true" for the cluster-wide operator, "false" for the namespaced operator.
• Optional. K3D_PRIMARY: "primary" on exactly one row: the version that runs automatically on merge request pipelines.
• Optional. TEST_CR_FILES_DIR and INSTALL_ENVOY_GATEWAY: "true" for the Gateway API/Envoy variant instead of the default NGINX Ingress overlay.
• Optional. RUNNER_TAG to run the row on a runner fleet other than the default e2e, for example saas-linux-large-arm64 for the arm64 row.

MicroShift cluster test

review_microshift tests the operator on OpenShift without the shared cluster. To run it, start the microshift job, which runs it in a child pipeline from .gitlab/ci/microshift-pipeline.gitlab-ci.yml. It runs a single-use MicroShift cluster as a privileged container in the job’s Docker-in-Docker environment, then runs the same deploy and QA smoke steps as review_k3d. It runs on the e2e runner fleet.

MicroShift serves the SecurityContextConstraints and Route APIs, so the operator takes its OpenShift code paths, and every GitLab pod is admitted through an SCC. GitLab is served over plain HTTP through Gateway API, including Git over SSH. MicroShift gives the Gateway’s LoadBalancer Service the node IP, on which the job publishes ports 80 and 22.

qa_microshift_full, which is manual in the same child pipeline, runs the full suite in 7 self-contained workers, like qa_k3d_full.

.gitlab/ci/scripts/microshift.sh creates the cluster:

• It fails unless SCC admission rejects a pod with a fixed UID outside the namespace range.
• It skips the TopoLVM manifests of the image, which need an LVM volume group, and installs local-path-provisioner instead.
• It removes the OpenShift router, which would hold ports 80 and 443, and installs Envoy Gateway and the ci-stack GatewayClass. Envoy Gateway may use the nonroot-v2 SCC, because it runs as UID 65532.
• It lets the CloudNativePG operator use the nonroot-v2 SCC, because the operator chart pins a UID that restricted-v2 rejects.
• It adds a ClientTrafficPolicy that keeps %2F in paths for the Gateway, which the chart renders only when it configures Envoy Gateway itself.

Update the MicroShift image

MICROSHIFT_IMAGE in .gitlab/ci/scripts/microshift.sh pins a multi-architecture image of ghcr.io/microshift-io/microshift by digest. The MicroShift upstream releases list its tags, for example 4.22.0_202610071017_g0a516b4eb_4.22.0_okd_scos.ec.16. If no image exists for the OpenShift version you want, pin the nearest one and record the choice in ADR 22.

microshift_create applies only the kindnet and kube-proxy manifests of the image. When you update the image, compare /usr/lib/microshift/manifests.d/ in the new image with that list.

End-to-end suite jobs

.gitlab/ci/e2e.gitlab-ci.yml runs the black-box suites in test/e2e, one suite per job, each in a single-use k3d cluster inside its own Docker-in-Docker environment. They reuse .gitlab/ci/scripts/k3d.sh and k3d_collect_debug, and they pull the bridge image build_bridge_image already published rather than building their own.

They are peers of the k3d cluster tests above, not part of them: those deploy a whole GitLab and run gitlab-qa, while these drive custom resources through the API server. e2e_siphon runs on two Kubernetes versions to cover the CEL validation and the schema defaults of both API servers. e2e_upgrades deploys a whole GitLab: it installs a Static version and follows it through Patch and Latest upgrades. It is manual, because it takes most of an hour. e2e_migrate deploys a whole GitLab through the v1beta1 resource and migrates it to a GitLabCore. It runs on scheduled pipelines, and is manual on branch and merge request pipelines, because it rolls the instance out twice.

Both k3d-debug/ and e2e-artifacts/ are uploaded whatever the outcome. The second is written by the harness itself, before it deletes the namespace, which is the one k3d_collect_debug cannot do. See Tests.

QA pipelines

By default, QA pipelines run the smoke suite, a small subset of fast end-to-end functional tests that quickly checks basic functionality. To run the full suite of end-to-end tests instead, trigger the manual full-suite job for the cluster: qa_k3d_full for k3d (see k3d cluster tests), microshift and then qa_microshift_full in its child pipeline for MicroShift (see MicroShift cluster test), or, for the shared OpenShift cluster, ocp and then qa_ocp_full_suite_trigger in its child pipeline. qa_ocp_full then runs after review_ocp.

To debug failures in tests, please follow investigate QA failures guide.

Container image tags

CI pushes the Operator image to the project repository, $CI_REGISTRY_IMAGE (registry.gitlab.com/gitlab-org/cloud-native/gitlab-operator). It pushes the bridge image, built with the bridge Go build tag, to its own repository, $BRIDGE_REGISTRY_IMAGE ($CI_REGISTRY_IMAGE/bridge-staging), so the project repository holds only public images:

Bridge artifacts of a release

Every release tag in the canonical project also builds Bridge, and keeps it out of the release:

• build_bridge_image pushes bridge-staging:<version>.
• build_kubectl_bridge builds the kubectl bridge plugin for each platform in KUBECTL_BRIDGE_PLATFORMS: Linux and macOS, on amd64 and arm64. Each platform gets a kubectl-bridge-<os>-<arch>.tar.gz that holds the binary and the license, the archive format krew installs. It also runs scripts/v2alpha1_manifests.sh, which writes v2alpha1-crds.yaml and v2alpha1-rbac.yaml: the v2alpha1 definitions and the manager RBAC that the Helm chart does not ship, the same files task install_v2alpha1_crds applies.
• upload_kubectl_bridge uploads the archives, the two manifests, and a SHA256SUMS file to the kubectl-bridge generic package, with the tag as its version.

build_kubectl_bridge also runs on the default branch, and in branch and merge request pipelines that change the SPA, cmd/kubectl-bridge, Dockerfile.kubectl-bridge, go.mod, go.sum, the v2alpha1 definitions or RBAC, or the CI configuration. There it keeps its output as job artifacts, so a broken build shows before a tag. bridge_unit_tests compiles the plugin for any other Go change. The job runs Dockerfile.kubectl-bridge with buildx, like the container builds, but exports the files instead of pushing an image. The Dockerfile builds the SPA and cross-compiles the plugin with it embedded. To build the same binaries locally:

docker buildx build -f Dockerfile.kubectl-bridge \
  --platform linux/amd64,darwin/arm64 --output type=local,dest=out .

The Bridge jobs are allowed to fail on a release tag. build_bundle_image and image_certification wait for every earlier stage, so a failing Bridge job would otherwise hold back the release bundle and certification. The Bridge jobs include pull_addon_charts, which fetches the Siphon and Orbit charts from other projects’ registries. If it fails, build_bridge_image still runs, and then fails before it pushes an image without those charts.

On a release tag, build_kubectl_bridge passes the tag to the plugin build as its version. The kubectl bridge migrate preflight compares that version with the Operator image tag.

publish_release links only the manifests of the gitlab-operator package, so the release description names none of them. To install them, see Install from a release.

Container builds

The Operator image is built for every platform in BUILD_PLATFORMS (linux/amd64 and linux/arm64), on a single amd64 runner, with no emulation. .docker_build_job creates a docker-container buildx driver in its own DinD service, because the default docker driver cannot produce an image for a foreign platform.

One Dockerfile builds both images. build_bridge_image sets BUILD_TAGS: bridge, which docker_build_and_push passes as a build argument, so the manager is compiled with the bridge Go build tag. The public image jobs leave it empty.

The image is pushed with --output type=image,push=true,oci-artifact=false. BuildKit 0.32 began exporting attestation manifests as OCI artifacts, which name the image manifest in a subject field. The GitLab registry rejects a manifest whose subject is not present yet, and BuildKit pushes the attestation concurrently with the manifest it names, so a plain --push fails intermittently with blob unknown to registry naming that manifest. oci-artifact=false keeps the legacy attestation format, which has no subject, so there is nothing to race. The published index is unchanged: two platform manifests and two attestation manifests.

Do not drop the flag for a newer BuildKit. The OCI distribution spec requires a registry to accept a manifest whose subject is absent, so the v0.32.1 push-order workaround was reverted in v0.32.2 and v0.33.0 pushes like the version that failed. The fix belongs to the registry and is tracked in container-registry#2375; the flag can go once that ships and its feature flag is enabled.

The multi-node Kubernetes driver this replaced never hit it, because a node per architecture forces buildx to push each platform separately, committing every subject before the attestation that names it.

docker_build_and_push writes the pushed index digest to --metadata-file, and sign reads it back from there. cosign stores a signature against the digest, at <repo>:sha256-<digest>.sig, and cosign verify <repo>:<tag> resolves the tag to its digest first — so one signature covers every tag the build pushed, and the digest never has to be re-resolved from a tag another pipeline could have moved.

Every stage that executes anything is pinned to the build platform with --platform=${BUILDPLATFORM}. The Go stage cross-compiles through GOOS/GOARCH. The certs and runtime stages are deliberately left unpinned so that the published image is the target architecture; they only COPY, so no foreign binary is ever executed.

Adding a RUN to a runtime stage would make that stage need emulation. Build it on a native runner instead, for example with the saas-linux-large-arm64 tag, and join the results with docker buildx imagetools create.

image_certification certifies one architecture per entry in BUILD_PLATFORMS, so the platform list has a single definition in the repository.

build_bundle_image is OLM metadata rather than a runnable workload, and stays single-arch on podman.

DockerHub rate limits

By default, CI uses images from DockerHub. The shared runners by default use a mirror to avoid hitting DockerHub rate limits. If you use custom runnners, that don’t use caching or mirroring, you should enable the dependency proxy by setting the DOCKERHUB_PREFIX to your proxy, for example DOCKERHUB_PREFIX: ${CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX}, and DEPENDENCY_PROXY_LOGIN="true".

The container build context by default uses the gcr DockerHub mirror. This behavior can be changed by overriding the DOCKER_OPTIONS or DOCKER_MIRROR variables.