The GitLabCore reconciler

internal/controller/gitlabcore reconciles the apps.gitlab.com/v2alpha1 GitLabCore resource, the resource Bridge configures a GitLab instance through. For the design of the resource, see ADR 26. For what the reconciler does, see the GitLabCore reconciler spec.

The reconciler is alpha. The controllers and helm packages keep serving the v1beta1 GitLab resource, and the two paths share no code.

Enable the reconciler

The reconciler rides with Bridge and is gated twice over, like the Bridge server itself:

  • The bridge build tag compiles it in. Without the tag, gitlabcore_stub.go takes its place and neither the reconciler nor the v2alpha1 types reach the binary, so public images carry neither.
  • ENABLE_BRIDGE=true registers it at runtime, even in a tagged build. This gate is not only a feature switch: the GitLabCore definition ships in no release, and a watch on a definition the cluster does not serve fails the manager on start.

task install_v2alpha1_crds installs the definitions and grants the manager ServiceAccount what the reconciler needs beyond the chart permissions: gitlabcores, its status and finalizers subresources, and PodDisruptionBudgets. The chart grants the v1beta1 GitLab permissions only, and no PodDisruptionBudget permissions at all, because the v1beta1 controller applies none. The grant goes into a role of its own, so task deploy_operator does not revert it.

Install both and run the Operator against a development cluster:

shell
task install_v2alpha1_crds
ENABLE_BRIDGE=true HELM_CHARTS=$(pwd)/charts task run_bridge
kubectl apply -f config/samples/gitlabcore_v2alpha1.yaml

task run builds without the tag, so the reconciler is absent there. Build an image with task docker-build-bridge, which uses Dockerfile.bridge.

The sample points at no infrastructure. For a resource that is wired to PostgreSQL, Redis, and object storage, run bash scripts/dev_dependencies.sh setup: it provisions them and writes an external-deps-v2alpha1.yaml to apply as it is. For more information, see External dependencies.

Reconciler settings

These controllers/settings environment variables configure the dynamic chart pull (see What one reconcile does). None of them apply to the v1beta1 controller, which never reaches out to a chart repository.

VariableDefaultMeaning
ENABLE_DYNAMIC_CHART_PULLtrueFalls back to a pull when HELM_CHARTS does not carry the target version. Set to false for a cluster that must not reach out to the network. The version is then a configuration error instead.
DYNAMIC_CHART_REPOSITORYhttps://charts.gitlab.io/The Helm chart repository (an index.yaml repository, not an OCI registry) a pull downloads from. Must be an https:// URL unless DYNAMIC_CHART_ALLOW_HTTP overrides that.
DYNAMIC_CHART_ALLOW_HTTPfalseAllows DYNAMIC_CHART_REPOSITORY to be a plain http:// URL. Leave this off unless the repository is a disconnected cluster’s own internal mirror that only serves plain HTTP.
DYNAMIC_CHART_CACHE_DIRECTORY<os.TempDir()>/gitlab-operator-chartsWhere a pulled chart is cached on disk, so a repeated reconcile does not download it again. The manager container must be able to create and write to this directory. A containerSecurityContext.readOnlyRootFilesystem: true manager needs a writable volume mounted there; the chart mounts an emptyDir at the default path when bridge.enabled. A custom directory needs a volume of its own.
DYNAMIC_CHART_CACHE_TTL30mHow long a chart may sit in the cache directory, unused, before it is pruned. A Go duration, for example 72h. 0 disables pruning. A value with no unit, such as 30, fails to parse. Load logs that on stderr and keeps the previous value rather than silently taking it.
DYNAMIC_CHART_VERIFYtrue (chart: dynamicChartPull.verify)Verifies a pulled chart against its provenance file, <archive URL>.prov, before it is cached or rendered. Set to false for a mirror that serves no provenance files; the manager then logs on start that pulled charts are not verified.
DYNAMIC_CHART_KEYRINGemptyA keyring file (binary, GnuPG keybox, or ASCII-armored with any number of exports one after another), the only keys a pulled chart may be signed with. The Operator binary carries none: empty, with DYNAMIC_CHART_VERIFY on, fails every pull. With bridge.enabled, the Operator chart sets it to /etc/gitlab-operator/chart-keys/keyring.asc, the keyring it renders into a ConfigMap from dynamicChartPull.trustedKeys. Add an entry there for a mirror that signs the charts again or for a rotated GitLab key, or set gitlab-charts.asc to null to stop trusting the GitLab key.

What one reconcile does

  1. Reads the capabilities of the cluster with internal/render/capabilities.
  2. Resolves spec.version into status.targetVersion; see Version.
  3. Renders the GitLab chart with internal/render, using the release name and namespace of the resource.
  4. Runs the pre-install hooks with internal/render/hookexec, RBAC excluded, and only when the hooks of this release have not run yet.
  5. Applies the rendered objects server-side, definitions and RBAC excluded.
  6. Reports the readiness of the rendered Deployments and StatefulSets through status.conditions.

The hooks run once per release, not once per pass. ConditionInitialized carries the generation whose hooks completed, and status.hooksVersion the chart version. The hooks run again when either changes, or when a failed pass set Initialized to False. A multi-minor upgrade therefore runs the hooks of each intermediate chart, so a secret a newer chart generates exists before its pods start. An apply that fails is retried without the hooks.

Every successful reconcile asks for the next one 30 seconds later. The reconciler watches the GitLabCore resource and the ConfigMap its spec.version.versionRef names, and none of the objects it applies, so an object deleted or edited by hand is restored on the next pass. Every object is applied under the field manager gitlabcore-controller, with ownership forced. For the rules a consumer of internal/render follows, see Apply rendered objects. For why the loop works this way, see ADR 30.

The chart comes from the charts directory that HELM_CHARTS points at and the image bakes in. When that directory lacks the target version, the reconciler pulls it from a chart repository. For the settings, see Reconciler settings, and for the design, see ADR 32. A pulled chart is verified against the GitLab chart signing key; see ADR 35.

Pruning what a render stops producing

Every non-upgrade pass ends with a prune (release.Pruner.PruneOrphans), which deletes what the render no longer produces. The finalizer runs the same prune against an empty release. For what cleanup does, see Object cleanup, and for why it works this way, see ADR 29.

The prune lists every kind it covers, and records them in status.renderedKinds as <apiVersion>/<Kind>. The next pass lists those kinds again, so it also finds a kind the new render dropped entirely:

yaml
status:
  renderedKinds:
    - apps/v1/Deployment
    - gateway.networking.k8s.io/v1/HTTPRoute
    - v1/ConfigMap

A full GitLab release records 17 kinds. A kind skipped because the cluster does not serve it, or because it is RBAC the reconciler never applies, is left out of the record.

Effective values

EffectiveValues builds the values in three layers: the values derived from the structured fields, then spec.chart.values merged over them, then the Operator overrides.

FieldChart values
spec.hostnameglobal.hosts.gitlab.name, and global.hosts.domain from the parent domain
spec.editionglobal.edition
spec.license.secretRefglobal.gitlab.license.secret, global.gitlab.license.key
spec.postgresqlglobal.psql.host, global.psql.password.secret, global.psql.password.key
spec.redisglobal.redis.host, global.redis.auth.secret, global.redis.auth.key
spec.objectStorageglobal.appConfig.object_store.enabled, global.appConfig.object_store.connection.secret, global.appConfig.object_store.connection.key
spec.openbaoglobal.openbao.enabled, openbao.install, global.openbao.psql.host, global.openbao.psql.password.secret, global.openbao.psql.password.key, and optionally global.openbao.psql.port/database/username; openbao.serviceAccount.name, openbao.serviceAccount.create: false, openbao.role.create: false
spec.networkingglobal.gatewayApi.enabled, global.gatewayApi.configureEnvoy, gatewayApiResources.class.name, gatewayApiResources.class.enabled, gatewayApiResources.class.controllerName, global.gatewayApi.gatewayRef.name, global.gatewayApi.gatewayRef.namespace, gatewayApiResources.gateway.annotations

A hostname of gitlab.example.com therefore yields a domain of example.com, and with it the sibling hosts registry.example.com and kas.example.com. An apex hostname is its own domain, because stripping its first label would leave the public suffix.

spec.edition selects the image repository every component pulls from. It defaults to ee, which runs the Free feature set until a license activates more. Pick ce only for an instance that must carry no proprietary code.

Since chart version 10, the chart bundles neither PostgreSQL nor Redis, so spec.postgresql and spec.redis point at servers you run. Valkey stands in for Redis. Each maps a host and the Secret that holds the password. The port, the database, and the user keep their chart defaults, so an instance that needs another one sets it in the free-form values. For the versions and extensions GitLab requires, see the PostgreSQL requirements.

spec.objectStorage names the Secret that holds the object storage connection, and naming it also turns the consolidated object storage on: the connection configures nothing while it is off. The chart enables object storage for artifacts, LFS, uploads, and packages, and gives them no connection of their own, so a resource without this field and without the equivalent free-form values fails its checkConfig with the connection property can not be empty. The Secret carries the endpoint, the region, and the credentials in the connection format of the chart, and the Operator passes it through without reading it. The registry, the Pages daemon, and the backup toolbox read settings of their own, which stay in the free-form values.

spec.openbao backs the GitLab Secret Manager with an OpenBao instance, and naming it also turns on both the GitLab-side integration (global.openbao.enabled) and the bundled OpenBao subchart (openbao.install). OpenBao needs a PostgreSQL database of its own: it does not inherit the password of spec.postgresql, so spec.openbao.postgresql is a connection of its own, with passwordSecretRef required rather than defaulted. The port, the database, and the username fall back to the chart’s own defaults (5432, openbao, openbao) when left unset. See the OpenBao chart setup for what those values configure; the database and its role are still an administrator prerequisite, the same way spec.postgresql’s database is.

spec.openbao.serviceAccount.name names the ServiceAccount the OpenBao pod runs as. The Operator defaults openbao.serviceAccount.create and openbao.role.create to false, so the Role that grants get, update, and patch on Pods, and its RoleBinding, are an administrator prerequisite. For why, see ADR 31.

spec.networking selects at most one of three Gateway API modes. Leaving it unset turns the Gateway API off, through setGatewayDefault, and leaves networking to the free-form values.

ModeNamesThe chart provisions
envoyGatewayControllerThe controller of an installed Envoy Gateway. Immutable.The GatewayClass, named <namespace>-<name>-gw by gatewayClassName, the Gateway, and the Envoy extensions.
gatewayClassAn existing GatewayClass.The Gateway. The Envoy extensions only with configureEnvoy.
gatewayAn existing Gateway.The routes only. The Envoy extensions only with configureEnvoy.

gatewayClass and gateway set gatewayApiResources.class.enabled: false. envoyGatewayController and gatewayClass take a certManager.issuerRef, which setCertManagerValues writes as an annotation on gatewayApiResources.gateway.annotations. Both modes need chart 10.4.0 or later, so the renderRelease specs that cover them run only at or above that version (supportsConfigureEnvoy). Switching modes removes what the previous mode rendered on the next pass. For why the modes work this way, see ADR 33.

The derived layer also mirrors the shared secrets defaults of the v1beta1 controller:

DefaultReason
shared-secrets.serviceAccount.create: false, name: $GITLAB_MANAGER_SERVICE_ACCOUNTThe Job runs under the ServiceAccount of the Operator, which the Operator installation provisions.
shared-secrets.rbac.create: falseThe Operator creates no RBAC, and the Job needs none: its account already has it.
shared-secrets.securityContext.runAsUser: "", fsGroup: ""Keeps the Job compatible with the OpenShift nonroot SecurityContextConstraint, which assigns both itself.
registry.enabled: falseThe container registry keeps its images in object storage of its own, through registry.storage, which no structured field covers. An instance that wants one turns it back on in spec.chart.values, where it also supplies the storage.

That ServiceAccount has to exist in the namespace of the resource, with permission to manage Secrets there, before the first reconcile.

The free-form values win over the derived ones, as ADR 26 decides, which keeps them a working escape hatch. The Operator overrides win over both:

OverrideReason
installCertmanager: falsecert-manager is a prerequisite of the Operator. The cluster administrator installs it once, and it serves every instance.
gitlab-runner.install: falseThe GitLab Runner has a lifecycle of its own and is deployed through the Runner Operator.
nginx-ingress.enabled: false, nginx-ingress-geo.enabled: false, haproxy.install: false, traefik.install: false, global.gatewayApi.installEnvoy: falseThe Operator installs no networking controllers/Operators.
global.gatewayApi.configureCertmanager: falseThe chart’s own Let’s Encrypt ACME Issuer would write a second cert-manager.io/issuer annotation onto the one Gateway setCertManagerValues already annotates. An instance points at an existing Issuer or ClusterIssuer through spec.networking.*.certManager.issuerRef instead. The Ingress equivalent is not overridden; see ADR 33.

For why these are overrides, see ADR 31 and ADR 33.

Version

What unattended upgrades do is in the unattended upgrades spec, and why they work this way is in ADR 28. This section is the reference for what the reconciler reads and reports.

spec.version is a union discriminated by type, and the definition enforces it with CEL rules:

TypeFieldRuns
Staticversion (x.y.z), or versionRefThat version. An upgrade is a change of it.
Patchminor (x.y)The latest patch of that minor. Raising minor upgrades to it.
Latestmajor (x)The latest minor (N) of that major, with its latest patch.
Previousmajor (x)The minor before the newest release (N-1), within that major.
OldestSupportedmajor (x)The oldest minor GitLab still supports (N-2), within that major.

Previous and OldestSupported count minors across majors. A major with too few minors resolves to its first one.

A Static version can come from a ConfigMap in the namespace of the resource instead of the spec, so a platform that owns the ConfigMap sets the version while someone else owns the resource. The spec is in Version reference:

yaml
spec:
  version:
    type: Static
    versionRef:
      kind: ConfigMap
      name: platform-versions
      key: chartVersion

version and versionRef are mutually exclusive. The value is trimmed and must be a semantic version. It behaves as a literal version: a change of it upgrades, and a value below the deployed version is refused. The reconciler watches ConfigMaps, and a change to the referenced one reconciles every GitLabCore in the namespace that names it.

Every type but Static resolves against the index of DYNAMIC_CHART_REPOSITORY, cached for 30 minutes, or against the bundled charts with ENABLE_DYNAMIC_CHART_PULL=false. An instance ahead of the minor its type tracks keeps taking the patches of its own minor. A failed instance whose type follows the repository requeues every 10 minutes.

The reconciler reports the resolution through these fields, reasons, and events:

SignalSet when
status.availableVersionEvery pass: what spec.version resolves to now.
status.targetVersionThe version the instance converges to. It moves once a candidate passes its preflight, and holds while an upgrade runs.
status.rejectedVersionThe last candidate that failed its preflight for good.
Upgradeable=False, DowngradeRefusedspec.version resolves below the deployed version. Also a DowngradeRefused event.
Upgradeable=False, VersionUnresolvedspec.version does not resolve, for example a Patch minor without a release, or a versionRef whose ConfigMap or key is missing or holds no semantic version. The instance stays on its target, and an upgrade in progress carries on toward it. An unreachable repository sets nothing. A versionRef also records a VersionUnresolved event.
Initialized=False, VersionUnresolvedAs above, for an instance with no target yet, which fails. Also a VersionUnresolved event.
Upgradeable=False, MissingIntermediateChartA step of a multi-minor upgrade is not available. Also an UpgradeBlocked event.
Preflight=False, PreflightFailed or PreflightInconclusiveA candidate failed its preflight, or it could not be checked. PreflightFailed also records an event that names the check.
AutoUpgrade eventAn unattended upgrade starts, or a failed one moves on to a newer patch.
VersionChangeDeferred eventThe resolution changes while the target is held.

spec.upgrade.skipBatchedMigrationCheck lets an upgrade step advance without waiting for the batched background migrations of the previous version to finish. GitLab requires them complete before the next upgrade, so leave it off except on development or test instances.

Preflight

CheckCatches
Every step of the upgrade path resolvesA missing intermediate chart, anywhere on the path
Every step pulls and renders with the values of the instanceA missing or unreachable chart, the fail guards of the chart, and Operator overrides the chart no longer accepts
The objects of the first step pass a server-side dry-run applyAdmission policies, changed immutable fields, and APIs the cluster does not serve

The dry run applies what the first pass of reconcileUpgrade applies: the gated workloads paused, and the pre-migrations Job. For what it leaves out, see ADR 28.

Status

ConditionMeaning
InitializedThe chart resolved and rendered, and its hooks ran.
AvailableEvery rendered workload has its desired replicas ready.
ProgressingAn upgrade step is under way. The reason names the stage, such as RunningPreMigrations, RollingOutWorkloads, or WaitingForBatchedMigrations.
UpgradeableThe upgrade toward the target can be carried out; False when an intermediate chart is missing, a downgrade is refused, or the version does not resolve.
PreflightThe last candidate of an unattended upgrade passed its preflight (PreflightPassed); False when it failed (PreflightFailed) or could not complete (PreflightInconclusive). See Preflight.

status.phase reports Preparing, Running, or Failed, status.version records the chart version that was applied, and status.gitlabVersion the version of GitLab that chart deploys, such as 19.3.2. Both describe what was applied rather than what the spec asks for, so a resource stepping through a multi-minor upgrade reports the version it has converged to so far. For the version fields, see Version.

The application version is read off the render, not from the catalog of charts the Operator carries: a release can be rendered from a chart pulled at reconcile time, which the catalog knows nothing about. Chart 10.3 and later label the migrations Job gitlab.com/target-version, which is authoritative because the chart computes it and it accounts for a global.gitlabVersion override; earlier charts, and releases with the migrations component disabled, fall back to the appVersion the chart declares. This is the only place the application version is published, and Siphon reads it to pin its table definitions.

The reconciler installs no definitions and no RBAC

The reconciler never applies a CustomResourceDefinition, whether rendered from a template or carried in Result.CRDs, and nothing in the rbac.authorization.k8s.io group, hooks included. The cluster administrator provisions both. For why, see ADR 31.

An object that needs an API the cluster does not serve fails to apply, with the kind named in the error and in the Available condition. Either have the cluster administrator install that API, or turn the component off in spec.chart.values. The chart routes through the Gateway API by default, so a cluster without the Gateway API and Envoy Gateway needs global.gatewayApi.enabled: false.

A release whose hook RBAC was applied would fail on the first hook, because the Operator has no permission to manage it:

roles.rbac.authorization.k8s.io "gitlab-shared-secrets" is forbidden: User
"system:serviceaccount:gitlab-system:gitlab-manager" cannot delete resource "roles"

Deletion

Objects in the namespace of the resource carry a controller reference to it, so Kubernetes deletes them. A cluster-scoped object, in practice only the GatewayClass, can hold no such reference. The finalizer prunes it by the release labels internal/render stamps:

operator.gitlab.com/release-name        the name of the resource
operator.gitlab.com/release-namespace   its namespace

The finalizer lists the kinds in status.renderedKinds, so deletion never resolves or renders the chart. It lists the matches and deletes them one at a time rather than with DeleteAllOf, because the ClusterRole the Operator ships grants list and delete on gatewayclasses, but not deletecollection. A failure is logged and never blocks the deletion.

Known limits

  • The effective values are checked against the chart values.schema.json when the chart renders, not when the resource is written. A violation fails the reconcile. Rejecting it at admission is the job of the validating webhook ADR 26 describes, which is not built.
  • For the limits of the loop and its hooks, see the Consequences of ADR 30.
  • For the limits of cleanup, see the Consequences of ADR 29.
  • For the trust placed in a pulled chart, see ADR 32.

Run the tests

The unit tests are Ginkgo specs that need the chart archive on disk and no cluster:

shell
task retrieve-charts
HELM_CHARTS=$(pwd)/charts CHART_VERSION=$(head -n1 CHART_VERSIONS) \
  task unit-tests TEST_PKGS="./internal/controller/..."

Run the end-to-end test

internal/controller/gitlabcore/e2e_test.go creates a GitLabCore in a throwaway namespace, calls Reconcile the way the manager does, and asserts on what reaches the cluster: the secrets the hooks generated, the owner references and release labels of the applied objects, the status conditions, and the prune the finalizer performs. It is gated behind the e2e build tag, so the unit tests never pick it up.

Point kubectl at a throwaway cluster and install the definitions first, otherwise the test skips:

shell
task install_v2alpha1_crds
task e2e-tests TEST_PKGS="./internal/controller/..."

The release renders into the test namespace and nowhere else: NGINX, Prometheus, and the Gateway API are off through the values, and cert-manager and the Runner through the Operator overrides, so every object is namespaced and of a built-in kind. The pods never become ready, because there is no PostgreSQL, Redis, or object storage, which the test asserts rather than waits for.

VariableDescription
E2E_CLUSTER_SCOPED=1Enables the NGINX Ingress controller of the chart, so the release also carries RBAC and cluster-scoped objects. The IngressClass and the admission webhook are applied, the RBAC is skipped, and the finalizer prunes what it applied.
E2E_KEEP_NAMESPACE=1Keeps the namespace and the cluster-scoped objects for inspection.

After a run with E2E_KEEP_NAMESPACE=1, delete the cluster-scoped objects by hand. An admission webhook whose backing service is gone rejects unrelated writes across the whole cluster.