The GitLabCore reconciler
internal/controller/gitlabcore reconciles the apps.gitlab.com/v2alpha1 GitLabCore resource,
the resource Bridge configures a GitLab instance through. For the design of the resource, see
ADR 26. For what the reconciler does, see
the GitLabCore reconciler spec.
The reconciler is alpha. The controllers and helm packages keep serving the v1beta1 GitLab
resource, and the two paths share no code.
Enable the reconciler
The reconciler rides with Bridge and is gated twice over, like the Bridge server itself:
- The
bridgebuild tag compiles it in. Without the tag,gitlabcore_stub.gotakes its place and neither the reconciler nor thev2alpha1types reach the binary, so public images carry neither. ENABLE_BRIDGE=trueregisters it at runtime, even in a tagged build. This gate is not only a feature switch: theGitLabCoredefinition ships in no release, and a watch on a definition the cluster does not serve fails the manager on start.
task install_v2alpha1_crds installs the definitions and grants the manager ServiceAccount what
the reconciler needs beyond the chart permissions: gitlabcores, its status and finalizers
subresources, and PodDisruptionBudgets. The chart grants the v1beta1 GitLab permissions only, and
no PodDisruptionBudget permissions at all, because the v1beta1 controller applies none. The grant
goes into a role of its own, so task deploy_operator does not revert it.
Install both and run the Operator against a development cluster:
task install_v2alpha1_crds
ENABLE_BRIDGE=true HELM_CHARTS=$(pwd)/charts task run_bridge
kubectl apply -f config/samples/gitlabcore_v2alpha1.yamltask run builds without the tag, so the reconciler is absent there. Build an image with
task docker-build-bridge, which uses Dockerfile.bridge.
The sample points at no infrastructure. For a resource that is wired to PostgreSQL, Redis, and object
storage, run bash scripts/dev_dependencies.sh setup: it provisions them and writes an
external-deps-v2alpha1.yaml to apply as it is. For more information, see
External dependencies.
Reconciler settings
These controllers/settings environment variables configure the dynamic chart pull (see
What one reconcile does). None of them apply to the v1beta1
controller, which never reaches out to a chart repository.
| Variable | Default | Meaning |
|---|---|---|
ENABLE_DYNAMIC_CHART_PULL | true | Falls back to a pull when HELM_CHARTS does not carry the target version. Set to false for a cluster that must not reach out to the network. The version is then a configuration error instead. |
DYNAMIC_CHART_REPOSITORY | https://charts.gitlab.io/ | The Helm chart repository (an index.yaml repository, not an OCI registry) a pull downloads from. Must be an https:// URL unless DYNAMIC_CHART_ALLOW_HTTP overrides that. |
DYNAMIC_CHART_ALLOW_HTTP | false | Allows DYNAMIC_CHART_REPOSITORY to be a plain http:// URL. Leave this off unless the repository is a disconnected cluster’s own internal mirror that only serves plain HTTP. |
DYNAMIC_CHART_CACHE_DIRECTORY | <os.TempDir()>/gitlab-operator-charts | Where a pulled chart is cached on disk, so a repeated reconcile does not download it again. The manager container must be able to create and write to this directory. A containerSecurityContext.readOnlyRootFilesystem: true manager needs a writable volume mounted there; the chart mounts an emptyDir at the default path when bridge.enabled. A custom directory needs a volume of its own. |
DYNAMIC_CHART_CACHE_TTL | 30m | How long a chart may sit in the cache directory, unused, before it is pruned. A Go duration, for example 72h. 0 disables pruning. A value with no unit, such as 30, fails to parse. Load logs that on stderr and keeps the previous value rather than silently taking it. |
DYNAMIC_CHART_VERIFY | true (chart: dynamicChartPull.verify) | Verifies a pulled chart against its provenance file, <archive URL>.prov, before it is cached or rendered. Set to false for a mirror that serves no provenance files; the manager then logs on start that pulled charts are not verified. |
DYNAMIC_CHART_KEYRING | empty | A keyring file (binary, GnuPG keybox, or ASCII-armored with any number of exports one after another), the only keys a pulled chart may be signed with. The Operator binary carries none: empty, with DYNAMIC_CHART_VERIFY on, fails every pull. With bridge.enabled, the Operator chart sets it to /etc/gitlab-operator/chart-keys/keyring.asc, the keyring it renders into a ConfigMap from dynamicChartPull.trustedKeys. Add an entry there for a mirror that signs the charts again or for a rotated GitLab key, or set gitlab-charts.asc to null to stop trusting the GitLab key. |
What one reconcile does
- Reads the capabilities of the cluster with
internal/render/capabilities. - Resolves
spec.versionintostatus.targetVersion; see Version. - Renders the GitLab chart with
internal/render, using the release name and namespace of the resource. - Runs the
pre-installhooks withinternal/render/hookexec, RBAC excluded, and only when the hooks of this release have not run yet. - Applies the rendered objects server-side, definitions and RBAC excluded.
- Reports the readiness of the rendered
DeploymentsandStatefulSetsthroughstatus.conditions.
The hooks run once per release, not once per pass. ConditionInitialized carries the generation
whose hooks completed, and status.hooksVersion the chart version. The hooks run again when either
changes, or when a failed pass set Initialized to False. A multi-minor upgrade therefore runs
the hooks of each intermediate chart, so a secret a newer chart generates exists before its pods
start. An apply that fails is retried without the hooks.
Every successful reconcile asks for the next one 30 seconds later. The reconciler watches the
GitLabCore resource and the ConfigMap its spec.version.versionRef names, and none of the
objects it applies, so an object deleted or edited by hand is restored on the next pass. Every
object is applied under the field manager gitlabcore-controller, with ownership forced. For the
rules a consumer of internal/render follows, see
Apply rendered objects. For why the loop works this way, see
ADR 30.
The chart comes from the charts directory that HELM_CHARTS points at and the image bakes in. When
that directory lacks the target version, the reconciler pulls it from a chart repository. For the
settings, see Reconciler settings, and for the design, see
ADR 32. A pulled chart is verified against the GitLab chart
signing key; see ADR 35.
Pruning what a render stops producing
Every non-upgrade pass ends with a prune (release.Pruner.PruneOrphans), which deletes what the
render no longer produces. The finalizer runs the same prune against an empty release. For what
cleanup does, see Object cleanup, and for why it works this
way, see ADR 29.
The prune lists every kind it covers, and records them in status.renderedKinds as
<apiVersion>/<Kind>. The next pass lists those kinds again, so it also finds a kind the new render
dropped entirely:
status:
renderedKinds:
- apps/v1/Deployment
- gateway.networking.k8s.io/v1/HTTPRoute
- v1/ConfigMapA full GitLab release records 17 kinds. A kind skipped because the cluster does not serve it, or because it is RBAC the reconciler never applies, is left out of the record.
Effective values
EffectiveValues builds the values in three layers: the values derived from the structured fields,
then spec.chart.values merged over them, then the Operator overrides.
| Field | Chart values |
|---|---|
spec.hostname | global.hosts.gitlab.name, and global.hosts.domain from the parent domain |
spec.edition | global.edition |
spec.license.secretRef | global.gitlab.license.secret, global.gitlab.license.key |
spec.postgresql | global.psql.host, global.psql.password.secret, global.psql.password.key |
spec.redis | global.redis.host, global.redis.auth.secret, global.redis.auth.key |
spec.objectStorage | global.appConfig.object_store.enabled, global.appConfig.object_store.connection.secret, global.appConfig.object_store.connection.key |
spec.openbao | global.openbao.enabled, openbao.install, global.openbao.psql.host, global.openbao.psql.password.secret, global.openbao.psql.password.key, and optionally global.openbao.psql.port/database/username; openbao.serviceAccount.name, openbao.serviceAccount.create: false, openbao.role.create: false |
spec.networking | global.gatewayApi.enabled, global.gatewayApi.configureEnvoy, gatewayApiResources.class.name, gatewayApiResources.class.enabled, gatewayApiResources.class.controllerName, global.gatewayApi.gatewayRef.name, global.gatewayApi.gatewayRef.namespace, gatewayApiResources.gateway.annotations |
A hostname of gitlab.example.com therefore yields a domain of example.com, and with it the
sibling hosts registry.example.com and kas.example.com. An apex hostname is its own domain,
because stripping its first label would leave the public suffix.
spec.edition selects the image repository every component pulls from. It defaults to ee, which
runs the Free feature set until a license activates more. Pick ce only for an instance that must
carry no proprietary code.
Since chart version 10, the chart bundles neither PostgreSQL nor Redis, so spec.postgresql and
spec.redis point at servers you run. Valkey stands in for Redis. Each maps a host and the Secret
that holds the password. The port, the database, and the user keep their chart defaults, so an
instance that needs another one sets it in the free-form values. For the versions and extensions
GitLab requires, see the PostgreSQL requirements.
spec.objectStorage names the Secret that holds the object storage connection, and naming it also
turns the consolidated object storage on: the connection configures nothing while it is off. The
chart enables object storage for artifacts, LFS, uploads, and packages, and gives them no connection
of their own, so a resource without this field and without the equivalent free-form values fails its
checkConfig with the connection property can not be empty. The
Secret carries the endpoint, the region, and the credentials in the
connection format of the chart, and the
Operator passes it through without reading it. The registry, the Pages daemon, and the backup
toolbox read settings of their own, which stay in the free-form values.
spec.openbao backs the GitLab Secret Manager with an OpenBao instance, and naming it also turns on
both the GitLab-side integration (global.openbao.enabled) and the bundled OpenBao subchart
(openbao.install). OpenBao needs a PostgreSQL database of its own: it does not inherit the password
of spec.postgresql, so spec.openbao.postgresql is a connection of its own, with passwordSecretRef
required rather than defaulted. The port, the database, and the username fall back to the chart’s own
defaults (5432, openbao, openbao) when left unset. See
the OpenBao chart setup
for what those values configure; the database and its role are still an administrator prerequisite,
the same way spec.postgresql’s database is.
spec.openbao.serviceAccount.name names the ServiceAccount the OpenBao pod runs as. The Operator
defaults openbao.serviceAccount.create and openbao.role.create to false, so the Role that grants
get, update, and patch on Pods, and its RoleBinding, are an administrator prerequisite. For why,
see ADR 31.
spec.networking selects at most one of three Gateway API modes. Leaving it unset turns the Gateway
API off, through setGatewayDefault, and leaves networking to the free-form values.
| Mode | Names | The chart provisions |
|---|---|---|
envoyGatewayController | The controller of an installed Envoy Gateway. Immutable. | The GatewayClass, named <namespace>-<name>-gw by gatewayClassName, the Gateway, and the Envoy extensions. |
gatewayClass | An existing GatewayClass. | The Gateway. The Envoy extensions only with configureEnvoy. |
gateway | An existing Gateway. | The routes only. The Envoy extensions only with configureEnvoy. |
gatewayClass and gateway set gatewayApiResources.class.enabled: false. envoyGatewayController
and gatewayClass take a certManager.issuerRef, which setCertManagerValues writes as an
annotation on gatewayApiResources.gateway.annotations. Both modes need chart 10.4.0 or later, so
the renderRelease specs that cover them run only at or above that version
(supportsConfigureEnvoy). Switching modes removes what the previous mode rendered on the next
pass. For why the modes work this way, see ADR 33.
The derived layer also mirrors the shared secrets defaults of the v1beta1 controller:
| Default | Reason |
|---|---|
shared-secrets.serviceAccount.create: false, name: $GITLAB_MANAGER_SERVICE_ACCOUNT | The Job runs under the ServiceAccount of the Operator, which the Operator installation provisions. |
shared-secrets.rbac.create: false | The Operator creates no RBAC, and the Job needs none: its account already has it. |
shared-secrets.securityContext.runAsUser: "", fsGroup: "" | Keeps the Job compatible with the OpenShift nonroot SecurityContextConstraint, which assigns both itself. |
registry.enabled: false | The container registry keeps its images in object storage of its own, through registry.storage, which no structured field covers. An instance that wants one turns it back on in spec.chart.values, where it also supplies the storage. |
That ServiceAccount has to exist in the namespace of the resource, with permission to manage Secrets there, before the first reconcile.
The free-form values win over the derived ones, as ADR 26 decides, which keeps them a working escape hatch. The Operator overrides win over both:
| Override | Reason |
|---|---|
installCertmanager: false | cert-manager is a prerequisite of the Operator. The cluster administrator installs it once, and it serves every instance. |
gitlab-runner.install: false | The GitLab Runner has a lifecycle of its own and is deployed through the Runner Operator. |
nginx-ingress.enabled: false, nginx-ingress-geo.enabled: false, haproxy.install: false, traefik.install: false, global.gatewayApi.installEnvoy: false | The Operator installs no networking controllers/Operators. |
global.gatewayApi.configureCertmanager: false | The chart’s own Let’s Encrypt ACME Issuer would write a second cert-manager.io/issuer annotation onto the one Gateway setCertManagerValues already annotates. An instance points at an existing Issuer or ClusterIssuer through spec.networking.*.certManager.issuerRef instead. The Ingress equivalent is not overridden; see ADR 33. |
For why these are overrides, see ADR 31 and ADR 33.
Version
What unattended upgrades do is in the unattended upgrades spec, and why they work this way is in ADR 28. This section is the reference for what the reconciler reads and reports.
spec.version is a union discriminated by type, and the definition enforces it with CEL rules:
| Type | Field | Runs |
|---|---|---|
Static | version (x.y.z), or versionRef | That version. An upgrade is a change of it. |
Patch | minor (x.y) | The latest patch of that minor. Raising minor upgrades to it. |
Latest | major (x) | The latest minor (N) of that major, with its latest patch. |
Previous | major (x) | The minor before the newest release (N-1), within that major. |
OldestSupported | major (x) | The oldest minor GitLab still supports (N-2), within that major. |
Previous and OldestSupported count minors across majors. A major with too few minors resolves to
its first one.
A Static version can come from a ConfigMap in the namespace of the resource instead of the spec,
so a platform that owns the ConfigMap sets the version while someone else owns the resource. The
spec is in Version reference:
spec:
version:
type: Static
versionRef:
kind: ConfigMap
name: platform-versions
key: chartVersionversion and versionRef are mutually exclusive. The value is trimmed and must be a semantic
version. It behaves as a literal version: a change of it upgrades, and a value below the deployed
version is refused. The reconciler watches ConfigMaps, and a change to the referenced one reconciles
every GitLabCore in the namespace that names it.
Every type but Static resolves against the index of DYNAMIC_CHART_REPOSITORY, cached for 30
minutes, or against the bundled charts with ENABLE_DYNAMIC_CHART_PULL=false. An instance ahead of
the minor its type tracks keeps taking the patches of its own minor. A failed instance whose type
follows the repository requeues every 10 minutes.
The reconciler reports the resolution through these fields, reasons, and events:
| Signal | Set when |
|---|---|
status.availableVersion | Every pass: what spec.version resolves to now. |
status.targetVersion | The version the instance converges to. It moves once a candidate passes its preflight, and holds while an upgrade runs. |
status.rejectedVersion | The last candidate that failed its preflight for good. |
Upgradeable=False, DowngradeRefused | spec.version resolves below the deployed version. Also a DowngradeRefused event. |
Upgradeable=False, VersionUnresolved | spec.version does not resolve, for example a Patch minor without a release, or a versionRef whose ConfigMap or key is missing or holds no semantic version. The instance stays on its target, and an upgrade in progress carries on toward it. An unreachable repository sets nothing. A versionRef also records a VersionUnresolved event. |
Initialized=False, VersionUnresolved | As above, for an instance with no target yet, which fails. Also a VersionUnresolved event. |
Upgradeable=False, MissingIntermediateChart | A step of a multi-minor upgrade is not available. Also an UpgradeBlocked event. |
Preflight=False, PreflightFailed or PreflightInconclusive | A candidate failed its preflight, or it could not be checked. PreflightFailed also records an event that names the check. |
AutoUpgrade event | An unattended upgrade starts, or a failed one moves on to a newer patch. |
VersionChangeDeferred event | The resolution changes while the target is held. |
spec.upgrade.skipBatchedMigrationCheck lets an upgrade step advance without waiting for the
batched background migrations of the previous version to finish. GitLab requires them complete
before the next upgrade, so leave it off except on development or test instances.
Preflight
| Check | Catches |
|---|---|
| Every step of the upgrade path resolves | A missing intermediate chart, anywhere on the path |
| Every step pulls and renders with the values of the instance | A missing or unreachable chart, the fail guards of the chart, and Operator overrides the chart no longer accepts |
| The objects of the first step pass a server-side dry-run apply | Admission policies, changed immutable fields, and APIs the cluster does not serve |
The dry run applies what the first pass of reconcileUpgrade applies: the gated workloads paused,
and the pre-migrations Job. For what it leaves out, see ADR 28.
Status
| Condition | Meaning |
|---|---|
Initialized | The chart resolved and rendered, and its hooks ran. |
Available | Every rendered workload has its desired replicas ready. |
Progressing | An upgrade step is under way. The reason names the stage, such as RunningPreMigrations, RollingOutWorkloads, or WaitingForBatchedMigrations. |
Upgradeable | The upgrade toward the target can be carried out; False when an intermediate chart is missing, a downgrade is refused, or the version does not resolve. |
Preflight | The last candidate of an unattended upgrade passed its preflight (PreflightPassed); False when it failed (PreflightFailed) or could not complete (PreflightInconclusive). See Preflight. |
status.phase reports Preparing, Running, or Failed, status.version records the chart
version that was applied, and status.gitlabVersion the version of GitLab that chart deploys, such
as 19.3.2. Both describe what was applied rather than what the spec asks for, so a resource
stepping through a multi-minor upgrade reports the version it has converged to so far.
For the version fields, see Version.
The application version is read off the render, not from the catalog of charts the Operator carries:
a release can be rendered from a chart pulled at reconcile time, which the catalog knows nothing
about. Chart 10.3 and later label the migrations Job gitlab.com/target-version, which is
authoritative because the chart computes it and it accounts for a global.gitlabVersion override;
earlier charts, and releases with the migrations component disabled, fall back to the appVersion
the chart declares. This is the only place the application version is published, and Siphon reads
it to pin its table definitions.
The reconciler installs no definitions and no RBAC
The reconciler never applies a CustomResourceDefinition, whether rendered from a template or
carried in Result.CRDs, and nothing in the rbac.authorization.k8s.io group, hooks included. The
cluster administrator provisions both. For why, see
ADR 31.
An object that needs an API the cluster does not serve fails to apply, with the kind named in the
error and in the Available condition. Either have the cluster administrator install that API, or
turn the component off in spec.chart.values. The chart routes through the Gateway API by default,
so a cluster without the Gateway API and Envoy Gateway needs global.gatewayApi.enabled: false.
A release whose hook RBAC was applied would fail on the first hook, because the Operator has no permission to manage it:
roles.rbac.authorization.k8s.io "gitlab-shared-secrets" is forbidden: User
"system:serviceaccount:gitlab-system:gitlab-manager" cannot delete resource "roles"Deletion
Objects in the namespace of the resource carry a controller reference to it, so Kubernetes deletes
them. A cluster-scoped object, in practice only the GatewayClass, can hold no such reference. The
finalizer prunes it by the release labels internal/render stamps:
operator.gitlab.com/release-name the name of the resource
operator.gitlab.com/release-namespace its namespaceThe finalizer lists the kinds in status.renderedKinds, so deletion never resolves or renders the
chart. It lists the matches and deletes them one at a time rather than with DeleteAllOf, because
the ClusterRole the Operator ships grants list and delete on gatewayclasses, but not
deletecollection. A failure is logged and never blocks the deletion.
Known limits
- The effective values are checked against the chart
values.schema.jsonwhen the chart renders, not when the resource is written. A violation fails the reconcile. Rejecting it at admission is the job of the validating webhook ADR 26 describes, which is not built. - For the limits of the loop and its hooks, see the Consequences of ADR 30.
- For the limits of cleanup, see the Consequences of ADR 29.
- For the trust placed in a pulled chart, see ADR 32.
Run the tests
The unit tests are Ginkgo specs that need the chart archive on disk and no cluster:
task retrieve-charts
HELM_CHARTS=$(pwd)/charts CHART_VERSION=$(head -n1 CHART_VERSIONS) \
task unit-tests TEST_PKGS="./internal/controller/..."Run the end-to-end test
internal/controller/gitlabcore/e2e_test.go creates a GitLabCore in a throwaway namespace, calls
Reconcile the way the manager does, and asserts on what reaches the cluster: the secrets the hooks
generated, the owner references and release labels of the applied objects, the status conditions, and
the prune the finalizer performs. It is gated behind the e2e build tag, so the unit tests never
pick it up.
Point kubectl at a throwaway cluster and install the definitions first, otherwise the test skips:
task install_v2alpha1_crds
task e2e-tests TEST_PKGS="./internal/controller/..."The release renders into the test namespace and nowhere else: NGINX, Prometheus, and the Gateway API are off through the values, and cert-manager and the Runner through the Operator overrides, so every object is namespaced and of a built-in kind. The pods never become ready, because there is no PostgreSQL, Redis, or object storage, which the test asserts rather than waits for.
| Variable | Description |
|---|---|
E2E_CLUSTER_SCOPED=1 | Enables the NGINX Ingress controller of the chart, so the release also carries RBAC and cluster-scoped objects. The IngressClass and the admission webhook are applied, the RBAC is skipped, and the finalizer prunes what it applied. |
E2E_KEEP_NAMESPACE=1 | Keeps the namespace and the cluster-scoped objects for inspection. |
After a run with E2E_KEEP_NAMESPACE=1, delete the cluster-scoped objects by hand. An admission
webhook whose backing service is gone rejects unrelated writes across the whole cluster.