Object cleanup

Status: Implemented Issue: none Parent: none

Goal

An administrator who turns a component off, or deletes an instance, expects its objects to be gone. Garbage collection fires only on deletion, and an apply says nothing about the objects it is not given, so a dropped component otherwise keeps running. For the design, see ADR 29.

Requirements

  • A render that stops producing an object deletes it, whether its kind stays in use or goes with it.
  • Cleanup reaches a cluster-scoped object of the release, which the resource cannot own.
  • Cleanup never touches an object of another release, including a namesake resource of another kind.
  • An object that holds data, or that opts out, survives cleanup even when nothing renders it.
  • An object a reconciler applies beside the render survives while that reconciler still wants it.
  • Deleting a resource removes what the Operator applied for it, with no chart and no repository.
  • Cleanup neither fails nor blocks a reconcile. What it cannot delete is logged, and tried again on the next pass while the render still uses its kind.

Out of scope

  • Any namespace but the one of the resource.
  • The definitions and the RBAC of a chart.
  • Data, and state outside the cluster.
  • The deprecated v1beta1 GitLab resource.
  • Objects applied by an Operator version that recorded no kinds.
  • Retrying a failed sweep of a kind the render dropped, or of a resource being deleted.

FAQ

  • Why not delete every object that carries the release labels and let the apply put back what is still wanted? Because the window in between is an outage, and a delete that is not followed by a successful apply is a lost instance. Cleanup compares names per kind instead, and deletes only what the render no longer asks for.
  • What does cleanup never delete? A PersistentVolumeClaim, PersistentVolume, VolumeSnapshot or Secret, and anything annotated helm.sh/resource-policy: keep, which is the opt-out Helm defines. A Secret may hold a credential that nothing derives again, and a claim whose name legitimately changes, such as one fixed in a nameOverride, must not read as an abandoned object.
  • What happens to the objects that depend on a deleted one? They go with it. A Job otherwise orphans its Pods, and the name of the migrations Job carries a hash of the values, so every edit of the chart values leaves the Pods of the previous one behind.
  • How is an object of a namesake resource recognized? By the controller reference the apply leaves. The release labels carry the name of the resource and its namespace but nothing about its kind, so a GitLabCore and a Siphon that are both called gitlab in one namespace stamp the same labels. A cluster-scoped object can hold no reference, so there the labels decide alone, which is one reason cleanup goes no further than the namespace of the resource.
  • Why does cleanup stop at one namespace? Because the record it sweeps by holds a kind and not a namespace, a namespace-scoped Operator has no permission in another namespace, and an object it cannot own carries nothing to tell one resource from another. For more information, see ADR 29.
  • Why is the chart RBAC out of scope? The Operator never applies it, and never applies a definition either, so neither is its to remove. RBAC is also the only thing the charts render into another namespace, which is why nothing reaches that limit today. For more information, see ADR 31.
  • Why is the state of a pipeline out of scope? A replication slot, a publication and a NATS stream live outside the cluster, and the Operator did not create them. For more information, see Siphon.
  • Why does deleting a resource not render its chart to know what to delete? Because a deletion that depends on a reachable chart repository can fail permanently, and the resource would then be undeletable. The kinds the last render used are on the status instead.
  • Does an upgrade clean up as it goes? No. An upgrade applies the release in phases, and a phase names a subset of the release. The first ordinary reconcile after the upgrade catches up.
  • What about an instance from an Operator version that recorded no kinds? Cleanup can only reach a kind it recorded, so an object of a kind dropped before the upgrade stays behind, and an administrator deletes it by its release labels. For the deferred work, see issue 2242.