Migrations Job failures

Status: Implemented Issue: none Parent: GitLabCore reconciler

Goal

Each step of a zero-downtime upgrade gates on three Jobs: the pre-deployment migrations, the full migrations, and the batched background migrations wait. A Job retries a failed attempt on its own. The chart restarts a failed container in place, and the Job replaces a pod that failed as a whole, for example an evicted one. An administrator watching the upgrade sees a retry as a retry, and gets what they need to act on a Job that ran out of retries.

Requirements

  • A Job of an upgrade has failed only when it reports the condition Failed with status True.
  • A Job retries when its containers restarted in place, or when pods of it failed as a whole. The instance then stays in phase Upgrading, and its gate keeps waiting. Progressing is True, with the reason RetryingPreMigrations, RetryingPostMigrations, or RetryingBatchedMigrationsCheck.
  • The message of a retry counts the failed attempts against the limits of the Job controller. Container restarts count against spec.backoffLimit, and failed pods against spec.backoffLimit + 1. A failed run counts as soon as it fails, so during its back-off the count is one restart ahead of the Job controller. The message reports the count nearer its limit: the failed attempt, the attempts left, and the last failure, with its reason, exit code, and message.
  • The message of a restart gives kubectl -n <namespace> logs job/<name> -c <container> --previous. The message of a failed pod gives kubectl -n <namespace> describe job/<name>.
  • A retrying upgrade holds its target. It does not move on to a newer patch, and starts no other Job for the gate.
  • After the Job of a gate exists, the gate reports it, even while the workloads of the release are not ready. A pod that restarts or a scale-up changes neither the condition nor the phase.
  • A Job that reports FailureTarget without Failed is failing, and its pods may still run. The instance stays in phase Upgrading and holds its target. Progressing keeps the reason of the gate, and its message says that the Job is failing and why.
  • A Job that reports Failed fails the upgrade: phase Failed, Progressing False with the reason MigrationsJobFailed or BatchedMigrationsCheckFailed. The message carries the reason and message of the Failed condition, the last failure, and kubectl -n <namespace> describe job/<name>. The last failure is the one a retry recorded, even after the pods are gone, or the evicted or preempted pod that took the failed pods past their limit. The message says that a change to the spec reruns the upgrade with new Jobs.
  • A pass that sees a failed count rise records one Warning event on the GitLabCore, MigrationsRetrying or BatchedMigrationsCheckRetrying. A failure records one, MigrationsFailed or BatchedMigrationsCheckFailed. Other passes record none.
  • A retry or a failure shows up on the first pass after the Job reports it, at most 30 seconds later.
  • The web UI marks a retrying gate apart from a running one, and shows its message. The command of a retrying gate or a failed upgrade is shown as code.

Out of scope

  • The deadline of the batched background migrations wait Job. DeadlineExceeded stays terminal.
  • A procedure to recover a failed upgrade, and the BYPASS_SCHEMA_VERSION the pods keep meanwhile.
  • Changing restartPolicy, backoffLimit, or any other part of the chart.
  • The migrations Job of the deprecated v1beta1 GitLab controller. For a defect there, see https://gitlab.com/gitlab-org/cloud-native/gitlab-operator/-/work_items/2309.

FAQ

  • Where are the logs of a Job that failed? Often gone. The Job controller deletes the pod when it gives up, so the condition keeps the last failure, and kubectl describe shows the events of the Job. For the design, see ADR 45.
  • How soon does a retry show up? The reconciler does not watch Jobs or pods, so the next pass reports it, at most 30 seconds later.
  • How is a failed upgrade retried? A change to the GitLabCore spec raises the generation, which names new Jobs, so the upgrade runs again from its first gate. An unattended upgrade also moves on to a newer patch of the same minor once one is released.