Namespace state inheritance propagation

GitLab represents the namespace hierarchy as a tree. The root element is the top-level namespace, and the child elements are subgroups and Namespaces::ProjectNamespace records. Each namespace has a state, such as ancestor_inherited (the default, active state), archived, or deletion_scheduled. When you set a state on a namespace, GitLab applies that state to every descendant in the tree.

For the structure of the namespace tree itself, see Namespaces. For the underlying batch iteration algorithm, see Batch iteration in a tree hierarchy.

Why propagation instead of lookup

The original model determined the effective state of a namespace by traversing up the ancestor chain at read time. A read of a deeply nested project checked each ancestor until it found a state. This approach had several problems:

• Every state check added query overhead because it walked the hierarchy.
• The effective state was inferred differently across descendants.
• Caching was required to keep read performance acceptable, which added complexity.

The propagation model writes the resulting state to every descendant when the state changes on a namespace. Reads then become a single-column comparison, with no ancestor traversal. The trade-off is that writes do more work, and that work happens asynchronously.

For more information, see ADR 003: State propagation model.

States that propagate

These states participate in propagation:

The precedence values mirror the canonical namespace state enum, so maintenance uses 6 rather than a contiguous rank. Only the ordering matters for precedence comparisons, and maintenance has the highest precedence.

A higher precedence state wins over a lower one. A lower-precedence propagation does not overwrite a descendant that is already in a higher-precedence state. For example, a namespace that is deletion_scheduled does not change when an ancestor propagates archived.

The maintenance state puts a namespace and its descendants into a read-only mode so the whole hierarchy is prepared for an organization transfer. The maintenance state is not yet used, and its propagation is not yet fully wired up.

Other states, such as transfer_in_progress and transfer_scheduled, do not propagate. A namespace-level transfer affects only the selected namespace, not the effective state of its descendants.

Components

Propagation uses four components that work together.

%%{init: { "flowchart": { "useMaxWidth": true } }}%%
flowchart TD
  accTitle: Namespace state propagation components
  accDescr: A state change writes an outbox record, a worker reads the outbox and drives the iterator, and the iterator returns descendant IDs to update.

  A[State change on a namespace] --> B[Outbox record in namespace_state_propagations]
  B --> C[StatePropagationWorker]
  C --> D[StatePropagationIterator]
  D --> E[Conditional UPDATE of descendant states]
  F[StatePropagationCronWorker] -.recovers lost or stalled records.-> B

Transactional outbox

When a namespace state changes, GitLab writes the state change and an outbox record in the same database transaction. The outbox record is a row in the namespace_state_propagations table. Because both writes commit together, the propagation request cannot be lost. If the worker that processes the record crashes, the record remains and is retried.

This pattern is the transactional outbox. The Namespaces::StatePropagation model stores the originating namespace_id, the source_state, the target_state, and a status of pending or processing. A unique partial index prevents more than one in-flight record for the same namespace and target state.

State precedence

The Namespaces::Stateful::StatePrecedence module decides which descendant states a propagation may overwrite. The overwritable_states(source_state, target_state) method returns the descendant states that the propagation can change, based on the precedence table.

The set of overwritable states depends on both the target state and the state the namespace is leaving:

When the source and target states have equal precedence, or the combination is unsupported, the method returns an empty array and no descendant changes.

State propagation worker

Namespaces::StatePropagationWorker performs the propagation. For a given namespace_id and target_state, the worker:

1. Finds the pending outbox record and marks it processing.
2. Computes the overwritable states with StatePrecedence.
3. Drives the iterator over descendants in batches.
4. Applies a conditional update for each batch.
5. Deletes the outbox record when propagation completes.

The conditional update only changes descendants that are still in an overwritable state:

UPDATE namespaces
SET state = :target_state
WHERE id IN (:batch_ids)
  AND state IN (:overwritable_states)

If the worker fails partway through, the outbox record stays in the table and the reconciliation worker re-enqueues the record.

State-aware iterator

Gitlab::Database::Namespaces::StatePropagationIterator is a subclass of the namespace tree iterator described in Batch iteration in a tree hierarchy. The base iterator walks the hierarchy depth-first and returns descendant IDs in batches. The subclass adds a state_filter that constrains the walk to namespaces whose state is in the filter.

The worker passes the overwritable states plus the target state as the filter. The target state is included so the walk does not stop at the origin namespace, which the caller already transitioned to the target state. The conditional update then guarantees that only descendants in an overwritable state change.

State alone can’t tell a descendant that scheduled its own deletion apart from one that inherited deletion_scheduled through propagation. The difference is namespace_details.deletion_scheduled_at. The schedule_deletion transition sets the column, and propagation never does.

When an ancestor cancels its deletion (deletion_scheduled to ancestor_inherited or archived), the worker passes options: { preserve_descendants_deletion: true }. With this option, the iterator treats any descendant with deletion_scheduled_at set as a boundary. The iterator skips that descendant and its subtree, so they keep their deletion schedule. The propagation root is never a boundary. Other state changes don’t set the option and don’t check namespace_details.

Reconciliation worker

Namespaces::StatePropagationCronWorker recovers lost or stalled propagation jobs. On a fixed schedule defined in config/schedule.yml, the worker:

1. Counts pending records that exceeded the stale threshold, and adds the count to stale_pending_total. The count runs before the reset, so records reset in the next step are not also counted as stale.
2. Resets processing records that exceeded the stuck threshold back to pending, clears started_at, and adds the number of reset records to stuck_processing_total.
3. Re-enqueues all pending records with StatePropagationWorker. The counters are updated before this step, so a failed re-enqueue doesn’t hide stale or stuck records.
4. Refreshes the pending and processing outbox gauges described in Metrics.

Deduplication on StatePropagationWorker ensures that a record still being processed is not run twice.

Metrics

Gitlab::NamespaceStatePropagation::Metrics defines the Prometheus metrics for propagation. The target_state label is the propagated state, such as archived. No metric has a namespace_id label, to keep cardinality bounded.

The duration histogram uses buckets of 0.1, 0.5, 1, 5, 15, 30, 60, 120, 300, 600, and 1800 seconds.

StatePropagationCronWorker refreshes both gauges at the end of every run. The gauges use the max multiprocess mode, because each count is table-wide and summing it across Sidekiq processes would over-count. A process keeps the last value it set until the cron worker runs on it again or the process restarts. As a result, after the outbox drains, a gauge can report a stale value from a process that ran the cron worker earlier.

To alert on stale records, use the stuck_processing_total and stale_pending_total counters instead of the gauges. Counters are monotonic and summed across processes, so they don’t report stale values after the outbox drains. Alert on increase(...[window]) > 0 instead of the absolute value. Every Sidekiq process creates both counters at zero when it starts. increase() ignores the first sample of a series, so a counter created by its first real increment would miss that increment. Because stale_pending_total adds the stale count on every run, its rate depends on the cron schedule.

StatePropagationCronWorker failures are not counted separately. Use the standard sidekiq_jobs_failed_total metric for the worker instead.

Logging

Propagation writes structured log entries to Gitlab::AppLogger. Each event has a distinct message value that you can search for:

In these entries:

• gl_namespace_id is the namespace where the propagation started.
• descendants_updated counts only the rows that the conditional update changed, so the value can be lower than the number of namespaces visited.
• duration_s covers only the batch walk.
• depth_before and depth_after are the cursor path lengths before and after the iterator rewinds past a boundary.

No entry is logged when the outbox record does not exist, when no descendant states can be overwritten, or when another job has already claimed the record. After the failure is logged, the error is raised again, so retry and reconciliation behave as before.

Walkthrough

Consider this hierarchy, where each node shows its current state:

%%{init: { "flowchart": { "useMaxWidth": true } }}%%
flowchart TD
  accTitle: Example namespace hierarchy before propagation
  accDescr: A top-level group that is active, with one active subgroup, one archived subgroup, and one subgroup scheduled for deletion.

  Root["Top-level group (ancestor_inherited)"] --> A["Subgroup A (ancestor_inherited)"]
  Root --> B["Subgroup B (archived)"]
  Root --> C["Subgroup C (deletion_scheduled)"]
  A --> A1["Project A1 (ancestor_inherited)"]
  B --> B1["Project B1 (archived)"]

When you archive the top-level group, the propagation runs with source_state of ancestor_inherited and target_state of archived. StatePrecedence.overwritable_states(:ancestor_inherited, :archived) returns [:ancestor_inherited], so only active descendants change.

The result is:

%%{init: { "flowchart": { "useMaxWidth": true } }}%%
flowchart TD
  accTitle: Example namespace hierarchy after archiving the top-level group
  accDescr: The top-level group, subgroup A, and project A1 become archived. Subgroup B stays archived. Subgroup C stays scheduled for deletion.

  Root["Top-level group (archived)"] --> A["Subgroup A (archived)"]
  Root --> B["Subgroup B (archived)"]
  Root --> C["Subgroup C (deletion_scheduled)"]
  A --> A1["Project A1 (archived)"]
  B --> B1["Project B1 (archived)"]

In this example:

• Subgroup A and project A1 change from ancestor_inherited to archived.
• Subgroup B and project B1 are already archived, so they do not change.
• Subgroup C stays deletion_scheduled because deletion_scheduled has higher precedence than archived.

Foundational merge requests

The propagation algorithm was built across several merge requests:

• Add the namespace state propagation iterator adds the StatePropagationIterator.
• Add Namespaces::Stateful::StatePrecedence adds the overwritable states logic.
• Add Namespaces::StatePropagationWorker adds the propagation worker.
• Add Namespaces::StatePropagationCronWorker adds the reconciliation worker.

For the overall plan, see epic 21607.