Chart provenance verification

Status: Implemented Issue: https://gitlab.com/gitlab-org/cloud-native/gitlab-operator/-/work_items/2213 Parent: GitLabCore reconciler

Goal

An administrator whose GitLabCore runs a chart version the Operator does not bundle knows that the chart is the one GitLab signed. A compromised or tampered repository, mirror, or cache cannot change what the Operator applies.

Requirements

  • A pulled chart renders only when its provenance file, the archive URL with .prov appended, is signed by a trusted key and records the SHA-256 digest of the archive. The chart and version it signs must be the ones requested.
  • The only trusted keys are those in the keyring file DYNAMIC_CHART_KEYRING names. The Operator binary carries no key: with verification on and no keyring, every pull fails.
  • With bridge.enabled, the Operator chart renders every dynamicChartPull.trustedKeys entry into one keyring in a ConfigMap, mounts it, and points DYNAMIC_CHART_KEYRING at it. The default entry is the GitLab chart signing key, 5E46F79EF5836E986A663B4AE30F9C687683D663. Without bridge.enabled, the chart renders none of these.
  • A key added to or removed from that ConfigMap applies to the next pull, without a restart.
  • A chart that fails verification, or keys that do not load, are never cached or rendered. The pass sets Initialized to False with reason ChartPullFailed and a message that names the version, and the phase to Failed. The resource pulls again every 10 minutes, even at a Static version.
  • A provenance file the repository fails to serve right now (408, 429, 5xx, or no response) is retried on the next pass, and does not fail the resource.
  • A missing or empty key ConfigMap does not stop the manager. It logs on start that pulled charts cannot be verified.
  • A cached chart renders only if it is the archive the same keyring file verified since the manager started. A key removed from the keyring, or replaced by its revoked export, stops trusting it. Any other cached chart is pulled again. The render uses the bytes that were checked.
  • DYNAMIC_CHART_VERIFY=false turns verification off, and the manager logs that on start.
  • A chart from the charts directory of the Operator is not verified. It ships in the Operator image.

Out of scope

  • OCI registries, which dynamic chart pull does not support.
  • Verifying a Siphon chart, which is never pulled.
  • Fetching or refreshing keys from a key server at run time.

FAQ

  • Why is verification on by default? The official repository signs every chart release since 7.8.0, and every release the Operator supports is newer. A repository that is compromised, or an HTTP mirror, could otherwise change what the Operator applies. For more information, see ADR 35.
  • Why can verification be turned off? An internal mirror might not serve the provenance files, and a chart older than 7.8.0 has none.
  • Why can the keys be changed? A mirror that signs the charts again with its own key needs it, and so does a rotated GitLab key before the next Operator release.
  • Why does the binary not embed the GitLab key? The bridge build, the only one that pulls, is deployed with the Operator chart, which always mounts the keys. A second copy in the binary would be a trust policy that only a run outside the chart uses. A local run can set DYNAMIC_CHART_VERIFY=false, or point DYNAMIC_CHART_KEYRING at an exported key.
  • Why is a cached chart not trusted across a restart? The cache directory can be a volume that outlives the manager, or that something else writes to. The manager pulls each cached version once more after it starts.
  • Is the key fetched from keys.openpgp.org? No. The key ships in the Operator chart, so a cluster needs no egress to a key server, and a compromised key server cannot change it.