GitLab ClickHouse integration

Status: Implemented Issue: https://gitlab.com/gitlab-org/cloud-native/gitlab-operator/-/work_items/2264 Parent: GitLabCore reconciler

Goal

An administrator connects GitLab to ClickHouse through a typed field or the Bridge form, without writing chart values. A Siphon streams into the tables the GitLab ClickHouse migrations create, so it builds on that connection and waits for it. For the design, see ADR 41. For what deleting a Siphon leaves in place, see Siphon and Orbit deprovisioning.

Requirements

  • spec.clickhouse configures the ClickHouse connection of GitLab: the URL of the HTTP interface, the database, the user, and a reference to the Secret key holding the password. All four are required. No password is stored in the resource.
  • The API server rejects a spec.clickhouse.url that does not start with http:// or https://.
  • Setting spec.clickhouse turns the ClickHouse integration of GitLab on, and spec.chart.values win over it. So a GitLabCore has ClickHouse configured when spec.chart.values set global.clickhouse.enabled to true, or set none and spec.clickhouse is set.
  • Nothing of a Siphon renders while its GitLabCore has no ClickHouse configured. The Siphon reports Initialized=False with reason GitLabClickHouseNotConfigured, and resolves in 30 seconds of the instance gaining it, without a change to the Siphon.
  • A Siphon whose instance loses ClickHouse reports the same reason, and leaves what it applied running.
  • spec.sink of a Siphon stays its own connection. The Operator derives none of it from spec.clickhouse.
  • The Bridge API reads and writes spec.clickhouse with the instance. The form edits it in a ClickHouse group of Dependencies, and refuses Siphon without it. For the form, see GitLab instance form.

Out of scope

  • Provisioning ClickHouse, its databases, or its users.
  • Running or checking the GitLab ClickHouse migrations.
  • Refusing in the Bridge API, or at admission, a Siphon for an instance with no ClickHouse.
  • ClickHouse connections other than main.

FAQ

  • Why does Siphon need its own ClickHouse connection? GitLab talks to the HTTP interface, and Siphon to the native protocol on another port. Siphon also writes as a user of its own, with INSERT on the target tables only.
  • Why does a Siphon wait for the GitLab connection? The GitLab ClickHouse migrations create the tables Siphon writes into. Without the connection, nothing creates them, and Siphon fails on every insert. The requester asked for the dependency in both the form and the reconciler.
  • Why does a Siphon keep running when its instance loses ClickHouse? The requester asked for it. Stopping a running pipeline over an edit of another resource is worse than a condition that names it. The tables stay where they are.
  • Do chart values count as configured? Yes. spec.chart.values are the escape hatch, and an instance that set global.clickhouse there before this field existed keeps its Siphon.