clickhouse:migrate:multi-version-upgrade job
This job runs ClickHouse migrations the way a real GitLab upgrade runs them, so
that migrations are exercised against populated tables. It is the ClickHouse
counterpart of db:migrate:multi-version-upgrade
for PostgreSQL.
The existing clickhouse:check-schema job runs every migration from scratch
against an empty database in a single pass. That cannot reproduce three
conditions that real upgrades have:
- The schema starts at an older required upgrade stop, not at the current version.
- Post-deployment migrations run after newer regular migrations have already changed a table. Omnibus and Helm upgrades run migrations in these two phases.
- Tables already contain rows, so a swap, a projection rebuild or a column drop has parts on disk to rewrite.
Issue #593129 needed all three at once and reached a customer because no job covered them.
What the job does
scripts/click_house/multi_version_upgrade.sh:
- Resolves the latest required upgrade stop with
Gitlab::Utils::UpgradePath, then picks that release’s highest patch tag. - Loads the ClickHouse schema as it was at that tag.
db/click_house/main.sqlanddb/click_house/schema_migrations/are committed, so this is agit checkoutof two paths and needs no database dump. - Seeds 10 rows into every MergeTree table except
AggregatingMergeTreetables, with the ClickHousegenerateRandomtable function, which takes its structure from the table. Integer columns are kept small, like real IDs, because batched data migrations walk ID ranges in fixed steps.AggregatingMergeTreetables get their rows from their materialized views, the way they do in production. An aggregated table stays empty when random source rows do not pass its view’sWHEREfilter. - Runs regular migrations with
SKIP_POST_DEPLOYMENT_MIGRATIONS=truein one log section, then the post-deployment migrations in a second. - Runs
scripts/validate_clickhouse_schema.rb, the same check asclickhouse:check-schema, to confirm the dumped schema, the schema cache and the migration marker files all match what is committed.
Seeding is derived from the schema rather than from hand-written fixtures on purpose. The migrations in the tested window change every release, so a fixed fixture set would drift away from the tables that actually get migrated.
When it runs
On merge request pipelines that change config/upgrade_path.yml, VERSION,
db/click_house/**, db/siphon/**, lib/click_house/**,
lib/tasks/gitlab/click_house/**, scripts/click_house/**, or the job
definition itself. It is skipped on
forks, on merge requests targeting stable branches, and on expedited pipelines.
The job is allowed to fail while it proves stable, so a failure does not block the merge. Issue 635169 tracks making it required.
Apply ~"pipeline:run-all-rspec" to force it to run.
Troubleshooting
The job is red on an unrelated merge request
Check which step failed:
- Resolving the upgrade stop: A
git ls-remoteorgit fetchfailure is usually transient. The error includes the output from Git. Retry the job. - Seeding: A
WARNING: could not seed <table>line includes the ClickHouse error. The job skips that table and continues, and only fails if no rows were seeded at all. A warning from a materialized view fed by the table is expected, because the view can reject random values. - Migrating: This is a real migration failure against populated tables and is what the job exists to catch. The two log sections show whether it failed in the regular or the post-deployment phase. Reproduce it locally with the steps below.
- Verifying: The schema drifted from
db/click_house/main.sql. For a confirmed false positive, the~"pipeline:skip-check-clickhouse-schema"label also applies here, the same way it does forclickhouse:check-schema.
To disable the job while a fix is prepared, set
DISABLE_CLICKHOUSE_MULTI_VERSION_UPGRADE to true in the project CI/CD
settings, and announce it in #master-broken.
Running it locally
Point the test ClickHouse configuration at a throwaway database, then:
CLICKHOUSE_URL=http://localhost:8123 \
CLICKHOUSE_DATABASE=gitlab_clickhouse_upgrade_test \
RAILS_ENV=test DISABLE_WEBMOCK=true ENABLE_CLICKHOUSE_DB_DUMP=true \
bash scripts/click_house/multi_version_upgrade.shThe script restores the branch’s main.sql and marker files on exit, so an
interrupted run does not leave the worktree on the upgrade stop’s schema. It
fetches the stop tag with --depth 1 only on CI, because a shallow fetch into
a full local clone marks that clone as shallow.