db:check-migrations job

This job runs on the test stage of a merge request pipeline. It checks:

  1. A schema dump comparison between the author’s working branch and the target branch, after executing a rollback of your new migrations. This check validates that the schema properly resets to what it was before executing new migrations.
  2. A schema dump comparison between the author’s working branch and the db/structure.sql file that the author committed. This check validates that it contains all the expected changes in the migration.
  3. A Git diff between the db/schema_migrations that the author committed and the one that the script generated after running migrations. This check validates that everything was properly committed.

Troubleshooting

False positives

This job is not allowed to fail, but it can throw some false positives.

Examples:

  1. When we drop a column and then roll back, the column is always re-added at the end of the list of columns. If the column was previously not the last column, the rollback can’t return the schema back exactly to its previous state. Reference: job failure.
  2. Sometimes pg_dump can change how it orders the constraints and other database objects in minor PostgreSQL version upgrades. This causes the job to fail by complaining about the ordering of statements in the schema that are not related to the MR. Reference: job failure.
    • Do report this in #database Slack channel or create an issue, if not already done by someone else. Since this will affect all the feature MRs.

In such cases it’s safer to add pipeline:skip-check-migrations label to the MR to skip this job.

Schema dump comparison fails after rollback

This failure often happens if your working branch is behind the target branch. A real scenario:

graph LR
    Main((main<br>commit A)) ===> |remove constraint<br>fk_rails_dbebdaa8fe| MainB((main<br>commit B))
    Main((main<br>commit A)) --> |checkout<br>dev| DevA((dev<br>commit A)):::dev
    DevA((dev<br>commit A)) --> |add column<br>dependency_proxy_size| DevC((dev<br>commit C)):::dev
    DevC -.-> |CI pipeline<br>executes| JOB-FAILED((JOB FAILED!)):::error

    classDef main fill:#f4f0ff,stroke:#7b58cf
    classDef dev fill:#e9f3fc,stroke:#1f75cb
    classDef error fill:#f15146,stroke:#d4121a
  1. You check out the dev working branch from the main target branch. At this point, each branch has their HEAD at commit A.
  2. Someone works on the main branch and drops the fk_rails_dbebdaa8fe constraint, thus creating commit B on main.
  3. You add column dependency_proxy_size to your dev branch.
  4. The db:check-migrations job fails for your dev branch’s CI/CD pipeline, because the structure.sql file did not rollback to its expected state.

This happened because branch dev contained commits A and C, not B. Its database schema did not know about the removal of the fk_rails_dbebdaa8fe constraint. When comparing the two schemas, the dev branch contained this constraint while the main branch didn’t.

This example really happened. Read the job failure logs.

To fix these kind of issues, rebase the working branch onto the target branch to get the latest changes.