CI configuration internals

Workflow rules

Pipelines for the GitLab project are created using the workflow:rules keyword feature of the GitLab CI/CD.

Pipelines are always created for the following scenarios:

  • main branch, including on schedules, pushes, merges, and so on.
  • Merge requests.
  • Tags.
  • Stable, auto-deploy, and security branches.

Pipeline creation is also affected by the following CI/CD variables:

  • If $FORCE_GITLAB_CI is set, pipelines are created.
  • If $GITLAB_INTERNAL is not set, pipelines are not created.

No pipeline is created in any other cases (for example, when pushing a branch with no MR for it).

The source of truth for these workflow rules is defined in .gitlab-ci.yml.

Default image

The default image is defined in .gitlab-ci.yml.

It includes Ruby, Go, Git, Git LFS, Chrome, Node, Yarn, PostgreSQL, and Graphics Magick.

The images used in our pipelines are configured in the gitlab-org/gitlab-build-images project, which is push-mirrored to gitlab/gitlab-build-images for redundancy.

The current version of the build images can be found in the “Used by GitLab section”.

Default variables

In addition to the predefined CI/CD variables, each pipeline includes default variables defined in .gitlab-ci.yml.

Stages

The current stages are:

  • sync: This stage is used to synchronize changes from gitlab-org/gitlab to gitlab-org/gitlab-foss.
  • prepare: This stage includes jobs that prepare artifacts that are needed by jobs in subsequent stages.
  • build-images: This stage includes jobs that prepare Docker images that are needed by jobs in subsequent stages or downstream pipelines.
  • fixtures: This stage includes jobs that prepare fixtures needed by frontend tests.
  • lint: This stage includes linting and static analysis jobs.
  • test: This stage includes most of the tests, and DB/migration jobs.
  • post-test: This stage includes jobs that build reports or gather data from the test stage’s jobs (for example, coverage, Knapsack metadata, and so on).
  • review: This stage includes jobs that build the CNG images, deploy them, and run end-to-end tests against Review Apps (see Review Apps for details). It also includes Docs Review App jobs.
  • qa: This stage includes jobs that perform QA tasks against the Review App that is deployed in stage review.
  • post-qa: This stage includes jobs that build reports or gather data from the qa stage’s jobs (for example, Review App performance report).
  • pages: This stage includes a job that deploys the various reports as GitLab Pages (for example, coverage-ruby, and webpack-report (found at https://gitlab-org.gitlab.io/gitlab/webpack-report/, but there is an issue with the deployment).
  • notify: This stage includes jobs that notify various failures to Slack.

Dependency Proxy

Some of the jobs are using images from Docker Hub, where we also use ${GITLAB_DEPENDENCY_PROXY_ADDRESS} as a prefix to the image path, so that we pull images from our Dependency Proxy. By default, this variable is set from the value of ${GITLAB_DEPENDENCY_PROXY}.

${GITLAB_DEPENDENCY_PROXY} is a group CI/CD variable defined in gitlab-org as ${CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX}/. This means when we use an image defined as:

image: ${GITLAB_DEPENDENCY_PROXY_ADDRESS}alpine:edge

Projects in the gitlab-org group pull from the Dependency Proxy, while forks that reside on any other personal namespaces or groups fall back to Docker Hub unless ${GITLAB_DEPENDENCY_PROXY} is also defined there.

Work around for when a pipeline is started by a Project access token user

When a pipeline is started by a Project access token user (e.g. the release-tools approver bot user which automatically updates the Gitaly version used in the main project), the Dependency proxy isn’t accessible and the job fails at the Preparing the "docker+machine" executor step. To work around that, we have a special workflow rule, that overrides the ${GITLAB_DEPENDENCY_PROXY_ADDRESS} variable so that Dependency proxy isn’t used in that case:

- if: '$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH && $GITLAB_USER_LOGIN =~ /project_\d+_bot\d*/'
  variables:
    GITLAB_DEPENDENCY_PROXY_ADDRESS: ""
note
We don’t directly override the ${GITLAB_DEPENDENCY_PROXY} variable because group-level variables have higher precedence over .gitlab-ci.yml variables.

Common job definitions

Most of the jobs extend from a few CI definitions defined in .gitlab/ci/global.gitlab-ci.yml that are scoped to a single configuration keyword.

Job definitionsDescription
.default-retryAllows a job to retry upon unknown_failure, api_failure, runner_system_failure, job_execution_timeout, or stuck_or_timeout_failure.
.default-before_scriptAllows a job to use a default before_script definition suitable for Ruby/Rails tasks that may need a database running (for example, tests).
.setup-test-env-cacheAllows a job to use a default cache definition suitable for setting up test environment for subsequent Ruby/Rails tasks.
.rails-cacheAllows a job to use a default cache definition suitable for Ruby/Rails tasks.
.static-analysis-cacheAllows a job to use a default cache definition suitable for static analysis tasks.
.coverage-cacheAllows a job to use a default cache definition suitable for coverage tasks.
.qa-cacheAllows a job to use a default cache definition suitable for QA tasks.
.yarn-cacheAllows a job to use a default cache definition suitable for frontend jobs that do a yarn install.
.assets-compile-cacheAllows a job to use a default cache definition suitable for frontend jobs that compile assets.
.use-pg11Allows a job to run the postgres 11 and redis services (see .gitlab/ci/global.gitlab-ci.yml for the specific versions of the services).
.use-pg11-eeSame as .use-pg11 but also use an elasticsearch service (see .gitlab/ci/global.gitlab-ci.yml for the specific version of the service).
.use-pg12Allows a job to use the postgres 12 and redis services (see .gitlab/ci/global.gitlab-ci.yml for the specific versions of the services).
.use-pg12-eeSame as .use-pg12 but also use an elasticsearch service (see .gitlab/ci/global.gitlab-ci.yml for the specific version of the service).
.use-pg13Allows a job to use the postgres 13 and redis services (see .gitlab/ci/global.gitlab-ci.yml for the specific versions of the services).
.use-pg13-eeSame as .use-pg13 but also use an elasticsearch service (see .gitlab/ci/global.gitlab-ci.yml for the specific version of the service).
.use-kanikoAllows a job to use the kaniko tool to build Docker images.
.as-if-fossSimulate the FOSS project by setting the FOSS_ONLY='1' CI/CD variable.
.use-docker-in-dockerAllows a job to use Docker in Docker.

rules, if: conditions and changes: patterns

We’re using the rules keyword extensively.

All rules definitions are defined in rules.gitlab-ci.yml, then included in individual jobs via extends.

The rules definitions are composed of if: conditions and changes: patterns, which are also defined in rules.gitlab-ci.yml and included in rules definitions via YAML anchors

if: conditions

if: conditionsDescriptionNotes
if-not-canonical-namespaceMatches if the project isn’t in the canonical (gitlab-org/) or security (gitlab-org/security) namespace.Use to create a job for forks (by using when: on_success|manual), or not create a job for forks (by using when: never).
if-not-eeMatches if the project isn’t EE (that is, project name isn’t gitlab or gitlab-ee).Use to create a job only in the FOSS project (by using when: on_success|manual), or not create a job if the project is EE (by using when: never).
if-not-fossMatches if the project isn’t FOSS (that is, project name isn’t gitlab-foss, gitlab-ce, or gitlabhq).Use to create a job only in the EE project (by using when: on_success|manual), or not create a job if the project is FOSS (by using when: never).
if-default-refsMatches if the pipeline is for master, main, /^[\d-]+-stable(-ee)?$/ (stable branches), /^\d+-\d+-auto-deploy-\d+$/ (auto-deploy branches), /^security\// (security branches), merge requests, and tags.Note that jobs aren’t created for branches with this default configuration.
if-master-refsMatches if the current branch is master or main. 
if-master-pushMatches if the current branch is master or main and pipeline source is push. 
if-master-schedule-maintenanceMatches if the current branch is master or main and pipeline runs on a 2-hourly schedule. 
if-master-schedule-nightlyMatches if the current branch is master or main and pipeline runs on a nightly schedule. 
if-auto-deploy-branchesMatches if the current branch is an auto-deploy one. 
if-master-or-tagMatches if the pipeline is for the master or main branch or for a tag. 
if-merge-requestMatches if the pipeline is for a merge request. 
if-merge-request-title-as-if-fossMatches if the pipeline is for a merge request and the MR has label ~"pipeline:run-as-if-foss" 
if-merge-request-title-update-cachesMatches if the pipeline is for a merge request and the MR has label ~"pipeline:update-cache". 
if-merge-request-title-run-all-rspecMatches if the pipeline is for a merge request and the MR has label ~"pipeline:run-all-rspec". 
if-security-merge-requestMatches if the pipeline is for a security merge request. 
if-security-scheduleMatches if the pipeline is for a security scheduled pipeline. 
if-nightly-master-scheduleMatches if the pipeline is for a master scheduled pipeline with $NIGHTLY set. 
if-dot-com-gitlab-org-scheduleLimits jobs creation to scheduled pipelines for the gitlab-org group on GitLab.com. 
if-dot-com-gitlab-org-masterLimits jobs creation to the master or main branch for the gitlab-org group on GitLab.com. 
if-dot-com-gitlab-org-merge-requestLimits jobs creation to merge requests for the gitlab-org group on GitLab.com. 
if-dot-com-gitlab-org-and-security-tagLimits job creation to tags for the gitlab-org and gitlab-org/security groups on GitLab.com. 
if-dot-com-gitlab-org-and-security-merge-requestLimit jobs creation to merge requests for the gitlab-org and gitlab-org/security groups on GitLab.com. 
if-dot-com-gitlab-org-and-security-tagLimit jobs creation to tags for the gitlab-org and gitlab-org/security groups on GitLab.com. 
if-dot-com-ee-scheduleLimits jobs to scheduled pipelines for the gitlab-org/gitlab project on GitLab.com. 
if-security-pipeline-merge-resultMatches if the pipeline is for a security merge request triggered by @gitlab-release-tools-bot. 

changes: patterns

changes: patternsDescription
ci-patternsOnly create job for CI configuration-related changes.
ci-build-images-patternsOnly create job for CI configuration-related changes related to the build-images stage.
ci-review-patternsOnly create job for CI configuration-related changes related to the review stage.
ci-qa-patternsOnly create job for CI configuration-related changes related to the qa stage.
yaml-lint-patternsOnly create job for YAML-related changes.
docs-patternsOnly create job for docs-related changes.
frontend-dependency-patternsOnly create job when frontend dependencies are updated (for example, package.json, and yarn.lock) changes.
frontend-patterns-for-as-if-fossOnly create job for frontend-related changes that have impact on FOSS.
backend-patternsOnly create job for backend-related changes.
db-patternsOnly create job for DB-related changes.
backstage-patternsOnly create job for backstage-related changes (that is, Danger, fixtures, RuboCop, specs).
code-patternsOnly create job for code-related changes.
qa-patternsOnly create job for QA-related changes.
code-backstage-patternsCombination of code-patterns and backstage-patterns.
code-qa-patternsCombination of code-patterns and qa-patterns.
code-backstage-qa-patternsCombination of code-patterns, backstage-patterns, and qa-patterns.
static-analysis-patternsOnly create jobs for Static Analytics configuration-related changes.