• Branch and tag protection
  • Protected branches
  • Protected Tags
• Mirroring between repositories
• Types of pipelines
  • Branch pipelines
  • Tag pipelines
  • Scheduled pipelines
  • Auto-deploy pipelines
  • Triggered pipelines
• CI Jobs
  • Development Jobs
    • danger-review
    • rubocop
    • docs-lint
    • review-docs-deploy
    • review-docs-cleanup
    • <OS_NAME> knapsack
    • <OS_NAME> specs
    • update-knapsack
    • Trigger:ce-package
    • Trigger:ee-package
    • fetch-assets
    • Trigger:package
    • Trigger:gitlab-docker
    • Trigger:qa-docker
    • Trigger:qa-test
    • RAT
    • <OS_NAME>-branch
    • Docker-branch
    • QA-branch
• Release jobs
  • <OS_NAME>
  • <OS_NAME>-staging
  • <OS_NAME>-release
  • Docker
  • Docker-Release
  • Docker-QA
  • QA-Tag
  • AWS
  • AWS-Ultimate
  • AWS-Premium
  • AWS-CE-Release
  • AWS-Ultimate-Release
  • AWS-Premium-Release
  • create_omnibus_manifest
  • dependency_scanning
  • RAT-Nightly
  • RAT-Tag
  • license-upload
• Housekeeping Jobs
  • update-gems-cache and update-trigger-package-cache
  • dependency_update
  • dependencies_io_check
  • valdiate_packer_changes
  • pages

Repositories, branches, and CI pipelines

omnibus-gitlab CI pipeline is a moderately complex one with pipelines split between all the mirrors of the project:

1. Development repository for normal feature development.
2. Release mirror for building release artifacts.
3. Security mirror for security development.
4. QA mirror for developers to run a package build and QA for development purposes.

Branch and tag protection

Protected branches

1. Development repository
   1. master: Maintainers, Delivery team
   2. *-stable : Delivery team, Release Managers
   3. *-stable-ee : Delivery team, Release Managers
   4. *-auto-deploy-* : Maintainers, delivery group, managers group
2. Release mirror:
   1. master: Maintainers
   2. *-stable : Maintainers
   3. *-stable-ee : Maintainers
   4. *-auto-deploy-* : Maintainers
3. Security mirror:
   1. master: GitLab Release Tools Bot, GitLab Bot, Delivery team, Release Managers
   2. *-stable: GitLab Release Tools Bot, GitLab Bot, Delivery team, Release Managers
   3. *-stable-ee: GitLab Release Tools Bot, GitLab Bot, Delivery team, Release Managers
   4. *-auto-deploy-*: GitLab Release Tools Bot, GitLab Bot, Delivery team, Release Managers
4. QA mirror:
   1. master: Developers (Merge only), Maintainers

Protected Tags

1. Development repository:
   1. * : Maintainers, Delivery team, Release Managers
2. Release mirror:
   1. *: Maintainers

3. Security mirror: Nil

4. QA mirror: Nil

Mirroring between repositories

Most of the development happens in the Development repository and security related changes go to Security mirror. These changes then gets mirrored to Release mirror and QA mirror. The following diagram describes what all gets mirrored between each of these repositories.

Types of pipelines

Branch pipelines

omnibus-gitlab doesn’t make use of detached merge request pipelines yet. So, pipelines caused by branch pushes are the common pipelines encountered in this project. These pipelines are used in Development repository, Security mirror, and Release mirror.

Tag pipelines

Tag pushes to Release mirror starts a pipeline with the release jobs. Tag pushes to Development repository and Security mirror behaves as regular branch pushes (except that they don’t have an option to start a package-and-qa pipeline) and run basic stylistic checks and unit tests.

Scheduled pipelines

In Development repository, there are two scheduled pipelines.

1. Dependencies Update - Pipeline to check for outdated dependencies using the dependency_update job.
2. Generate license pages - Pipeline to populate License collection webpage with license information from the S3 bucket.

In Release mirror, there are two scheduled pipelines

1. CE nightly - Pipeline to build nightly packages and Docker images for GitLab CE
2. EE nightly - Pipeline to build nightly packages and Docker images for GitLab EE

Other mirrors doesn’t have any scheduled pipelines

Auto-deploy pipelines

GitLab uses Auto deploy branches and tags for its release process. These branches are named <MAJOR>-<MINOR>-auto-deploy-<TIMESTAMP> and tags have the format <MAJOR>.<MINOR>.<TIMESTAMP>+<gitlab sha>.<omnibus-gitlab sha>.

Only certain jobs - that are needed to deploy the changes to GitLab environments - will be part of this pipeline.

Triggered pipelines

We use triggered pipelines to run a “package-and-qa” pipeline in the QA mirror. These can be triggered either from a pipeline in the Development repository or GitLab project.

This pipeline is intended to give the developers a package and an image to test their changes in addition to automatically doing a QA run against these artifacts. It also provides an option to do a QA run against an HA instance spun up using these artifacts.

CI Jobs

Development Jobs

danger-review

This job runs a check against the merge request to ensure it satisfies some basic requirements using the Danger tool.

This job is run only on Development repository and Security mirror on branch and tag pipelines.

rubocop

Check the source code files to satisfy certain stylistic requirements.

This job is run only on Development repository and Security mirror on branch and tag pipelines.

docs-lint

Check the docs files to satisfy certain stylistic requirements.

This job is run only on Development repository and Security mirror on branch and tag pipelines.

review-docs-deploy

A manual job that will trigger a docs build in gitlab-docs, which will deploy a review app for the GitLab Docs with changes from the current commit.

This job is run only on Development repository on branch pipelines.

review-docs-cleanup

A manual job to stop the environment created by review-docs-deploy. Automatically ran on merging a merge request.

This job is run only on Development repository on branch pipelines.

<OS_NAME> knapsack

We have tests covering our Chef recipes and libraries using RSpec and Chefspec. They are parallelized with the help of knapsack.

This job pulls in the knapsack reports of previous pipeline runs from the cache and gets them in place for the current spec runs.

These jobs are run only on Development repository and Security mirror on branch and tag pipelines.

<OS_NAME> specs

Jobs that actually run rspec via knapsack. Parallelized to a count of 6 using the parallel keyword

These jobs are run only on Development repository and Security mirror on branch and tag pipelines.

update-knapsack

This job combines the reports from each individual parallel runs of unit tests and prepares a final JSON report. This report is used by the MR widget to display spec status, and is also uploaded to the cache for the next pipeline run to use.

This job is run only on Development repository and Security mirror on branch and tag pipelines.

Trigger:ce-package

This is a manual job which when played triggers a pipeline in the QA mirror to run a package build and QA for development purposes. This can be used by developers to get a package or Docker image for testing, or to run a full QA suite against their MR’s changes.

This job is run only on Development repository and Security mirror on branch pipelines.

Trigger:ee-package

Same as Trigger:ce-package, but builds an EE package.