End-to-end test pipelines
Common architecture
All E2E tests are executed within separate child pipelines. To support different dynamic features of E2E testing pipelines, all child pipeline YAML files are generated by e2e-test-pipeline-generate
CI/CD job and triggered by respective trigger job.
e2e-test-pipeline-generate
The e2e-test-pipeline-generate
job generates CI/CD YAML file definitions which are used to trigger child pipelines that run E2E tests.
The generate_e2e_pipelines
Rake task:
- Determines which e2e specs should be executed in a particular merge request pipeline.
- Generates a CI/CD YAML file definition for each E2E test pipeline type.
This Rake task:
- Analyzes changes in a particular merge request and determines which specs must be executed using selective test execution with
these criteria. Based on that, a
dry-run
of every scenario executes and determines if a scenario contains any executable tests. - The total run time for each scenario is calculated.
- Based on the run time value, dynamic job scaling calculates the necessary number of parallel CI/CD jobs for each scenario type and generates pipeline YAML file with appropriate values.
e2e:test-on-cng
The e2e:test-on-cng
child pipeline runs tests against a Cloud Native GitLab installation.
Deployment is managed by the cng
orchestrator tool, which you can also use to locally recreate CI/CD deployments.
The e2e:test-on-cng
child pipeline is executed in merge requests and is part of pre-merge validation lifecycle. If any test fails, you can’t merge introduced
code changes.
Setup
This E2E test child pipeline is triggered by the e2e:test-on-cng
job using dynamically generated CI/CD YAML file stored as artifacts in the e2e-test-pipeline-generate
CI/CD job. CI/CD YAML files are generated using a template.
child pipeline jobs
The child pipeline consists of several stages that support E2E test execution.
.pre
build-cng-env
job is responsible for setting up all environment variables for CNG downstream pipelinebuild-cng
job triggersCNG
downstream pipeline which is responsible for building all necessary images
test
Jobs in test
stage perform following actions:
- local k8s cluster setup using
kind
- GitLab installation using official
helm
chart - E2E test execution against performed deployment
report
This stage is responsible for allure test report generation.
Debugging
To help with debugging:
- Each test job prints a list of arguments that you can pass to the
cng
orchestrator to exactly recreate the same deployment for local debugging. - Cluster events log and all pod logs are saved in E2E test job artifacts.
cng
orchestrator automatically outputs all cluster events with errors in the case of failed deployment.
e2e:test-on-omnibus
The e2e:test-on-omnibus
child pipeline runs tests against an Omnibus installation. This pipeline type is not executed
in merge request pipelines by default and can be triggered manually by triggering the e2e:test-on-omnibus-ee
job.
This pipeline type is allowed to fail and even in the case of a manual trigger inside of a merge request pipeline, failing tests will not block the ability to merge.
Linux package deployment is managed by gitlab-qa
.
Setup
This E2E test child pipeline is triggered by the e2e:test-on-omnibus-ee
job using dynamically generated CI/CD YAML file stored as artifacts in
e2e-test-pipeline-generate
CI/CD jobs. The CI/CD YAML file is generated using a
template.
child pipeline jobs
E2E test execution pipeline consists of several stages which all support execution of E2E tests.
.pre
This stage is responsible for the following tasks:
- Triggering downstream pipeline which builds the
omnibus-gitlab
Docker image.
test
Jobs in test
stage perform following actions:
- Installs GitLab by using a built Linux package.
- Executes E2E tests against the Linux package installation.
report
This stage is responsible for allure test report generation.
e2e:test-on-gdk
The e2e:test-on-gdk
child pipeline supports development of the GitLab platform by providing feedback to engineers on
end-to-end test execution faster than via e2e:test-on-omnibus
.
This is achieved by running tests against the GitLab Development Kit (GDK), which can be built and installed in less time than when testing against Omnibus GitLab. The trade-off is that Omnibus GitLab can be used to deploy a production installation, whereas the GDK is a development environment. Tests that run against the GDK might not catch bugs that depend on part of the process of preparing GitLab to run in a production environment, including pre-compiling assets, assigning configuration defaults as part of an official installation package, deploying GitLab services to multiple servers, and more. On the other hand, engineers who use the GDK day-to-day can benefit from automated tests catching bugs that only appear on the GDK.
Setup
This E2E test child pipeline is triggered by the e2e:test-on-gdk
job using dynamically generated CI/CD YAML file stored as artifacts in the e2e-test-pipeline-generate
CI/CD job. The CI/CD YAML file is generated by using a template.
build-gdk-image
The build-gdk-image
job
uses the code from the merge request to build a Docker image that can be used in test jobs to launch a GDK instance in a container. The image is pushed to the container registry.
The job also runs in pipelines on the default branch to build a base image that includes the GDK and GitLab components. This avoids building the entire image from scratch in merge requests. However, if the merge request includes changes to certain GitLab components or code the job will rebuild the base image before building the image that will be used in the test jobs.
child pipeline jobs
The child pipeline consists of several stages that support E2E test execution.
test
Jobs in the test
stage perform following actions:
- Start an instance of GDK by using the Docker image built by the
build-gdk-image
job. - Run E2E tests against the running instance of GDK.
report
This stage is responsible for allure test report generation.
Test Licenses
Please see the Test Licenses runbook for more information on the licenses used by these pipelines.
Adding new jobs to E2E test pipelines
E2E test pipelines use dynamic scaling of jobs based on their runtime. To create a mapping between job definitions in pipeline definition YAML files and
a particular test scenario, scenario
classes are used. These classes are located in qa/qa/scenario
folder.
A typical job definition in one of the e2e test pipeline definition YAML files would look like:
my-new-test-job:
...
variables:
QA_SCENARIO: Test::Integration::MyNewTestScenario
In this example:
QA_SCENARIO: Test::Integration::MyNewTestScenario
: name of the scenario class that is passed to theqa/bin/qa
test execution script. While the full class name would beQA::Scenario::Test:Integration::MyNewTestScenario
,QA::Scenario
is omitted to have shorted definitions.
Considering example above, perform the following steps to create a new job:
Create a new scenario
my_new_job.rb
in theintegration
directory of thee2e
test framework. The scenario class should define a pipeline mapping that couples the scenario to a specific job in a specific pipeline type. If job was added to the test-on-cng pipeline, this scenario would define RSpec tags that should be executed and pipeline mapping:module QA module Scenario module Test module Integration class MyNewJob < Test::Instance::All tags :some_special_tag pipeline_mappings test_on_cng: %w[my-new-test-job] end end end end end
Add the new job definition in the
main.gitlab-ci.yml
pipeline definition:my-new-test-job: extends: - .cng-test variables: QA_SCENARIO: Test::Integration::MyNewTestScenario
Such a definition ensures that my-new-test-job
has automatic parallel job scaling based on predefined runtime threshold.
Docs
Edit this page to fix an error or add an improvement in a merge request.
Create an issue to suggest an improvement to this page.
Product
Create an issue if there's something you don't like about this feature.
Propose functionality by submitting a feature request.
Feature availability and product trials
View pricing to see all GitLab tiers and features, or to upgrade.
Try GitLab for free with access to all features for 30 days.
Get help
If you didn't find what you were looking for, search the docs.
If you want help with something specific and could use community support, post on the GitLab forum.
For problems setting up or using this feature (depending on your GitLab subscription).
Request support