e2e:package-and-test child pipeline is the main executor of E2E testing for the GitLab platform. The pipeline definition has several dynamic
components to reduce the number of tests being executed in merge request pipelines.
Pipeline setup consists of:
e2e-test-pipeline-generatejob in the
preparestage of the main GitLab pipeline.
e2e:package-and-testjob in the
qastage, which triggers the child pipeline that is responsible for building the
omnibuspackage and running E2E tests.
This job consists of two components that implement selective test execution:
detect_changesRake task determines which e2e specs should be executed in a particular merge request pipeline. This task analyzes changes in a particular merge request and determines which specs must be executed. Based on that, a
dry-runof every scenario executes and determines if a scenario contains any executable tests. Selective test execution uses these criteria to determine which specific tests to execute.
generate-e2e-pipelineis executed, which generates a child pipeline YAML definition file with appropriate environment variables.
E2E test execution pipeline consists of several stages which all support execution of E2E tests.
This stage is responsible for the following tasks:
knapsackreports that support parallel test execution.
- Triggering downstream pipeline which builds the
This stage runs e2e tests against different types of GitLab configurations. The number of jobs executed is determined dynamically by
This stage is responsible for allure test report generation.
Adding new jobs
Selective test execution depends on a set of rules present in every job definition. A typical job contains the following attributes:
variables: QA_SCENARIO: Test::Integration::MyNewJob rules: - !reference [.rules:test:qa, rules] - if: $QA_SUITES =~ /Test::Integration::MyNewJob/ - !reference [.rules:test:manual, rules]
In this example:
QA_SCENARIO: Test::Integration::MyNewJob: name of the scenario class that is passed to the
!reference [.rules:test:qa, rules]: main rule definition that is matched for pipelines that should execute all tests. For example, when changes to
qaframework are present.
if: $QA_SUITES =~ /Test::Integration::MyNewJob/: main rule responsible for selective test execution.
QA_SUITEis the name of the scenario abstraction located in
QA_SUITEis not the same as
QA_SCENARIO, which is passed to the
gitlab-qaexecutor. For consistency, it usually has the same name.
QA_SUITEabstraction class usually contains information on what tags to run and optionally some additional setup steps.
!reference [.rules:test:manual, rules]: final rule that is always matched and sets the job to
manualso it can still be executed on demand, even if not set to execute by selective test execution.
Considering example above, perform the following steps to create a new job:
- Create new scenario type
integrationdirectory of the
gitlab-qaproject and release new version so it’s generally available.
Create new scenario
integrationdirectory of the
qaframework. In the most simple case, this scenario would define RSpec tags that should be executed:
module QA module Scenario module Test module Integration class MyNewJob < Test::Instance::All tags :some_special_tag end end end end end
Add new job definition in the
ee:my-new-job: extends: .qa variables: QA_SCENARIO: Test::Integration::MyNewJob rules: - !reference [.rules:test:qa, rules] - if: $QA_SUITES =~ /Test::Integration::MyNewJob/ - !reference [.rules:test:manual, rules]
For selective execution to work correctly with job types that require running multiple parallel jobs, a job definition typically must be split into parallel and selective variants. Splitting is necessary so that when selective execution executes only a single spec, multiple unnecessary jobs are not spawned. For example:
ee:my-new-job-selective: extends: .qa variables: QA_SCENARIO: Test::Integration::MyNewJob rules: - !reference [.rules:test:qa-selective, rules] - if: $QA_SUITES =~ /Test::Integration::MyNewJob/ ee:my-new-job: extends: - .parallel - ee:my-new-job-selective rules: - !reference [.rules:test:qa-parallel, rules] - if: $QA_SUITES =~ /Test::Integration::MyNewJob/