Test Report Generation in Omnibus-GitLab

The following three pipelines are created while generating the allure-report

  • Omnibus pipeline
  • TRIGGERED_CE/EE_PIPELINE child pipeline (Manually Triggered)
  • QA-SUBSET-TEST child pipeline

Omnibus MR Pipeline

An Omnibus-GitLab project MR pipeline can be triggered in two ways

  • manually running the pipeline
  • a MR exists and a commit is pushed to the repository

The tests in the pipeline are currently triggered manually by

  • Trigger:ce-package job
  • Trigger:ee-package job

Trigger:ce/ee-package job

These jobs can be triggered manually after the generate-facts job is completed. On triggering these jobs, a child pipeline is created.

The child pipeline, called TRIGGERED_CE/EE_PIPELINE is generated in the Omnibus-GitLab repository

TRIGGERED_CE/EE_PIPELINE child pipeline

This child pipeline consists of a job called qa-subset-test which uses the test-on-omnibus/main.gitlab-ci.yml file of the main GitLab project.

qa-subset-test job

The qa-subset-test job triggers another child pipeline in the Omnibus-GitLab repository To get an allure report snapshot as a comment in the MR, following environment variables need to be passed to qa-subset-test

Environment Variable Description
GITLAB_AUTH_TOKEN This is used to give access to the Danger bot to post comment in omnibus-gitlab repository. We are using $DANGER_GITLAB_API_TOKEN which is also being used for other Danger bot related access in omnibugs-gitlab as mentioned ci-variable
ALLURE_MERGE_REQUEST_IID This denotes the MR ID which will be used by e2e-test-report-job which inturn used allure-report-publisher to post message to MR with provided ID e.g. !6190

qa-remaining-test-manual

The qa-remaining-test-manual job is a manual trigger pipeline. It triggers the same pipeline as qa-subset-test but runs the tests which aren’t run as a part of qa-subset-test job.

The environment variables used in qa-subset-test are the same that are used in this job to generate the allure report.

QA-SUBSET-TEST child pipeline

This pipeline runs a subset of all the orchestrated tests using GitLab QA project which in turn uses allure gem to generate report source files for each test that is executed and stores the files in a common folder. Certain orchestrated jobs like instance, decomposition-single-db, decomposition-multiple-db and praefect run only smoke tests which initially used to run the entire suite.

e2e-test-report job

The e2e-test-report job includes .generate-allure-report-base job which uses the allure-report-publisher gem to collate all the report in the mentioned folder into a single report and uploads it to the s3 bucket.

It also posts the allure report as a comment on the MR having the ID passed in ALLURE_MERGE_REQUEST_IID variable in the qa-subset-test.

allure-report-publisher is a gem which uses allure in the backend. It has been catered for GitLab to upload the report and post the comment to MR.

The entire flow of QA in omnibus MR pipeline is as follows

QA flow in omnibus pipeline
Omnibus Parent Pipeline
Trigger:CE/EE-job Child Pipeline
qa-subset-test Child Pipeline
MR Pipeline Triggered on each commit
Creates Child Pipeline
Creates Child Pipelines
Once tests are successful
calls e2e-test-report job
includes
Runs
includes
uploads report
Posts report link as a comment on MR
pulls
Merge
Request

trigger-package stage Unsupported markdown: html Manual Trigger:ce/ee-package job kicked off

trigger-qa stage Unsupported markdown: html qa-subset-test job

from Unsupported markdown: html test-on-omnibus/main.gitlab-ci.yml in gitlab-org/gitlab

report stage Unsupported markdown: html e2e-test-report job

.generate-allure-report-base job fromUnsupported markdown: html quality/pipeline-common

allure-report-publisher gem

test-on-omnibus/main.gitlab-ci.yml Unsupported markdown: html from gitlab-org/gitlab

AWS S3 Unsupported markdown: html gitlab-qa-allure-report Unsupported markdown: html in Unsupported markdown: html eng-quality-ops-ci-cd-shared-infra Unsupported markdown: html project

pulls Unsupported markdown: html image andrcuns/allure-report-publisher:1.6.0

Demo for Allure report & QA pipelines

An in-depth video walkthrough of the pipeline and how to use Allure report is available on YouTube.