Pipeline secret detection

Pipeline secret detection scans files after they are committed to a Git repository and pushed to GitLab.

After you enable pipeline secret detection, scans run in a CI/CD job named secret_detection. You can run scans and view pipeline secret detection JSON report artifacts in any GitLab tier.

With GitLab Ultimate, pipeline secret detection results are also processed so you can:

• See them in the merge request widget, pipeline security report, and vulnerability report.
• Use them in approval workflows.
• Review them in the security dashboard.
• Automatically respond to leaks in public repositories.
• Enforce consistent secret detection rules across projects using security policies.

For an interactive reading and how-to demo of this pipeline secret detection documentation see:

• How to enable secret detection in GitLab Application Security Part 1/2
• How to enable secret detection in GitLab Application Security Part 2/2

For other interactive reading and how-to demos, see the Get Started With GitLab Application Security Playlist.

Availability

Different features are available in different GitLab tiers.

Coverage

Pipeline secret detection is optimized to balance coverage and run time. Only the current state of the repository and future commits are scanned for secrets. To identify secrets already present in the repository’s history, run an historic scan once after enabling pipeline secret detection. Scan results are available only after the pipeline is completed.

Exactly what is scanned for secrets depends on the type of pipeline, and whether any additional configuration is set.

By default, when you run a pipeline:

• On the default branch, the Git working tree is scanned. This means the entire repository is scanned as though it were a typical directory.
• On a new, non-default branch, the content of all commits from the most recent commit on the parent branch to the latest commit is scanned.
• On an existing, non-default branch, the content of all commits from the most recent branch commit to the latest commit is scanned.
• On a merge request, the content of all commits on the branch is scanned. If the analyzer can’t access every commit, the content of all commits from the parent to the latest commit is scanned. To use merge request pipelines, you must use the latest pipeline secret detection template.

To override the default behavior, use the available CI/CD variables.

Historic scan

By default, pipeline secret detection scans only the current state of the Git repository. Any secrets contained in the repository’s history are not detected. Run a historic scan to check for secrets from all commits and branches in the Git repository.

You should do a historic scan only once, after enabling pipeline secret detection. Historic scans can take a long time, especially for larger repositories with lengthy Git histories. After completing an initial historic scan, use only standard pipeline secret detection as part of your pipeline.

Advanced vulnerability tracking

When developers make changes to a file with identified secrets, it’s likely that the positions of these secrets will also change. Pipeline secret detection may have already flagged these secrets as vulnerabilities, tracked in the vulnerability report. These vulnerabilities are associated with specific secrets for easy identification and action. However, if the detected secrets aren’t accurately tracked as they shift, managing vulnerabilities becomes challenging, potentially resulting in duplicate vulnerability reports.

Pipeline secret detection uses an advanced vulnerability tracking algorithm to more accurately identify when the same secret has moved within a file due to refactoring or unrelated changes.

For more information, see the confidential project https://gitlab.com/gitlab-org/security-products/post-analyzers/tracking-calculator. The content of this project is available only to GitLab team members.

Unsupported workflows

• The algorithm does not support the workflow where the existing finding lacks a tracking signature and does not share the same location as the newly detected finding.
• For some rule types, such as cryptographic keys, pipeline secret detection identifies leaks by matching prefix of the secret rather than the entire secret value. In this scenario, the algorithm consolidates different secrets of the same rule type in a file into a single finding, rather than treating each distinct secret as a separate finding. For example, the SSH Private Key rule type matches only the -----BEGIN OPENSSH PRIVATE KEY----- prefix of a value to confirm the presence of a SSH private key. If there are two distinct SSH Private Keys within the same file, the algorithm considers both values as identical and reports only one finding instead of two.
• The algorithm’s scope is limited to a per-file basis, meaning that the same secret appearing in two different files is treated as two distinct findings.

Detected secrets

Pipeline secret detection scans the repository’s content for specific patterns. Each pattern matches a specific type of secret and is specified in a rule by using a TOML syntax. GitLab maintains the default set of rules.

With GitLab Ultimate you can extend these rules to suit your needs. For example, while personal access tokens that use a custom prefix are not detected by default, you can customize the rules to identify these tokens. For details, see Customize analyzer rulesets.

To confirm which secrets are detected by pipeline secret detection, see Detected secrets. To provide reliable, high-confidence results, pipeline secret detection only looks for passwords or other unstructured secrets in specific contexts like URLs.

When a secret is detected a vulnerability is created for it. The vulnerability remains as “Still detected” even if the secret is removed from the scanned file and pipeline secret detection has been run again. This is because the secret remains in the Git repository’s history. To remove a secret from the Git repository’s history, see Redact text from repository.

Enable the analyzer

Enable the analyzer to use pipeline secret detection. After you enable it, you can customize the analyzer settings.

Prerequisites:

• Linux-based GitLab Runner with the docker or kubernetes executor. If you’re using hosted runners for GitLab.com, this is enabled by default.
  • Windows Runners are not supported.
  • CPU architectures other than amd64 are not supported.
• GitLab CI/CD configuration (.gitlab-ci.yml) must include the test stage.

To enable pipeline secret detection, either:

• Enable Auto DevOps, which includes Auto Secret Detection.
• Edit the .gitlab-ci.yml file manually. Use this method if your .gitlab-ci.yml file is complex.
• Use an automatically configured merge request.

Edit the .gitlab-ci.yml file manually

This method requires you to manually edit the existing .gitlab-ci.yml file. Use this method if your GitLab CI/CD configuration file is complex.

1. On the left sidebar, select Search or go to and find your project.

2. Select Build > Pipeline editor.

3. Copy and paste the following to the bottom of the .gitlab-ci.yml file:

   YAML Copy to clipboard

   include:
     - template: Jobs/Secret-Detection.gitlab-ci.yml

4. Select the Validate tab, then select Validate pipeline. The message Simulation completed successfully indicates the file is valid.

5. Select the Edit tab.

6. Optional. In the Commit message text box, customize the commit message.

7. In the Branch text box, enter the name of the default branch.

8. Select Commit changes.

Pipelines now include a pipeline secret detection job.

Use an automatically configured merge request

This method automatically prepares a merge request, with the pipeline secret detection template included in the .gitlab-ci.yml file. You then merge the merge request to enable pipeline secret detection.

To enable pipeline secret detection:

1. On the left sidebar, select Search or go to and find your project.
2. Select Secure > Security configuration.
3. In the Pipeline secret detection row, select Configure with a merge request.
4. Optional. Complete the fields.
5. Select Create merge request.
6. Review and merge the merge request.

Pipelines now include a pipeline secret detection job.

Output

Pipeline secret detection outputs the file gl-secret-detection-report.json as a job artifact. The file contains detected secrets. You can download the file for processing outside GitLab.

For more information, see:

• Report file schema
• Example report file

Remediate a leaked secret

When a secret is detected, you should rotate it immediately. GitLab attempts to automatically revoke some types of leaked secrets. For those that are not automatically revoked, you must do so manually.

Purging a secret from the repository’s history does not fully address the leak. The original secret remains in any existing forks or clones of the repository.

For instructions on how to respond to a leaked secret, select the vulnerability in the vulnerability report.

FIPS-enabled images

The default scanner images are built off a base Alpine image for size and maintainability. GitLab offers Red Hat UBI versions of the images that are FIPS-enabled.

To use the FIPS-enabled images, either:

• Set the SECRET_DETECTION_IMAGE_SUFFIX CI/CD variable to -fips.
• Add the -fips extension to the default image name.

For example:

variables:
  SECRET_DETECTION_IMAGE_SUFFIX: '-fips'

include:
  - template: Jobs/Secret-Detection.gitlab-ci.yml

Troubleshooting

Debug-level logging

Debug-level logging can help when troubleshooting. For details, see debug-level logging.

Warning: gl-secret-detection-report.json: no matching files

For information on this, see the general Application Security troubleshooting section.

Error: Couldn't run the gitleaks command: exit status 2

The pipeline secret detection analyzer relies on generating patches between commits to scan content for secrets. If the number of commits in a merge request is greater than the value of the GIT_DEPTH CI/CD variable, Secret Detection fails to detect secrets.

For example, you could have a pipeline triggered from a merge request containing 60 commits and the GIT_DEPTH variable set to less than 60. In that case the pipeline secret detection job fails because the clone is not deep enough to contain all of the relevant commits. To verify the current value, see pipeline configuration.

To confirm this as the cause of the error, enable debug-level logging, then rerun the pipeline. The logs should look similar to the following example. The text “object not found” is a symptom of this error.

ERRO[2020-11-18T18:05:52Z] object not found
[ERRO] [secrets] [2020-11-18T18:05:52Z] ▶ Couldn't run the gitleaks command: exit status 2
[ERRO] [secrets] [2020-11-18T18:05:52Z] ▶ Gitleaks analysis failed: exit status 2

To resolve the issue, set the GIT_DEPTH CI/CD variable to a higher value. To apply this only to the pipeline secret detection job, the following can be added to your .gitlab-ci.yml file:

secret_detection:
  variables:
    GIT_DEPTH: 100

Error: ERR fatal: ambiguous argument

Pipeline secret detection can fail with the message ERR fatal: ambiguous argument error if your repository’s default branch is unrelated to the branch the job was triggered for. See issue !352014 for more details.

To resolve the issue, make sure to correctly set your default branch on your repository. You should set it to a branch that has related history with the branch you run the secret-detection job on.

exec /bin/sh: exec format error message in job log

The GitLab pipeline secret detection analyzer only supports running on the amd64 CPU architecture. This message indicates that the job is being run on a different architecture, such as arm.

Error: fatal: detected dubious ownership in repository at '/builds/<project dir>'

Secret detection might fail with an exit status of 128. This can be caused by a change to the user on the Docker image.

For example:

$ /analyzer run
[INFO] [secrets] [2024-06-06T07:28:13Z] ▶ GitLab secrets analyzer v6.0.1
[INFO] [secrets] [2024-06-06T07:28:13Z] ▶ Detecting project
[INFO] [secrets] [2024-06-06T07:28:13Z] ▶ Analyzer will attempt to analyze all projects in the repository
[INFO] [secrets] [2024-06-06T07:28:13Z] ▶ Loading ruleset for /builds....
[WARN] [secrets] [2024-06-06T07:28:13Z] ▶ /builds/....secret-detection-ruleset.toml not found, ruleset support will be disabled.
[INFO] [secrets] [2024-06-06T07:28:13Z] ▶ Running analyzer
[FATA] [secrets] [2024-06-06T07:28:13Z] ▶ get commit count: exit status 128

To work around this issue, add a before_script with the following:

before_script:
    - git config --global --add safe.directory "$CI_PROJECT_DIR"

For more information about this issue, see issue 465974.