• Use cases
• Requirements
  • Making Secret Detection available to all GitLab tiers
    • Summary of features per tier
• Configuration
  • Customizing settings
    • Available variables
  • Custom rulesets
  • Logging level
• Full History Secret Scan
  • Make GitLab Secret Detection analyzer image available inside your Docker registry
    • If support for Custom Certificate Authorities are needed
• Troubleshooting
  • Getting warning message gl-secret-detection-report.json: no matching files

Secret Detection

A recurring problem when developing applications is that developers may unintentionally commit secrets and credentials to their remote repositories. If other people have access to the source, or if the project is public, the sensitive information is then exposed and can be leveraged by malicious users to gain access to resources like deployment environments.

GitLab 11.9 includes a new check called Secret Detection. It scans the content of the repository to find API keys and other information that should not be there.

GitLab displays identified secrets visibly in a few places:

• Security Dashboard
• Pipelines’ Security tab
• Report in the merge request widget

Use cases

• Detecting unintentional commit of secrets like keys, passwords, and API tokens.
• Performing a single or recurring scan of the full history of your repository for secrets.

Requirements

To run Secret Detection jobs, by default, you need GitLab Runner with the docker or kubernetes executor. If you’re using the shared runners on GitLab.com, this is enabled by default.

Making Secret Detection available to all GitLab tiers

To make Secret Detection available to as many customers as possible, we have enabled it for all GitLab tiers. However not all features are available on every tier. See the breakdown below for more details.

Summary of features per tier

Different features are available in different GitLab tiers, as shown in the following table:

Configuration

GitLab 13.1 splits Secret Detection from the SAST configuration into its own CI/CD template. If you’re using GitLab 13.0 or earlier and SAST is enabled, then Secret Detection is already enabled.

Secret Detection is performed by a specific analyzer during the secret-detection job. It runs regardless of your app’s programming language.

The Secret Detection analyzer includes Gitleaks and TruffleHog checks.

Note that the Secret Detection analyzer ignores Password-in-URL vulnerabilities if the password begins with a dollar sign ($), as this likely indicates the password is an environment variable. For example, https://username:$password@example.com/path/to/repo isn’t detected, while https://username:password@example.com/path/to/repo is.

To enable Secret Detection for GitLab 13.1 and later, you must include the Secret-Detection.gitlab-ci.yml template that’s provided as a part of your GitLab installation. For GitLab versions earlier than 11.9, you can copy and use the job as defined in that template.

Add the following to your .gitlab-ci.yml file:

include:
  - template: Secret-Detection.gitlab-ci.yml

The included template creates Secret Detection jobs in your CI/CD pipeline and scans your project’s source code for secrets.

The results are saved as a Secret Detection report artifact that you can later download and analyze. Due to implementation limitations, we always take the latest Secret Detection artifact available.

Customizing settings

The Secret Detection scan settings can be changed through environment variables by using the variables parameter in .gitlab-ci.yml.

To override a job definition, (for example, change properties like variables or dependencies), declare a job with the same name as the SAST job to override. Place this new job after the template inclusion and specify any additional keys under it.

In the following example, we include the Secret Detection template and at the same time we override the secret_detection job with the SECRET_DETECTION_HISTORIC_SCAN variable to true:

include:
  - template: Secret-Detection.gitlab-ci.yml

secret_detection:
  variables:
    SECRET_DETECTION_HISTORIC_SCAN: "true"

Because the template is evaluated before the pipeline configuration, the last mention of the variable takes precedence.

Available variables

Secret Detection can be customized by defining available variables:

Custom rulesets

You can customize the default secret detection rules provided with GitLab. Customization allows you to exclude rules and add new rules.

To create a custom ruleset:

1. Create a .gitlab directory at the root of your project, if one doesn’t already exist.
2. Create a custom ruleset file named secret-detection-ruleset.toml in the .gitlab directory.

3. In the secret-detection-ruleset.toml file, do one of the following:

   • Define a custom ruleset:

     [secrets]
       description = 'secrets custom rules configuration'
     
       [[secrets.passthrough]]
         type  = "raw"
         target = "gitleaks.toml"
         value = """\
     title = "gitleaks config"
     # add regexes to the regex table
     [[rules]]
     description = "Test for Raw Custom Rulesets"
     regex = '''Custom Raw Ruleset T[est]{3}'''
     """
     

   • Provide the name of the file containing a custom ruleset:

     [secrets]
       description = 'secrets custom rules configuration'
     
       [[secrets.passthrough]]
         type  = "file"
         target = "gitleaks.toml"
         value = "config/gitleaks.toml"
     

Logging level

To control the verbosity of logs set the SECURE_LOG_LEVEL environment variable. Messages of this logging level or higher are output. Introduced in GitLab 13.1.

From highest to lowest severity, the logging levels are:

• fatal
• error
• warn
• info (default)
• debug

Full History Secret Scan

GitLab 12.11 introduced support for scanning the full history of a repository. This new functionality is particularly useful when you are enabling Secret Detection in a repository for the first time and you want to perform a full secret scan. Running a secret scan on the full history can take a long time, especially for larger repositories with lengthy Git histories. We recommend not setting this variable as part of your normal job definition.

A new configuration variable (SECRET_DETECTION_HISTORIC_SCAN) can be set to change the behavior of the GitLab Secret Detection scan to run on the entire Git history of a repository.

We have created a short video walkthrough showcasing how you can perform a full history secret scan.

Make GitLab Secret Detection analyzer image available inside your Docker registry

Import the following default Secret Detection analyzer images from registry.gitlab.com into your local Docker container registry:

registry.gitlab.com/gitlab-org/security-products/analyzers/secrets:3

The process for importing Docker images into a local offline Docker registry depends on your network security policy. Please consult your IT staff to find an accepted and approved process by which external resources can be imported or temporarily accessed. Note that these scanners are updated periodically with new definitions, so consider if you’re able to make periodic updates yourself.

For details on saving and transporting Docker images as a file, see Docker’s documentation on docker save, docker load, docker export, and docker import.

If support for Custom Certificate Authorities are needed

Support for custom certificate authorities was introduced in the following versions.

Troubleshooting

Getting warning message gl-secret-detection-report.json: no matching files

For information on this, see the general Application Security troubleshooting section.