Customize pipeline secret detection

Depending on your subscription tier and configuration method, you can change how pipeline secret detection works.

Availability

Different features are available in different GitLab tiers.

Customize analyzer settings

The pipeline secret detection scan settings can be changed through CI/CD variables by using the variables parameter in .gitlab-ci.yml.

Add new patterns

To search for other types of secrets in your repositories, you can customize analyzer rulesets.

To propose a new detection rule for all users of pipeline secret detection, see our single source of truth for our rules and follow the guidance to create a merge request.

If you operate a cloud or SaaS product and you’re interested in partnering with GitLab to better protect your users, learn more about our partner program for leaked credential notifications.

Pin to specific analyzer version

The GitLab-managed CI/CD template specifies a major version and automatically pulls the latest analyzer release within that major version.

In some cases, you may need to use a specific version. For example, you might need to avoid a regression in a later release.

To override the automatic update behavior, set the SECRETS_ANALYZER_VERSION CI/CD variable in your CI/CD configuration file after you include the Secret-Detection.gitlab-ci.yml template.

You can set the tag to:

• A major version, like 4. Your pipelines use any minor or patch updates that are released within this major version.
• A minor version, like 4.5. Your pipelines use any patch updates that are released within this minor version.
• A patch version, like 4.5.0. Your pipelines don’t receive any updates.

This example uses a specific minor version of the analyzer:

include:
  - template: Jobs/Secret-Detection.gitlab-ci.yml

secret_detection:
  variables:
    SECRETS_ANALYZER_VERSION: "4.5"

Enable full history scan

To enable full history scan, set the variable SECRET_DETECTION_HISTORIC_SCAN to true in your .gitlab-ci.yml file.

Run jobs in merge request pipelines

See Use security scanning tools with merge request pipelines.

Override the analyzer jobs

To override a job definition, (for example, change properties like variables or dependencies), declare a job with the same name as the secret_detection job to override. Place this new job after the template inclusion and specify any additional keys under it.

In the following example extract of a .gitlab-ci.yml file:

• The Jobs/Secret-Detection CI template is included.
• In the secret_detection job, the CI/CD variable SECRET_DETECTION_HISTORIC_SCAN is set to true. Because the template is evaluated before the pipeline configuration, the last mention of the variable takes precedence, so an historic scan is performed.

include:
  - template: Jobs/Secret-Detection.gitlab-ci.yml

secret_detection:
  variables:
    SECRET_DETECTION_HISTORIC_SCAN: "true"

Customize analyzer rulesets

You can customize the behavior of pipeline secret detection by creating a ruleset configuration file, either in the repository being scanned or a remote repository. Customization enables you to modify, replace, or extend the default ruleset.

There are multiple kinds of customizations available:

• Modify the behavior of rules predefined in the default ruleset. This includes:
  • Override a rule from the default ruleset.
  • Disable a rule from the default ruleset.
  • Disable or override a rule with a remote ruleset.
• Replace the default ruleset with a custom ruleset using passthroughs. This includes:
  • Use configuration from an inline ruleset.
  • Use configuration from a local ruleset.
  • Use configuration from a remote ruleset.
  • Use configuration from a private remote ruleset
• Extend the behavior of the default ruleset using passthroughs. This includes:
  • Use configuration from a local ruleset.
  • Use configuration from a remote ruleset.
• Ignore secrets and paths using Gitleaks-native functionality. This includes:
  • Use Gitleaks' [allowlist] directive to ignore patterns and paths.
  • Use gitleaks:allow comment to ignore secrets inline.

Create a ruleset configuration file

To create a ruleset configuration file:

1. Create a .gitlab directory at the root of your project, if one doesn’t already exist.
2. Create a file named secret-detection-ruleset.toml in the .gitlab directory.

Modify rules from the default ruleset

You can modify rules predefined in the default ruleset.

Modifying rules can help you adapt pipeline secret detection to an existing workflow or tool. For example you may want to override the severity of a detected secret or disable a rule from being detected at all.

You can also use a ruleset configuration file stored remotely (that is, a remote Git repository or website) to modify predefined rules. New rules must use the custom rule format.

Disable a rule

You can disable rules that you don’t want active. To disable rules from the analyzer default ruleset:

1. Create a ruleset configuration file, if one doesn’t exist already.
2. Set the disabled flag to true in the context of a ruleset section.
3. In one or more ruleset.identifier subsections, list the rules to disable. Every ruleset.identifier section has:
   • A type field for the predefined rule identifier.
   • A value field for the rule name.

In the following example secret-detection-ruleset.toml file, the disabled rules are matched by the type and value of identifiers:

[secrets]
  [[secrets.ruleset]]
    disable = true
    [secrets.ruleset.identifier]
      type  = "gitleaks_rule_id"
      value = "RSA private key"

Override a rule

If there are specific rules to customize, you can override them. For example, you may increase the severity of a specific type of secret because leaking it would have a higher impact on your workflow.

To override rules from the analyzer default ruleset:

1. Create a ruleset configuration file, if one doesn’t exist already.
2. In one or more ruleset.identifier subsections, list the rules to override. Every ruleset.identifier section has:
   • A type field for the predefined rule identifier.
   • A value field for the rule name.
3. In the ruleset.override context of a ruleset section, provide the keys to override. Any combination of keys can be overridden. Valid keys are:
   • description
   • message
   • name
   • severity (valid options are: Critical, High, Medium, Low, Unknown, Info)

In the following secret-detection-ruleset.toml file, rules are matched by the type and value of identifiers and then overridden:

[secrets]
  [[secrets.ruleset]]
    [secrets.ruleset.identifier]
      type  = "gitleaks_rule_id"
      value = "RSA private key"
    [secrets.ruleset.override]
      description = "OVERRIDDEN description"
      message     = "OVERRIDDEN message"
      name        = "OVERRIDDEN name"
      severity    = "Info"

With a remote ruleset

A remote ruleset is a configuration file stored outside the current repository. It can be used to modify rules across multiple projects.

To modify a predefined rule with a remote ruleset, you can use the SECRET_DETECTION_RULESET_GIT_REFERENCE CI/CD variable:

include:
  - template: Jobs/Secret-Detection.gitlab-ci.yml

variables:
  SECRET_DETECTION_RULESET_GIT_REFERENCE: "gitlab.com/example-group/remote-ruleset-project"

Pipeline secret detection assumes the configuration is defined in .gitlab/secret-detection-ruleset.toml file in the repository referenced by the CI variable where the remote ruleset is stored. If that file doesn’t exist, please make sure to create one and follow the steps to override or disable a predefined rule as outlined above.

The SECRET_DETECTION_RULESET_GIT_REFERENCE variable uses a format similar to Git URLs for specifying a URI, optional authentication, and optional Git SHA. The variable uses the following format:

<AUTH_USER>:<AUTH_PASSWORD>@<PROJECT_PATH>@<GIT_SHA>

If the configuration file is stored in a private project that requires authentication, you may use a Group Access Token securely stored in a CI variable to load the remote ruleset:

include:
  - template: Jobs/Secret-Detection.gitlab-ci.yml

variables:
  SECRET_DETECTION_RULESET_GIT_REFERENCE: "group_2504721_bot_7c9311ffb83f2850e794d478ccee36f5:$GROUP_ACCESS_TOKEN@gitlab.com/example-group/remote-ruleset-project"

The group access token must have the read_repository scope and at least the Reporter role. For details, see Repository permissions.

See bot users for groups to learn how to find the username associated with a group access token.

Replace the default ruleset

You can replace the default ruleset configuration using a number of customizations. Those can be combined using passthroughs into a single configuration.

Using passthroughs, you can:

• Chain up to 20 passthroughs into a single configuration to replace or extend predefined rules.
• Include environment variables in passthroughs.
• Set a timeout for evaluating passthroughs.
• Validate TOML syntax used in each defined passthrough.

With an inline ruleset

You can use raw passthrough to replace default ruleset with configuration provided inline.

To do so, add the following in the .gitlab/secret-detection-ruleset.toml configuration file stored in the same repository, and adjust the rule defined under [[rules]] as appropriate:

[secrets]
  [[secrets.passthrough]]
    type   = "raw"
    target = "gitleaks.toml"
    value  = """
title = "replace default ruleset with a raw passthrough"

[[rules]]
description = "Test for Raw Custom Rulesets"
regex = '''Custom Raw Ruleset T[est]{3}'''
"""

The above example replaces the default ruleset with a rule that checks for the regex defined - Custom Raw Ruleset T with a suffix of 3 characters from either one of e, s, or t letters.

For more information on the passthrough syntax to use, see Schema.

With a local ruleset

You can use file passthrough to replace the default ruleset with another file committed to the current repository.

To do so, add the following in the .gitlab/secret-detection-ruleset.toml configuration file stored in the same repository and adjust the value as appropriate to point to the path of the file with the local ruleset configuration:

[secrets]
  [[secrets.passthrough]]
    type   = "file"
    target = "gitleaks.toml"
    value  = "config/gitleaks.toml"

This would replace the default ruleset with the configuration defined in config/gitleaks.toml file.

For more information on the passthrough syntax to use, see Schema.

With a remote ruleset

You can replace the default ruleset with configuration defined in a remote Git repository or a file stored somewhere online using the git and url passthroughs, respectively.

A remote ruleset can be used across multiple projects. For example, you may want to apply the same ruleset to a number of projects in one of your namespaces, in such case, you may use either type of passthrough to load up that remote ruleset and have it used by multiple projects. It also enables centralized management of a ruleset, with only authorized people able to edit.

To use git passthrough, add the following to the .gitlab/secret-detection-ruleset.toml configuration file stored in a repository and adjust the value to point to the address of the Git repository:

# .gitlab/secret-detection-ruleset.toml in https://gitlab.com/user_group/basic_repository
[secrets]
  [[secrets.passthrough]]
    type   = "git"
    ref    = "main"
    subdir = "config"
    value  = "https://gitlab.com/user_group/central_repository_with_shared_ruleset"

In this configuration the analyzer loads the ruleset from the gitleaks.toml file inside the config directory in the main branch of the repository stored at user_group/central_repository_with_shared_ruleset. You can then proceed to include the same configuration in projects other than user_group/basic_repository.

Alternatively, you may use the url passthrough to replace the default ruleset with a remote ruleset configuration.

To use the url passthrough, add the following to the .gitlab/secret-detection-ruleset.toml configuration file stored in a repository and adjust the value to point to the address of the remote file:

# .gitlab/secret-detection-ruleset.toml in https://gitlab.com/user_group/basic_repository
[secrets]
  [[secrets.passthrough]]
    type   = "url"
    target = "gitleaks.toml"
    value  = "https://example.com/gitleaks.toml"

In this configuration the analyzer loads the ruleset configuration from gitleaks.toml file stored at the address provided.

For more information on the passthrough syntax to use, see Schema.

With a private remote ruleset

If a ruleset configuration is stored in a private repository you must provide the credentials to access the repository by using the passthrough’s auth setting.

To use a remote ruleset stored in a private repository, add the following to the .gitlab/secret-detection-ruleset.toml configuration file stored in a repository, adjust the value to point to the address of the Git repository, and update auth to use the appropriate credentials:

[secrets]
  [[secrets.passthrough]]
    type   = "git"
    ref    = "main"
    auth   = "USERNAME:PASSWORD" # replace USERNAME and PASSWORD as appropriate
    subdir = "config"
    value  = "https://gitlab.com/user_group/central_repository_with_shared_ruleset"

For more information on the passthrough syntax to use, see Schema.

Extend the default ruleset

You can also extend the default ruleset configuration with additional rules as appropriate. This can be helpful when you would still like to benefit from the high-confidence predefined rules maintained by GitLab in the default ruleset, but also want to add rules for types of secrets that may be used in your own projects and namespaces. New rules must follow the custom rule format.

With a local ruleset

You can use a file passthrough to extend the default ruleset to add additional rules.

Add the following to the .gitlab/secret-detection-ruleset.toml configuration file stored in the same repository, and adjust the value as appropriate to point to the path of the extended configuration file:

# .gitlab/secret-detection-ruleset.toml
[secrets]
  [[secrets.passthrough]]
    type   = "file"
    target = "gitleaks.toml"
    value  = "extended-gitleaks-config.toml"

The extended configuration stored in extended-gitleaks-config.toml is included in the configuration used by the analyzer in the CI/CD pipeline.

In the example below, we add a couple of new [[rules]] sections that define a number of regular expressions to be detected:

# extended-gitleaks-config.toml
[extend]
# Extends default packaged ruleset, NOTE: do not change the path.
path = "/gitleaks.toml"

[[rules]]
  id = "example_api_key"
  description = "Example Service API Key"
  regex = '''example_api_key'''

[[rules]]
  id = "example_api_secret"
  description = "Example Service API Secret"
  regex = '''example_api_secret'''

With this ruleset configuration the analyzer detects any strings matching with those two defined regex patterns.

For more information on the passthrough syntax to use, see Schema.

With a remote ruleset

Similar to how you can replace the default ruleset with a remote ruleset, you can also extend the default ruleset with configuration stored in a remote Git repository or file stored external to the repository in which you have the .gitlab/secret-detection-ruleset.toml configuration file.

This can be achieved by using either of the git or url passthroughs as discussed previously.

To do that with a git passthrough, add the following to .gitlab/secret-detection-ruleset.toml configuration file stored in the same repository, and adjust the value, ref, and subdir as appropriate to point to the path of the extended configuration file:

# .gitlab/secret-detection-ruleset.toml in https://gitlab.com/user_group/basic_repository
[secrets]
  [[secrets.passthrough]]
    type   = "git"
    ref    = "main"
    subdir = "config"
    value  = "https://gitlab.com/user_group/central_repository_with_shared_ruleset"

Pipeline secret detection assumes the remote ruleset configuration file is called gitleaks.toml, and is stored in config directory on the main branch of the referenced repository.

To extend the default ruleset, the gitleaks.toml file should use [extend] directive similar to the example above:

# https://gitlab.com/user_group/central_repository_with_shared_ruleset/-/raw/main/config/gitleaks.toml
[extend]
# Extends default packaged ruleset, NOTE: do not change the path.
path = "/gitleaks.toml"

[[rules]]
  id = "example_api_key"
  description = "Example Service API Key"
  regex = '''example_api_key'''

[[rules]]
  id = "example_api_secret"
  description = "Example Service API Secret"
  regex = '''example_api_secret'''

To use a url passthrough, add the following to .gitlab/secret-detection-ruleset.toml configuration file stored in the same repository, and adjust the value as appropriate to point to the path of the extended configuration file

# .gitlab/secret-detection-ruleset.toml in https://gitlab.com/user_group/basic_repository
[secrets]
  [[secrets.passthrough]]
    type   = "url"
    target = "gitleaks.toml"
    value  = "https://example.com/gitleaks.toml"

For more information on the passthrough syntax to use, see Schema.

Ignore patterns and paths

There may be situations in which you need to ignore a certain pattern or path from being detected by pipeline secret detection. For example, you may have a file including fake secrets to be used in a test suite.

In that case, you can utilize Gitleaks’ native [allowlist] directive to ignore specific patterns or paths.

To ignore a pattern, add the following to the .gitlab/secret-detection-ruleset.toml configuration file stored in the same repository, and adjust the value as appropriate to point to the path of the extended configuration file:

# .gitlab/secret-detection-ruleset.toml
[secrets]
  [[secrets.passthrough]]
    type   = "file"
    target = "gitleaks.toml"
    value  = "extended-gitleaks-config.toml"

The extended configuration stored in extended-gitleaks-config.toml will be included in the configuration used by the analyzer.

In the example below, we add an [allowlist] directive that defines a regex that matches the secret to be ignored (“allowed”):

# extended-gitleaks-config.toml
[extend]
# Extends default packaged ruleset, NOTE: do not change the path.
path = "/gitleaks.toml"

[allowlist]
  description = "allowlist of patterns to ignore in detection"
  regexTarget = "match"
  regexes = [
    '''glpat-[0-9a-zA-Z_\\-]{20}'''
  ]

This ignores any string matching glpat- with a suffix of 20 characters of digits and letters.

Similarly, you can exclude specific paths from being scanned. In the example below, we define an array of paths to ignore under the [allowlist] directive. A path could either be a regular expression, or a specific file path:

# extended-gitleaks-config.toml
[extend]
# Extends default packaged ruleset, NOTE: do not change the path.
path = "/gitleaks.toml"

[allowlist]
  description = "allowlist of patterns to ignore in detection"
  paths = [
    '''/gitleaks.toml''',
    '''(.*?)(jpg|gif|doc|pdf|bin|svg|socket)'''
  ]

This ignores any secrets detected in either /gitleaks.toml file or any file ending with one of the specified extensions.

For more information on the passthrough syntax to use, see Schema.

Ignore secrets inline

In some instances, you might want to ignore a secret inline. For example, you may have a fake secret in an example or a test suite. In these instances, you will want to ignore the secret instead of having it reported as a vulnerability.

To ignore a secret, add gitleaks:allow as a comment to the line that contains the secret.

For example:

"A personal token for GitLab will look like glpat-JUST20LETTERSANDNUMB"  # gitleaks:allow

Detecting complex strings

The default ruleset provides patterns to detect structured strings with a low rate of false positives. However, you might want to detect more complex strings like passwords. Because Gitleaks doesn’t support lookahead or lookbehind, writing a high-confidence, general rule to detect unstructured strings is not possible.

Although you can’t detect every complex string, you can extend your ruleset to meet specific use cases.

For example, this rule modifies the generic-api-key rule from the Gitleaks default ruleset:

(?i)(?:pwd|passwd|password)(?:[0-9a-z\-_\t .]{0,20})(?:[\s|']|[\s|"]){0,3}(?:=|>|=:|:{1,3}=|\|\|:|<=|=>|:|\?=)(?:'|\"|\s|=|\x60){0,5}([0-9a-z\-_.=\S_]{3,50})(?:['|\"|\n|\r|\s|\x60|;]|$)

This regular expression matches:

1. A case-insensitive identifier that starts with pwd, or passwd or password. You can adjust this with other variations like secret or key.
2. A suffix that follows the identifier. The suffix is a combination of digits, letters, and symbols, and is between zero and 23 characters long.
3. Commonly used assignment operators, like =, :=, :, or =>.
4. A secret prefix, often used as a boundary to help with detecting the secret.
5. A string of digits, letters, and symbols, which is between three and 50 characters long. This is the secret itself. If you expect longer strings, you can adjust the length.
6. A secret suffix, often used as a boundary. This matches common endings like ticks, line breaks, and new lines.

Here are example strings which are matched by this regular expression:

pwd = password1234
passwd = 'p@ssW0rd1234'
password = thisismyverylongpassword
password => mypassword
password := mypassword
password: password1234
"password" = "p%ssward1234"
'password': 'p@ssW0rd1234'

To use this regex, extend your ruleset with one of the methods documented on this page.

For example, imagine you wish to extend the default ruleset with a local ruleset that includes this rule.

Add the following to a .gitlab/secret-detection-ruleset.toml configuration file stored in the same repository. Adjust the value to point to the path of the extended configuration file:

# .gitlab/secret-detection-ruleset.toml
[secrets]
  [[secrets.passthrough]]
    type   = "file"
    target = "gitleaks.toml"
    value  = "extended-gitleaks-config.toml"

In extended-gitleaks-config.toml file, add a new [[rules]] section with the regular expression you want to use:

# extended-gitleaks-config.toml
[extend]
# Extends default packaged ruleset, NOTE: do not change the path.
path = "/gitleaks.toml"

[[rules]]
  description = "Generic Password Rule"
  id = "generic-password"
  regex = '''(?i)(?:pwd|passwd|password)(?:[0-9a-z\-_\t .]{0,20})(?:[\s|']|[\s|"]){0,3}(?:=|>|=:|:{1,3}=|\|\|:|<=|=>|:|\?=)(?:'|\"|\s|=|\x60){0,5}([0-9a-z\-_.=\S_]{3,50})(?:['|\"|\n|\r|\s|\x60|;]|$)'''
  entropy = 3.5
  keywords = ["pwd", "passwd", "password"]

Available CI/CD variables

Pipeline secret detection can be customized by defining available CI/CD variables:

In previous GitLab versions, the following variables were also available:

Offline configuration

An offline environment has limited, restricted, or intermittent access to external resources through the internet. For instances in such an environment, pipeline secret detection requires some configuration changes. The instructions in this section must be completed together with the instructions detailed in offline environments.

Configure GitLab Runner

By default, a runner tries to pull Docker images from the GitLab container registry even if a local copy is available. You should use this default setting, to ensure Docker images remain current. However, if no network connectivity is available, you must change the default GitLab Runner pull_policy variable.

Configure the GitLab Runner CI/CD variable pull_policy to if-not-present.

Use local pipeline secret detection analyzer image

Use a local pipeline secret detection analyzer image if you want to obtain the image from a local Docker registry instead of the GitLab container registry.

Prerequisites:

• Importing Docker images into a local offline Docker registry depends on your network security policy. Consult your IT staff to find an accepted and approved process to import or temporarily access external resources.

1. Import the default pipeline secret detection analyzer image from registry.gitlab.com into your local Docker container registry:

   registry.gitlab.com/security-products/secrets:6

   The pipeline secret detection analyzer’s image is periodically updated so you should periodically update the local copy.

2. Set the CI/CD variable SECURE_ANALYZERS_PREFIX to the local Docker container registry.

   include:
     - template: Jobs/Secret-Detection.gitlab-ci.yml
   
   variables:
     SECURE_ANALYZERS_PREFIX: "localhost:5000/analyzers"

The pipeline secret detection job should now use the local copy of the analyzer Docker image, without requiring internet access.

Using a custom SSL CA certificate authority

To trust a custom Certificate Authority, set the ADDITIONAL_CA_CERT_BUNDLE variable to the bundle of CA certificates that you trust. Do this either in the .gitlab-ci.yml file, in a file variable, or as a CI/CD variable.

• In the .gitlab-ci.yml file, the ADDITIONAL_CA_CERT_BUNDLE value must contain the text representation of the X.509 PEM public-key certificate.

  For example:

  variables:
    ADDITIONAL_CA_CERT_BUNDLE: |
        -----BEGIN CERTIFICATE-----
        MIIGqTCCBJGgAwIBAgIQI7AVxxVwg2kch4d56XNdDjANBgkqhkiG9w0BAQsFADCB
        ...
        jWgmPqF3vUbZE0EyScetPJquRFRKIesyJuBFMAs=
        -----END CERTIFICATE-----

• If using a file variable, set the value of ADDITIONAL_CA_CERT_BUNDLE to the path to the certificate.

• If using a variable, set the value of ADDITIONAL_CA_CERT_BUNDLE to the text representation of the certificate.

Demos

There are demonstration projects that illustrate some of these configuration options.

Below is a table with the demonstration projects and their associated workflows:

There are also some video demonstrations walking through setting up remote rulesets:

• Secret detection with local and remote ruleset