Troubleshooting application security

Tier: Ultimate Offering: GitLab.com, Self-managed, GitLab Dedicated

When working with application security features, you might encounter the following issues.

Logging level

The verbosity of logs output by GitLab analyzers is determined by the SECURE_LOG_LEVEL environment variable. Messages of this logging level or higher are output.

From highest to lowest severity, the logging levels are:

  • fatal
  • error
  • warn
  • info (default)
  • debug

Debug-level logging

caution
Debug logging can be a serious security risk. The output may contain the content of environment variables and other secrets available to the job. The output is uploaded to the GitLab server and is visible in job logs.

To enable debug-level logging, add the following to your .gitlab-ci.yml file: 

variables:
  SECURE_LOG_LEVEL: "debug"

This indicates to all GitLab analyzers that they are to output all messages. For more details, see logging level.

Secure job failing with exit code 1

If a Secure job is failing and it’s unclear why:

  1. Enable debug-level logging.
  2. Run the job.
  3. Examine the job’s output.
  4. Remove the debug log level to return to the default info value.

Outdated security reports

When a security report generated for a merge request becomes outdated, the merge request shows a warning message in the security widget and prompts you to take an appropriate action.

This can happen in two scenarios:

Source branch is behind the target branch

A security report can be out of date when the most recent common ancestor commit between the target branch and the source branch is not the most recent commit on the target branch.

To fix this issue, rebase or merge to incorporate the changes from the target branch.

Incorporate target branch changes

Target branch security report is out of date

This can happen for many reasons, including failed jobs or new advisories. When the merge request shows that a security report is out of date, you must run a new pipeline on the target branch. Select new pipeline to run a new pipeline.

Run a new pipeline

Getting warning messages … report.json: no matching files

caution
Debug logging can be a serious security risk. The output may contain the content of environment variables and other secrets available to the job. The output is uploaded to the GitLab server and visible in job logs.

This message is often followed by the error No files to upload, and preceded by other errors or warnings that indicate why the JSON report wasn’t generated. Check the entire job log for such messages. If you don’t find these messages, retry the failed job after setting SECURE_LOG_LEVEL: "debug" as a custom CI/CD variable. This provides extra information to investigate further.

Getting error message sast job: config key may not be used with 'rules': only/except

When including a .gitlab-ci.yml template like SAST.gitlab-ci.yml, the following error may occur, depending on your GitLab CI/CD configuration: 

Unable to create pipeline

    jobs:sast config key may not be used with `rules`: only/except

This error appears when the included job’s rules configuration has been overridden with the deprecated only or except syntax. To fix this issue, you must either:

For more information, see Overriding SAST jobs.

Transitioning your only/except syntax to rules

When overriding the template to control job execution, previous instances of only or except are no longer compatible and must be transitioned to the rules syntax.

If your override is aimed at limiting jobs to only run on main, the previous syntax would look similar to: 

include:
  - template: Jobs/SAST.gitlab-ci.yml

# Ensure that the scanning is only executed on main or merge requests
spotbugs-sast:
  only:
    refs:
      - main
      - merge_requests

To transition the above configuration to the new rules syntax, the override would be written as follows: 

include:
  - template: Jobs/SAST.gitlab-ci.yml

# Ensure that the scanning is only executed on main or merge requests
spotbugs-sast:
  rules:
    - if: $CI_COMMIT_BRANCH == "main"
    - if: $CI_MERGE_REQUEST_ID

If your override is aimed at limiting jobs to only run on branches, not tags, it would look similar to: 

include:
  - template: Jobs/SAST.gitlab-ci.yml

# Ensure that the scanning is not executed on tags
spotbugs-sast:
  except:
    - tags

To transition to the new rules syntax, the override would be rewritten as: 

include:
  - template: Jobs/SAST.gitlab-ci.yml

# Ensure that the scanning is not executed on tags
spotbugs-sast:
  rules:
    - if: $CI_COMMIT_TAG == null

For more information, see rules.

Pin your templates to the deprecated versions

To ensure the latest support, we strongly recommend that you migrate to rules.

If you’re unable to immediately update your CI configuration, there are several workarounds that involve pinning to the previous template versions, for example: 

  include:
    remote: 'https://gitlab.com/gitlab-org/gitlab/-/raw/12-10-stable-ee/lib/gitlab/ci/templates/Security/SAST.gitlab-ci.yml'

Additionally, we provide a dedicated project containing the versioned legacy templates. This can be used for offline setups or for anyone wishing to use Auto DevOps.

Instructions are available in the legacy template project.

Vulnerabilities are found, but the job succeeds. How can you have a pipeline fail instead?

In these circumstances, that the job succeeds is the default behavior. The job’s status indicates success or failure of the analyzer itself. Analyzer results are displayed in the job logs, merge request widget, or security dashboard.

Error: job is used for configuration only, and its script should not be executed

Changes made in GitLab 13.4 to the Security/Dependency-Scanning.gitlab-ci.yml and Security/SAST.gitlab-ci.yml templates mean that if you enable the sast or dependency_scanning jobs by setting the rules attribute, they fail with the error (job) is used for configuration only, and its script should not be executed.

The sast or dependency_scanning stanzas can be used to make changes to all SAST or Dependency Scanning, such as changing variables or the stage, but they cannot be used to define shared rules.

There is an issue open to improve extendability. You can upvote the issue to help with prioritization, and contributions are welcomed.

Empty Vulnerability Report, Dependency List pages

If the pipeline has manual steps with a job that has the allow_failure: false option, and this job is not finished, GitLab can’t populate listed pages with the data from security reports. In this case, the Vulnerability Report and the Dependency List pages are empty. These security pages can be populated by running the jobs from the manual step of the pipeline.

There is an issue open to handle this scenario. You can upvote the issue to help with prioritization, and contributions are welcomed.