• Choose a test codebase
  • Benchmarks and example projects
  • AI-generated test code
• Conduct the test

Evaluate GitLab SAST

You might choose to evaluate GitLab SAST before using it in your organization. Consider the following guidance as you plan and conduct your evaluation.

Choose a test codebase

When choosing a codebase to test SAST, you should:

• Test in a repository where you can safely modify the CI/CD configuration without getting in the way of normal development activities. SAST scans run in your CI/CD pipeline, so you’ll need to make a small edit to the CI/CD configuration to enable SAST.
  • You can make a fork or copy of an existing repository for testing. This way, you can set up your testing environment without any chance of interrupting normal development.
• Use a codebase that matches your organization’s typical technology stack.
• Use a language that Advanced SAST supports. Advanced SAST produces more accurate results than other analyzers.

Your test project must have GitLab Ultimate. Only Ultimate includes features like:

• Proprietary cross-file, cross-function scanning with GitLab Advanced SAST.
• The merge request widget, pipeline security report, and default-branch vulnerability report that makes scan results visible and actionable.

Benchmarks and example projects

If you choose to use a benchmark or an intentionally vulnerable application for testing, remember that these applications:

• Focus on specific vulnerability types. The benchmark’s focus may be different from the vulnerability types your organization prioritizes for discovery and remediation.
• Use specific technologies in specific ways that may differ from how your organization builds software.
• Report results in ways that may implicitly emphasize certain criteria over others. For example, you may prioritize precision (fewer false-positive results) while the benchmark only scores based on recall (fewer false-negative results).

Epic 15296 tracks work to recommend specific projects for testing.

AI-generated test code

You should not use AI tools to create vulnerable code for testing SAST. AI models often return code that is not truly exploitable.

For example:

• AI tools often write small functions that take a parameter and use it in a sensitive context (called a “sink”), without actually receiving any user input. This can be a safe design if the function is only called with program-controlled values, like constants. The code is not vulnerable unless user input is allowed to flow to these sinks without first being sanitized or validated.
• AI tools may comment out part of the vulnerability to prevent you from accidentally running the code.

Reporting vulnerabilities in these unrealistic examples would cause false-positive results in real-world code. GitLab SAST is not designed to report vulnerabilities in these cases.

Conduct the test

After you choose a codebase to test with, you’re ready to conduct the test. You can follow these steps:

1. Enable SAST by creating a merge request (MR) that adds SAST to the CI/CD configuration.
   • Be sure to set the CI/CD variable to enable Advanced SAST for more accurate results.
2. Merge the MR to the repository’s default branch.
3. Open the Vulnerability Report to see the vulnerabilities found on the default branch.
   • If you’re using Advanced SAST, you can use the Tool filter to show results only from that scanner.
4. Review vulnerability results.
   • Check the code flow view for Advanced SAST vulnerabilities that involve tainted user input, like SQL injection or path traversal.
   • If you have GitLab Duo Enterprise, explain or resolve a vulnerability.
5. To see how scanning works as new code is developed, create a new merge request that changes application code and adds a new vulnerability or weakness.