License scanning of CycloneDX files

Tier: Ultimate Offering: GitLab.com, Self-managed, GitLab Dedicated
History
  • Introduced in GitLab 15.9 for GitLab SaaS with two flags named license_scanning_sbom_scanner and package_metadata_synchronization. Both flags disabled by default.
  • Generally available in GitLab 16.4. Feature flags license_scanning_sbom_scanner and package_metadata_synchronization removed.
  • The legacy License Compliance analyzer (License-Scanning.gitlab-ci.yml) was removed in GitLab 17.0.
  • In GitLab 17.5 we introduced the ability to use a CycloneDX report artifact as a source of data for license information behind the feature flag license_scanning_with_sbom_licenses, disabled by default.

To detect the licenses in use, License Compliance relies on running the Dependency Scanning CI Jobs, and analyzing the CycloneDX Software Bill of Materials (SBOM) generated by those jobs. This method of scanning is capable of parsing and identifying over 600 different types of licenses, as defined in the SPDX list. Third-party scanners may be used to generate the list of dependencies, as long as they produce a CycloneDX report artifact for one of our supported languages and follow the GitLab CycloneDX property taxonomy. The ability to provide other licenses is tracked in epic 10861.

note
The License Scanning feature relies on publicly available package metadata collected in an external database and synced with the GitLab instance automatically. This database is a multi-region Google Cloud Storage bucket hosted in the United States. The scan is executed exclusively within the GitLab instance. No contextual information (for example, a list of project dependencies) is sent to the external service.

Configuration

To enable License scanning of CycloneDX files:

  • Using the Dependency Scanning template
  • Or use the CI/CD component for applicable package registries.

Supported languages and package managers

License scanning is supported for the following languages and package managers:

Language Package Manager Dependency Scanning Template CI/CD Component
.NET NuGet Yes No
C# Yes No
C Conan Yes No
C++ Yes No
Go1 Go Yes No
Java Gradle Yes No
Maven Yes No
Android Yes Yes
JavaScript and TypeScript npm Yes No
pnpm Yes No
yarn Yes No
PHP Composer Yes No
Python setuptools Yes No
pip Yes No
Pipenv Yes No
Poetry Yes No
Ruby Bundler Yes No
Scala sbt Yes No
Rust cargo No Yes

  1. Go standard libraries such as `stdlib` are not supported and will appear with an `unknown` license. Support for these is tracked in issue 480305.

The supported files and versions are the ones supported by Dependency Scanning.

Data sources

License information for supported packages is obtained from the sources below. GitLab does additional processing on the original data, which includes mapping variations to the canonical license names.

Package manager Source
Cargo https://deps.dev/
Conan https://github.com/conan-io/conan-center-index
Go https://index.golang.org/
Maven https://storage.googleapis.com/maven-central
npm https://deps.dev/
NuGet https://api.nuget.org/v3/catalog0/index.json
Packagist https://packagist.org/packages/list.json
PyPI https://warehouse.pypa.io/api-reference/bigquery-datasets.html
Rubygems https://rubygems.org/versions

License expressions

The License Scanning of CycloneDX files does not support composite licenses. Adding this capability is tracked in issue 336878.

Blocking merge requests based on detected licenses

Users can require approval for merge requests based on the licenses that are detected by configuring a license approval policy.

Running in an offline environment

For self-managed GitLab instances in an environment with limited, restricted, or intermittent access to external resources through the internet, some adjustments are required to successfully scan CycloneDX reports for licenses. For more information, see the offline quick start guide.

Use CycloneDX report as a source of license information

The ability to use a CI report artifact as a source of license information data was introduced in GitLab 17.5 behind the feature flag license_scanning_with_sbom_licenses.

When the feature flag license_scanning_with_sbom_licenses is enabled, the License Scanning uses the licenses field of the CycloneDX JSON SBOM when available. If the license information is unavailable, the license information imported from the external license database will be used(current behavior). License information can be provided using a valid SPDX identifier or a license name. However, providing a license using an SPDX License Expression is not supported. More information about the license field format can be found on the CycloneDX specification.

Compatible CycloneDX SBOM generators that provide the licenses field can be found in the CycloneDX Tool Center.

Troubleshooting

A CycloneDX file is not being scanned and appears to provide no results

Ensure that the CycloneDX file adheres to the CycloneDX JSON specification. This specification does not permit duplicate entries. Projects that contain multiple SBOM files should either report each SBOM file up as individual CI report artifacts or they should ensure that duplicates are removed if the SBOMs are merged as part of the CI pipeline.

You can validate CycloneDX SBOM files against the CycloneDX JSON specification as follows: 

$ docker run -it --rm -v "$PWD:/my-cyclonedx-sboms" -w /my-cyclonedx-sboms cyclonedx/cyclonedx-cli:latest cyclonedx validate --input-version v1_4 --input-file gl-sbom-all.cdx.json

Validating JSON BOM...
BOM validated successfully.

If the JSON BOM fails validation, for example, because there are duplicate components: 

Validation failed: Found duplicates at the following index pairs: "(A, B), (C, D)"
#/properties/components/uniqueItems

This issue can be fixed by updating the CI template to use jq to remove the duplicate components from the gl-sbom-*.cdx.json report by overriding the job definition that produces the duplicate components. For example, the following removes duplicate components from the gl-sbom-gem-bundler.cdx.json report file produced by the gemnasium-dependency_scanning job: 

include:
  - template: Jobs/Dependency-Scanning.gitlab-ci.yml

gemnasium-dependency_scanning:
  after_script:
    - apk update && apk add jq
    - jq '.components |= unique' gl-sbom-gem-bundler.cdx.json > tmp.json && mv tmp.json gl-sbom-gem-bundler.cdx.json

Remove unused license data

License scanning changes (released in GitLab 15.9) required a significant amount of additional disk space to be available on the instances. This issue was resolved in GitLab 16.3 by the Reduce package metadata table on-disk footprint epic. But if your instance was running license scanning between GitLab 15.9 and 16.3, you may want to remove the unneeded data.

To remove the unneeded data:

  1. Check if the package_metadata_synchronization feature flag is currently, or was previously enabled, and if so, disable it. Use Rails console to execute the following commands. 

    Feature.enabled?(:package_metadata_synchronization) && Feature.disable(:package_metadata_synchronization)

  2. Check if there is deprecated data in the database: 

    PackageMetadata::PackageVersionLicense.count
PackageMetadata::PackageVersion.count

  3. If there is deprecated data in the database, remove it by running the following commands in order: 

    ActiveRecord::Base.connection.execute('SET statement_timeout TO 0')
PackageMetadata::PackageVersionLicense.delete_all
PackageMetadata::PackageVersion.delete_all

Dependency licenses are unknown

Open source license information is stored in the database and used to resolve licenses for a project’s dependencies. A dependency’s license may appear as unknown if license information does not exist or if that data is not yet available in the database.

Lookups for a dependency’s licenses are done upon pipeline completion, so if this data was not available at that time an unknown license is recorded. This license is shown up until a subsequent pipeline is executed at which point another license lookup is made. If a lookup confirms the dependency’s license has changed, the new license is shown at this time.