Operational container scanning

Supported architectures

Operational container scanning (OCS) is supported for linux/arm64 and linux/amd64.

Enable operational container scanning

You can use OCS to scan container images in your cluster for security vulnerabilities. OCS uses a wrapper image around Trivy to scan images for vulnerabilities.

OCS can be configured to run on a cadence by using agent config or a project’s scan execution policy.

Enable via agent configuration

To enable scanning of images within your Kubernetes cluster via the agent configuration, add a container_scanning configuration block to your agent configuration with a cadence field containing a CRON expression for when the scans are run.

container_scanning:
  cadence: '0 0 * * *' # Daily at 00:00 (Kubernetes cluster time)

The cadence field is required. GitLab supports the following types of CRON syntax for the cadence field:

• A daily cadence of once per hour at a specified hour, for example: 0 18 * * *
• A weekly cadence of once per week on a specified day and at a specified hour, for example: 0 13 * * 0

By default, operational container scanning does not scan any workloads for vulnerabilities. You can set the vulnerability_report block with the namespaces field which can be used to select which namespaces are scanned. For example, if you would like to scan only the default, kube-system namespaces, you can use this configuration:

container_scanning:
  cadence: '0 0 * * *'
  vulnerability_report:
    namespaces:
      - default
      - kube-system

For every target namespace, all images in the following workload resources are scanned by default:

• Pod
• ReplicaSet
• ReplicationController
• StatefulSet
• DaemonSet
• CronJob
• Job

This can be customized by configuring the Trivy Kubernetes Resource Detection.

Enable via scan execution policies

To enable scanning of images in your Kubernetes cluster by using scan execution policies, use the scan execution policy editor to create a new schedule rule.

Operational container scanning operates independently of GitLab pipelines. It is fully automated and managed by the Kubernetes Agent, which initiates new scans at the scheduled time configured in the Scan Execution Policy. The agent creates a dedicated Job within your cluster to perform the scan and report findings back to GitLab.

Here is an example of a policy which enables operational container scanning within the cluster the Kubernetes agent is attached to:

- name: Enforce container scanning in cluster connected through my-gitlab-agent for default and kube-system namespaces
  enabled: true
  rules:
  - type: schedule
    cadence: '0 10 * * *'
    agents:
      <agent-name>:
        namespaces:
        - 'default'
        - 'kube-system'
  actions:
  - scan: container_scanning

The keys for a schedule rule are:

• cadence (required): a CRON expression for when the scans are run
• agents:<agent-name> (required): The name of the agent to use for scanning
• agents:<agent-name>:namespaces (required): The Kubernetes namespaces to scan.

You can view the complete schema within the scan execution policy documentation.

OCS vulnerability resolution for multi cluster configuration

To ensure accurate vulnerability tracking with OCS, you should create a separate GitLab project with OCS enabled for each cluster. If you have multiple clusters, be sure to use one project for each cluster.

OCS resolves vulnerabilities that are no longer found in your cluster after each scan by comparing the current scan vulnerabilities with those previously detected. Any vulnerabilities from earlier scans that are no longer present in the current scan are resolved for the GitLab project.

If multiple clusters are configured in the same project, an OCS scan in one cluster (for example, Project A) would resolve previously detected vulnerabilities from another cluster (for example, Project B), leading to incorrect vulnerability reporting.

Each vulnerability is associated with the agent that reported it. If you register a new agent for a cluster, scans from the new agent do not resolve vulnerabilities reported by the previous agent. To clean up these vulnerabilities, change their status manually. If you delete the previous agent, these vulnerabilities no longer show which agent reported them.

Permissions for reporting vulnerabilities

OCS creates and resolves vulnerabilities on behalf of the user who registered the agent. This user must have the Security Manager, Maintainer, or Owner role for the project that contains the agent.

If this user loses access to the project or no longer has one of these roles (for example, because their account is blocked or removed from the group), GitLab rejects the scan results from the agent. Vulnerabilities are no longer created or resolved for that cluster. The agent logs create starboard vulnerability api request failed with status: 403 for each vulnerability, followed by Some vulnerabilities failed to create, skipping vulnerability resolution.

Register the agent with a service account

To avoid depending on an individual user account, register the agent with a service account.

Prerequisites:

• You must have permission to create a service account and add it to the project.
• You must have the Maintainer or Owner role for the project.

To register the agent with a service account:

1. Create a service account and add it to the project with the Maintainer role.
2. Create a personal access token for the service account with the api scope.
3. With the service account’s token, register the agent and create an agent token.
4. Install the agent in your cluster with the agent token.

After you create the agent token, you can revoke the service account’s personal access token. OCS uses the service account’s role in the project, not its personal access token.

Configure scanner resource requirements

By default the scanner pod’s default resource requirements are:

requests:
  cpu: 100m
  memory: 100Mi
  ephemeral_storage: 1Gi
limits:
  cpu: 500m
  memory: 500Mi
  ephemeral_storage: 3Gi

You can customize it with a resource_requirements field.

container_scanning:
  resource_requirements:
    requests:
      cpu: '0.2'
      memory: 200Mi
      ephemeral_storage: 2Gi
    limits:
      cpu: '0.7'
      memory: 700Mi
      ephemeral_storage: 4Gi

When using a fractional value for CPU, format the value as a string.

Custom repository for Trivy K8s Wrapper

During a scan, OCS deploys pods using an image from the Trivy K8s Wrapper repository, which transmits the vulnerability report generated by Trivy Kubernetes to OCS.

If your cluster’s firewall restricts access to the Trivy K8s Wrapper repository, you can configure OCS to pull the image from a custom repository. Ensure that the custom repository mirrors the Trivy K8s Wrapper repository for compatibility.

container_scanning:
  trivy_k8s_wrapper_image:
    repository: "your-custom-registry/your-image-path"

Configure scan timeout

By default, the Trivy scan times out after five minutes. The agent itself provides an extra 15 minutes to read the chained configmaps and transmit the vulnerabilities.

To customize the Trivy timeout duration:

• Specify the duration in seconds with the scanner_timeout field.

For example:

container_scanning:
  scanner_timeout: "3600s" # 60 minutes

Configure Trivy report size

By default, the Trivy report is limited to 100 MB, which is sufficient for most scans. However, if you have a lot of workloads, you might need to increase the limit.

To do this:

• Specify the limit in bytes with the report_max_size field.

For example:

container_scanning:
  report_max_size: "300000000" # 300 MB

Configure Trivy Kubernetes resource detection

By default, Trivy looks for the following Kubernetes resource types to discover scannable images:

• Pod
• ReplicaSet
• ReplicationController
• StatefulSet
• DaemonSet
• CronJob
• Job
• Deployment

You can limit the Kubernetes resource types that Trivy discovers, for example to only scan “active” images.

To do this:

• Specify the resource types with the resource_types field:

  yaml

  container_scanning:
    vulnerability_report:
      resource_types:
        - Deployment
        - Pod
        - Job

Configure Trivy report artifact deletion

By default, the GitLab agent for Kubernetes deletes the Trivy report artifact after a scan has completed.

You can configure the agent to preserve the report artifact, so you can view the report in its raw state.

To do this:

• Set delete_report_artifact to false:

  yaml

  container_scanning:
    delete_report_artifact: false

Configure Trivy severity threshold filter

OCS scans for vulnerabilities for all severity levels by default.

To report only vulnerabilities at or above a specific severity level, set the configuration variable severity_threshold to that value. After you set a severity threshold, vulnerabilities below the severity you choose are no longer returned in the Vulnerability Report, API payloads, and other reporting mechanisms.

This allows you to focus on vulnerabilities that meet your organization’s risk tolerance needs.

Supported threshold values are UNKNOWN, LOW, MEDIUM, HIGH, and CRITICAL.

For example, to report vulnerabilities of high and critical severity:

container_scanning:
  severity_threshold: "HIGH"

View cluster vulnerabilities

To view vulnerability information in GitLab:

1. In the top bar, select Search or go to and find the project that contains the agent configuration file.
2. Select Operate > Kubernetes clusters.
3. Select the Agent tab.
4. Select an agent to view the cluster vulnerabilities.

This information can also be found under operational vulnerabilities.

Scanning private images

To scan private images, the scanner relies on the image pull secrets (direct references and from the service account) to pull the image.

Known issues

In GitLab agent for Kubernetes 16.9 and later, operational container scanning:

• Handles Trivy reports of up to 100 MB. For previous releases, this limit is 10 MB.
• Is disabled when the GitLab agent for Kubernetes runs in fips mode.

Troubleshooting

Error running Trivy scan. Container terminated reason: OOMKilled

OCS might fail with an OOM error if there are too many resources to be scanned or if the images being scanned are large.

To resolve this, configure the resource requirement to increase the amount of memory available.

Pod ephemeral local storage usage exceeds the total limit of containers

OCS scans could fail for Kubernetes clusters that have low default ephemeral storage. For example, GKE autopilot sets the default ephemeral storage to 1 GB. This is an issue for OCS when scanning namespaces with large images, as there may not be enough space to store all data necessary for OCS.

To resolve this, configure the resource requirement to increase the amount of ephemeral storage available.

Another message indicative of this issue may be: OCS Scanning pod evicted due to low resources. Please configure higher resource limits.

Error running Trivy scan due to context timeout

OCS might fail to complete a scan if it takes Trivy too long to complete the scan. The default scan timeout is 5 minutes, with an extra 15 minutes for the agent to read the results and transmit the vulnerabilities.

To resolve this, configure the scanner timeout to increase the time available for the scan.

trivy report size limit exceeded

OCS might fail with this error if the generated Trivy report size is larger than the default maximum limit.

To resolve this, configure the max Trivy report size to increase the maximum allowed size of the Trivy report.