Pipeline security

Secrets Management

Secrets management is the systems that developers use to securely store sensitive data in a secure environment with strict access controls. A secret is a sensitive credential that should be kept confidential. Examples of a secret include:

• Passwords
• SSH keys
• Access tokens
• Any other types of credentials where exposure would be harmful to an organization

Secrets storage

Secrets management providers

Secrets that are the most sensitive and under the strictest policies should be stored in a secrets manager. When using a secrets manager solution, secrets are stored outside of the GitLab instance. There are a number of providers in this space, including HashiCorp’s Vault, Azure Key Vault, and Google Cloud Secret Manager.

You can use the GitLab native integrations for certain external secret management providers to retrieve those secrets in CI/CD pipelines when they are needed.

CI/CD variables

CI/CD Variables are a convenient way to store and reuse data in a CI/CD pipeline, but variables are less secure than secrets management providers. Variable values:

• Are stored in the GitLab project, group, or instance settings. Users with access to the settings have access to variables values that are not hidden.
• Can be overridden, making it hard to determine which value was used.
• Can be exposed by accidental pipeline misconfiguration.

Information suitable for storage in a variable should be data that can be exposed without risk of exploitation (non-sensitive).

Sensitive data should be stored in a secrets management solution. If there is low sensitivity data that you want to store in a CI/CD variable, be sure to always:

• Mask the variables or Mask and hide the variable.
• Protect the variables when possible.
• Hide the variables when possible.

Pipeline Integrity

The key security principals of ensuring pipeline integrity include:

• Supply Chain Security: Assets should be obtained from trusted sources and their integrity verified.
• Reproducibility: Pipelines should produce consistent results when using the same inputs.
• Auditability: All pipeline dependencies should be traceable and their provenance verifiable.
• Version Control: Changes to pipeline dependencies should be tracked and controlled.

Docker images

Always use SHA digests for Docker images to ensure client-side integrity verification. For example:

• Node:
  • Use: image: node@sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef
  • Instead of: image: node:latest
• Python:
  • Use image: python@sha256:9876543210abcdef9876543210abcdef9876543210abcdef9876543210abcdef
  • Instead of: image: python:3.9

You can find the SHA digest of an image with a specific tag using:

docker pull node:18.17.1
docker images --digests node:18.17.1

Prefer to pull from container registries that protect image integrity:

• Use protected container repositories to restrict which users can make changes to container images in your container repository.
• Use protected tags to control who can push and delete container tags.

When possible, avoid using variables in container references as they can be modified to point to malicious images. For example:

• Prefer:
  • image: my-registry.example.com/node:18.17.1
• Instead of:
  • image: ${CUSTOM_REGISTRY}/node:latest
  • image: node:${VERSION}

Package dependencies

You should lock down package dependencies in your jobs. Use exact versions, defined in lock files:

• npm:
  • Use: npm ci
  • Instead of: npm install
• yarn:
  • Use: yarn install --frozen-lockfile
  • Instead of: yarn install
• Python:
  • Use:
    • pip install -r requirements.txt --require-hashes
    • pip install -r requirements.lock
  • Instead of: pip install -r requirements.txt
• Go:
  • Use exact versions from go.sum:
    • go mod verify
    • go mod download
  • Instead of: go get ./...

For example, in a CI/CD job:

javascript-job:
  script:
    - npm ci

Shell commands and scripts

When installing tools in a job, always specify and verify exact versions. For example, in a Terraform job:

terraform_job:
  script:
    # Download specific version
    - |
      wget https://releases.hashicorp.com/terraform/1.5.7/terraform_1.5.7_linux_amd64.zip
      # IMPORTANT: Always verify checksums
      echo "c0ed7bc32ee52ae255af9982c8c88a7a4c610485cf1d55feeb037eab75fa082c terraform_1.5.7_linux_amd64.zip" | sha256sum -c
      unzip terraform_1.5.7_linux_amd64.zip
      mv terraform /usr/local/bin/
    # Use the installed version
    - terraform init
    - terraform plan

Version management tools

Use version managers when possible:

node_build:
  script:
    # Use nvm to install and use a specific Node version
    - |
      nvm install 16.15.1
      nvm use 16.15.1
    - node --version  # Verify version
    - npm ci
    - npm run build

Included configurations

When using the include keyword to add configuration or CI/CD components to your pipeline, use a specific ref when possible. For example:

include:
  - project: 'my-group/my-project'
    ref: 8b0c8b318857c8211c15c6643b0894345a238c4e  # Pin to a specific commit
    file: '/templates/build.yml'
  - project: 'my-group/security'
    ref: v2.1.0                                    # Pin to a protected tag
    file: '/templates/scan.yml'
  - component: 'my-group/security-scans'           # Pin to a specific version
    version: '1.2.3'

Avoid versionless includes:

include:
  - project: 'my-group/my-project'                   # Unsafe
    file: '/templates/build.yml'
  - component: 'my-group/security-scans'             # Unsafe
  - remote: 'https://example.com/security-scan.yml'  # Unsafe

Instead of including remote files, download the file and save it in your repository. Then you can include the local copy:

include:
  - local: '/ci/security-scan.yml'  # Verified and stored in the repository

Related topics

1. CIS Docker Benchmarks
2. Google Cloud: Design secure deployment pipelines