Kubernetes integration tests

Kubernetes integration tests run in GitLab Runner’s CI/CD pipeline. These tests verify that the GitLab Runner works correctly with Kubernetes clusters. These tests run against a dedicated Kubernetes cluster managed by the runner-Kubernetes-infra repository.

Test infrastructure

Runner Kubernetes infrastructure repository

The test infrastructure is hosted at:

The infrastructure uses a blue-green deployment model with two separate clusters to enable zero-downtime updates.

Cluster configuration

For detailed cluster configuration including node pools, resource limits, and autoscaling settings, see the cluster configuration section in the infrastructure repository.

Pipeline structure

Test pipeline stages

The integration tests run through the following GitLab CI/CD stages:

  1. Provision integration Kubernetes (provision integration kubernetes):

    • Provisions test-specific RBAC resources
    • Creates service account k8s-runner-integration-tests-runner-$CI_PIPELINE_ID
    • Executes mage k8s:provisionIntegrationKubernetes $CI_PIPELINE_ID

  2. Integration test jobs (parallel execution):

    • integration kubernetes: Standard integration tests
    • integration kubernetes exec legacy: Tests with legacy execution strategy
    • integration kubernetes attach: Tests with attach execution strategy

  3. Cleanup (destroy integration kubernetes):

    • Destroys test-specific resources
    • Executes mage k8s:destroyIntegrationKubernetes $CI_PIPELINE_ID

Pipeline configuration

The pipeline is defined in .gitlab/ci/test-kubernetes-integration.gitlab-ci.yml:

.integration kubernetes:
  extends:
    - .rules:merge_request_pipelines:no_docs:no-community-mr
  tags:
    - $KUBERNETES_RUNNER_INTEGRATION_TAG
  stage: test kubernetes integration
  variables:
    KUBERNETES_SERVICE_ACCOUNT_OVERWRITE: "k8s-runner-integration-tests-runner-$CI_PIPELINE_ID"

Test execution

Integration tests are executed using gotestsum:

gotestsum --format=testname --format-hide-empty-pkg --rerun-fails=3 \
  --hide-summary=output --packages=gitlab.com/gitlab-org/gitlab-runner/executors/kubernetes \
  --junitfile=junit_report.xml --junitfile-hide-empty-pkg -- \
  -timeout=10m -parallel=20 $EXTRA_GO_TEST_FLAGS \
  -tags=integration,kubernetes ./executors/kubernetes/...

Key parameters:

  • Timeout: 10 minutes per test
  • Parallel execution: Up to 20 tests simultaneously
  • Retry logic: Failing tests are retried up to 3 times
  • Build tags: integration,kubernetes

Test categories

Standard integration tests

  • Job: integration kubernetes
  • Purpose: Main integration test suite
  • Feature flags: Uses default feature flag configuration
  • Filter: Excludes feature flag-specific tests with -skip=TestRunIntegrationTestsWithFeatureFlag

Legacy execution strategy tests

  • Job: integration kubernetes exec legacy
  • Feature flag: FF_USE_LEGACY_KUBERNETES_EXECUTION_STRATEGY=true
  • Filter: Only runs TestRunIntegrationTestsWithFeatureFlag
  • Purpose: Validates backward compatibility

Attach strategy tests

  • Job: integration kubernetes attach
  • Feature flag: FF_USE_LEGACY_KUBERNETES_EXECUTION_STRATEGY=false
  • Filter: Only runs TestRunIntegrationTestsWithFeatureFlag
  • Purpose: Tests the newer attach-based execution strategy

RBAC and permissions

Dynamic permission provisioning

The provisioning system (mage k8s:provisionIntegrationKubernetes) analyzes the codebase to generate the minimal required RBAC permissions:

  1. Code analysis: Scans /executors/kubernetes/ for Kubernetes API calls
  2. Permission generation: Creates the role YAML with only required permissions
  3. Resource creation: Applies the generated RBAC to the k8s-runner-integration-tests namespace

This system ensures tests use the same permissions as the code under test.

Test-specific service accounts

Each pipeline creates unique resources:

  • Service account: k8s-runner-integration-tests-runner-$CI_PIPELINE_ID
  • Role: Generated based on code analysis
  • Role binding: Links service account to generated role

Administrative permissions

Integration tests also use administrative RBAC for test management:

  • Service account: integration-tests-admin
  • Purpose: Create/delete test resources, observe cluster state
  • Scope: Additional permissions beyond normal runner operations

Test implementation

Test environment

Tests run with the following environment variables:

  • KUBERNETES_SERVICE_ACCOUNT_OVERWRITE: Pipeline-specific service account
  • Feature flag variables (for feature flag tests)
  • Cluster connection details (managed by infrastructure)

Resource management

Automated cleanup

The infrastructure includes automated cleanup mechanisms. For detailed information about CronJobs, scheduling, and configuration, see the operational automation section in the infrastructure repository.

Resource isolation

Tests use resource groups to prevent conflicts:

  • "$CI_COMMIT_REF_SLUG-k8s-integration"
  • "$CI_COMMIT_REF_SLUG-k8s-integration-exec-legacy"
  • "$CI_COMMIT_REF_SLUG-k8s-integration-attach"

Monitoring and observability

Metrics and logging

The test infrastructure includes comprehensive monitoring and logging. For information on accessing Grafana, Prometheus dashboards, log aggregation with Loki, and the available make commands, see the metrics and log collection sections in the infrastructure repository.

Troubleshooting

Common issues

  • Test timeouts:

    • Check cluster resource availability.
    • Verify worker pool scaling (0-6 nodes).
    • Review test parallelism settings.

  • RBAC permissions:

    • Ensure provisioning job succeeded.
    • Verify service account creation.
    • Check generated Role matches code requirements.

  • Resource conflicts:

    • Check resource group isolation.
    • Verify cleanup job execution.
    • Review pipeline-specific naming.

Debugging steps

  1. Check the infrastructure status. For more information about the make commands and infrastructure management, see blue-green deployment.

  2. Review test logs:

    • Check pipeline job logs for specific failures.
    • Use Grafana dashboard for aggregated logs. For more information, see log collection.
    • Review gotestsum output for test-specific issues.

  3. Validate RBAC:

    kubectl get sa,role,rolebinding -n k8s-runner-integration-tests
kubectl describe role k8s-runner-integration-tests-runner-$CI_PIPELINE_ID -n k8s-runner-integration-tests

Running tests locally

Integration tests are designed to run in the CI/CD environment with the dedicated infrastructure. Local execution requires:

  1. Access to the GKE cluster.
  2. Appropriate RBAC permissions.
  3. Environment variables that match the CI/CD configuration.

For local development, use unit tests or a local Kubernetes cluster (kind/minikube) with appropriate setup.