Slot-based cgroup support

Slot-based cgroup support improves resource isolation and management when you use GitLab Runner with autoscaling. Slot-based cgroups automatically assign jobs to specific control groups (cgroups) based on the slot number allocated by the autoscaler.

Benefits

Better resource isolation: Prevents resource interference between concurrent jobs on the same instance.
Easier monitoring: Per-slot resource usage can be tracked independently.
Improved debugging: Cgroup-based metrics help identify resource-hungry jobs.
Fine-grained control: Set resource limits per slot for predictable performance.

Supported executors

Slot-based cgroups work with autoscaling executors that use taskscaler for slot management:

Prerequisites

Linux host with cgroup v2 support
Root access for initial cgroup hierarchy setup
GitLab Runner with autoscaler functionality
Taskscaler for slot assignment (automatically provided by autoscaler)

Configuration

To enable slot-based cgroup support, add the following to your config.toml.

For Docker with `systemd` cgroup driver

If Docker is using the systemd cgroup driver (most common), use the systemd slice format:

[[runners]]
  name = "my-autoscaler-runner"
  executor = "docker-autoscaler"
  use_slot_cgroups = true
  slot_cgroup_template = "runner-slot-${slot}.slice"

  [runners.autoscaler]
    capacity_per_instance = 4

For Docker with `cgroupfs` driver

If Docker is using the cgroupfs driver, use the raw cgroup path format:

[[runners]]
  name = "my-autoscaler-runner"
  executor = "docker-autoscaler"
  use_slot_cgroups = true
  slot_cgroup_template = "gitlab-runner/slot-${slot}"

  [runners.autoscaler]
    capacity_per_instance = 4

Configuration options

Setting	Description	Default
`use_slot_cgroups`	Enable slot-based cgroup assignment	`false`
`slot_cgroup_template`	Template for cgroup paths. Use `${slot}` as placeholder. Format depends on Docker’s cgroup driver (systemd: `runner-slot-${slot}.slice`, cgroupfs: `gitlab-runner/slot-${slot}`)	`"gitlab-runner/slot-${slot}"`

Templates use bash-style variable expansion with ${slot} as the placeholder for the slot number. For example:

With systemd driver: runner-slot-${slot}.slice becomes runner-slot-5.slice for slot 5
With cgroupfs driver: gitlab-runner/slot-${slot} becomes gitlab-runner/slot-5 for slot 5

Check your Docker cgroup driver with: docker info | grep "Cgroup Driver"

Docker-specific configuration

When using the Docker Autoscaler executor, you can specify a separate template for service containers:

[[runners]]
  executor = "docker-autoscaler"
  use_slot_cgroups = true
  slot_cgroup_template = "runner-slot-${slot}.slice"

  [runners.docker]
    service_slot_cgroup_template = "runner-slot-${slot}.slice"

Setting	Description	Default
`service_slot_cgroup_template`	Template for service container cgroup paths. Must match Docker’s cgroup driver format	Same as `slot_cgroup_template`

Environment setup

Before enabling slot-based cgroups, prepare the cgroup hierarchy on your runner hosts.

Setup script for systemd cgroup driver

If Docker is using the systemd cgroup driver (check with docker info | grep "Cgroup Driver"), you must create systemd slices instead of raw cgroup directories.

Create a setup script (gitlab-runner-systemd-slice-setup.sh):

#!/bin/bash
# gitlab-runner-systemd-slice-setup.sh
# Script to set up systemd slices for GitLab Runner slot-based cgroups
# This example configures 4 slots on an 8-core machine, with each slot pinned to 2 CPUs

set -e

MAX_SLOTS=4  # Adjust based on your capacity_per_instance configuration

# CPU pinning configuration (2 CPUs per slot on an 8-core machine)
# Format: comma-separated CPU list for systemd AllowedCPUs
declare -a CPU_ASSIGNMENTS=(
    "0,1"    # Slot 0: CPUs 0 and 1
    "2,3"    # Slot 1: CPUs 2 and 3
    "4,5"    # Slot 2: CPUs 4 and 5
    "6,7"    # Slot 3: CPUs 6 and 7
)

# Check if running as root
if [[ $EUID -ne 0 ]]; then
   echo "This script must be run as root for systemd slice setup"
   exit 1
fi

# Verify systemd is available
if ! command -v systemctl &> /dev/null; then
    echo "Error: systemctl not found. This script requires systemd."
    exit 1
fi

echo "Setting up systemd slices for GitLab Runner"
echo "Configuration: $MAX_SLOTS slots on an 8-core machine (2 CPUs per slot)"

for ((slot=0; slot<MAX_SLOTS; slot++)); do
    slice_name="runner-slot-${slot}.slice"
    echo "Creating systemd slice: $slice_name (CPUs: ${CPU_ASSIGNMENTS[$slot]})"

    # Create systemd slice configuration
    cat > "/etc/systemd/system/$slice_name" <<EOF
[Unit]
Description=GitLab Runner Slot $slot
Before=slices.target

[Slice]
CPUAccounting=true
MemoryAccounting=true
AllowedCPUs=${CPU_ASSIGNMENTS[$slot]}
EOF

done

# Reload systemd to pick up new slice units
systemctl daemon-reload

# Start all slices
for ((slot=0; slot<MAX_SLOTS; slot++)); do
    slice_name="runner-slot-${slot}.slice"
    systemctl start "$slice_name"
done

echo ""
echo "Systemd slices created successfully!"
echo ""
echo "Verifying slices:"
for ((slot=0; slot<MAX_SLOTS; slot++)); do
    slice_name="runner-slot-${slot}.slice"
    status=$(systemctl is-active "$slice_name" 2>/dev/null || echo "inactive")
    echo "  $slice_name: $status"
done

echo ""
echo "To verify CPU assignments, check:"
echo "  systemctl show runner-slot-0.slice | grep AllowedCPUs"

Run the setup script:

chmod +x gitlab-runner-systemd-slice-setup.sh
sudo ./gitlab-runner-systemd-slice-setup.sh

Setup script for `cgroupfs` driver (alternative)

If Docker is using the cgroupfs driver instead of systemd, use this alternative script that creates raw cgroup directories:

#!/bin/bash
# gitlab-runner-cgroup-setup.sh
# Script to set up cgroup v2 hierarchy for GitLab Runner slot-based cgroups
# This example configures 4 slots on an 8-core machine, with each slot pinned to 2 CPUs
# Use this script only if Docker is using the cgroupfs driver (not systemd)

set -e

CGROUP_ROOT="/sys/fs/cgroup"
RUNNER_CGROUP="gitlab-runner"
MAX_SLOTS=4  # Adjust based on your capacity_per_instance configuration

# CPU pinning configuration (2 CPUs per slot on an 8-core machine)
# Format: "cpu_list" - adjust based on your CPU topology
declare -a CPU_ASSIGNMENTS=(
    "0-1"    # Slot 0: CPUs 0 and 1
    "2-3"    # Slot 1: CPUs 2 and 3
    "4-5"    # Slot 2: CPUs 4 and 5
    "6-7"    # Slot 3: CPUs 6 and 7
)

# Check if running as root
if [[ $EUID -ne 0 ]]; then
   echo "This script must be run as root for cgroup setup"
   exit 1
fi

# Verify cgroup v2 is available
if [[ ! -f "$CGROUP_ROOT/cgroup.controllers" ]]; then
    echo "Error: cgroup v2 not detected. This script requires cgroup v2."
    exit 1
fi

echo "Setting up cgroup v2 hierarchy for GitLab Runner"
echo "Configuration: $MAX_SLOTS slots on an 8-core machine (2 CPUs per slot)"

# Create base runner cgroup
mkdir -p "$CGROUP_ROOT/$RUNNER_CGROUP"

# Enable controllers if available
if [[ -f "$CGROUP_ROOT/cgroup.controllers" ]]; then
    echo "+memory +cpu +cpuset" > "$CGROUP_ROOT/cgroup.subtree_control" 2>/dev/null || true
fi

# Create slot-specific cgroups
for ((slot=0; slot<MAX_SLOTS; slot++)); do
    slot_path="$CGROUP_ROOT/$RUNNER_CGROUP/slot-$slot"
    echo "Creating cgroup for slot $slot (CPUs: ${CPU_ASSIGNMENTS[$slot]})"

    mkdir -p "$slot_path"

    # Enable controllers for this slot
    if [[ -f "$CGROUP_ROOT/$RUNNER_CGROUP/cgroup.controllers" ]]; then
        echo "+memory +cpu +cpuset" > "$CGROUP_ROOT/$RUNNER_CGROUP/cgroup.subtree_control" 2>/dev/null || true
    fi

    # Pin slot to specific CPUs
    echo "${CPU_ASSIGNMENTS[$slot]}" > "$slot_path/cpuset.cpus"

    # Set memory nodes (usually 0 for single NUMA node systems)
    echo "0" > "$slot_path/cpuset.mems"

    # Set permissions for GitLab Runner user
    chown -R gitlab-runner:gitlab-runner "$slot_path" 2>/dev/null || true
done

echo "Cgroup setup complete!"

# Verify setup
echo ""
echo "Verifying cgroup setup:"
for ((slot=0; slot<MAX_SLOTS; slot++)); do
    slot_path="$CGROUP_ROOT/$RUNNER_CGROUP/slot-$slot"
    cpus=$(cat "$slot_path/cpuset.cpus")
    echo "  Slot $slot: CPUs $cpus"
done

Run the setup script:

chmod +x gitlab-runner-cgroup-setup.sh
sudo ./gitlab-runner-cgroup-setup.sh

How it works

Docker Autoscaler executor

The Docker Autoscaler executor automatically applies slot-based cgroup paths to Docker containers using the --cgroup-parent flag. Both build containers and service containers are assigned to their slot-specific cgroups without requiring any changes to your job scripts.

Instance executor

The Instance executor provides the GITLAB_RUNNER_SLOT_CGROUP environment variable to jobs. You can use this variable in your job scripts to run processes under the slot-specific cgroup.

Using `systemd-run`

job:
  script:
    - echo "Running in cgroup $GITLAB_RUNNER_SLOT_CGROUP"
    - systemd-run --scope --slice=$GITLAB_RUNNER_SLOT_CGROUP ./my-process

Using `cgexec`

job:
  script:
    - echo "Running in cgroup $GITLAB_RUNNER_SLOT_CGROUP"
    - cgexec -g cpu,memory:$GITLAB_RUNNER_SLOT_CGROUP ./my-process

Setting cgroup limits

You can set resource limits on the cgroup before running your job processes:

job:
  script:
    - echo "Configuring cgroup limits"
    - echo "100M" > /sys/fs/cgroup/$GITLAB_RUNNER_SLOT_CGROUP/memory.max
    - echo "50000" > /sys/fs/cgroup/$GITLAB_RUNNER_SLOT_CGROUP/cpu.max
    - ./my-process

Troubleshooting

Containers fail to start with cgroup errors

Check that the cgroup paths exist under /sys/fs/cgroup/:
```
ls -la /sys/fs/cgroup/gitlab-runner/
```
Ensure the GitLab Runner user has write access to the cgroup directories:
```
ls -la /sys/fs/cgroup/gitlab-runner/slot-0/
```
Confirm slot_cgroup_template uses the correct format with ${slot} placeholder:
Check GitLab Runner logs for specific cgroup creation errors:

Test manually:

For Docker Autoscaler executor:

docker run --rm --cgroup-parent=gitlab-runner/slot-0 alpine echo "test"

For Instance executor:

job:
  script:
    - echo "Slot cgroup: $GITLAB_RUNNER_SLOT_CGROUP"

Jobs use the same cgroup

If you see a warning in the logs about templates not containing ${slot} placeholder:

level=warning msg="Slot cgroup template does not contain ${slot} placeholder.
All jobs will use the same cgroup, defeating the purpose of slot-based isolation."

This means your slot_cgroup_template is missing the ${slot} variable. Update your configuration to include the placeholder:

[[runners]]
  slot_cgroup_template = "gitlab-runner/slot-${slot}"

Cgroup v2 not available

If the setup script reports that cgroup v2 is not detected, you might need to enable it on your system. Check your Linux distribution’s documentation for enabling cgroup v2. Modern distributions typically enable it by default.