GitLab Orbit sizing

Size GitLab Orbit next to your GitLab reference architecture. Find the size of your reference architecture, then add the GitLab Orbit components for that size to your environment.

GitLab Orbit load depends more on the groups that you index, your Gitaly nodes, and your PostgreSQL change rate than on user count. After you choose a size, read Adjust for your workload.

The values are starting points, and they can be more than your workload needs. Monitor the signals in Monitor and scale, then decrease the values.

Choose a size

Use the size that matches your GitLab reference architecture. For fewer than 2,000 users, use the 2,000 users size.

Components

Kubernetes

Node pools

Put the GitLab Orbit pods on their own node pool, and NATS and Siphon on a second node pool. The maximum node counts leave room for the extra pods that a chart upgrade starts.

Machine type examples are for illustration. Other machine types that meet the requirements are supported, including Arm types. Do not use burstable instance types, because their CPU performance is not consistent.

Pods

The following tables give the pod count, and the request and the limit for one pod. The indexers use most of their memory limit during backfill, so their memory request equals the limit. GitLab Orbit sets CPU limits on the code indexer, because the code indexer sets its indexing lanes from them.

Webserver:

SDLC indexer:

Code indexer:

Each indexing lane needs about 2 vCPU and 5 GiB of memory. Keep the CPU limit below the usable vCPU of one node.

Configure the pools

The chart defaults do not match these tables. To configure the pools, set the values in orbit-values.yaml, then upgrade the chart. For example, for 10,000 users:

webserver:
  replicas: 2
  resources:
    requests:
      cpu: 500m
      memory: 2Gi
    limits:
      cpu: "2"
      memory: 6Gi
indexer:
  pools:
    sdlc:
      replicas: 2
      resources:
        requests:
          cpu: 500m
          memory: 24Gi
        limits:
          cpu: "4"
          memory: 24Gi
      engine:
        max_concurrent_workers: 8
        concurrency_groups:
          sdlc: 8
    code:
      replicas: 3
      tmpSizeLimit: 54Gi
      resources:
        requests:
          cpu: "6"
          memory: 26Gi
          ephemeral-storage: 30Gi
        limits:
          cpu: "12"
          memory: 26Gi
          ephemeral-storage: 60Gi
engine:
  handlers:
    code-indexing-task:
      pipeline:
        archive_storage_budget: 4
        archive_tenant_budget: 4

• For the SDLC indexer, set max_concurrent_workers and concurrency_groups.sdlc to the same value.
• Keep tmpSizeLimit below the ephemeral storage limit, because logs and the container layer use the same disk.
• Without autoscaling, set the code indexer replicas to the maximum of the pod range.
• Before you turn on indexing for groups, apply the rules in Limit the archive load on Gitaly to archive_storage_budget and archive_tenant_budget.
• To put the pods on the node pool, set nodeSelector and tolerations for the webserver, the dispatcher, the health check, and each indexer pool.

Autoscale the code indexer

Prerequisites:

• The KEDA operator runs in the cluster.

To autoscale the code indexer on the count of pending code indexing tasks:

1. Add these values to orbit-values.yaml:

   yaml

   healthCheck:
     enabled: true
   indexer:
     pools:
       code:
         keda:
           enabled: true
           minReplicaCount: <code_min_pods>
           maxReplicaCount: <code_max_pods>

   Replace the following values:

   • <code_min_pods>: the minimum of the code indexer pod range in Pods.
   • <code_max_pods>: the maximum of the code indexer pod range. The chart default is 4.

2. Upgrade the chart.

ClickHouse

The sizes are for the ClickHouse instance that holds the GitLab Orbit data lake and graph. If GitLab Orbit shares the ClickHouse instance of GitLab, use these sizes for the shared instance. They are larger than the sizes for GitLab ClickHouse features, because the data lake holds a copy of the GitLab database.

The examples are Arm machine types. x86 machine types of the same size are also supported.

For the high availability setup and its load balancer, see High availability. On ClickHouse Cloud, choose a service with at least the memory of one node in the table.

If you run ClickHouse on VMs:

• Give each node this much disk:

  disk = 2 x (GitLab PostgreSQL database size + 1 MiB x indexed projects)

  The factor of 2 keeps free space for merges.

• Set the memory and external sort settings in Memory and disk. Without them, large sorts fail instead of spilling to disk. The example there is for 32 GiB of memory. For other sizes, set max_memory_usage to about 62% of the memory, and the two external limits to 25%. On a 64 GiB node, use 42949672960 for max_memory_usage, and 17179869184 for the external limits.

• Set a TTL on the system log tables, as in System log retention.

NATS

For fewer than 25,000 users, run NATS as one node. One node has no redundancy. If you lose its volume, Siphon needs a fresh snapshot, and GitLab Orbit can stop with Error: ontology archive vNN is missing.

For 25,000 users or more, run NATS as a 3-node JetStream cluster. For the steps, see Configure a NATS cluster.

Calculate the volume with this formula. Use the larger of the result and the table value.

volume = retention_seconds x peak_changes_per_second x 300 bytes

• retention_seconds is the Siphon max_age_seconds. The example in Install Siphon keeps 15 days, which is 1296000 seconds.

• peak_changes_per_second is the PostgreSQL change rate at peak. To measure it, run this query twice during peak, then divide the difference by the seconds between the two runs:

  sql

  SELECT sum(n_tup_ins + n_tup_upd + n_tup_del) FROM pg_stat_user_tables;

Configure a NATS cluster

Prerequisites:

• Three nodes in the NATS and Siphon node pool.

To run NATS as a 3-node JetStream cluster:

1. Add the cluster settings to the NATS chart values, then upgrade the NATS release:

   yaml

   config:
     cluster:
       enabled: true
       replicas: 3

2. Set nats_config.replicas: 3 in siphon-values.yaml, as in Install Siphon, then upgrade the Siphon release.

The cluster protects the Siphon stream only. GitLab Orbit creates its own streams and key-value buckets, including the schema archive catalog, with one replica. If you lose the node that holds them, GitLab Orbit can stop with Error: ontology archive vNN is missing.

Siphon

Each Siphon deployment runs one pod.

For CPU, give each pod a request of 250m vCPU and a limit of 2 vCPU.

PostgreSQL disk for the replication slot

Siphon reads changes through a PostgreSQL logical replication slot. While Siphon is stopped, the slot holds write-ahead log (WAL) on the GitLab PostgreSQL disk.

Keep enough free disk for the longest Siphon outage that you accept, and at least 24 hours:

free disk = WAL bytes per hour x outage hours

To measure the WAL rate:

1. Run this query, and note the result:

   sql

   SELECT pg_current_wal_lsn();

2. One hour later, run the same query again.

3. Calculate the WAL bytes for the hour:

   sql

   SELECT pg_wal_lsn_diff('<second_lsn>', '<first_lsn>');

   Replace the following values:

   • <first_lsn>: the result of the first query.
   • <second_lsn>: the result of the second query.

To protect the disk, you can set max_slot_wal_keep_size in PostgreSQL to less than the free disk. When a slot exceeds this limit, PostgreSQL invalidates it, and Siphon needs a fresh snapshot.

For more information, see Verify replication.

Adjust for your workload

The first code backfill puts the most load on GitLab Orbit and on Gitaly. Estimate its length, and limit its load on Gitaly before you turn on indexing for groups.

Estimate the backfill time

To estimate the time for the first code backfill, use this formula:

parallel_archives = min(archive slots x Gitaly storages, code indexer pods x indexing lanes per pod)
backfill_hours = indexed projects x 30 seconds / parallel_archives / 3600

For example, 100,000 projects with 4 parallel archives take about 9 days. The formula assumes that the repositories are spread evenly across the storages. If one storage holds most of them, that storage sets the pace. Large repositories take longer.

The smaller term in parallel_archives sets the pace:

• If archive slots x Gitaly storages is smaller, the archive slots set the pace. Add slots only as the rules in Limit the archive load on Gitaly allow. More code indexer pods do not help.
• If code indexer pods x indexing lanes per pod is smaller, add code indexer pods.

Limit the archive load on Gitaly

Gitaly builds an archive of every repository for the code backfill. The load goes to the Gitaly node that holds the repositories.

Before you turn on indexing for groups:

1. Turn on the GitLab rate limits.
2. Balance your repositories across your Gitaly nodes as evenly as you can.
3. Find the vCPU count of the Gitaly node that holds the most repositories. For Gitaly Cluster (Praefect), use the smallest node in the cluster.
4. Calculate the archive slots: 1 for each 4 vCPU of that node, up to 16. If the node has more than one storage, divide by the number of storages on the node.
5. Set archive_storage_budget and archive_tenant_budget to the smaller of this value and the archive slots in Pods, then upgrade the chart.

archive_storage_budget limits the archive downloads in flight for each Gitaly storage, across all code indexer pods. archive_tenant_budget is the same limit for projects whose Gitaly storage GitLab Orbit cannot find.

During the backfill:

1. Turn on indexing for a few groups at a time.
2. Monitor the CPU of the Gitaly node with the most repositories. If its CPU is more than 70%, or its 1-minute load average is more than its vCPU count, pause code indexing until the load drops. To pause it, set indexer.pools.code.replicas: 0 and upgrade the chart, or with autoscaling, set the autoscaling.keda.sh/paused-replicas: "0" annotation on the code indexer ScaledObject.
3. Monitor the p95 of grpc_server_handling_seconds on Gitaly for GetArchive, which is the backfill load. PackObjectsHookWithSidechannel, PostUploadPackWithSidechannel, and FindCommit show the effect on your users. Gitaly exports these histograms only when you set grpc_latency_buckets in its Prometheus configuration.

Monitor and scale

After the first backfill, monitor these signals and adjust the values. For the metrics and dashboards, see Monitor GitLab Orbit.

Related topics

• GitLab reference architectures
• GitLab rate limits
• Set up GitLab Orbit
• ClickHouse for GitLab Orbit
• Monitor GitLab Orbit