• Requirements
• Design Choices
• Configuration
• Installation command line options
• Chart configuration examples
  • resources
  • extraEnv
  • extraEnvFrom
  • extraVolumes
  • extraVolumeMounts
  • image.pullSecrets
  • serviceAccount
  • tolerations
  • annotations
• Using the Community Edition of this chart
• External Services
  • Redis
  • PostgreSQL
  • Gitaly
• Metrics
• Chart-wide defaults
• Per-pod Settings
  • queues
  • Example pod entry
  • Full example of Sidekiq configuration
• Configuring the networkpolicy
  • Example Network Policy
• Configuring KEDA

Using the GitLab-Sidekiq chart

The sidekiq sub-chart provides configurable deployment of Sidekiq workers, explicitly designed to provide separation of queues across multiple Deployments with individual scalability and configuration.

While this chart provides a default pods: declaration, if you provide an empty definition, you will have no workers.

Requirements

This chart depends on access to Redis, PostgreSQL, and Gitaly services, either as part of the complete GitLab chart or provided as external services reachable from the Kubernetes cluster this chart is deployed onto.

Design Choices

This chart creates multiple Deployments and associated ConfigMaps. It was decided that it would be clearer to make use of ConfigMap behaviours instead of using environment attributes or additional arguments to the command for the containers, in order to avoid any concerns about command length. This choice results in a large number of ConfigMaps, but provides very clear definitions of what each pod should be doing.

Configuration

The sidekiq chart is configured in three parts: chart-wide external services, chart-wide defaults, and per-pod definitions.

Installation command line options

The table below contains all the possible charts configurations that can be supplied to the helm install command using the --set flags:

Chart configuration examples

resources

resources allows you to configure the minimum and maximum amount of resources (memory and CPU) a Sidekiq pod can consume.

Sidekiq pod workloads vary greatly between deployments. Generally speaking, it is understood that each Sidekiq process consumes approximately 1 vCPU and 2 GB of memory. Vertical scaling should generally align to this 1:2 ratio of vCPU:Memory.

Below is an example use of resources:

resources:
  limits:
    memory: 5G
  requests:
    memory: 2G
    cpu: 900m

extraEnv

extraEnv allows you to expose additional environment variables in the dependencies container.

Below is an example use of extraEnv:

extraEnv:
  SOME_KEY: some_value
  SOME_OTHER_KEY: some_other_value

When the container is started, you can confirm that the environment variables are exposed:

env | grep SOME
SOME_KEY=some_value
SOME_OTHER_KEY=some_other_value

You can also set extraEnv for a specific pod:

extraEnv:
  SOME_KEY: some_value
  SOME_OTHER_KEY: some_other_value
pods:
  - name: mailers
    queues: mailers
    extraEnv:
      SOME_POD_KEY: some_pod_value
  - name: catchall

This will set SOME_POD_KEY only for application containers in the mailers pod. Pod-level extraEnv settings are not added to init containers.

extraEnvFrom

extraEnvFrom allows you to expose additional environment variables from other data sources in all containers in the pods. Subsequent variables can be overridden per Sidekiq pod.

Below is an example use of extraEnvFrom:

extraEnvFrom:
  MY_NODE_NAME:
    fieldRef:
      fieldPath: spec.nodeName
  MY_CPU_REQUEST:
    resourceFieldRef:
      containerName: test-container
      resource: requests.cpu
  SECRET_THING:
    secretKeyRef:
      name: special-secret
      key: special_token
      # optional: boolean
pods:
  - name: immediate
    extraEnvFrom:
      CONFIG_STRING:
        configMapKeyRef:
          name: useful-config
          key: some-string
          # optional: boolean

extraVolumes

extraVolumes allows you to configure extra volumes chart-wide.

Below is an example use of extraVolumes:

extraVolumes: |
  - name: example-volume
    persistentVolumeClaim:
      claimName: example-pvc

extraVolumeMounts

extraVolumeMounts allows you to configure extra volumeMounts on all containers chart-wide.

Below is an example use of extraVolumeMounts:

extraVolumeMounts: |
  - name: example-volume-mount
    mountPath: /etc/example

image.pullSecrets

pullSecrets allows you to authenticate to a private registry to pull images for a pod.

Additional details about private registries and their authentication methods can be found in the Kubernetes documentation.

Below is an example use of pullSecrets:

image:
  repository: my.sidekiq.repository
  pullPolicy: Always
  pullSecrets:
  - name: my-secret-name
  - name: my-secondary-secret-name

serviceAccount

This section controls if a ServiceAccount should be created and if the default access token should be mounted in pods.

tolerations

tolerations allow you schedule pods on tainted worker nodes

Below is an example use of tolerations:

tolerations:
- key: "node_label"
  operator: "Equal"
  value: "true"
  effect: "NoSchedule"
- key: "node_label"
  operator: "Equal"
  value: "true"
  effect: "NoExecute"

annotations

annotations allows you to add annotations to the Sidekiq pods.

Below is an example use of annotations:

annotations:
  kubernetes.io/example-annotation: annotation-value

Using the Community Edition of this chart

By default, the Helm charts use the Enterprise Edition of GitLab. If desired, you can use the Community Edition instead. Learn more about the differences between the two.

In order to use the Community Edition, set image.repository to registry.gitlab.com/gitlab-org/build/cng/gitlab-sidekiq-ce.

External Services

This chart should be attached to the same Redis, PostgreSQL, and Gitaly instances as the Webservice chart. The values of external services will be populated into a ConfigMap that is shared across all Sidekiq pods.

Redis

redis:
  host: rank-racoon-redis
  port: 6379
  sentinels:
    - host: sentinel1.example.com
      port: 26379
  password:
    secret: gitlab-redis
    key: redis-password

PostgreSQL

psql:
  host: rank-racoon-psql
  serviceName: pgbouncer
  port: 5432
  database: gitlabhq_production
  username: gitlab
  preparedStatements: false
  password:
    secret: gitlab-postgres
    key: psql-password

Gitaly

gitaly:
  internal:
    names:
      - default
      - default2
  external:
    - name: node1
      hostname: node1.example.com
      port: 8079
  authToken:
    secret: gitaly-secret
    key: token

Metrics

By default, a Prometheus metrics exporter is enabled per pod. Metrics are only available when GitLab Prometheus metrics are enabled in the Admin area. The exporter exposes a /metrics endpoint on port 3807. When metrics are enabled, annotations are added to each pod allowing a Prometheus server to discover and scrape the exposed metrics.

Chart-wide defaults

The following values will be used chart-wide, in the event that a value is not presented on a per-pod basis.

Per-pod Settings

The pods declaration provides for the declaration of all attributes for a worker pod. These will be templated to Deployments, with individual ConfigMaps for their Sidekiq instances.

queues

The queues value is a string containing a comma-separated list of queues to be processed. By default, it is not set, meaning that all queues will be processed.

The string should not contain spaces: merge,post_receive,process_commit will work, but merge, post_receive, process_commit will not.

Any queue to which jobs are added but are not represented as a part of at least one pod item will not be processed. For a complete list of all queues, see these files in the GitLab source:

1. app/workers/all_queues.yml
2. ee/app/workers/all_queues.yml

In addition to configuring gitlab.sidekiq.pods[].queues, you must also configure global.appConfig.sidekiq.routingRules. For more information, see Sidekiq routing rules settings.

Example pod entry

pods:
  - name: immediate
    concurrency: 10
    minReplicas: 2  # defaults to inherited value
    maxReplicas: 10 # defaults to inherited value
    maxUnavailable: 5 # defaults to inherited value
    queues: merge,post_receive,process_commit
    extraVolumeMounts: |
      - name: example-volume-mount
        mountPath: /etc/example
    extraVolumes: |
      - name: example-volume
        persistentVolumeClaim:
          claimName: example-pvc
    resources:
      limits:
        cpu: 800m
        memory: 2Gi
    hpa:
      cpu:
        targetType: Value
        targetAverageValue: 350m

Full example of Sidekiq configuration

The following is a full example of Sidekiq configuration using a separate Sidekiq pod for import-related jobs, a Sidekiq pod for export-related jobs using a separate Redis instance and another pod for everything else.

...
global:
  appConfig:
    sidekiq:
      routingRules:
      - ["feature_category=importers", "import"]
      - ["feature_category=exporters", "export", "queues_shard_extra_shard"]
      - ["*", "default"]
  redis:
    redisYmlOverride:
      queues_shard_extra_shard: ...
...
gitlab:
  sidekiq:
    pods:
    - name: import
      queues: import
    - name: export
      queues: export
      extraEnv:
        SIDEKIQ_SHARD_NAME: queues_shard_extra_shard # to match key in global.redis.redisYmlOverride
    - name: default
...

Configuring the networkpolicy

This section controls the NetworkPolicy. This configuration is optional and is used to limit Egress and Ingress of the Pods to specific endpoints.

Example Network Policy

The Sidekiq service requires Ingress connections for only the Prometheus exporter if enabled, and normally requires Egress connections to various places. This examples adds the following network policy:

• All Ingress requests from the network on TCP 10.0.0.0/8 port 3807 are allowed for metrics exporting
• All Egress requests to the network on UDP 10.0.0.0/8 port 53 are allowed for DNS
• All Egress requests to the network on TCP 10.0.0.0/8 port 5432 are allowed for PostgreSQL
• All Egress requests to the network on TCP 10.0.0.0/8 port 6379 are allowed for Redis
• Other Egress requests to the local network on 10.0.0.0/8 are restricted
• Egress requests outside of the 10.0.0.0/8 are allowed

Note the example provided is only an example and may not be complete

Note that the Sidekiq service requires outbound connectivity to the public internet for images on external object storage

networkpolicy:
  enabled: true
  ingress:
    enabled: true
    rules:
      - from:
        - ipBlock:
            cidr: 10.0.0.0/8
        ports:
        - port: 3807
  egress:
    enabled: true
    rules:
      - to:
        - ipBlock:
            cidr: 10.0.0.0/8
        ports:
        - port: 53
          protocol: UDP
      - to:
        - ipBlock:
            cidr: 10.0.0.0/8
        ports:
        - port: 5432
          protocol: TCP
      - to:
        - ipBlock:
            cidr: 10.0.0.0/8
        ports:
        - port: 6379
          protocol: TCP
      - to:
        - ipBlock:
            cidr: 0.0.0.0/0
            except:
            - 10.0.0.0/8

Configuring KEDA

This keda section enables the installation of KEDA ScaledObjects instead of regular HorizontalPodAutoscalers. This configuration is optional and can be used when there is a need for autoscaling based on custom or external metrics.

Most settings default to the values set in the hpa section where applicable.

If the following are true, CPU and memory triggers are added automatically based on the CPU and memory thresholds set in the hpa section:

• triggers is not set.
• The corresponding request.cpu.request or request.memory.request setting is also set to a non-zero value.

If no triggers are set, the ScaledObject is not created.

Refer to the KEDA documentation for more details about those settings.