- Dedicated Queues
- Queue Namespaces
- Latency Sensitive Jobs
- Jobs with External Dependencies
- CPU-bound and Memory-bound Workers
- Declaring a Job as CPU-bound
- Determining whether a worker is CPU-bound
- Feature Categorization
- Sidekiq Compatibility across Updates
This document outlines various guidelines that should be followed when adding or modifying Sidekiq workers.
All workers should include
ApplicationWorker instead of
which adds some convenience methods and automatically sets the queue based on
the worker’s name.
All workers should use their own queue, which is automatically set based on the
worker class name. For a worker named
ProcessSomethingWorker, the queue name
process_something. If you’re not sure what queue a worker uses,
you can find it using
SomeWorker.queue. There is almost never a reason to
manually override the queue name using
sidekiq_options queue: :some_queue.
You must always add any new queues to
otherwise your worker will not run.
While different workers cannot share a queue, they can share a queue namespace.
Defining a queue namespace for a worker makes it possible to start a Sidekiq
process that automatically handles jobs for all workers in that namespace,
without needing to explicitly list all their queue names. If, for example, all
workers that are managed by
sidekiq-cron use the
cronjob queue namespace, we
can spin up a Sidekiq process specifically for these kinds of scheduled jobs.
If a new worker using the
cronjob namespace is added later on, the Sidekiq
process will automatically pick up jobs for that worker too (after having been
restarted), without the need to change any configuration.
A queue namespace can be set using the
queue_namespace DSL class method:
class SomeScheduledTaskWorker include ApplicationWorker queue_namespace :cronjob # ... end
Behind the scenes, this will set
cronjob:some_scheduled_task. Commonly used namespaces will have their own
concern module that can easily be included into the worker class, and that may
set other Sidekiq options besides the queue namespace.
example, sets the namespace, but also disables retries.
bundle exec sidekiq is namespace-aware, and will automatically listen on all
queues in a namespace (technically: all queues prefixed with the namespace name)
when a namespace is provided instead of a simple queue name in the
-q) option, or in the
:queues: section in
Note that adding a worker to an existing namespace should be done with care, as the extra jobs will take resources away from jobs from workers that were already there, if the resources available to the Sidekiq process handling the namespace are not adjusted appropriately.
If a large number of background jobs get scheduled at once, queueing of jobs may occur while jobs wait for a worker node to be become available. This is normal and gives the system resilience by allowing it to gracefully handle spikes in traffic. Some jobs, however, are more sensitive to latency than others. Examples of these jobs include:
- A job which updates a merge request following a push to a branch.
- A job which invalidates a cache of known branches for a project after a push to the branch.
- A job which recalculates the groups and projects a user can see after a change in permissions.
- A job which updates the status of a CI pipeline after a state change to a job in the pipeline.
When these jobs are delayed, the user may perceive the delay as a bug: for
example, they may push a branch and then attempt to create a merge request for
that branch, but be told in the UI that the branch does not exist. We deem these
jobs to be
Extra effort is made to ensure that these jobs are started within a very short period of time after being scheduled. However, in order to ensure throughput, these jobs also have very strict execution duration requirements:
- The median job execution time should be less than 1 second.
- 99% of jobs should complete within 10 seconds.
If a worker cannot meet these expectations, then it cannot be treated as a
latency_sensitive worker: consider redesigning the worker, or splitting the
work between two different workers, one with
latency_sensitive code that
executes quickly, and the other with non-
latency_sensitive, which has no
execution latency requirements (but also has lower scheduling targets).
This can be summed up in the following table:
|Latency Sensitivity||Queue Scheduling Target||Execution Latency Requirement|
|Not ||1 minute||Maximum run time of 1 hour|
|100 milliseconds||p50 of 1 second, p99 of 10 seconds|
To mark a worker as being
latency_sensitive, use the
latency_sensitive_worker! attribute, as shown in this example:
class LatencySensitiveWorker include ApplicationWorker latency_sensitive_worker! # ... end
Most background jobs in the GitLab application communicate with other GitLab services, eg Postgres, Redis, Gitaly and Object Storage. These are considered to be “internal” dependencies for a job.
However, some jobs will be dependent on external services in order to complete successfully. Some examples include:
- Jobs which call web-hooks configured by a user.
- Jobs which deploy an application to a k8s cluster configured by a user.
These jobs have “external dependencies”. This is important for the operation of the background processing cluster in several ways:
- Most external dependencies (such as web-hooks) do not provide SLOs, and
therefore we cannot guarantee the execution latencies on these jobs. Since we
cannot guarantee execution latency, we cannot ensure throughput and
therefore, in high-traffic environments, we need to ensure that jobs with
external dependencies are separated from
latency_sensitivejobs, to ensure throughput on those queues.
- Errors in jobs with external dependencies have higher alerting thresholds as there is a likelihood that the cause of the error is external.
class ExternalDependencyWorker include ApplicationWorker # Declares that this worker depends on # third-party, external services in order # to complete successfully worker_has_external_dependencies! # ... end
Workers that are constrained by CPU or memory resource limitations should be
annotated with the
Most workers tend to spend most of their time blocked, wait on network responses from other services such as Redis, Postgres and Gitaly. Since Sidekiq is a multithreaded environment, these jobs can be scheduled with high concurrency.
Some workers, however, spend large amounts of time on-cpu running logic in Ruby. Ruby MRI does not support true multithreading - it relies on the GIL to greatly simplify application development by only allowing one section of Ruby code in a process to run at a time, no matter how many cores the machine hosting the process has. For IO bound workers, this is not a problem, since most of the threads are blocked in underlying libraries (which are outside of the GIL).
If many threads are attempting to run Ruby code simultaneously, this will lead to contention on the GIL which will have the affect of slowing down all processes.
In high-traffic environments, knowing that a worker is CPU-bound allows us to run it on a different fleet with lower concurrency. This ensures optimal performance.
Likewise, if a worker uses large amounts of memory, we can run these on a bespoke low concurrency, high memory fleet.
Note that Memory-bound workers create heavy GC workloads, with pauses of
10-50ms. This will have an impact on the latency requirements for the
worker. For this reason,
latency_sensitive jobs are not
permitted and will fail CI. In general,
memory bound workers are
discouraged, and alternative approaches to processing the work should be
This example shows how to declare a job as being CPU-bound.
class CPUIntensiveWorker include ApplicationWorker # Declares that this worker will perform a lot of # calculations on-CPU. worker_resource_boundary :cpu # ... end
We use the following approach to determine whether a worker is CPU-bound:
- In the Sidekiq structured JSON logs, aggregate the worker
durationrefers to the total job execution duration, in seconds
cpu_sis derived from the
Process::CLOCK_THREAD_CPUTIME_IDcounter, and is a measure of time spent by the job on-CPU.
durationto get the percentage time spend on-CPU.
- If this ratio exceeds 33%, the worker is considered CPU-bound and should be annotated as such.
- Note that these values should not be used over small sample sizes, but rather over fairly large aggregates.
Each Sidekiq worker, or one of its ancestor classes, must declare a
feature_category attribute. This attribute maps each worker to a feature
category. This is done for error budgeting, alert routing, and team attribution
for Sidekiq workers.
The declaration uses the
feature_category class method, as shown below.
class SomeScheduledTaskWorker include ApplicationWorker # Declares that this worker is part of the # `continuous_integration` feature category feature_category :continuous_integration # ... end
The list of value values can be found in the file
This file is, in turn generated from the
stages.yml from the GitLab Company Handbook
Occasionally new features will be added to GitLab stages. When this occurs, you
can automatically update
config/feature_categories.yml by running
scripts/update-feature-categories. This script will fetch and parse
and generate a new version of the file, which needs to be checked into source control.
A few Sidekiq workers, that are used across all features, cannot be mapped to a
single category. These should be declared as such using the
declaration, as shown below:
class SomeCrossCuttingConcernWorker include ApplicationWorker # Declares that this worker does not map to a feature category feature_category_not_owned! # ... end
Each Sidekiq worker must be tested using RSpec, just like any other class. These
tests should be placed in
Keep in mind that the arguments for a Sidekiq job are stored in a queue while it is scheduled for execution. During a online update, this could lead to several possible situations:
- An older version of the application publishes a job, which is executed by an upgraded Sidekiq node.
- A job is queued before an upgrade, but executed after an upgrade.
- A job is queued by a node running the newer version of the application, but executed on a node running an older version of the application.
Jobs need to be backwards- and forwards-compatible between consecutive versions of the application.
This can be done by following this process:
- Do not remove arguments from the
performfunction.. Instead, use the following approach
- Provide a default value (usually
nil) and use a comment to mark the argument as deprecated
- Stop using the argument in
- Ignore the value in the worker class, but do not remove it until the next major release.
- Provide a default value (usually
Try to avoid removing workers and their queues in minor and patch releases.
During online update instance can have pending jobs and removing the queue can lead to those jobs being stuck forever. If you can’t write migration for those Sidekiq jobs, please consider removing the worker in a major release only.
For the same reasons that removing workers is dangerous, care should be taken when renaming queues.
When renaming queues, use the
sidekiq_queue_migrate helper migration method,
as show in this example:
class MigrateTheRenamedSidekiqQueue < ActiveRecord::Migration[5.0] include Gitlab::Database::MigrationHelpers DOWNTIME = false def up sidekiq_queue_migrate 'old_queue_name', to: 'new_queue_name' end def down sidekiq_queue_migrate 'new_queue_name', to: 'old_queue_name' end end