- Routing rules
- Queue selectors (deprecated)
- Migrating from queue selectors to routing rules
- Worker matching query
Processing specific job classes
GitLab has two options for creating Sidekiq processes that only handle specific job classes:
- Routing rules are used on GitLab.com. They direct jobs inside the application to queue names configured by administrators. This lowers the load on Redis, which is important on very large-scale deployments.
- Queue selectors perform the job selection outside the application, when starting the Sidekiq process. This was used on GitLab.com until September 2021, and is retained for compatibility reasons.
Both of these use the same worker matching query syntax. While they can technically be used together, most deployments should choose one or the other; there is no particular benefit in combining them.
Routing rules
- Default routing rule value introduced in GitLab 15.4.
mailers
queue. When using routing rules, ensure that at least one process is
listening to the mailers
queue. Typically this can be placed alongside the
default
queue.We recommend most GitLab instances using routing rules to manage their Sidekiq
queues. This allows administrators to choose single queue names for groups of
job classes based on their attributes. The syntax is an ordered array of pairs of [query, queue]
:
- The query is a worker matching query.
- The queue name must be a valid Sidekiq queue name. If the queue name
is
nil
, or an empty string, the worker is routed to the queue generated by the name of the worker instead. (See list of available job classes for more information). The queue name does not have to match any existing queue name in the list of available job classes. - The first query matching a worker is chosen for that worker; later rules are ignored.
Routing rules migration
After the Sidekiq routing rules are changed, administrators must take care with the migration to avoid losing jobs entirely, especially in a system with long queues of jobs. The migration can be done by following the migration steps mentioned in Sidekiq job migration.
Routing rules in a scaled architecture
Routing rules must be the same across all GitLab nodes (especially GitLab Rails and Sidekiq nodes) as they are part of the application configuration. Queue selectors can be different across GitLab nodes because they only change the arguments to the launched Sidekiq process.
Detailed example
This is a comprehensive example intended to show different possibilities. A Helm chart example is also available. They are not recommendations.
-
Edit
/etc/gitlab/gitlab.rb
:sidekiq['routing_rules'] = [ # Route all non-CPU-bound workers that are high urgency to `high-urgency` queue ['resource_boundary!=cpu&urgency=high', 'high-urgency'], # Route all database, gitaly and global search workers that are throttled to `throttled` queue ['feature_category=database,gitaly,global_search&urgency=throttled', 'throttled'], # Route all workers having contact with outside world to a `network-intenstive` queue ['has_external_dependencies=true|feature_category=hooks|tags=network', 'network-intensive'], # Wildcard matching, route the rest to `default` queue ['*', 'default'] ]
The
queue_groups
can then be set to match these generated queue names. For instance:sidekiq['queue_selector'] = false sidekiq['queue_groups'] = [ # Run two high-urgency processes 'high-urgency', 'high-urgency', # Run one process for throttled, network-intensive, import 'throttled,network-intensive,import', # Run one 'catchall' process on the default and mailers queues 'default,mailers' ]
-
Save the file and reconfigure GitLab:
sudo gitlab-ctl reconfigure
Queue selectors (deprecated)
The queue_selector
option allows queue groups to be selected in a more general
way using a worker matching query. After
queue_selector
is set, all queue_groups
must follow the aforementioned
syntax.
Using queue selectors
-
Edit
/etc/gitlab/gitlab.rb
:sidekiq['enable'] = true sidekiq['routing_rules'] = [['*', nil]] sidekiq['queue_selector'] = true sidekiq['queue_groups'] = [ # Run all non-CPU-bound queues that are high urgency 'resource_boundary!=cpu&urgency=high', # Run all continuous integration and pages queues that are not high urgency 'feature_category=continuous_integration,pages&urgency!=high', # Run all queues '*' ]
-
Save the file and reconfigure GitLab:
sudo gitlab-ctl reconfigure
Negate settings (deprecated)
This allows you to have the Sidekiq process work on every queue except the ones you list. This is generally only used when there are multiple Sidekiq nodes. In this example, we exclude all import-related jobs from a Sidekiq node.
-
Edit
/etc/gitlab/gitlab.rb
:sidekiq['routing_rules'] = [['*', nil]] sidekiq['negate'] = true sidekiq['queue_selector'] = true sidekiq['queue_groups'] = [ "feature_category=importers" ]
-
Save the file and reconfigure GitLab:
sudo gitlab-ctl reconfigure
Migrating from queue selectors to routing rules
We recommend GitLab deployments add more Sidekiq processes listening to all queues, as in the Reference Architectures. For very large-scale deployments, we recommend routing rules instead of queue selectors. We use routing rules on GitLab.com as it helps to lower the load on Redis.
Single node setup
To migrate from queue selectors to routing rules in a single node setup:
- Open
/etc/gitlab/gitlab.rb
. - Set
sidekiq['queue_selector']
tofalse
. - Take all queue
selector
s in thesidekiq['queue_groups']
. - Give each
selector
aqueue_name
and put them in[selector, queue_name]
format. - Replace
sidekiq['routing_rules']
with an array of[selector, queue_name]
entries. - Add a wildcard match of
['*', 'default']
as the last entry insidekiq['routing_rules']
. This “catchall” queue has to be named asdefault
. - Replace
sidekiq['queue_groups']
withqueue_name
s. - Add at least one
default
queue and at least onemailers
queue to thesidekiq['queue_groups']
. -
Save the file and reconfigure GitLab:
sudo gitlab-ctl reconfigure
-
Run the Rake task to migrate existing jobs:
sudo gitlab-rake gitlab:sidekiq:migrate_jobs:retry gitlab:sidekiq:migrate_jobs:schedule gitlab:sidekiq:migrate_jobs:queued
Migration example
The following example better illustrates the migration process above:
-
In
/etc/gitlab/gitlab.rb
, check theurgency
queries in thesidekiq['queue_groups']
. For example:sidekiq['routing_rules'] = [] sidekiq['queue_selector'] = true sidekiq['queue_groups'] = [ 'urgency=high', 'urgency=low', 'urgency=throttled', '*' ]
-
Use these same
urgency
queries to update/etc/gitlab/gitlab.rb
to use routing rules:sidekiq['min_concurrency'] = 20 sidekiq['max_concurrency'] = 20 sidekiq['routing_rules'] = [ ['urgency=high', 'high_urgency'], ['urgency=low', 'low_urgency'], ['urgency=throttled', 'throttled_urgency'], # Wildcard matching, route the rest to `default` queue ['*', 'default'] ] sidekiq['queue_selector'] = false sidekiq['queue_groups'] = [ 'high_urgency', 'low_urgency', 'throttled_urgency', 'default,mailers' ]
-
Save the file and reconfigure GitLab:
sudo gitlab-ctl reconfigure
-
Run the Rake task to migrate existing jobs:
sudo gitlab-rake gitlab:sidekiq:migrate_jobs:retry gitlab:sidekiq:migrate_jobs:schedule gitlab:sidekiq:migrate_jobs:queued
min_concurrency
and max_concurrency
to the same value. For example, if the number of queues
in a queue group entry is 1, while min_concurrency
is set to 0
, and max_concurrency
is set to 20
, the resulting
concurrency is set to 2
instead. A concurrency of 2
might be too low in most cases, except for very highly-CPU
bound tasks.Multiple node setup
For a multiple node setup:
- Reconfigure all GitLab Rails and Sidekiq nodes with the same
sidekiq['routing_rules']
setting. - Alternate between GitLab Rails and Sidekiq nodes as you update and reconfigure the nodes. This ensures the newly configured Sidekiq is ready to consume jobs from the new set of queues during the migration. Otherwise, the new jobs hang until the end of the migration.
Consider the following example of three GitLab Rails nodes and two Sidekiq nodes. To migrate from queue selectors to routing rules:
- In Sidekiq 1, follow all steps but one in single node setup. Do not run the Rake task to migrate existing jobs.
- Configure the external load balancer to remove Rails 1 from accepting traffic. This step ensures Rails 1 is not serving any request while the Rails process is restarting. For more information, see issue 428794.
- In Rails 1, update
/etc/gitlab/gitlab.rb
to use the samesidekiq['routing_rules']
setting as Sidekiq 1. Onlysidekiq['routing_rules']
is required in Rails nodes. - Configure the external load balancer to register Rails 1 back.
- Repeat steps 1 to 4 for Sidekiq 2 and Rails 2.
- Repeat steps 2 to 4 for Rails 3.
- If there are more Sidekiq nodes than Rails nodes, follow step 1 on the remaining Sidekiq nodes.
-
Run the Rake task to migrate existing jobs:
sudo gitlab-rake gitlab:sidekiq:migrate_jobs:retry gitlab:sidekiq:migrate_jobs:schedule gitlab:sidekiq:migrate_jobs:queued
Worker matching query
GitLab provides a query syntax to match a worker based on its attributes. This query syntax is employed by both routing rules and queue selectors. A query includes two components:
- Attributes that can be selected.
- Operators used to construct a query.
Available attributes
Queue matching query works upon the worker attributes, described in Sidekiq style guide. We support querying based on a subset of worker attributes:
-
feature_category
- the GitLab feature category the queue belongs to. For example, themerge
queue belongs to thesource_code_management
category. -
has_external_dependencies
- whether or not the queue connects to external services. For example, all importers have this set totrue
. -
urgency
- how important it is that this queue’s jobs run quickly. Can behigh
,low
, orthrottled
. For example, theauthorized_projects
queue is used to refresh user permissions, and ishigh
urgency. -
worker_name
- the worker name. Use this attribute to select a specific worker. Find all available names in the job classes lists below. -
name
- the queue name generated from the worker name. Use this attribute to select a specific queue. Because this is generated from the worker name, it does not change based on the result of other routing rules. -
resource_boundary
- if the queue is bound bycpu
,memory
, orunknown
. For example, theProjectExportWorker
is memory bound as it has to load data in memory before saving it for export. -
tags
- short-lived annotations for queues. These are expected to frequently change from release to release, and may be removed entirely.
has_external_dependencies
is a boolean attribute: only the exact
string true
is considered true, and everything else is considered
false.
tags
is a set, which means that =
checks for intersecting sets, and
!=
checks for disjoint sets. For example, tags=a,b
selects queues
that have tags a
, b
, or both. tags!=a,b
selects queues that have
neither of those tags.
Available operators
Routing rules and queue selectors support the following operators, listed from highest to lowest precedence:
-
|
- the logicalOR
operator. For example,query_a|query_b
(wherequery_a
andquery_b
are queries made up of the other operators here) includes queues that match either query. -
&
- the logicalAND
operator. For example,query_a&query_b
(wherequery_a
andquery_b
are queries made up of the other operators here) include only queues that match both queries. -
!=
- theNOT IN
operator. For example,feature_category!=issue_tracking
excludes all queues from theissue_tracking
feature category. -
=
- theIN
operator. For example,resource_boundary=cpu
includes all queues that are CPU bound. -
,
- the concatenate set operator. For example,feature_category=continuous_integration,pages
includes all queues from either thecontinuous_integration
category or thepages
category. This example is also possible using the OR operator, but allows greater brevity, as well as being lower precedence.
The operator precedence for this syntax is fixed: it’s not possible to make AND
have higher precedence than OR
.
As with the standard queue group syntax above, a single *
as the
entire queue group selects all queues.
List of available job classes
For a list of the existing Sidekiq job classes and queues, check the following files: