- Ensuring a worker is idempotent
- Declaring a worker as idempotent
- Setting the deduplication time-to-live (TTL)
It’s known that a job can fail for multiple reasons. For example, network outages or bugs. In order to address this, Sidekiq has a built-in retry mechanism that is used by default by most workers within GitLab.
It’s expected that a job can run again after a failure without major side-effects for the application or users, which is why Sidekiq encourages jobs to be idempotent and transactional.
As a general rule, a worker can be considered idempotent if:
- It can safely run multiple times with the same arguments.
- Application side-effects are expected to happen only once (or side-effects of a second run do not have an effect).
A good example of that would be a cache expiration worker.
A job scheduled for an idempotent worker is deduplicated when an unstarted job with the same arguments is already in the queue.
Make sure the worker tests pass using the following shared example:
include_examples 'an idempotent worker' do it 'marks the MR as merged' do # Using subject inside this block will process the job multiple times subject expect(merge_request.state).to eq('merged') end end
perform_multiple method directly instead of
helper method is automatically included for workers).
class IdempotentWorker include ApplicationWorker # Declares a worker is idempotent and can # safely run multiple times. idempotent! # ... end
It’s encouraged to only have the
idempotent! call in the top-most worker class, even if
perform method is defined in another class or module.
If the worker class isn’t marked as idempotent, a cop fails. Consider skipping the cop if you’re not confident your job can safely run multiple times.
When a job for an idempotent worker is enqueued while another unstarted job is already in the queue, GitLab drops the second job. The work is skipped because the same work would be done by the job that was scheduled first; by the time the second job executed, the first job would do nothing.
GitLab supports two deduplication strategies:
until_executing, which is the default strategy
More deduplication strategies have been suggested. If you are implementing a worker that could benefit from a different strategy, please comment in the issue.
This strategy takes a lock when a job is added to the queue, and removes that lock before the job starts.
AuthorizedProjectsWorker takes a user ID. When the
worker runs, it recalculates a user’s authorizations. GitLab schedules
this job each time an action potentially changes a user’s
authorizations. If the same user is added to two projects at the
same time, the second job can be skipped if the first job hasn’t
begun, because when the first job runs, it creates the
authorizations for both projects.
module AuthorizedProjectUpdate class UserRefreshOverUserRangeWorker include ApplicationWorker deduplicate :until_executing idempotent! # ... end end
This strategy takes a lock when a job is added to the queue, and removes that lock after the job finishes. It can be used to prevent jobs from running simultaneously multiple times.
module Ci class BuildTraceChunkFlushWorker include ApplicationWorker deduplicate :until_executed idempotent! # ... end end
Also, you can pass
if_deduplicated: :reschedule_once option to re-run a job once after
the currently running job finished and deduplication happened at least once.
This ensures that the latest result is always produced even if a race condition
happened. See this issue for more information.
GitLab doesn’t skip jobs scheduled in the future, as we assume that
the state has changed by the time the job is scheduled to
execute. Deduplication of jobs scheduled in the future is possible
If you do want to deduplicate jobs scheduled in the future,
this can be specified on the worker by passing
including_scheduled: true argument
when defining deduplication strategy:
module AuthorizedProjectUpdate class UserRefreshOverUserRangeWorker include ApplicationWorker deduplicate :until_executing, including_scheduled: true idempotent! # ... end end
Deduplication depends on an idempotency key that is stored in Redis. This is normally cleared by the configured deduplication strategy.
However, the key can remain until its TTL in certain cases like:
until_executingis used but the job was never enqueued or executed after the Sidekiq client middleware was run.
until_executedis used but the job fails to finish due to retry exhaustion, gets interrupted the maximum number of times, or gets lost.
The default value is 6 hours. During this time, jobs won’t be enqueued even if the first job never executed or finished.
The TTL can be configured with:
class ProjectImportScheduleWorker include ApplicationWorker idempotent! deduplicate :until_executing, ttl: 5.minutes end
Duplicate jobs can happen when the TTL is reached, so make sure you lower this only for jobs that can tolerate some duplication.
- Introduced in GitLab 14.3.
- Enabled on GitLab.com in GitLab 14.4.
- Enabled on self-managed in GitLab 14.6.
- Generally available in GitLab 14.9. Feature flag preserve_latest_wal_locations_for_idempotent_jobs removed.
The deduplication always take into account the latest binary replication pointer, not the first one. This happens because we drop the same job scheduled for the second time and the Write-Ahead Log (WAL) is lost. This could lead to comparing the old WAL location and reading from a stale replica.
To support both deduplication and maintaining data consistency with load balancing, we are preserving the latest WAL location for idempotent jobs in Redis. This way we are always comparing the latest binary replication pointer, making sure that we read from the replica that is fully caught up.