Repository mirroring

Deep Dive

In December 2018, Tiago Botelho hosted a Deep Dive (GitLab team members only: https://gitlab.com/gitlab-org/create-stage/-/issues/1) on the GitLab Pull Repository Mirroring functionality to share his domain specific knowledge with anyone who may work in this part of the codebase in the future. You can find the recording on YouTube, and the slides in PDF. Specific details may have changed since then, but it should still serve as a good introduction.

Explanation of mirroring process

GitLab performs these steps when an API call triggers a pull mirror. Scheduled mirror updates are similar, but do not start with the API call:

  1. The request originates from an API call, and triggers the start_pull_mirroring_service in project_mirror.rb.
  2. The pull mirroring service (start_pull_mirroring_service.rb) starts. It updates the project state, and forces the job to start immediately.
  3. The project import state is updated, and then triggers an update_all_mirrors_worker in project_import_state.rb.
  4. The update all mirrors worker (update_all_mirrors_worker.rb) attempts to avoid stampedes by calling the project_import_schedule worker.
  5. The project import schedule worker (project_import_schedule_worker.rb) updates the state of the project, and starts a Ruby state_machine to manage the import transition process.
  6. While updating the project state, this call in project.rb starts the repository_update_mirror worker.
  7. The Sidekiq background mirror workers (repository_update_mirror_worker.rb) track the state of the mirroring task, and provide good error state information. Processes can hang here, because this step manages the Git steps.
  8. The update mirror service (update_mirror_service.rb) performs the Git operations.

The import and mirror update processes are complete after the update mirror service step. However, depending on the changes included, more tasks (such as pipelines for commits) can be triggered.