- Moving data within a GitLab instance
Migrating to another GitLab instance
- Recommended approach in all cases
Target directory is empty: use a
The target directory contains an outdated copy of the repositories: use
Thousands of Git repositories: use one
Sometimes you need to move all repositories managed by GitLab to another file system or another server.
The GitLab API is the recommended way to move Git repositories:
- Between servers.
- Between different storage.
- From single-node Gitaly to Gitaly Cluster.
For more information, see:
Configuring additional storage for Gitaly. This
example configures additional storage called
- The API documentation details the endpoints for querying and scheduling project repository moves.
- The API documentation details the endpoints for querying and scheduling snippet repository moves.
- The API documentation details the endpoints for querying and scheduling group repository moves .
- Migrate existing repositories to Gitaly Cluster.
Using the API isn’t an option if you are migrating to a new GitLab environment, for example:
- From a single-node GitLab to a scaled-out architecture.
- From a GitLab instance in your private data center to a cloud provider.
The rest of the document looks
at some of the ways you can copy all your repositories from
We look at three scenarios:
- The target directory is empty.
- The target directory contains an outdated copy of the repositories.
- How to deal with thousands of repositories.
/mnt/gitlab/repositories. Do not mix up the source and the target.
The GitLab backup and restore capability should be used. Git
repositories are accessed, managed, and stored on GitLab servers by Gitaly as a database. Data loss
can result from directly accessing and copying Gitaly’s files using tools like
- From GitLab 13.3, backup performance can be improved by processing multiple repositories concurrently.
- Backups can be created of just the repositories using the skip feature.
If the target directory
/mnt/gitlab/repositories is empty the
simplest thing to do is to use a
tar pipe. This method has low
tar is almost always already installed on your system.
However, it is not possible to resume an interrupted
tar pipe: if
that happens then all data must be copied again.
sudo -u git sh -c 'tar -C /var/opt/gitlab/git-data/repositories -cf - -- . |\ tar -C /mnt/gitlab/repositories -xf -'
If you want to see progress, replace
You can also use a
tar pipe to copy data to another server. If your
git user has SSH access to the new server as
can pipe the data through SSH.
sudo -u git sh -c 'tar -C /var/opt/gitlab/git-data/repositories -cf - -- . |\ ssh git@newserver tar -C /mnt/gitlab/repositories -xf -'
If you want to compress the data before it goes over the network
(which costs you CPU cycles) you can replace
If the target directory already contains a partial / outdated copy
of the repositories it may be wasteful to copy all the data again
tar. In this scenario it is better to use
rsync. This utility
is either already installed on your system, or installable
sudo -u git sh -c 'rsync -a --delete /var/opt/gitlab/git-data/repositories/. \ /mnt/gitlab/repositories'
/. in the command above is very important, without it you can
get the wrong directory structure in the target directory.
If you want to see progress, replace
git user on your source system has SSH access to the target
server you can send the repositories over the network with
sudo -u git sh -c 'rsync -a --delete /var/opt/gitlab/git-data/repositories/. \ git@newserver:/mnt/gitlab/repositories'
Every time you start an
rsync job it must:
- Inspect all files in the source directory.
- Inspect all files in the target directory.
- Decide whether or not to copy files.
If the source or target directory
has many contents, this startup phase of
rsync can become a burden
for your GitLab server. You can reduce the workload of
rsync by dividing its
work in smaller pieces, and sync one repository at a time.
In addition to
rsync we use GNU Parallel.
This utility is not included in GitLab, so you must install it yourself with
This process does not clean up repositories at the target location that no longer exist at the source.
This syncs repositories with 10
rsync processes at a time. We keep
track of progress so that the transfer can be restarted if necessary.
First we create a new directory, owned by
git, to hold transfer
logs. We assume the directory is empty before we start the transfer
procedure, and that we are the only ones writing files in it.
# Omnibus sudo mkdir /var/opt/gitlab/transfer-logs sudo chown git:git /var/opt/gitlab/transfer-logs # Source sudo -u git -H mkdir /home/git/transfer-logs
We seed the process with a list of the directories we want to copy.
# Omnibus sudo -u git sh -c 'gitlab-rake gitlab:list_repos > /var/opt/gitlab/transfer-logs/all-repos-$(date +%s).txt' # Source cd /home/git/gitlab sudo -u git -H sh -c 'bundle exec rake gitlab:list_repos > /home/git/transfer-logs/all-repos-$(date +%s).txt'
Now we can start the transfer. The command below is idempotent, and
the number of jobs done by GNU Parallel should converge to zero. If it
does not, some repositories listed in
all-repos-1234.txt may have been
deleted/renamed before they could be copied.
# Omnibus sudo -u git sh -c ' cat /var/opt/gitlab/transfer-logs/* | sort | uniq -u |\ /usr/bin/env JOBS=10 \ /opt/gitlab/embedded/service/gitlab-rails/bin/parallel-rsync-repos \ /var/opt/gitlab/transfer-logs/success-$(date +%s).log \ /var/opt/gitlab/git-data/repositories \ /mnt/gitlab/repositories ' # Source cd /home/git/gitlab sudo -u git -H sh -c ' cat /home/git/transfer-logs/* | sort | uniq -u |\ /usr/bin/env JOBS=10 \ bin/parallel-rsync-repos \ /home/git/transfer-logs/success-$(date +%s).log \ /home/git/repositories \ /mnt/gitlab/repositories `
Suppose you have already done one sync that started after 2015-10-1 12:00 UTC.
Then you might only want to sync repositories that were changed by using GitLab
after that time. You can use the
SINCE variable to tell
gitlab:list_repos to only print repositories with recent activity.
# Omnibus sudo gitlab-rake gitlab:list_repos SINCE='2015-10-1 12:00 UTC' |\ sudo -u git \ /usr/bin/env JOBS=10 \ /opt/gitlab/embedded/service/gitlab-rails/bin/parallel-rsync-repos \ success-$(date +%s).log \ /var/opt/gitlab/git-data/repositories \ /mnt/gitlab/repositories # Source cd /home/git/gitlab sudo -u git -H bundle exec rake gitlab:list_repos SINCE='2015-10-1 12:00 UTC' |\ sudo -u git -H \ /usr/bin/env JOBS=10 \ bin/parallel-rsync-repos \ success-$(date +%s).log \ /home/git/repositories \ /mnt/gitlab/repositories