Updating the Geo nodes

Depending on which version of Geo you are updating to/from, there may be different steps.

General update steps

In order to update the Geo nodes when a new GitLab version is released, all you need to do is update GitLab itself:

  1. Log into each node (primary and secondary nodes).
  2. Update GitLab.
  3. Test primary and secondary nodes, and check version in each.

Check status after updating

Now that the update process is complete, you may want to check whether everything is working correctly:

  1. Run the Geo raketask on all nodes, everything should be green:

    sudo gitlab-rake gitlab:geo:check
  2. Check the primary node’s Geo dashboard for any errors.
  3. Test the data replication by pushing code to the primary node and see if it is received by secondary nodes.

Upgrading to GitLab 12.1

By default, GitLab 12.1 will attempt to automatically upgrade the embedded PostgreSQL server to 10.7 from 9.6. Please see the omnibus documentation for the recommended procedure.

This can be temporarily disabled by running the following before ugprading:

sudo touch /etc/gitlab/disable-postgresql-upgrade

Upgrading to GitLab 10.8

Before 10.8, broadcast messages would not propagate without flushing the cache on the secondary nodes. This has been fixed in 10.8, but requires one last cache flush on each secondary node:

sudo gitlab-rake cache:clear

Upgrading to GitLab 10.6

In 10.4, we started to recommend that you define a password for database user (gitlab).

We now require this change as we use this password to enable the Foreign Data Wrapper, as a way to optimize the Geo Tracking Database. We are also improving security by disabling the use of trust authentication method.

  1. (primary) Login to your primary node and run:

    gitlab-ctl pg-password-md5 gitlab
# Enter password: <your_password_here>
# Confirm password: <your_password_here>
# fca0b89a972d69f00eb3ec98a5838484

    Copy the generated hash and edit /etc/gitlab/gitlab.rb:

    # Fill with the hash generated by `gitlab-ctl pg-password-md5 gitlab`
postgresql['sql_user_password'] = '<md5_hash_of_your_password>'

# Every node that runs Unicorn or Sidekiq needs to have the database
# password specified as below. If you have a high-availability setup, this
# must be present in all application nodes.
gitlab_rails['db_password'] = '<your_password_here>'

    Still in the configuration file, locate and remove the trust_auth_cidr_address:

    postgresql['trust_auth_cidr_addresses'] = ['127.0.0.1/32','1.2.3.4/32'] # <- Remove this

  2. (primary) Reconfigure and restart:

    sudo gitlab-ctl reconfigure
sudo gitlab-ctl restart

  3. (secondary) Login to all secondary nodes and edit /etc/gitlab/gitlab.rb:

    # Fill with the hash generated by `gitlab-ctl pg-password-md5 gitlab`
postgresql['sql_user_password'] = '<md5_hash_of_your_password>'

# Every node that runs Unicorn or Sidekiq needs to have the database
# password specified as below. If you have a high-availability setup, this
# must be present in all application nodes.
gitlab_rails['db_password'] = '<your_password_here>'

# Enable Foreign Data Wrapper
geo_secondary['db_fdw'] = true

# Secondary address in CIDR format, for example '5.6.7.8/32'
postgresql['md5_auth_cidr_addresses'] = ['<secondary_node_ip>/32']

    Still in the configuration file, locate and remove the trust_auth_cidr_address:

    postgresql['trust_auth_cidr_addresses'] = ['127.0.0.1/32','5.6.7.8/32'] # <- Remove this

  4. (secondary) Reconfigure and restart:

    sudo gitlab-ctl reconfigure
sudo gitlab-ctl restart

Upgrading to GitLab 10.5

For Geo Disaster Recovery to work with minimum downtime, your secondary node should use the same set of secrets as the primary node. However, setup instructions prior to the 10.5 release only synchronized the db_key_base secret.

To rectify this error on existing installations, you should overwrite the contents of /etc/gitlab/gitlab-secrets.json on each secondary node with the contents of /etc/gitlab/gitlab-secrets.json on the primary node, then run the following command on each secondary node:

sudo gitlab-ctl reconfigure

If you do not perform this step, you may find that two-factor authentication is broken following DR.

To prevent SSH requests to the newly promoted primary node from failing due to SSH host key mismatch when updating the primary node domain’s DNS record you should perform the step to Manually replicate primary SSH host keys in each secondary node.

Upgrading to GitLab 10.4

There are no Geo-specific steps to take!

Upgrading to GitLab 10.3

Support for SSH repository synchronization removed

In GitLab 10.2, synchronizing secondaries over SSH was deprecated. In 10.3, support is removed entirely. All installations will switch to the HTTP/HTTPS cloning method instead. Before upgrading, ensure that all your Geo nodes are configured to use this method and that it works for your installation. In particular, ensure that Git access over HTTP/HTTPS is enabled.

Synchronizing repositories over the public Internet using HTTP is insecure, so you should ensure that you have HTTPS configured before upgrading. Note that file synchronization is also insecure in these cases!

Upgrading to GitLab 10.2

Secure PostgreSQL replication

Support for TLS-secured PostgreSQL replication has been added. If you are currently using PostgreSQL replication across the open internet without an external means of securing the connection (e.g., a site-to-site VPN), then you should immediately reconfigure your primary and secondary PostgreSQL instances according to the updated instructions.

If you are securing the connections externally and wish to continue doing so, ensure you include the new option --sslmode=prefer in future invocations of gitlab-ctl replicate-geo-database.

HTTPS repository sync

Support for replicating repositories and wikis over HTTP/HTTPS has been added. Replicating over SSH has been deprecated, and support for this option will be removed in a future release.

To switch to HTTP/HTTPS replication, log into the primary node as an admin and visit Admin Area > Geo (/admin/geo/nodes). For each secondary node listed, press the “Edit” button, change the “Repository cloning” setting from “SSH (deprecated)” to “HTTP/HTTPS”, and press “Save changes”. This should take effect immediately.

Any new secondaries should be created using HTTP/HTTPS replication - this is the default setting.

After you’ve verified that HTTP/HTTPS replication is working, you should remove the now-unused SSH keys from your secondaries, as they may cause problems if the secondary node if ever promoted to a primary node:

  1. (secondary) Login to all your secondary nodes and run:

    sudo -u git -H rm ~git/.ssh/id_rsa ~git/.ssh/id_rsa.pub

Hashed Storage

Warning: Hashed storage is in Alpha. It is considered experimental and not production-ready. See Hashed Storage for more detail.

If you previously enabled Hashed Storage and migrated all your existing projects to Hashed Storage, disabling hashed storage will not migrate projects to their previous project based storage path. As such, once enabled and migrated we recommend leaving Hashed Storage enabled.

Upgrading to GitLab 10.1

Warning: Hashed storage is in Alpha. It is considered experimental and not production-ready. See Hashed Storage for more detail.

Hashed storage was introduced in GitLab 10.0, and a migration path for existing repositories was added in GitLab 10.1.

Upgrading to GitLab 10.0

Since GitLab 10.0, we require all Geo systems to use SSH key lookups via the database to avoid having to maintain consistency of the authorized_keys file for SSH access. Failing to do this will prevent users from being able to clone via SSH.

Note that in older versions of Geo, attachments downloaded on the secondary nodes would be saved to the wrong directory. We recommend that you do the following to clean this up.

On the secondary Geo nodes, run as root:

mv /var/opt/gitlab/gitlab-rails/working /var/opt/gitlab/gitlab-rails/working.old
mkdir /var/opt/gitlab/gitlab-rails/working
chmod 700 /var/opt/gitlab/gitlab-rails/working
chown git:git /var/opt/gitlab/gitlab-rails/working

You may delete /var/opt/gitlab/gitlab-rails/working.old any time.

Once this is done, we advise restarting GitLab on the secondary nodes for the new working directory to be used:

sudo gitlab-ctl restart

Upgrading from GitLab 9.3 or older

If you started running Geo on GitLab 9.3 or older, we recommend that you resync your secondary PostgreSQL databases to use replication slots. If you started using Geo with GitLab 9.4 or 10.x, no further action should be required because replication slots are used by default. However, if you started with GitLab 9.3 and upgraded later, you should still follow the instructions below.

When in doubt, it does not hurt to do a resync. The easiest way to do this in Omnibus is the following:

  1. Make sure you have Omnibus GitLab on the primary server.
  2. Run gitlab-ctl reconfigure and gitlab-ctl restart postgresql. This will enable replication slots on the primary database.
  3. Check the steps about defining postgresql['sql_user_password'], gitlab_rails['db_password'].
  4. Make sure postgresql['max_replication_slots'] matches the number of secondary Geo nodes locations.
  5. Install GitLab on the secondary server.
  6. Re-run the database replication process.

Special update notes for 9.0.x

IMPORTANT: With GitLab 9.0, the PostgreSQL version is upgraded to 9.6 and manual steps are required in order to update the secondary nodes and keep the Streaming Replication working. Downtime is required, so plan ahead.

The following steps apply only if you upgrade from a 8.17 GitLab version to 9.0+. For previous versions, update to GitLab 8.17 first before attempting to upgrade to 9.0+.

Make sure to follow the steps in the exact order as they appear below and pay extra attention in what node (either primary or secondary) you execute them! Each step is prepended with the relevant node for better clarity:

  1. (secondary) Login to all your secondary nodes and stop all services:

    sudo gitlab-ctl stop

  2. (secondary) Make a backup of the recovery.conf file on all secondary nodes to preserve PostgreSQL’s credentials:

    sudo cp /var/opt/gitlab/postgresql/data/recovery.conf /var/opt/gitlab/

  3. (primary) Update the primary node to GitLab 9.0 following the regular update docs. At the end of the update, the primary node will be running with PostgreSQL 9.6.

  4. (primary) To prevent a de-synchronization of the repository replication, stop all services except postgresql as we will use it to re-initialize the secondary node’s database:

    sudo gitlab-ctl stop
sudo gitlab-ctl start postgresql

  5. (secondary) Run the following steps on each of the secondary nodes:

    1. (secondary) Stop all services:

      sudo gitlab-ctl stop

    2. (secondary) Prevent running database migrations:

      sudo touch /etc/gitlab/skip-auto-migrations

    3. (secondary) Move the old database to another directory:

      sudo mv /var/opt/gitlab/postgresql{,.bak}

    4. (secondary) Update to GitLab 9.0 following the regular update docs. At the end of the update, the node will be running with PostgreSQL 9.6.

    5. (secondary) Make sure all services are up:

      sudo gitlab-ctl start

    6. (secondary) Reconfigure GitLab:

      sudo gitlab-ctl reconfigure

    7. (secondary) Run the PostgreSQL upgrade command:

      sudo gitlab-ctl pg-upgrade

    8. (secondary) See the stored credentials for the database that you will need to re-initialize the replication:

      sudo grep -s primary_conninfo /var/opt/gitlab/recovery.conf

    9. (secondary) Save the snippet below in a file, let’s say /tmp/replica.sh. Modify the embedded paths if necessary:

      #!/bin/bash

PORT="5432"
USER="gitlab_replicator"
echo ---------------------------------------------------------------
echo WARNING: Make sure this script is run from the secondary server
echo ---------------------------------------------------------------
echo
echo Enter the IP or FQDN of the primary PostgreSQL server
read HOST
echo Enter the password for $USER@$HOST
read -s PASSWORD
echo Enter the required sslmode
read SSLMODE

echo Stopping PostgreSQL and all GitLab services
sudo service gitlab stop
sudo service postgresql stop

echo Backing up postgresql.conf
sudo -u postgres mv /var/opt/gitlab/postgresql/data/postgresql.conf /var/opt/gitlab/postgresql/

echo Cleaning up old cluster directory
sudo -u postgres rm -rf /var/opt/gitlab/postgresql/data

echo Starting base backup as the replicator user
echo Enter the password for $USER@$HOST
sudo -u postgres /opt/gitlab/embedded/bin/pg_basebackup -h $HOST -D /var/opt/gitlab/postgresql/data -U gitlab_replicator -v -x -P

echo Writing recovery.conf file
sudo -u postgres bash -c "cat > /var/opt/gitlab/postgresql/data/recovery.conf <<- _EOF1_
  standby_mode = 'on'
  primary_conninfo = 'host=$HOST port=$PORT user=$USER password=$PASSWORD sslmode=$SSLMODE'
_EOF1_
"

echo Restoring postgresql.conf
sudo -u postgres mv /var/opt/gitlab/postgresql/postgresql.conf /var/opt/gitlab/postgresql/data/

echo Starting PostgreSQL
sudo service postgresql start

    10. (secondary) Run the recovery script using the credentials from the previous step:

      sudo bash /tmp/replica.sh

    11. (secondary) Reconfigure GitLab:

      sudo gitlab-ctl reconfigure

    12. (secondary) Start all services:

      sudo gitlab-ctl start

    13. (secondary) Repeat the steps for the remaining secondary nodes.

  6. (primary) After all secondary nodes are updated, start all services in primary node:

    sudo gitlab-ctl start

Update tracking database on secondary node

After updating a secondary node, you might need to run migrations on the tracking database. The tracking database was added in GitLab 9.1, and it is required since 10.0.

  1. Run database migrations on tracking database:

    sudo gitlab-rake geo:db:migrate

  2. Repeat this step for each secondary node.