Geo Troubleshooting

Note: This list is an attempt to document all the moving parts that can go wrong. We are working into getting all this steps verified automatically in a rake task in the future.

Setting up Geo requires careful attention to details and sometimes it’s easy to miss a step. Here is a list of questions you should ask to try to detect what you need to fix (all commands and path locations are for Omnibus installs):

First check the health of the secondary node

Visit the primary node’s Admin Area > Geo (/admin/geo/nodes) in your browser. We perform the following health checks on each secondary node to help identify if something is wrong:

  • Is the node running?
  • Is the node’s secondary database configured for streaming replication?
  • Is the node’s secondary tracking database configured?
  • Is the node’s secondary tracking database connected?
  • Is the node’s secondary tracking database up-to-date?

Geo health check

If the UI is not working, or you are unable to log in, you can run the Geo health check manually to get this information as well as a few more details. This rake task can be run on an app node in the primary or secondary Geo nodes:

sudo gitlab-rake gitlab:geo:check

Example output:

Checking Geo ...

GitLab Geo is available ... yes
GitLab Geo is enabled ... yes
GitLab Geo secondary database is correctly configured ... yes
Using database streaming replication? ... yes
GitLab Geo tracking database is configured to use Foreign Data Wrapper? ... yes
GitLab Geo tracking database Foreign Data Wrapper schema is up-to-date? ... yes
GitLab Geo HTTP(S) connectivity ...
* Can connect to the primary node ... yes
HTTP/HTTPS repository cloning is enabled ... yes
Machine clock is synchronized ... yes
Git user has default SSH configuration? ... yes
OpenSSH configured to use AuthorizedKeysCommand ... yes
GitLab configured to disable writing to authorized_keys file ... yes
GitLab configured to store new projects in hashed storage? ... yes
All projects are in hashed storage? ... yes

Checking Geo ... Finished

Current sync information can be found manually by running this rake task on any secondary app node:

sudo gitlab-rake geo:status

Example output:
                        GitLab Version: 11.8.1-ee
                              Geo Role: Secondary
                         Health Status: Healthy
                          Repositories: 190/190 (100%)
                 Verified Repositories: 190/190 (100%)
                                 Wikis: 190/190 (100%)
                        Verified Wikis: 190/190 (100%)
                           LFS Objects: 35/35 (100%)
                           Attachments: 528/528 (100%)
                      CI job artifacts: 477/477 (100%)
                  Repositories Checked: 0/190 (0%)
                         Sync Settings: Full
              Database replication lag: 0 seconds
       Last event ID seen from primary: 2158 (about 2 minute ago)
     Last event ID processed by cursor: 2158 (about 2 minute ago)
                Last status report was: 4 minutes ago

Is Postgres replication working?

Are my nodes pointing to the correct database instance?

You should make sure your primary Geo node points to the instance with writing permissions.

Any secondary nodes should point only to read-only instances.

Can Geo detect my current node correctly?

Geo uses the defined node from the Admin Area > Geo screen, and tries to match it with the value defined in the /etc/gitlab/gitlab.rb configuration file. The relevant line looks like: external_url "".

To check if the node on the current machine is correctly detected type:

sudo gitlab-rails runner "puts Gitlab::Geo.current_node.inspect"

and expect something like:

#<GeoNode id: 2, schema: "https", host: "", port: 443, relative_url_root: "", primary: false, ...>

By running the command above, primary should be true when executed in the primary node, and false on any secondary node.

How do I fix the message, “ERROR: replication slots can only be used if max_replication_slots > 0”?

This means that the max_replication_slots PostgreSQL variable needs to be set on the primary database. In GitLab 9.4, we have made this setting default to 1. You may need to increase this value if you have more secondary nodes. Be sure to restart PostgreSQL for this to take effect. See the PostgreSQL replication setup guide for more details.

How do I fix the message, “FATAL: could not start WAL streaming: ERROR: replication slot “geo_secondary_my_domain_com” does not exist”?

This occurs when PostgreSQL does not have a replication slot for the secondary node by that name. You may want to rerun the replication process on the secondary node .

How do I fix the message, “Command exceeded allowed execution time” when setting up replication?

This may happen while initiating the replication process on the secondary node, and indicates that your initial dataset is too large to be replicated in the default timeout (30 minutes).

Re-run gitlab-ctl replicate-geo-database, but include a larger value for --backup-timeout:

sudo gitlab-ctl \
   replicate-geo-database \
   --host=<primary_node_hostname> \
   --slot-name=<secondary_slot_name> \

This will give the initial replication up to six hours to complete, rather than the default thirty minutes. Adjust as required for your installation.

How do I fix the message, “PANIC: could not write to file ‘pg_xlog/xlogtemp.123’: No space left on device”

Determine if you have any unused replication slots in the primary database. This can cause large amounts of log data to build up in pg_xlog. Removing the unused slots can reduce the amount of space used in the pg_xlog.

  1. Start a PostgreSQL console session:

     sudo gitlab-psql gitlabhq_production

    Note that using gitlab-rails dbconsole will not work, because managing replication slots requires superuser permissions.

  2. View your replication slots with:

     SELECT * FROM pg_replication_slots;

Slots where active is f are not active.

  • When this slot should be active, because you have a secondary node configured using that slot, log in to that secondary node and check the PostgreSQL logs why the replication is not running.

  • If you are no longer using the slot (e.g. you no longer have Geo enabled), you can remove it with in the PostgreSQL console session:

      SELECT pg_drop_replication_slot('<name_of_extra_slot>');

Very large repositories never successfully synchronize on the secondary node

GitLab places a timeout on all repository clones, including project imports and Geo synchronization operations. If a fresh git clone of a repository on the primary takes more than a few minutes, you may be affected by this. To increase the timeout, add the following line to /etc/gitlab/gitlab.rb on the secondary node:

gitlab_rails['gitlab_shell_git_timeout'] = 10800

Then reconfigure GitLab:

sudo gitlab-ctl reconfigure

This will increase the timeout to three hours (10800 seconds). Choose a time long enough to accommodate a full clone of your largest repositories.

How to reset Geo secondary node replication

If you get a secondary node in a broken state and want to reset the replication state, to start again from scratch, there are a few steps that can help you:

  1. Stop Sidekiq and the Geo LogCursor

    It’s possible to make Sidekiq stop gracefully, but making it stop getting new jobs and wait until the current jobs to finish processing.

    You need to send a SIGTSTP kill signal for the first phase and them a SIGTERM when all jobs have finished. Otherwise just use the gitlab-ctl stop commands.

     gitlab-ctl status sidekiq
     # run: sidekiq: (pid 10180) <- this is the PID you will use
     kill -TSTP 10180 # change to the correct PID
     gitlab-ctl stop sidekiq
     gitlab-ctl stop geo-logcursor

    You can watch sidekiq logs to know when sidekiq jobs processing have finished:

     gitlab-ctl tail sidekiq
  2. Rename repository storage folders and create new ones

     mv /var/opt/gitlab/git-data/repositories /var/opt/gitlab/git-data/repositories.old
     mkdir -p /var/opt/gitlab/git-data/repositories
     chown git:git /var/opt/gitlab/git-data/repositories
    Tip You may want to remove the /var/opt/gitlab/git-data/repositories.old in the future as soon as you confirmed that you don’t need it anymore, to save disk space.
  3. (Optional) Rename other data folders and create new ones

    Caution: You may still have files on the secondary node that have been removed from primary node but removal have not been reflected. If you skip this step, they will never be removed from this Geo node.

    Any uploaded content like file attachments, avatars or LFS objects are stored in a subfolder in one of the two paths below:

    1. /var/opt/gitlab/gitlab-rails/shared
    2. /var/opt/gitlab/gitlab-rails/uploads

    To rename all of them:

     gitlab-ctl stop
     mv /var/opt/gitlab/gitlab-rails/shared /var/opt/gitlab/gitlab-rails/shared.old
     mkdir -p /var/opt/gitlab/gitlab-rails/shared
     mv /var/opt/gitlab/gitlab-rails/uploads /var/opt/gitlab/gitlab-rails/uploads.old
     mkdir -p /var/opt/gitlab/gitlab-rails/uploads

    Reconfigure in order to recreate the folders and make sure permissions and ownership are correctly

     gitlab-ctl reconfigure
  4. Reset the Tracking Database

     gitlab-rake geo:db:reset
  5. Restart previously stopped services

     gitlab-ctl start

How do I fix a “Foreign Data Wrapper (FDW) is not configured” error?

When setting up Geo, you might see this warning in the gitlab-rake gitlab:geo:check output:

GitLab Geo tracking database Foreign Data Wrapper schema is up-to-date? ... foreign data wrapper is not configured

There are a few key points to remember:

  1. The FDW settings are configured on the Geo tracking database.
  2. The configured foreign server enables a login to the Geo secondary, read-only database.

By default, the Geo secondary and tracking database are running on the same host on different ports. That is, 5432 and 5431 respectively.

Checking configuration

Note: The following steps are for Omnibus installs only. Using Geo with source-based installs is deprecated.

To check the configuration:

  1. Enter the database console:

  2. Check whether any tables are present. If everything is working, you should see something like this:

     gitlabhq_geo_production=# SELECT * from information_schema.foreign_tables;
       foreign_table_catalog  | foreign_table_schema |               foreign_table_name                | foreign_server_catalog  | foreign_server_n
      gitlabhq_geo_production | gitlab_secondary     | abuse_reports                                   | gitlabhq_geo_production | gitlab_secondary
      gitlabhq_geo_production | gitlab_secondary     | appearances                                     | gitlabhq_geo_production | gitlab_secondary
      gitlabhq_geo_production | gitlab_secondary     | application_setting_terms                       | gitlabhq_geo_production | gitlab_secondary
      gitlabhq_geo_production | gitlab_secondary     | application_settings                            | gitlabhq_geo_production | gitlab_secondary

    However, if the query returns with 0 rows, then continue onto the next steps.

  3. Check that the foreign server mapping is correct via \des+. The results should look something like this:

     gitlabhq_geo_production=# \des+
     List of foreign servers
     -[ RECORD 1 ]--------+------------------------------------------------------------
     Name                 | gitlab_secondary
     Owner                | gitlab-psql
     Foreign-data wrapper | postgres_fdw
     Access privileges    | "gitlab-psql"=U/"gitlab-psql"                              +
                          | gitlab_geo=U/"gitlab-psql"
     Type                 |
     Version              |
     FDW Options          | (host '', port '5432', dbname 'gitlabhq_production')
     Description          |
    Note: Pay particular attention to the host and port under FDW options. That configuration should point to the Geo secondary database.

    If you need to experiment with changing the host or password, the following queries demonstrate how:

     ALTER SERVER gitlab_secondary OPTIONS (SET host '<my_new_host>');
     ALTER SERVER gitlab_secondary OPTIONS (SET port 5432);

    If you change the host and/or port, you will also have to adjust the following settings in /etc/gitlab/gitlab.rb and run gitlab-ctl reconfigure:

    • gitlab_rails['db_host']
    • gitlab_rails['db_port']
  4. Check that the user mapping is configured properly via \deu+:

     gitlabhq_geo_production=# \deu+
                                                  List of user mappings
           Server      | User name  |                                  FDW Options
      gitlab_secondary | gitlab_geo | ("user" 'gitlab', password 'YOUR-PASSWORD-HERE')
     (1 row)

    Make sure the password is correct. You can test that logins work by running psql:

     # Connect to the tracking database as the `gitlab_geo` user
     sudo \
        -u git /opt/gitlab/embedded/bin/psql \
        -h /var/opt/gitlab/geo-postgresql \
        -p 5431 \
        -U gitlab_geo \
        -W \
        -d gitlabhq_geo_production

    If you need to correct the password, the following query shows how:

     ALTER USER MAPPING FOR gitlab_geo SERVER gitlab_secondary OPTIONS (SET password '<my_new_password>');

    If you change the user or password, you will also have to adjust the following settings in /etc/gitlab/gitlab.rb and run gitlab-ctl reconfigure:

    • gitlab_rails['db_username']
    • gitlab_rails['db_password']

    If you are using PgBouncer in front of the secondary database, be sure to update the following settings:

    • geo_postgresql['fdw_external_user']
    • geo_postgresql['fdw_external_password']

Manual reload of FDW schema

If you’re still unable to get FDW working, you may want to try a manual reload of the FDW schema. To manually reload the FDW schema:

  1. On the node running the Geo tracking database, enter the PostgreSQL console via the gitlab_geo user:

     sudo \
        -u git /opt/gitlab/embedded/bin/psql \
        -h /var/opt/gitlab/geo-postgresql \
        -p 5431 \
        -U gitlab_geo \
        -W \
        -d gitlabhq_geo_production

    Be sure to adjust the port and hostname for your configuration. You may be asked to enter a password.

  2. Reload the schema via:

     DROP SCHEMA IF EXISTS gitlab_secondary CASCADE;
     CREATE SCHEMA gitlab_secondary;
     GRANT USAGE ON FOREIGN SERVER gitlab_secondary TO gitlab_geo;
     IMPORT FOREIGN SCHEMA public FROM SERVER gitlab_secondary INTO gitlab_secondary;
  3. Test that queries work:

     SELECT * from information_schema.foreign_tables;
     SELECT * FROM gitlab_secondary.projects limit 1;