Troubleshooting Zoekt

  • Tier: Premium, Ultimate
  • Offering: GitLab.com, GitLab Self-Managed
  • Status: Beta

When working with Zoekt, you might encounter the following issues. For preliminary debugging:

Namespace is not indexed

When you enable the setting, new namespaces get indexed automatically. If a namespace is not indexed automatically, inspect the Sidekiq logs to see if the jobs are being processed. Search::Zoekt::SchedulingWorker is responsible for indexing namespaces.

In a Rails console session, you can check:

  • Namespaces where Zoekt is not enabled:

    Namespace.group_namespaces.root_namespaces_without_zoekt_enabled_namespace

  • The status of Zoekt indices:

    Search::Zoekt::Index.all.pluck(:state, :namespace_id)

To index a namespace manually, see set up indexing.

Error: SilentModeBlockedError

You might get a SilentModeBlockedError when you try to run exact code search. This issue occurs when Silent Mode is enabled on the GitLab instance.

To resolve this issue, ensure Silent Mode is disabled.

Error: connections to all backends failing

In application_json.log, you might get the following error:

connections to all backends failing; last error: UNKNOWN: ipv4:1.2.3.4:5678: Trying to connect an http1.x server

To resolve this issue, check if you’re using any proxies. If you are, set the IP address of the GitLab server to no_proxy:

gitlab_rails['env'] = {
  "http_proxy" => "http://proxy.domain.com:1234",
  "https_proxy" => "http://proxy.domain.com:1234",
  "no_proxy" => ".domain.com,IP_OF_GITLAB_INSTANCE,127.0.0.1,localhost"
}

proxy.domain.com:1234 is the domain of the proxy instance and the port. IP_OF_GITLAB_INSTANCE points to the public IP address of the GitLab instance.

You can get this information by running ip a and checking one of the following:

  • The IP address of the appropriate network interface
  • The public IP address of any load balancer you’re using

Verify Zoekt node connections

To verify that your Zoekt nodes are properly configured and connected, in a Rails console session:

  • Check the total number of configured Zoekt nodes:

    Search::Zoekt::Node.count

  • Check how many nodes are online:

    Search::Zoekt::Node.online.count

Alternatively, you can use the gitlab:zoekt:info Rake task.

If the number of online nodes is lower than the number of configured nodes or is zero when nodes are configured, you might have connectivity issues between GitLab and your Zoekt nodes.

Error: TaskRequest responded with [401]

In your Zoekt indexer logs, you might see TaskRequest responded with [401]. This error indicates that the Zoekt indexer is failing to authenticate with GitLab.

To resolve this issue, verify that gitlab-shell-secret is correctly configured and matches between your GitLab instance and Zoekt indexer. For example, the output of the following command must match gitlab-shell-secret in your gitlab.rb:

kubectl get secret gitlab-shell-secret -o jsonpath='{.data.secret}' -n your_zoekt_namespace | base64 -d

Error: missing selected ALPN property

When you use an external load balancer in front of the Zoekt gateway, you might see the following error in your GitLab logs:

rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: credentials: cannot check peer: missing selected ALPN property"

This error occurs when the load balancer does not support or advertise ALPN (Application-Layer Protocol Negotiation) with HTTP/2. Zoekt relies on gRPC for communication between nodes, which requires HTTP/2 support.

To resolve this issue, do one of the following:

  • Enable HTTP/2 support on your load balancer (recommended):

    1. Configure your load balancer to support and advertise HTTP/2 through ALPN:

      • For HAProxy, in your backend, ensure alpn h2,http/1.1 is configured.
      • For NGINX, in your server block, use:
        • In NGINX 1.25.1 and later, http2 on;.
        • In NGINX 1.25.0 and earlier, listen 443 ssl http2;.

    2. Verify HTTP/2 support:

      curl --verbose --http2 "https://your-zoekt-gateway-url/health" 2>&1 | grep ALPN

      You should see output similar to:

      * ALPN, server accepted to use h2

  • Use TLS passthrough:

    If your load balancer cannot support HTTP/2, configure the balancer for TLS passthrough. The Zoekt gateway can then handle TLS termination directly, which ensures proper ALPN negotiation. To use TLS passthrough, configure a valid TLS certificate on the Zoekt gateway:

    1. For Helm chart deployments, in your values.yaml, configure the certificate:

      gateway:
  tls:
    certificate:
      enabled: true
      secretName: zoekt-gateway-cert

    2. Configure your load balancer to pass through encrypted traffic without terminating TLS.