Advanced search development tips

Kibana

Use Kibana to interact with your Elasticsearch cluster.

See the download instructions.

Viewing index status

Run

bundle exec rake gitlab:elastic:info

to see the status and information about your cluster.

Creating all indices from scratch and populating with local data

Option 1: Rake task

Run

bundle exec rake gitlab:elastic:index

which triggers Search::Elastic::TriggerIndexingWorker to run async.

Run

Elastic::ProcessInitialBookkeepingService.new.execute

until it shows [0, 0] meaning there are no more refs in the queue.

Option 2: manual

Manually execute the steps in Search::Elastic::TriggerIndexingWorker.

Sometimes Sidekiq doesn’t pick up jobs correctly, so you might need to restart Sidekiq or if you prefer to run through the steps in a Rails console:

task_executor_service = Search::RakeTaskExecutorService.new(logger: ::Gitlab::Elasticsearch::Logger.build)
task_executor_service.execute(:recreate_index)
task_executor_service.execute(:clear_index_status)
task_executor_service.execute(:clear_reindex_status)
task_executor_service.execute(:resume_indexing)
task_executor_service.execute(:index_namespaces)
task_executor_service.execute(:index_projects)
task_executor_service.execute(:index_snippets)
task_executor_service.execute(:index_users)

Run

Elastic::ProcessInitialBookkeepingService.new.execute

until it shows [0, 0] meaning there are no more refs in the queue.

Option 3: reindexing task

First delete the existing index, then create a ReindexingTask for the index you want to target. This creates a new index based on the current configuration, then copies the data over.

Search::Elastic::ReindexingTask.create!(targets: %w[MergeRequest])

Run

ElasticClusterReindexingCronWorker.new.perform

On repeat until

Search::Elastic::ReindexingTask.last.state

is success.

Index data

To add and index database records, call the track! method and execute the book keeper:

Elastic::ProcessBookkeepingService.track!(MergeRequest.first)
Elastic::ProcessBookkeepingService.track!(*MergeRequest.all)

Elastic::ProcessBookkeepingService.new.execute

Dependent association index updates

You can use elastic_index_dependant_association to automatically update associated records in the index when specific fields change. For example, to reindex all work items when a project’s visibility_level changes

  elastic_index_dependant_association :work_items, on_change: :visibility_level, depends_on_finished_migration: :add_mapping_migration

The depends_on_finished_migration parameter is optional and ensures the update only occurs after the specified advanced search migration has completed (such as a migration that added the necessary field to the mapping).

Testing

Advanced search migrations

Testing a migration that changes a mapping of an index

1. Make sure the index doesn’t already have the changes applied. Remember the migration cron worker runs in the background so it’s possible the migration was already applied.

   • Optional. In GitLab 18.0 and later, to disable the migration worker, run the following commands:

       settings = ApplicationSetting.last # Ensure this setting does not return `nil`
       settings.elastic_migration_worker_enabled = false
       settings.save!

   • See if the migration is pending: ::Elastic::DataMigrationService.pending_migrations.

   • Check that the migration is not completed: Elastic::DataMigrationService.pending_migrations.first.completed?.

   • Make sure the mappings aren’t already applied

     • either by checking in Kibana GET gitlab-development-some-index/_mapping
     • or sending a curl request curl "http://localhost:9200/gitlab-development-some-index/_mappings" | jq
2. Tail the logs to see logged messages: tail -f log/elasticsearch.log.
3. Execute the migration in one of the following ways:
   • Run the Elastic::MigrationWorker.new.perform migration worker. In GitLab 18.0 and later, the elastic_migration_worker_enabled application setting must be enabled.
   • Use pending migrations: ::Elastic::DataMigrationService.pending_migrations.first.migrate.
   • Use the version: Elastic::DataMigrationService[20250220214819].migrate, replacing the version with the migration version.
4. View the status of the migration.
   • View the migration record in Kibana: GET gitlab-development-migrations/_doc/20250220214819 (changing the version). This contains information like when it started and what the status is.
   • See if the mappings are changed in Kibana: GET gitlab-development-some-index/_mapping.

Analyze query changes

Developers can use the GitLab staging rails console to help in code reviews to compare before and after queries.

On the Rails console we can use the Gitlab::Search::Client to construct the queries.

An example query using the helper looks like:

  Gitlab::Search::Client.new.search(
    index: 'gitlab-production-vulnerabilities',
    routing: 'group_110', # data is distributed across shards and the query builder passes routing information.
    body: {
      query: {
        term: { vulnerability_id: 4356 }
      }
    }
  )

Vulnerability advanced search finders

The vulnerability advanced search finders in ee/lib/search/advanced_finders/security/vulnerability/ follow a different pattern from the generic global search framework.

Search level

The generic Search::Level class (in lib/search/level.rb) recognizes three levels: project, group, and global. Search::AdvancedFinders::Security::Vulnerability::BaseFinder introduces a fourth level, organization, which is set when the vulnerable object is an Organizations::Organization instance. The organization level is not recognized by Search::Level and is not used anywhere in the generic global search framework.

Authorization and scoping

The generic framework uses Filters.by_search_level_and_membership, by_user_accessible_namespaces, and search_level_filter to enforce visibility and membership rules. VulnerabilityQueryBuilder does not call any of these methods. The only method it uses from Search::Elastic::Filters is by_traversal_ids, which applies a prefix filter on the traversal_ids field and does not read search_level.

Avoiding those methods is intentional: Search::Elastic::Filters::ALLOWED_SEARCH_LEVELS is %i[global group project], and fetch_search_level! raises ArgumentError, 'search_level invalid' for any value outside that list. Wiring any Filters method that calls fetch_search_level! into VulnerabilityQueryBuilder would break at runtime for organization-level queries.

Scoping is handled by the finder and VulnerabilityFilters:

• For project and group levels, traversal_ids is set to the namespace ancestry prefix of the vulnerable object.
• For the organization level, traversal_ids is nil (no prefix filter). Scoping is provided by by_organization_id in VulnerabilityFilters, which is gated on the backfill_organization_id_in_vulnerabilities migration. Until that migration finishes, the finder sets project_id to a sentinel value ([0]) so the query returns no results rather than running unscoped.

Note that VulnerabilityFilters.by_archived_projects does read search_level to skip the archived filter at the :project level, so scoping is not handled entirely by the finder alone.

Shard routing

es_search_options passes root_ancestor_ids for Elasticsearch shard routing:

• For project and group levels, this is the single root namespace ID of the vulnerable object.
• For the organization level, this is the list of top-level namespace IDs belonging to the organization (capped at ES_ROUTING_MAX_COUNT + 1. Beyond that limit, routing is dropped and all shards are queried).

Sibling finders

The sibling finders (CountBySeverityFinder, CountByAgeFinder, CountOverTimeFinder, IdentifierNamesFinder, RiskScoresFinder, and TopCwesFinder) all inherit from BaseFinder and reuse search_params and es_search_options unchanged. VulnerabilitySorts and VulnerabilityAggregations do not reference search_level either, so the organization level has no effect on sorting or aggregation logic.

SearchFinder also inherits from BaseFinder but is the primary finder rather than a sibling, so it is not listed above.