ClickHouse integration guidelines
- Status: Experiment
This feature is an experiment.
Instructions about how to set up integration between GitLab and ClickHouse database.
Setup
To set up ClickHouse as the GitLab data storage:
- Run ClickHouse Cluster and configure database.
- Configure GitLab connection to ClickHouse.
- Run ClickHouse migrations.
Run and configure ClickHouse
The most straightforward way to run ClickHouse is with ClickHouse Cloud. You can also run ClickHouse on your own server. Refer to the ClickHouse documentation regarding recommendations for GitLab Self-Managed.
When you run ClickHouse on a hosted server, various data points might impact the resource consumption, like the number of builds that run on your instance each month, the selected hardware, the data center choice to host ClickHouse, and more. Regardless, the cost should not be significant.
ClickHouse is a secondary data store for GitLab. Only specific data is stored in ClickHouse for analytics purposes.
To create necessary user and database objects:
Generate a secure password and save it.
Sign in to the ClickHouse SQL console.
Execute the following command. Replace
PASSWORD_HERE
with the generated password.CREATE DATABASE gitlab_clickhouse_main_production; CREATE USER gitlab IDENTIFIED WITH sha256_password BY 'PASSWORD_HERE'; CREATE ROLE gitlab_app; GRANT SELECT, INSERT, ALTER, CREATE, UPDATE, DROP, TRUNCATE, OPTIMIZE ON gitlab_clickhouse_main_production.* TO gitlab_app; GRANT SELECT ON information_schema.* TO gitlab_app; GRANT gitlab_app TO gitlab;
Configure the GitLab connection to ClickHouse
To provide GitLab with ClickHouse credentials:
Edit
/etc/gitlab/gitlab.rb
:gitlab_rails['clickhouse_databases']['main']['database'] = 'gitlab_clickhouse_main_production' gitlab_rails['clickhouse_databases']['main']['url'] = 'https://example.com/path' gitlab_rails['clickhouse_databases']['main']['username'] = 'gitlab' gitlab_rails['clickhouse_databases']['main']['password'] = 'PASSWORD_HERE' # replace with the actual password
Save the file and reconfigure GitLab:
sudo gitlab-ctl reconfigure
Save the ClickHouse password as a Kubernetes Secret:
kubectl create secret generic gitlab-clickhouse-password --from-literal="main_password=PASSWORD_HERE"
Export the Helm values:
helm get values gitlab > gitlab_values.yaml
Edit
gitlab_values.yaml
:global: clickhouse: enabled: true main: username: default password: secret: gitlab-clickhouse-password key: main_password database: gitlab_clickhouse_main_production url: 'http://example.com'
Save the file and apply the new values:
helm upgrade -f gitlab_values.yaml gitlab gitlab/gitlab
To verify that your connection is set up successfully:
Sign in to Rails console
Execute the following:
ClickHouse::Client.select('SELECT 1', :main)
If successful, the command returns
[{"1"=>1}]
Run ClickHouse migrations
To create the required database objects execute:
sudo gitlab-rake gitlab:clickhouse:migrate
Enable ClickHouse for Analytics
Now that your GitLab instance is connected to ClickHouse, you can enable features to use ClickHouse by enabling ClickHouse for Analytics.
Docs
Edit this page to fix an error or add an improvement in a merge request.
Create an issue to suggest an improvement to this page.
Product
Create an issue if there's something you don't like about this feature.
Propose functionality by submitting a feature request.
Feature availability and product trials
View pricing to see all GitLab tiers and features, or to upgrade.
Try GitLab for free with access to all features for 30 days.
Get help
If you didn't find what you were looking for, search the docs.
If you want help with something specific and could use community support, post on the GitLab forum.
For problems setting up or using this feature (depending on your GitLab subscription).
Request support