- Log Levels
- Log Rotation
production_json.log
production.log
api_json.log
application.log
application_json.log
integrations_json.log
kubernetes.log
git_json.log
audit_json.log
- Sidekiq Logs
-
gitlab-shell.log
- Gitaly Logs
- Puma Logs
repocheck.log
importer.log
exporter.log
features_json.log
auth.log
graphql_json.log
migrations.log
mail_room_json.log
(default)- Reconfigure logs
sidekiq_exporter.log
andweb_exporter.log
database_load_balancing.log
elasticsearch.log
exceptions_json.log
service_measurement.log
geo.log
update_mirror_service_json.log
- Registry Logs
- NGINX Logs
- Pages Logs
- Mattermost Logs
- Workhorse Logs
- PostgreSQL Logs
- Prometheus Logs
- Redis Logs
- Alertmanager Logs
- Crond Logs
- Grafana Logs
- LogRotate Logs
- GitLab Monitor Logs
- GitLab Exporter
- GitLab agent server
- Praefect Logs
- Backup log
- Performance bar stats
- Gathering logs
Log system
GitLab has an advanced log system where everything is logged, so you can analyze your instance using various system log files. In addition to system log files, GitLab Enterprise Edition provides Audit Events.
System log files are typically plain text in a standard log file format. This guide talks about how to read and use these system log files.
Read more about the log system and using the logs:
- Customize logging on Omnibus GitLab installations including adjusting log retention, log forwarding, switching logs from JSON to plain text logging, and more.
- How to parse and analyze JSON logs.
Log Levels
Each log message has an assigned log level that indicates its importance and verbosity. Each logger has an assigned minimum log level. A logger emits a log message only if its log level is equal to or above the minimum log level.
The following log levels are supported:
Level | Name |
---|---|
0 | DEBUG |
1 | INFO |
2 | WARN |
3 | ERROR |
4 | FATAL |
5 | UNKNOWN |
GitLab loggers emit all log messages because they are set to DEBUG
by default.
Override default log level
You can override the minimum log level for GitLab loggers using the GITLAB_LOG_LEVEL
environment variable.
Valid values are either a value of 0
to 5
, or the name of the log level.
Example:
GITLAB_LOG_LEVEL=info
For some services, other log levels are in place that are not affected by this setting. Some of these services have their own environment variables to override the log level. For example:
Service | Log level | Environment variable |
---|---|---|
GitLab API | INFO |
|
GitLab Cleanup | INFO |
DEBUG |
GitLab Doctor | INFO |
VERBOSE |
GitLab Export | INFO |
EXPORT_DEBUG |
GitLab Geo | INFO |
|
GitLab Import | INFO |
IMPORT_DEBUG |
GitLab QA Runtime | INFO |
QA_LOG_LEVEL |
Google APIs | INFO |
|
Rack Timeout | ERROR |
|
Sidekiq (server) | INFO |
|
Snowplow Tracker | FATAL |
|
gRPC Client (Gitaly) | WARN |
GRPC_LOG_LEVEL |
Log Rotation
The logs for a given service may be managed and rotated by:
logrotate
-
svlogd
(runit
’s service logging daemon) -
logrotate
andsvlogd
- Or not at all
The following table includes information about what’s responsible for managing and rotating logs for
the included services. Logs
managed by svlogd
are written to a file called current
. The logrotate
service built into GitLab
manages all logs
except those captured by runit
.
Log type | Managed by logrotate | Managed by svlogd/runit |
---|---|---|
Alertmanager Logs | No | Yes |
Crond Logs | No | Yes |
Gitaly | Yes | Yes |
GitLab Exporter for Omnibus | No | Yes |
GitLab Pages Logs | Yes | Yes |
GitLab Rails | Yes | No |
GitLab Shell Logs | Yes | No |
Grafana Logs | No | Yes |
LogRotate Logs | No | Yes |
Mailroom | Yes | Yes |
NGINX | Yes | Yes |
PostgreSQL Logs | No | Yes |
Praefect Logs | Yes | Yes |
Prometheus Logs | No | Yes |
Puma | Yes | Yes |
Redis Logs | No | Yes |
Registry Logs | No | Yes |
Workhorse Logs | Yes | Yes |
production_json.log
Depending on your installation method, this file is located at:
- Omnibus GitLab:
/var/log/gitlab/gitlab-rails/production_json.log
- Installations from source:
/home/git/gitlab/log/production_json.log
It contains a structured log for Rails controller requests received from
GitLab, thanks to Lograge.
Requests from the API are logged to a separate file in api_json.log
.
Each line contains JSON that can be ingested by services like Elasticsearch and Splunk. Line breaks were added to examples for legibility:
{
"method":"GET",
"path":"/gitlab/gitlab-foss/issues/1234",
"format":"html",
"controller":"Projects::IssuesController",
"action":"show",
"status":200,
"time":"2017-08-08T20:15:54.821Z",
"params":[{"key":"param_key","value":"param_value"}],
"remote_ip":"18.245.0.1",
"user_id":1,
"username":"admin",
"queue_duration_s":0.0,
"gitaly_calls":16,
"gitaly_duration_s":<