Log system

GitLab has an advanced log system where everything is logged, so you can analyze your instance using various system log files. In addition to system log files, GitLab Enterprise Edition provides Audit Events.

System log files are typically plain text in a standard log file format. This guide talks about how to read and use these system log files.

Read more about the log system and using the logs:

Log Levels

Each log message has an assigned log level that indicates its importance and verbosity. Each logger has an assigned minimum log level. A logger emits a log message only if its log level is equal to or above the minimum log level.

The following log levels are supported:

Level Name
0 DEBUG
1 INFO
2 WARN
3 ERROR
4 FATAL
5 UNKNOWN

GitLab loggers emit all log messages because they are set to DEBUG by default.

Override default log level

You can override the minimum log level for GitLab loggers using the GITLAB_LOG_LEVEL environment variable. Valid values are either a value of 0 to 5, or the name of the log level.

Example:

GITLAB_LOG_LEVEL=info

For some services, other log levels are in place that are not affected by this setting. Some of these services have their own environment variables to override the log level. For example:

Service Log level Environment variable
GitLab API INFO  
GitLab Cleanup INFO DEBUG
GitLab Doctor INFO VERBOSE
GitLab Export INFO EXPORT_DEBUG
GitLab Geo INFO  
GitLab Import INFO IMPORT_DEBUG
GitLab QA Runtime INFO QA_LOG_LEVEL
Google APIs INFO  
Rack Timeout ERROR  
Sidekiq (server) INFO  
Snowplow Tracker FATAL  
gRPC Client (Gitaly) WARN GRPC_LOG_LEVEL

Log Rotation

The logs for a given service may be managed and rotated by:

  • logrotate
  • svlogd (runit’s service logging daemon)
  • logrotate and svlogd
  • Or not at all

The following table includes information about what’s responsible for managing and rotating logs for the included services. Logs managed by svlogd are written to a file called current. The logrotate service built into GitLab manages all logs except those captured by runit.

Log type Managed by logrotate Managed by svlogd/runit
Alertmanager Logs No Yes
Crond Logs No Yes
Gitaly Yes Yes
GitLab Exporter for Omnibus No Yes
GitLab Pages Logs Yes Yes
GitLab Rails Yes No
GitLab Shell Logs Yes No
Grafana Logs No Yes
LogRotate Logs No Yes
Mailroom Yes Yes
NGINX Yes Yes
PostgreSQL Logs No Yes
Praefect Logs Yes Yes
Prometheus Logs No Yes
Puma Yes Yes
Redis Logs No Yes
Registry Logs No Yes
Workhorse Logs Yes Yes

production_json.log

Depending on your installation method, this file is located at:

  • Omnibus GitLab: /var/log/gitlab/gitlab-rails/production_json.log
  • Installations from source: /home/git/gitlab/log/production_json.log

It contains a structured log for Rails controller requests received from GitLab, thanks to Lograge. Requests from the API are logged to a separate file in api_json.log.

Each line contains JSON that can be ingested by services like Elasticsearch and Splunk. Line breaks were added to examples for legibility:

{
  "method":"GET",
  "path":"/gitlab/gitlab-foss/issues/1234",
  "format":"html",
  "controller":"Projects::IssuesController",
  "action":"show",
  "status":200,
  "time":"2017-08-08T20:15:54.821Z",
  "params":[{"key":"param_key","value":"param_value"}],
  "remote_ip":"18.245.0.1",
  "user_id":1,
  "username":"admin",
  "queue_duration_s":0.0,
  "gitaly_calls":16,
  "gitaly_duration_s":<