- Managed Prometheus instances
- External Prometheus instances
- Trigger actions from alerts
Moved from Ultimate to Core in GitLab 12.10.
After configuring metrics for your CI/CD environment, you can set up alerting for Prometheus metrics depending on the location of your instances, and trigger actions from alerts to notify your team when environment performance falls outside of the boundaries you set.
- In your project, navigate to Operations > Metrics,
- Identify the metric you want to create the alert for, and click the ellipsis icon in the top right corner of the metric.
- Choose Alerts.
- Set threshold and operator.
- (Optional) Add a Runbook URL.
- Click Add to save and activate the alert.
To remove the alert, click back on the alert icon for the desired metric, and click Delete.
Runbook URLs introduced in GitLab 13.3.
When creating alerts from the metrics dashboard for managed Prometheus instances, you can also link a runbook. When the alert triggers, the chart context menu on the metrics chart links to the runbook, making it easy for you to locate and access the correct runbook as soon as the alert fires:
For manually configured Prometheus servers, GitLab provides a notify endpoint for use with Prometheus webhooks. If you have manual configuration enabled, an Alerts section is added to Settings > Integrations > Prometheus. This section contains the needed URL and Authorization Key. The Reset Key button invalidates the key and generates a new one.
To send GitLab alert notifications, copy the URL and Authorization Key into the
section of your Prometheus Alertmanager configuration:
receivers: name: gitlab webhook_configs: - http_config: bearer_token: 9e1cbfcd546896a9ea8be557caf13a76 send_resolved: true url: http://192.168.178.31:3001/root/manual_prometheus/prometheus/alerts/notify.json # Rest of configuration omitted # ...
For GitLab to associate your alerts with an environment,
you must configure a
gitlab_environment_name label on the alerts you set up in
Prometheus. The value of this should match the name of your environment in GitLab.
In GitLab versions 13.1 and greater, you can configure your manually configured Prometheus server to use the Generic alerts integration.
Alerts can be used to trigger actions, like opening an issue automatically
(disabled by default since
13.1). To configure the actions:
- Navigate to your project’s Settings > Operations > Incidents.
- Enable the option to create issues.
- Choose the issue template to create the issue from.
- Optionally, select whether to send an email notification to the developers of the project.
- Click Save changes.
After enabling, GitLab automatically opens an issue when an alert is triggered containing
values extracted from the
alerts field in webhook payload:
- Issue author:
GitLab Alert Bot
- Issue title: Extracted from the alert payload fields
Summary: A list of properties from the alert’s payload.
starts_at: Alert start time from the payload’s
full_query: Alert query extracted from the payload’s
- Optional list of attached annotations extracted from
- Alert GFM: GitLab Flavored Markdown from the payload’s
When GitLab receives a Recovery Alert, it closes the associated issue. This action is recorded as a system message on the issue indicating that it was closed automatically by the GitLab Alert bot.
To further customize the issue, you can add labels, mentions, or any other supported
quick action in the selected issue template,
which applies to all incidents. To limit quick actions or other information to
only specific types of alerts, use the
Since version 12.2,
GitLab tags each incident issue with the
incident label automatically. If the label
does not yet exist, it is also created automatically.
If the metric exceeds the threshold of the alert for over 5 minutes, GitLab sends an email to all Maintainers and Owners of the project.