Incident management

Introduced in GitLab 13.0.

Alert Management enables developers to easily discover and view the alerts generated by their application. By surfacing alert information where the code is being developed, efficiency and awareness can be increased.

GitLab offers solutions for handling incidents in your applications and services, such as setting up Prometheus alerts, displaying metrics, and sending notifications. While no configuration is required to use the manual features of incident management, both automation and configuration of incident management are only available in GitLab Ultimate and GitLab.com Gold.

For users with at least Developer permissions, the Incident Management list is available at Operations > Incidents in your project’s sidebar. The list contains the following metrics:

Incident Management List

  • Incident - The description of the incident, which attempts to capture the most meaningful data.
  • Date created - How long ago the incident was created. This field uses the standard GitLab pattern of X time ago, but is supported by a granular date/time tooltip depending on the user’s locale.
  • Assignees - The user assigned to the incident.
  • Published - Whether or not an incident is published to the Status Page.

The Incident Management list displays incidents sorted by incident created date. (Introduced to GitLab core in 13.3).) To see if a column is sortable, point your mouse at the header. Sortable columns display an arrow next to the column name.

The Incident list supports a simple free text search, which filters on the Title and Incident fields.

To filter incidents by their status, click Open, Closed, or All above the incident list.

Note: Incidents share the Issues API.

Enable Alert Management

Note: You will need at least Maintainer permissions to enable the Alert Management feature.

There are several ways to accept alerts into your GitLab project, as outlined below. Enabling any of these methods will allow the Alerts list to display. After configuring alerts, visit Operations > Alerts in your project’s sidebar to view the list of alerts.

Opsgenie integration

A new way of monitoring Alerts via a GitLab integration is with Opsgenie.

Note: If you enable the Opsgenie integration, you cannot have other GitLab alert services, such as Generic Alerts or Prometheus alerts, active at the same time.

To enable Opsgenie integration:

  1. Sign in as a user with Maintainer or Owner permissions.
  2. Navigate to Operations > Alerts.
  3. In the Integrations select box, select Opsgenie.
  4. Click the Active toggle.
  5. In the API URL, enter the base URL for your Opsgenie integration, such as https://app.opsgenie.com/alert/list.
  6. Click Save changes.

After enabling the integration, navigate to the Alerts list page at Operations > Alerts, and click View alerts in Opsgenie.

Enable a Generic Alerts endpoint

GitLab provides the Generic Alerts endpoint so you can accept alerts from a third-party alerts service. See the instructions for toggling generic alerts to add this option. After configuring the endpoint, the Alerts list is enabled.

To populate the alerts with data, see Customizing the payload for requests to the alerts endpoint.

Enable GitLab-managed Prometheus alerts

You can install the GitLab-managed Prometheus application on your Kubernetes cluster. For more information, see Managed Prometheus on Kubernetes. When GitLab-managed Prometheus is installed, the Alerts list is also enabled.

To populate the alerts with data, see GitLab-Managed Prometheus instances.

Enable external Prometheus alerts

You can configure an externally-managed Prometheus instance to send alerts to GitLab. To set up this configuration, see the configuring Prometheus documentation. Activating the external Prometheus configuration also enables the Alerts list.

To populate the alerts with data, see External Prometheus instances.

Alert Management severity

Each level of alert contains a uniquely shaped and color-coded icon to help you identify the severity of a particular alert. These severity icons help you immediately identify which alerts you should prioritize investigating:

Alert Management Severity System

Alerts contain one of the following icons:

Severity Icon Color (hexadecimal)
Critical #8b2615
High #c0341d
Medium #fca429
Low #fdbc60
Info #418cd8
Unknown #bababa

Alert Management list

Note: You will need at least Developer permissions to view the Alert Management list.

You can find the Alert Management list at Operations > Alerts in your project’s sidebar. Each alert contains the following metrics:

Alert Management List

  • Severity - The current importance of a alert and how much attention it should receive.
  • Start time - How long ago the alert fired. This field uses the standard GitLab pattern of X time ago, but is supported by a granular date/time tooltip depending on the user’s locale.
  • Alert description - The description of the alert, which attempts to capture the most meaningful data.
  • Event count - The number of times that an alert has fired.
  • Issue - A link to the incident issue that has been created for the alert.
  • Status - The current status of the alert.

Alert Management list sorting

Introduced in GitLab 13.1.

The Alert Management list displays alerts sorted by start time, but you can change the sort order by clicking the headers in the Alert Management list.

To see if a column is sortable, point your mouse at the header. Sortable columns display an arrow next to the column name, as shown in this example:

Alert Management List Sorting

Searching alerts

Introduced in GitLab 13.1.

The alert list supports a simple free text search.

Alert List Search

This search filters on the following fields:

  • Title
  • Description
  • Monitoring tool
  • Service

Alert Management statuses

Each alert contains a status dropdown to indicate which alerts need investigation. Standard alert statuses include triggered, acknowledged, and resolved:

  • Triggered: No one has begun investigation.
  • Acknowledged: Someone is actively investigating the problem.
  • Resolved: No further work is required.

Alert Management details

Note: You will need at least Developer permissions to view Alert Management details.

Navigate to the Alert Management detail view by visiting the Alert Management list and selecting an Alert from the list.

Alert Management Detail Overview

Alert Management Full Details

Update an Alert’s status

The Alert Management detail view enables you to update the Alert Status. See Alert Management statuses for more details.

Create an Issue from an Alert

Introduced in GitLab 13.1.

The Alert Management detail view enables you to create an issue with a description automatically populated from an alert. To create the issue, click the Create Issue button. You can then view the issue from the alert by clicking the View Issue button.

Closing a GitLab issue associated with an alert changes the alert’s status to Resolved. See Alert Management statuses for more details about statuses.

Update an Alert’s assignee

Introduced in GitLab 13.1.

The Alert Management detail view allows users to update the Alert assignee.

In large teams, where there is shared ownership of an alert, it can be difficult to track who is investigating and working on it. The Alert Management detail view enables you to update the Alert assignee:

Note: GitLab currently only supports a single assignee per alert.
  1. To display the list of current alerts, click Operations > Alerts:

    Alert Management List View Assignee(s)

  2. Select your desired alert to display its Alert Management Details View:

    Alert Management Details View Assignee(s)

  3. If the right sidebar is not expanded, click Expand sidebar to expand it.
  4. In the right sidebar, locate the Assignee and click Edit. From the dropdown menu, select each user you want to assign to the alert. GitLab creates a To-Do list item for each user.

    Alert Management Details View Assignee(s)

To remove an assignee, click Edit next to the Assignee dropdown menu and deselect the user from the list of assignees, or click Unassigned.

Alert system notes

Introduced in GitLab 13.1.

When you take action on an alert, this is logged as a system note, which is visible in the Alert Details view. This gives you a linear timeline of the alert’s investigation and assignment history.

The following actions will result in a system note:

Alert Management Details View System Notes

Create a To-Do from an Alert

Introduced in GitLab 13.1.

You can manually create To-Do list items for yourself from the Alert details screen, and view them later on your To-Do List. To add a To-Do:

  1. To display the list of current alerts, click Operations > Alerts.
  2. Select your desired alert to display its Alert Management Details View.
  3. Click the Add a To-Do button in the right sidebar:

    Alert Management Details Add A To Do

Click the To-Do in the navigation bar to view your current To-Do list.

Alert Management Details Added to Do

View an Alert’s metrics data

Introduced in GitLab 13.2.

To view the metrics for an alert:

  1. Sign in as a user with Developer or higher permissions.
  2. Navigate to Operations > Alerts.
  3. Click the alert you want to view.
  4. Below the title of the alert, click the Metrics tab.

Alert Management Metrics View

For GitLab-managed Prometheus instances, metrics data is automatically available for the alert, making it easy to see surrounding behavior. See Managed Prometheus instances for information on setting up alerts.

For externally-managed Prometheus instances, you can configure your alerting rules to display a chart in the alert. See Embedding metrics based on alerts in incident issues for information on how to appropriately configure your alerting rules. See External Prometheus instances for information on setting up alerts for your self-managed Prometheus instance.

Use cases for assigning alerts

Consider a team formed by different sections of monitoring, collaborating on a single application. After an alert surfaces, it’s extremely important to route the alert to the team members who can address and resolve the alert.

Assigning Alerts to multiple assignees eases collaboration and delegation. All assignees are shown in your team’s work-flows, and all assignees receive notifications, simplifying communication and ownership of the alert.

After completing their portion of investigating or fixing the alert, users can unassign their account from the alert when their role is complete. The alerts status can be updated to reflect if the alert has been resolved.

Slack Notifications

Introduced in GitLab 13.1.

You can be alerted via a Slack message when a new alert has been received.

See the Slack Notifications Service docs for information on how to set this up.

Configure incidents

Introduced in GitLab Ultimate 11.11.

With Maintainer or higher permissions, you can enable or disable Incident Management features in the GitLab user interface to create issues when alerts are triggered:

  1. Navigate to Settings > Operations > Incidents and expand Incidents:

    Incident Management Settings

  2. For GitLab versions 11.11 and greater, you can select the Create an issue checkbox to create an issue based on your own issue templates. For more information, see Trigger actions from alerts .
  3. To create issues from alerts, select the template in the Issue Template select box.
  4. To send separate email notifications to users with Developer permissions, select Send a separate email notification to Developers.
  5. Click Save changes.

Appropriately configured alerts include an embedded chart for the query corresponding to the alert. You can also configure GitLab to close issues when you receive notification that the alert is resolved.

Notify developers of alerts

GitLab can react to the alerts triggered from your applications and services by creating issues and alerting developers through email. By default, GitLab sends these emails to owners and maintainers of the project. These emails contain details of the alert, and a link for more information.

To send separate email notifications to users with Developer permissions, see Configure incidents.

Create an incident manually

Moved to GitLab core in 13.3.

For users with at least Developer permissions, to create a Incident you can take any of the following actions:

  • Navigate to Operations > Incidents and click Create Incident.
  • Create a new issue using the incident template available when creating it.
  • Create a new issue and assign the incident label to it.

Incident List Create

Configure PagerDuty integration

Introduced in GitLab 13.3.

You can set up a webhook with PagerDuty to automatically create a GitLab issue for each PagerDuty incident. This configuration requires you to make changes in both PagerDuty and GitLab:

  1. Sign in as a user with Maintainer permissions.
  2. Navigate to Settings > Operations > Incidents and expand Incidents.
  3. Select the PagerDuty integration tab:

    PagerDuty incidents integration

  4. Activate the integration, and save the changes in GitLab.
  5. Copy the value of Webhook URL for use in a later step.
  6. Follow the steps described in the PagerDuty documentation to add the webhook URL to a PagerDuty webhook integration.

To confirm the integration is successful, trigger a test incident from PagerDuty to confirm that a GitLab issue is created from the incident.

Configure Prometheus alerts

You can set up Prometheus alerts in:

Prometheus alerts are created by the special Alert Bot user. You can’t remove this user, but it does not count toward your license limit.

Configure external generic alerts

GitLab can accept alerts from any source through a generic webhook receiver. When configuring the generic alerts integration, GitLab creates a unique endpoint which receives a JSON-formatted, customizable payload.

Embed metrics in incidents and issues

You can embed metrics anywhere GitLab Markdown is used, such as descriptions, comments on issues, and merge requests. Embedding metrics helps you share them when discussing incidents or performance issues. You can output the dashboard directly into any issue, merge request, epic, or any other Markdown text field in GitLab by copying and pasting the link to the metrics dashboard.

You can embed both GitLab-hosted metrics and Grafana metrics in incidents and issue templates.

Context menu

You can view more details about an embedded metrics panel from the context menu. To access the context menu, click the More actions dropdown box above the upper right corner of the panel. For a list of options, see Chart context menu.

View logs from metrics panel

Version history

Viewing logs from a metrics panel can be useful if you’re triaging an application incident and need to explore logs from across your application. These logs help you understand what is affecting your application’s performance and resolve any problems.

Integrate incidents with Slack

Slack slash commands allow you to control GitLab and view GitLab content without leaving Slack.

Learn how to set up Slack slash commands and how to use the available slash commands.

Integrate issues with Zoom

GitLab enables you to associate a Zoom meeting with an issue for synchronous communication during incident management. After starting a Zoom call for an incident, you can associate the conference call with an issue. Your team members can join the Zoom call without requesting a link.