Restructure As code and developer resources (#113845)
This commit is contained in:
Jack Baldry
@@ -0,0 +1,357 @@
|
||||
---
|
||||
description: Learn how to create Grafana IRM integrations, escalation policies, and on-call schedules in Grafana Cloud using Terraform
|
||||
keywords:
|
||||
- Infrastructure as Code
|
||||
- Quickstart
|
||||
- Grafana Cloud
|
||||
- Terraform
|
||||
- Grafana Cloud IRM
|
||||
- OnCall
|
||||
title: Manage Grafana IRM in Grafana Cloud using Terraform
|
||||
weight: 120
|
||||
canonical: https://grafana.com/docs/grafana/latest/as-code/infrastructure-as-code/terraform/terraform-oncall/
|
||||
---
|
||||
|
||||
# Manage Grafana IRM in Grafana Cloud using Terraform
|
||||
|
||||
Learn how to use Terraform to manage [Grafana IRM](https://grafana.com/docs/grafana-cloud/alerting-and-irm/irm/) resources.
|
||||
This guide shows you how to connect an integration, configure escalation policies, and add on-call schedules using Terraform.
|
||||
|
||||
To illustrate the use of IRM across multiple teams, this guide features examples with two teams: `Devs` and `SREs`.
|
||||
Additionally, it includes the necessary steps to configure Slack for IRM.
|
||||
|
||||
{{< admonition type="note" >}}
|
||||
Grafana IRM supports Terraform-based configuration for a limited set of resources, primarily those related to OnCall functionality.
|
||||
These resources use the `grafana_oncall_` naming convention in Terraform. Additional IRM components are not yet configurable via Terraform.
|
||||
{{< /admonition >}}
|
||||
|
||||
## Before you begin
|
||||
|
||||
Before you begin, you should have the following:
|
||||
|
||||
- A Grafana Cloud account, as shown in [Get started](https://grafana.com/docs/grafana-cloud/get-started/)
|
||||
- [Terraform](https://www.terraform.io/downloads) installed on your machine
|
||||
- Administrator permissions in your Grafana instance
|
||||
- (Optional) Administrator permissions in your Slack workspace, if you plan to integrate Slack with Grafana IRM
|
||||
|
||||
{{< admonition type="note" >}}
|
||||
All of the following Terraform configuration files should be saved in the same directory.
|
||||
{{< /admonition >}}
|
||||
|
||||
## Connect Slack to Grafana IRM
|
||||
|
||||
Before including Slack settings in your Terraform setup, you must first configure the Slack integration with Grafana IRM.
|
||||
|
||||
To connect your Slack workspace to Grafana IRM, refer to the [Slack integration for Grafana IRM](https://grafana.com/docs/grafana-cloud/alerting-and-irm/irm/configure/integrations/irm-slack/) documentation.
|
||||
|
||||
## Configure the Grafana provider
|
||||
|
||||
This Terraform configuration sets up the [Grafana provider](https://registry.terraform.io/providers/grafana/grafana/latest/docs) to provide necessary authentication when managing resources for Grafana IRM.
|
||||
|
||||
You can reuse a similar setup to the one described in [Creating and managing a Grafana Cloud stack using Terraform](../terraform-cloud-stack/) to set up a service account and a token.
|
||||
|
||||
1. Create a Service account and token in Grafana. To create a new one, refer to [Service account tokens](https://grafana.com/docs/grafana/latest/administration/service-accounts/#service-account-tokens).
|
||||
|
||||
1. Create a file named `main.tf` and add the following:
|
||||
|
||||
```terraform
|
||||
terraform {
|
||||
required_providers {
|
||||
grafana = {
|
||||
source = "grafana/grafana"
|
||||
version = ">= 3.15.3"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
provider "grafana" {
|
||||
alias = "oncall"
|
||||
|
||||
url = "<Stack-URL>"
|
||||
auth = "<Service-account-token>"
|
||||
oncall_url = "<OnCall-URL>"
|
||||
}
|
||||
```
|
||||
|
||||
1. Replace the following field values:
|
||||
- `<Stack-URL>` with the URL of your Grafana stack
|
||||
- `<Service-account-token>` with the service account token that you created
|
||||
- `<OnCall-URL>` with the API URL found on the **Admin & API** tab of the IRM **Settings** page
|
||||
|
||||
{{< admonition type="note" >}}
|
||||
If the service account has the right permissions, this provider setup also allows you to manage other Grafana resources.
|
||||
{{< /admonition >}}
|
||||
|
||||
### Authentication via OnCall API tokens (deprecated)
|
||||
|
||||
OnCall API tokens are being deprecated.
|
||||
While existing tokens will continue to work, we recommend using
|
||||
[Grafana Cloud service account tokens](https://grafana.com/docs/grafana-cloud/security-and-account-management/authentication-and-permissions/service-accounts/) for all new API authentication.
|
||||
|
||||
{{< collapse title="Authentication via OnCall API tokens" >}}
|
||||
To use an existing OnCall API token:
|
||||
|
||||
1. Log into your Grafana Cloud instance
|
||||
1. Select **Alerts & IRM** > **IRM**
|
||||
1. Click **Settings**, and then select **Admin & API**
|
||||
1. Locate the **Grafana IRM API** section
|
||||
1. View, copy or revoke existing **OnCall API tokens**
|
||||
|
||||
1. Create a file named `main.tf` and add the following:
|
||||
|
||||
```terraform
|
||||
terraform {
|
||||
required_providers {
|
||||
grafana = {
|
||||
source = "grafana/grafana"
|
||||
version = ">= 2.9.0"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
provider "grafana" {
|
||||
alias = "oncall"
|
||||
|
||||
oncall_access_token = "<OnCall-API-Token>"
|
||||
oncall_url = "<OnCall-URL>"
|
||||
}
|
||||
```
|
||||
|
||||
1. Replace the following field values:
|
||||
- `<OnCall-API-Token>` with your existing OnCall API Token
|
||||
- `<OnCall-URL>` with the API URL found on the **Admin & API** tab of the IRM **Settings** page
|
||||
{{< /collapse >}}
|
||||
|
||||
## Add on-call schedules
|
||||
|
||||
This Terraform configuration sets up two on-call schedules, `SREs` and `Devs`, using the [`grafana_oncall_schedule` resource](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/oncall_schedule) to define the schedules within Grafana IRM.
|
||||
Additionally, this configuration includes Slack channels to receive notifications for the on-call schedules of each team.
|
||||
|
||||
To learn more about managing on-call schedules, refer to the [On-call schedules documentation](https://grafana.com/docs/grafana-cloud/alerting-and-irm/irm/manage/on-call-schedules/).
|
||||
|
||||
1. Create two new calendars in your calendar service, one for `Devs` and one for `SREs`
|
||||
|
||||
1. Locate and save the secret iCal URLs.
|
||||
For example, in a Google calendar, these URLs can be found in **Settings > Settings for my calendars > Integrate calendar**
|
||||
|
||||
1. Create a file named `schedule.tf` and add the following:
|
||||
|
||||
```terraform
|
||||
# Name of the Slack channel to notify about on-call schedules for Devs
|
||||
data "grafana_oncall_slack_channel" "Devs" {
|
||||
provider = grafana.oncall
|
||||
|
||||
name = "<Devs-channel-name>"
|
||||
}
|
||||
|
||||
# Name of the Slack channel to notify about on-call schedules for SREs
|
||||
data "grafana_oncall_slack_channel" "SREs" {
|
||||
provider = grafana.oncall
|
||||
|
||||
name = "<SREs-channel-name>"
|
||||
}
|
||||
|
||||
resource "grafana_oncall_schedule" "schedule_Devs" {
|
||||
provider = grafana.oncall
|
||||
|
||||
name = "Devs"
|
||||
type = "ical"
|
||||
ical_url_primary = "<secret-iCal-URL-for-devs-calendar>"
|
||||
slack {
|
||||
channel_id = data.grafana_oncall_slack_channel.Devs.slack_id
|
||||
}
|
||||
}
|
||||
|
||||
resource "grafana_oncall_schedule" "schedule_SREs" {
|
||||
provider = grafana.oncall
|
||||
|
||||
name = "SREs"
|
||||
type = "ical"
|
||||
ical_url_primary = "<secret-iCal-URL-for-SREs-calendar>"
|
||||
slack {
|
||||
channel_id = data.grafana_oncall_slack_channel.SREs.slack_id
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
1. Replace the following field values:
|
||||
- `<Devs-channel-name>` with name of the Slack channel to notify about on-call schedules for `Devs`
|
||||
- `<SREs-channel-name>` with name of the Slack channel to notify about on-call schedules for `SREs`
|
||||
- `<secret-iCal-URL-for-devs-calendar>` with the secret iCal URL created in the first step for `Devs` Calendar
|
||||
- `<secret-iCal-URL-for-SREs-calendar>` with the secret iCal URL created in the first step for `SREs` Calendar
|
||||
|
||||
## Add escalation chains
|
||||
|
||||
This Terraform configuration creates two escalation chains named `SREs` and `Devs` in Grafana IRM using the [`grafana_oncall_escalation_chain` (Resource)](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/oncall_escalation_chain).
|
||||
The configuration also adds the following three steps to each escalation chain using the [`grafana_oncall_escalation` (Resource)](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/oncall_escalation):
|
||||
|
||||
- Notify users from on-call schedule
|
||||
- Wait for 5 minutes
|
||||
- Notify default Slack channel
|
||||
|
||||
1. Create a file named `escalation-devs.tf` and add the following:
|
||||
|
||||
```terraform
|
||||
resource "grafana_oncall_escalation_chain" "Devs" {
|
||||
provider = grafana.oncall
|
||||
|
||||
name = "Devs"
|
||||
}
|
||||
|
||||
// Notify users from on-call schedule
|
||||
resource "grafana_oncall_escalation" "notify_schedule_step_Devs" {
|
||||
provider = grafana.oncall
|
||||
|
||||
escalation_chain_id = grafana_oncall_escalation_chain.Devs.id
|
||||
type = "notify_on_call_from_schedule"
|
||||
notify_on_call_from_schedule = grafana_oncall_schedule.schedule_Devs.id
|
||||
position = 0
|
||||
}
|
||||
|
||||
// Wait step for 5 Minutes
|
||||
resource "grafana_oncall_escalation" "wait_step_Devs" {
|
||||
provider = grafana.oncall
|
||||
|
||||
escalation_chain_id = grafana_oncall_escalation_chain.Devs.id
|
||||
type = "wait"
|
||||
duration = 300
|
||||
position = 1
|
||||
}
|
||||
|
||||
// Notify default Slack channel step
|
||||
resource "grafana_oncall_escalation" "notify_step_Devs" {
|
||||
provider = grafana.oncall
|
||||
|
||||
escalation_chain_id = grafana_oncall_escalation_chain.Devs.id
|
||||
type = "notify_whole_channel"
|
||||
important = true
|
||||
position = 2
|
||||
}
|
||||
```
|
||||
|
||||
2. Create a file named `escalation-sre.tf` and add the following:
|
||||
|
||||
```terraform
|
||||
resource "grafana_oncall_escalation_chain" "SREs" {
|
||||
provider = grafana.oncall
|
||||
|
||||
name = "SREs"
|
||||
}
|
||||
|
||||
// Notify users from on-call schedule
|
||||
resource "grafana_oncall_escalation" "notify_schedule_step_SREs" {
|
||||
provider = grafana.oncall
|
||||
|
||||
escalation_chain_id = grafana_oncall_escalation_chain.SREs.id
|
||||
type = "notify_on_call_from_schedule"
|
||||
notify_on_call_from_schedule = grafana_oncall_schedule.schedule_SREs.id
|
||||
position = 0
|
||||
}
|
||||
|
||||
// Wait step for 5 Minutes
|
||||
resource "grafana_oncall_escalation" "wait_step_SREs" {
|
||||
provider = grafana.oncall
|
||||
|
||||
escalation_chain_id = grafana_oncall_escalation_chain.SREs.id
|
||||
type = "wait"
|
||||
duration = 300
|
||||
position = 1
|
||||
}
|
||||
|
||||
// Notify default Slack channel step
|
||||
resource "grafana_oncall_escalation" "notify_step_SREs" {
|
||||
provider = grafana.oncall
|
||||
|
||||
escalation_chain_id = grafana_oncall_escalation_chain.SREs.id
|
||||
type = "notify_whole_channel"
|
||||
important = true
|
||||
position = 2
|
||||
}
|
||||
```
|
||||
|
||||
## Connect an integration to Grafana IRM
|
||||
|
||||
This Terraform configuration connects Alertmanager to Grafana IRM using the [`grafana_oncall_integration` (Resource)](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/oncall_integration).
|
||||
It also adds the `Devs` escalation chain as the default route for alerts.
|
||||
|
||||
1. Create a file named `integrations.tf` and add the following:
|
||||
|
||||
```terraform
|
||||
resource "grafana_oncall_integration" "AlertManager" {
|
||||
provider = grafana.oncall
|
||||
|
||||
name = "AlertManager"
|
||||
type = "alertmanager"
|
||||
default_route {
|
||||
escalation_chain_id = grafana_oncall_escalation_chain.Devs.id
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
1. To configure Alertmanager, refer to [Alertmanager integration for Grafana OnCall](https://grafana.com/docs/grafana-cloud/alerting-and-irm/oncall/integrations/alertmanager/)
|
||||
|
||||
## Set up a route to configure escalation behavior for alert group notifications
|
||||
|
||||
This Terraform configuration sets up a route to the Alertmanager integration using the `grafana_oncall_route` (Resource).
|
||||
This route ensures that notifications for alerts with `\"namespace\" *: *\"ops-.*\"` in the payload are escalated to the `SREs` escalation chain.
|
||||
|
||||
Create a file named `routes.tf` and add the following:
|
||||
|
||||
```terraform
|
||||
resource "grafana_oncall_route" "route_SREs" {
|
||||
provider = grafana.oncall
|
||||
|
||||
integration_id = grafana_oncall_integration.AlertManager.id
|
||||
escalation_chain_id = grafana_oncall_escalation_chain.SREs.id
|
||||
routing_regex = "\"namespace\" *: *\"ops-.*\""
|
||||
position = 0
|
||||
}
|
||||
```
|
||||
|
||||
## Apply the Terraform configuration
|
||||
|
||||
In a terminal, run the following commands from the directory where all of the configuration files are located.
|
||||
|
||||
1. Initialize a working directory containing Terraform configuration files.
|
||||
|
||||
```shell
|
||||
terraform init
|
||||
```
|
||||
|
||||
1. Preview the changes that Terraform will make.
|
||||
|
||||
```shell
|
||||
terraform plan
|
||||
```
|
||||
|
||||
1. Apply the configuration files.
|
||||
|
||||
```shell
|
||||
terraform apply
|
||||
```
|
||||
|
||||
## Validation
|
||||
|
||||
After you apply the changes in the Terraform configurations, you can verify the following:
|
||||
|
||||
- Two new Schedules named `Devs` and `SREs` are created in Grafana IRM:
|
||||
|
||||
|
||||
|
||||
- New Escalation chain named `SREs` is created in Grafana IRM:
|
||||
|
||||
|
||||
|
||||
- New Escalation chain named `Devs` is created in Grafana IRM:
|
||||
|
||||
|
||||
|
||||
- The Alertmanager integration is added and configured with escalation policies:
|
||||
|
||||
|
||||
|
||||
## Conclusion
|
||||
|
||||
In this guide, you learned how to use Terraform to manage Grafana IRM by connecting an integration, configuring escalation policies, and setting up on-call schedules.
|
||||
|
||||
To learn more about managing Grafana Cloud using Terraform, refer to [Grafana provider's documentation](https://registry.terraform.io/providers/grafana/grafana/latest/docs).
|
||||
Reference in New Issue
Block a user