Docs: Refactor alerting documentation (part 1) (#48664)

* Initial commit

* Moved files, ad fixed broken relrefs.

* Fixed other broken relrefs

* More changes.

* Fixing broken relrefs

* More changes.

* Fixed last of the broken links

* More re-org.

* Added aliases and some weight adjustments

* More aliases.

* Fix fundamentals topic.

* Fixed remaining metadata issues

* Ran prettier
This commit is contained in:
achatterjee-grafana
2022-05-04 08:22:11 -04:00
committed by GitHub
parent f00ffb190c
commit 082cfbdb06
49 changed files with 148 additions and 120 deletions
@@ -0,0 +1,27 @@
+++
title = " Alerting high availability"
description = "High availability"
keywords = ["grafana", "alerting", "tutorials", "ha", "high availability"]
aliases = ["/docs/grafana/latest/alerting/unified-alerting/high-availability/"]
weight = 450
+++
# About alerting high availability
The Grafana alerting system has two main components: a `Scheduler` and an internal `Alertmanager`. The `Scheduler` evaluates your [alert rules]({{< relref "../fundamentals/evaluate-grafana-alerts.md" >}}), while the internal Alertmanager manages **routing** and **grouping**.
When running Grafana alerting in high availability, the operational mode of the scheduler remains unaffected, and each Grafana instance evaluates all alerts. The operational change happens in the Alertmanager when it deduplicates alert notifications across Grafana instances.
{{< figure src="/static/img/docs/alerting/unified/high-availability-ua.png" class="docs-image--no-shadow" max-width= "750px" caption="High availability" >}}
The coordination between Grafana instances happens via [a Gossip protocol](https://en.wikipedia.org/wiki/Gossip_protocol). Alerts are not gossiped between instances and each scheduler delivers the same volume of alerts to each Alertmanager.
The two types of messages gossiped between Grafana instances are:
- Notification logs: Who (which instance) notified what (which alert).
- Silences: If an alert should fire or not.
The notification logs and silences are persisted in the database periodically and during a graceful Grafana shut down.
For configuration instructions, refer to [enable alerting high availability]({{< relref "./enable-alerting-ha.md" >}}).
@@ -0,0 +1,37 @@
+++
title = "Enable alerting high availability"
description = "Enable alerting high availability"
keywords = ["grafana", "alerting", "tutorials", "ha", "high availability"]
aliases = ["/docs/grafana/latest/alerting/unified-alerting/high-availability/"]
weight = 450
+++
# Enable alerting high availability
You can enable [alerting high availability]({{< relref "./_index.md" >}}) support by updating the Grafana configuration file. On Kubernetes, you can enable alerting high availability by updating the Kubernetes container definition.
## Update Grafana configuration file
### Before you begin
Since gossiping of notifications and silences uses both TCP and UDP port `9094`, ensure that each Grafana instance is able to accept incoming connections on these ports.
**To enable high availability support:**
1. In your custom configuration file ($WORKING_DIR/conf/custom.ini), go to the `[unified_alerting]` section.
2. Set `[ha_peers]` to the number of hosts for each Grafana instance in the cluster (using a format of host:port), for example, `ha_peers=10.0.0.5:9094,10.0.0.6:9094,10.0.0.7:9094`.
You must have at least one (1) Grafana instance added to the [`[ha_peer]` section.
3. Set `[ha_listen_address]` to the instance IP address using a format of `host:port` (or the [Pod's](https://kubernetes.io/docs/concepts/workloads/pods/) IP in the case of using Kubernetes).
By default, it is set to listen to all interfaces (`0.0.0.0`).
## Update Kubernetes container definition
If you are using Kubernetes, you can expose the pod IP [through an environment variable](https://kubernetes.io/docs/tasks/inject-data-application/environment-variable-expose-pod-information/) via the container definition such as:
```bash
env:
- name: POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
```