public/rancher-docs
1
0
Fork 0
mirror of https://github.com/rancher/rancher-docs.git synced 2026-09-23 03:28:07 +00:00
Code Issues Packages Projects Releases Wiki Activity

Alert enhancement

Browse Source 
Changes:
Alert group and alert
Use wechat as a Notifier
expression based alert

Issue:
https://github.com/rancher/docs/issues/1166
This commit is contained in:
Aiwantaozi
2019-03-22 10:04:25 -07:00
committed by Denise Schannon
parent f81579f4ec
commit 13d88761d5
2 changed files with 172 additions and 22 deletions
Download Patch File Download Diff File Expand all files Collapse all files
content/rancher/v2.x/en/tools/_index.md
+1
View File
@@ -43,6 +43,7 @@ _Notifiers_ are services that inform you of alert events. You can configure noti
- Email: Choose email recipients for alert notifications.
- PagerDuty: Route notifications to staff by phone, SMS, or personal email.
- Webhooks: Update a webpage with alert notifications.
- Wechat: Send alert notifications to your Wechat at Work contacts.
For more information, see [Notifiers]({{< baseurl >}}/rancher/v2.x/en/tools/notifiers-and-alerts/#notifiers).
content/rancher/v2.x/en/tools/notifiers-and-alerts/_index.md
+171 -22
View File
@@ -21,6 +21,7 @@ Rancher integrates with a variety of popular IT services, including:
- **Email**: Choose email recipients for alert notifications.
- **PagerDuty**: Route notifications to staff by phone, SMS, or personal email.
- **WebHooks**: Update a webpage with alert notifications.
- **Wechat**: Send alert notifications to your Wechat at Work contacts.
<br/>
<br/>
@@ -28,7 +29,7 @@ Rancher integrates with a variety of popular IT services, including:
Set up a notifier so that you can begin configuring and sending alerts.
1. From the **Global View**, open the cluster that you want to add a notifier to.
1. From the **Global View**, open the cluster that you want to add a notifier.
1. From the main menu, select **Tools > Notifiers**. Then click **Add Notifier**.
@@ -45,7 +46,7 @@ Set up a notifier so that you can begin configuring and sending alerts.
{{% accordion id="email" label="Email" %}}
1. Enter a **Name** for the notifier.
1. In the **Sender** field, enter an email address available on your mail server that you want to send the notification.
1. In the **Host** field, enter the IP address or host name for your SMTP server. Example: `smtp.email.com`
1. In the **Host** field, enter the IP address or hostname for your SMTP server. Example: `smtp.email.com`
1. In the **Port** field, enter the port used for email. Typically, TLS uses `587` and SSL uses `465`. If you're using TLS, make sure **Use TLS** is selected.
1. Enter a **Username** and **Password** that authenticate with the SMTP server.
1. In the **Default Recipient** field, enter the email address that you want to receive the notification.
@@ -64,6 +65,13 @@ Set up a notifier so that you can begin configuring and sending alerts.
1. Enter your webhook **URL**.
1. Click **Test**. If the test is successful, the URL you're configuring as a notifier outputs `Webhook setting validated`.
{{% /accordion %}}
{{% accordion id="Wechat" label="Wechat" %}}
1. Enter a **Name** for the notifier.
1. In the **Corporation ID** field, enter corporation id of your corporation, you could get it from [corporation info](https://work.weixin.qq.com/wework_admin/frame#profile).
1. From Wechat, create an application in the Wechat at Work [Application page](https://work.weixin.qq.com/wework_admin/frame#apps). Enter the **Application Agent ID** and **Application Secret** for this application.
1. In the **Recipient Type** field, select one of the recipient types.
1. In the **Default Recipient** field, enter the party name, tag name or user account that you want to receive the notification. For contact information, see [Wechat Contacts](https://work.weixin.qq.com/wework_admin/frame#contacts)
{{% /accordion %}}
1. Click **Add** to complete adding the notifier.
@@ -99,12 +107,14 @@ At the [cluster level](#adding-cluster-alerts), Rancher monitors components in y
- The state of your nodes.
- The system services that manage your Kubernetes cluster.
- The resource events from specific system services.
- The Prometheus expression cross the thresholds
At the [project level](#adding-project-alerts), Rancher monitors specific deployments and sends alerts for:
* Deployment availability
* Workloads status
* Pod status
* The Prometheus expression cross the thresholds
<br/>
<br/>
@@ -116,13 +126,13 @@ As a cluster owner, you can configure Rancher to send you alerts for cluster eve
1. From the **Global** view, open the cluster that you want to configure alerts for.
1. From the main menu, select **Tools > Alerts**. Then click **Add Alert**.
1. From the main menu, select **Tools > Alerts**. Then click **Add Alert Group**.
1. Enter a **Name** for the alert that describes its purpose.
1. Enter a **Name** for the alert that describes its purpose, you could group alert rules for the different purpose.
1. Based on the type of alert you want to create, complete one of the instruction subsets below.
{{% accordion id="system-service" label="System Service Alerts" %}}
This alert type monitors for events that affect one of the Kubernetes master components, regardless of the node it occurs on.
This alert type monitor for events that affect one of the Kubernetes master components, regardless of the node it occurs on.
1. Select the **System Services** option, and then select an option from the drop-down.
@@ -130,7 +140,7 @@ This alert type monitors for events that affect one of the Kubernetes master com
- [etcd](https://kubernetes.io/docs/concepts/overview/components/#etcd)
- [scheduler](https://kubernetes.io/docs/concepts/overview/components/#kube-scheduler)
1. Select the urgency level of the of alert. The options are:
1. Select the urgency level of the alert. The options are:
- **Critical**: Most urgent
- **Warning**: Normal urgency
@@ -138,6 +148,13 @@ This alert type monitors for events that affect one of the Kubernetes master com
<br/>
<br/>
Select the urgency level based on the importance of the service and how many nodes fill the role within your cluster. For example, if you're making an alert for the `etcd` service, select **Critical**. If you're making an alert for redundant schedulers, **Warning** is more appropriate.
1. Config advance options, it inheriting the advance option from group level by default, you configure alert rule use a customized advance option by disabling inherited.
- **Group Wait Time**: How long to wait to buffer alerts of the same group before sending initially.
- **Group Interval Time**: How long to wait before sending an alert that has been added to a group which contains already fired alerts.
- **Repeat Wait Time**: How long to wait before sending an alert that has been added to a group which contains already fired alerts.
{{% /accordion %}}
{{% accordion id="resource-event" label="Resource Event Alerts" %}}
This alert type monitors for specific events that are thrown from a resource type.
@@ -155,7 +172,7 @@ This alert type monitors for specific events that are thrown from a resource typ
- [Pod](https://kubernetes.io/docs/concepts/workloads/pods/pod/)
- [StatefulSet](https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/)
1. Select the urgency level of the of alert.
1. Select the urgency level of the alert.
- **Critical**: Most urgent
- **Warning**: Normal urgency
@@ -165,8 +182,13 @@ This alert type monitors for specific events that are thrown from a resource typ
Select the urgency level of the alert by considering factors such as how often the event occurs or its importance. For example:
- If you set a normal alert for pods, you're likely to receive alerts often, and individual pods usually self-heal, so select an urgency of **Info**.
- If you set a warning alert for StatefulSets, its very likely to impact operations, so select an urgency of **Critical**.
- If you set a warning alert for StatefulSets, it's very likely to impact operations, so select an urgency of **Critical**.
1. Config advance options, it inheriting the advance option from group level by default, you configure alert rule use a customized advance option by disabling inherited.
- **Group Wait Time**: How long to wait to buffer alerts of the same group before sending initially. The group wait time is configured to 1s to receive the alert when the event happened.
- **Group Interval Time**: How long to wait before sending an alert that has been added to a group which contains already fired alerts. The group wait time is configured to 1s to receive the alert when the event happened.
- **Repeat Wait Time**: How long to wait before sending an alert that has been added to a group which contains already fired alerts.
{{% /accordion %}}
{{% accordion id="node" label="Node Alerts" %}}
@@ -178,16 +200,23 @@ This alert type monitors for events that occur on a specific node.
- **Not Ready**: Sends you an alert when the node is unresponsive.
- **CPU usage over**: Sends you an alert when the node raises above an entered percentage of its processing allocation.
- **Mem usuage over**: Sends you an alert when the node raises above an entered percentage of its memory allocation.
- **Mem usage over**: Sends you an alert when the node raises above an entered percentage of its memory allocation.
1. Select the urgency level of the of alert.
1. Select the urgency level of the alert.
- **Critical**: Most urgent
- **Warning**: Normal urgency
- **Info**: Least urgent
<br/>
<br/>
Select the urgency level of the alert based on its impact on operations. For example, an alert triggered when a node's CPU raises above 60% deems a urgency of **Info**, but a node that is **Not Ready** deems an urgency of **Critical**.
Select the urgency level of the alert based on its impact on operations. For example, an alert triggered when a node's CPU raises above 60% deems an urgency of **Info**, but a node that is **Not Ready** deems an urgency of **Critical**.
1. Config advance options, it inheriting the advance option from group level by default, you configure alert rule use a customized advance option by disabling inherited.
- **Group Wait Time**: How long to wait to buffer alerts of the same group before sending initially.
- **Group Interval Time**: How long to wait before sending an alert that has been added to a group which contains already fired alerts.
- **Repeat Wait Time**: How long to wait before sending an alert that has been added to a group which contains already fired alerts.
{{% /accordion %}}
{{% accordion id="node-selector" label="Node Selector Alerts" %}}
This alert type monitors for events that occur on any node on marked with a label. For more information, see the Kubernetes documentation for [Labels](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/).
@@ -198,17 +227,72 @@ This alert type monitors for events that occur on any node on marked with a labe
- **Not Ready**: Sends you an alert when selected nodes are unresponsive.
- **CPU usage over**: Sends you an alert when selected nodes raise above an entered percentage of processing allocation.
- **Mem usuage over**: Sends you an alert when selected nodes raise above an entered percentage of memory allocation.
- **Mem usage over**: Sends you an alert when selected nodes raise above an entered percentage of memory allocation.
1. Select the urgency level of the of alert.
1. Select the urgency level of the alert.
- **Critical**: Most urgent
- **Warning**: Normal urgency
- **Info**: Least urgent
<br/>
<br/>
Select the urgency level of the alert based on its impact on operations. For example, an alert triggered when a node's CPU raises above 60% deems a urgency of **Info**, but a node that is **Not Ready** deems an urgency of **Critical**.
Select the urgency level of the alert based on its impact on operations. For example, an alert triggered when a node's CPU raises above 60% deems an urgency of **Info**, but a node that is **Not Ready** deems an urgency of **Critical**.
1. Config advance options, it inheriting the advance option from group level by default, you configure alert rule use a customized advance option by disabling inherited.
- **Group Wait Time**: How long to wait to buffer alerts of the same group before sending initially.
- **Group Interval Time**: How long to wait before sending an alert that has been added to a group which contains already fired alerts.
- **Repeat Wait Time**: How long to wait before sending an alert that has been added to a group which contains already fired alerts.
{{% /accordion %}}
{{% accordion id="cluster-expression" label="Metric Expression Alerts" %}}
This alert type monitors prometheus querying expressions crossed the threshold, it would available after you enable monitoring.
1. Input or select an **Expression**, the drop down shows the original expressions from prometheus, below monitoring information are exposed at the cluster level.
- **Kube State Metric**: Add-on agent to generate and expose cluster-level metrics. For more information, see [Kube-State-Metric](https://github.com/kubernetes/kube-state-metrics)
- **Kubelet Cadvisor**: Analyzes resource usage and performance characteristics of running containers. For more information, see [Cadvisor](https://github.com/google/cadvisor)
- **Node Exporter**: Expose machine monitoring information. For more information, see [Node Exporter](https://github.com/prometheus/node_exporter)
- **Kubernetes Metric**: Kubernetes related metrics. For more information, see [Kubernetes](https://github.com/kubernetes/metrics)
- **ETCD Metric**: Expose ETCD monitoring information. For cluster use rke to deploy, could get etcd metrics. For more information, see [Kubernetes](https://github.com/etcd-io/etcd/blob/master/Documentation/op-guide/monitoring.md)
- **Fluentd Metric**: Expose the Fluentd monitoring information. For more information, see [Fluentd](https://docs.fluentd.org/v1.0/articles/monitoring-prometheus)
- **Prometheus Metric**: Prometheus itself metrics.
- **Grafana Metric**: Expose Grafana monitoring information. For more information, see [Grafana](http://docs.grafana.org/administration/metrics/)
1. Choose a **Comparison**.
- **Equal**: Trigger alert when expression value equal to the threshold.
- **Not Equal**: Trigger alert when expression value not equal to the threshold.
- **Greater Than**: Trigger alert when expression value greater than to threshold.
- **Less Than**: Trigger alert when expression value equal or less than the threshold.
- **Greater or Equal**: Trigger alert when expression value greater to equal to the threshold.
- **Less or Equal**: Trigger alert when expression value less or equal to the threshold.
1. Input a **Threshold**, will trigger alert when the value of expression cross the threshold.
1. Choose a **Comparison**.
1. Select a duration, will trigger alert when expression value crosses the threshold longer than the configured duration.
1. Select the urgency level of the alert.
- **Critical**: Most urgent
- **Warning**: Normal urgency
- **Info**: Least urgent
<br/>
<br/>
Select the urgency level of the alert based on its impact on operations. For example, an alert triggered when a node's load expression ```sum(node_load5) / count(node_cpu_seconds_total{mode="system"})``` raises above 0.6 deems an urgency of **Info**, but 1 deems an urgency of **Critical**.
1. Config advance options, it inheriting the advance option from group level by default, you configure alert rule use a customized advance option by disabling inherited.
- **Group Wait Time**: How long to wait to buffer alerts of the same group before sending initially.
- **Group Interval Time**: How long to wait before sending an alert that has been added to a group which contains already fired alerts.
- **Repeat Wait Time**: How long to wait before sending an alert that has been added to a group which contains already fired alerts.
{{% /accordion %}}
1. Continue adding more **Alert Rule** to the group.
1. Finally, choose the notifiers that send you alerts.
- You can set up multiple notifiers.
@@ -223,6 +307,8 @@ After you set up cluster alerts, you can manage each alert object. To manage ale
- Deactivate/Reactive alerts
- Edit alert settings
- Delete unnecessary alerts
- Mute firing alerts
- Unmute muted alerts
#### Adding Project Alerts
@@ -230,28 +316,35 @@ After you set up cluster alerts, you can manage each alert object. To manage ale
1. From the **Global** view, open the project that you want to configure alerts for.
1. From the main menu, select **Resources > Alerts**. Then click **Add Alert**.
1. From the main menu, select **Tools > Alerts**. Then click **Add Alert Group**.
1. Enter a **Name** for the alert that describes its purpose.
1. Enter a **Name** for the alert that describes its purpose, you could group alert rules for the different purpose.
1. Based on the type of alert you want to create, complete one of the instruction subsets below.
{{% accordion id="pod" label="Pod Alerts" %}}
This alert type monitors for the status of a specific pod.
1. Select the **Pod** option, and then select a pod from the drop-down.
1. Select a pod status that triggers and alert:
1. Select a pod status that triggers an alert:
- **Not Running**
- **Not Scheduled**
- **Restarted `<x>` times with the last `<x>` Minutes**
1. Select the urgency level of the of alert. The options are:
1. Select the urgency level of the alert. The options are:
- **Critical**: Most urgent
- **Warning**: Normal urgency
- **Info**: Least urgent
Select the urgency level of the alert based on pod state and expendability. For example, an stateless pod that's not can be easily replaced, so select **Info**. However, if an important pod isn't scheduled, it may affect operations, so choose **Critical**.
Select the urgency level of the alert based on pod state and expandability. For example, a stateless pod that's not can be easily replaced, so select **Info**. However, if an important pod isn't scheduled, it may affect operations, so choose **Critical**.
1. Config advance options, it inheriting the advance option from group level by default, you configure alert rule use a customized advance option by disabling inherited.
- **Group Wait Time**: How long to wait to buffer alerts of the same group before sending initially.
- **Group Interval Time**: How long to wait before sending an alert that has been added to a group which contains already fired alerts.
- **Repeat Wait Time**: How long to wait before sending an alert that has been added to a group which contains already fired alerts.
{{% /accordion %}}
{{% accordion id="workload" label="Workload Alerts" %}}
This alert type monitors for the availability of a workload.
@@ -260,7 +353,7 @@ This alert type monitors for the availability of a workload.
1. Choose an availability percentage using the slider. The alert is triggered when the workload's availability on your cluster nodes drops below the set percentage.
1. Select the urgency level of the of alert.
1. Select the urgency level of the alert.
- **Critical**: Most urgent
- **Warning**: Normal urgency
@@ -268,21 +361,75 @@ This alert type monitors for the availability of a workload.
Select the urgency level of the alert based on the percentage you choose and the importance of the workload.
1. Config advance options, it inheriting the advance option from group level by default, you configure alert rule use a customized advance option by disabling inherited.
- **Group Wait Time**: How long to wait to buffer alerts of the same group before sending initially.
- **Group Interval Time**: How long to wait before sending an alert that has been added to a group which contains already fired alerts.
- **Repeat Wait Time**: How long to wait before sending an alert that has been added to a group which contains already fired alerts.
{{% /accordion %}}
{{% accordion id="workload-selector" label="Workload Selector Alerts" %}}
This alert type monitors for the availability of all workloads marked with tags that you've specified.
1. Select the **Workload Selector** option, and then click **Add Selector** to enter the key value pair for a label. If one of the workloads drops below your specifications, an alert is triggered. This label should be applied to one or more of your workloads.
1. Select the urgency level of the of alert.
1. Select the urgency level of the alert.
- **Critical**: Most urgent
- **Warning**: Normal urgency
- **Info**: Least urgent
Select the urgency level of the alert based on the percentage you choose and the importance of the workload.
1. Config advance options, it inheriting the advance option from group level by default, you configure alert rule use a customized advance option by disabling inherited.
- **Group Wait Time**: How long to wait to buffer alerts of the same group before sending initially.
- **Group Interval Time**: How long to wait before sending an alert that has been added to a group which contains already fired alerts.
- **Repeat Wait Time**: How long to wait before sending an alert that has been added to a group which contains already fired alerts.
{{% /accordion %}}
{{% accordion id="project-expression" label="Metric Expression" %}}
This alert type monitors prometheus querying expressions crossed the threshold, it would available after you enable monitoring.
1. Input or select an **Expression**, the drop down shows the original expressions from prometheus, below monitoring information are exposed at the project level.
- **Kube State Metric**: Add-on agent to generate and expose cluster-level metrics. For more information, see [Kube-State-Metric](https://github.com/kubernetes/kube-state-metrics)
- **Kubelet Cadvisor**: Analyzes resource usage and performance characteristics of running containers. For more information, see [Cadvisor](https://github.com/google/cadvisor)
- **Prometheus Metric**: Project level Prometheus itself metrics.
- **Grafana Metric**: Expose project level Grafana monitoring information. For more information, see [Grafana](http://docs.grafana.org/administration/metrics/)
1. Choose a comparison.
- **Equal**: Trigger alert when expression value equal to the threshold.
- **Not Equal**: Trigger alert when expression value not equal to the threshold.
- **Greater Than**: Trigger alert when expression value greater than to threshold.
- **Less Than**: Trigger alert when expression value equal or less than the threshold.
- **Greater or Equal**: Trigger alert when expression value greater to equal to the threshold.
- **Less or Equal**: Trigger alert when expression value less or equal to the threshold.
1. Input a **Threshold**, will trigger alert when the value of expression cross the threshold.
1. Choose a **Comparison**.
1. Select a **Duration**, will trigger alert when expression value crosses the threshold longer than the configured duration.
1. Select the urgency level of the alert.
- **Critical**: Most urgent
- **Warning**: Normal urgency
- **Info**: Least urgent
<br/>
<br/>
Select the urgency level of the alert based on its impact on operations. For example, an alert triggered when a expression for container memory close to the limit raises above 60% deems an urgency of **Info**, but raised about 95% deems an urgency of **Critical**.
1. Config advance options, it inheriting the advance option from group level by default, you configure alert rule use a customized advance option by disabling inherited.
- **Group Wait Time**: How long to wait to buffer alerts of the same group before sending initially.
- **Group Interval Time**: How long to wait before sending an alert that has been added to a group which contains already fired alerts.
- **Repeat Wait Time**: How long to wait before sending an alert that has been added to a group which contains already fired alerts.
{{% /accordion %}}
1. Continue adding more **Alert Rule** to the group.
1. Finally, choose the notifiers that send you alerts.
@@ -298,3 +445,5 @@ To manage project alerts, browse to the project that alerts you want to manage.
- Deactivate/Reactive alerts
- Edit alert settings
- Delete unnecessary alerts
- Mute firing alerts
- Unmute muted alerts