Convert h1 to h2 (2.0-2.4)

This commit is contained in:
Billy Tat
2022-10-28 18:48:50 -07:00
parent bb76e99d20
commit eaa89030f6
89 changed files with 377 additions and 354 deletions
@@ -6,7 +6,8 @@ When you create a cluster, some alert rules are predefined. These alerts notify
Several of the alerts use Prometheus expressions as the metric that triggers the alert. For more information on how expressions work, you can refer to the Rancher [documentation about Prometheus expressions](../cluster-monitoring/expression.md) or the Prometheus [documentation about querying metrics](https://prometheus.io/docs/prometheus/latest/querying/basics/).
# Alerts for etcd
## Alerts for etcd
Etcd is the key-value store that contains the state of the Kubernetes cluster. Rancher provides default alerts if the built-in monitoring detects a potential problem with etcd. You don't have to enable monitoring to receive these alerts.
A leader is the node that handles all client requests that need cluster consensus. For more information, you can refer to this [explanation of how etcd works.](https://rancher.com/blog/2019/2019-01-29-what-is-etcd/#how-does-etcd-work)
@@ -20,8 +21,7 @@ The leader of the cluster can change in response to certain events. It is normal
| Etcd is unavailable | A critical alert is triggered when etcd becomes unavailable. |
| Etcd member has no leader | A critical alert is triggered when the etcd cluster does not have a leader for at least three minutes. |
# Alerts for Kubernetes Components
## Alerts for Kubernetes Components
Rancher provides alerts when core Kubernetes system components become unhealthy.
Controllers update Kubernetes resources based on changes in etcd. The [controller manager](https://kubernetes.io/docs/reference/command-line-tools-reference/kube-controller-manager/) monitors the cluster desired state through the Kubernetes API server and makes the necessary changes to the current state to reach the desired state.
@@ -33,16 +33,16 @@ The [scheduler](https://kubernetes.io/docs/reference/command-line-tools-referenc
| Controller Manager is unavailable | A critical warning is triggered when the cluster’s controller-manager becomes unavailable. |
| Scheduler is unavailable | A critical warning is triggered when the cluster’s scheduler becomes unavailable. |
## Alerts for Events
# Alerts for Events
Kubernetes events are objects that provide insight into what is happening inside a cluster, such as what decisions were made by the scheduler or why some pods were evicted from the node. In the Rancher UI, from the project view, you can see events for each workload.
| Alert | Explanation |
|-------|-------------|
| Get warning deployment event | A warning alert is triggered when a warning event happens on a deployment. |
## Alerts for Nodes
# Alerts for Nodes
Alerts can be triggered based on node metrics. Each computing resource in a Kubernetes cluster is called a node. Nodes can be either bare-metal servers or virtual machines.
| Alert | Explanation |
@@ -51,5 +51,6 @@ Alerts can be triggered based on node metrics. Each computing resource in a Kube
| High node memory utilization | A warning alert is triggered if the node uses more than 80 percent of its available memory for at least three minutes. |
| Node disk is running full within 24 hours | A critical alert is triggered if the disk space on the node is expected to run out in the next 24 hours based on the disk growth over the last 6 hours. |
# Project-level Alerts
## Project-level Alerts
When you enable monitoring for the project, some project-level alerts are provided. For details, refer to the [section on project-level alerts.](../../../reference-guides/rancher-project-tools/project-alerts.md)
@@ -8,8 +8,7 @@ The PromQL expressions in this doc can be used to configure [alerts.](../../../p
For more information about querying Prometheus, refer to the official [Prometheus documentation.](https://prometheus.io/docs/prometheus/latest/querying/basics/)
# Cluster Metrics
## Cluster Metrics
### Cluster CPU Utilization
@@ -60,7 +59,7 @@ For more information about querying Prometheus, refer to the official [Prometheu
| Detail | <table><tr><td>receive</td><td><code>sum(rate(node_network_receive_bytes_total{device!~"lo &#124; veth.&ast; &#124; docker.&ast; &#124; flannel.&ast; &#124; cali.&ast; &#124; cbr.&ast;"}[5m])) by (instance)</code></td></tr><tr><td>transmit</td><td><code>sum(rate(node_network_transmit_bytes_total{device!~"lo &#124; veth.&ast; &#124; docker.&ast; &#124; flannel.&ast; &#124; cali.&ast; &#124; cbr.&ast;"}[5m])) by (instance)</code></td></tr></table> |
| Summary | <table><tr><td>receive</td><td><code>sum(rate(node_network_receive_bytes_total{device!~"lo &#124; veth.&ast; &#124; docker.&ast; &#124; flannel.&ast; &#124; cali.&ast; &#124; cbr.&ast;"}[5m]))</code></td></tr><tr><td>transmit</td><td><code>sum(rate(node_network_transmit_bytes_total{device!~"lo &#124; veth.&ast; &#124; docker.&ast; &#124; flannel.&ast; &#124; cali.&ast; &#124; cbr.&ast;"}[5m]))</code></td></tr></table> |
# Node Metrics
## Node Metrics
### Node CPU Utilization
@@ -111,7 +110,7 @@ For more information about querying Prometheus, refer to the official [Prometheu
| Detail | <table><tr><td>receive</td><td><code>sum(rate(node_network_receive_bytes_total{device!~"lo &#124; veth.&ast; &#124; docker.&ast; &#124; flannel.&ast; &#124; cali.&ast; &#124; cbr.&ast;",instance=~"$instance"}[5m])) by (device)</code></td></tr><tr><td>transmit</td><td><code>sum(rate(node_network_transmit_bytes_total{device!~"lo &#124; veth.&ast; &#124; docker.&ast; &#124; flannel.&ast; &#124; cali.&ast; &#124; cbr.&ast;",instance=~"$instance"}[5m])) by (device)</code></td></tr></table> |
| Summary | <table><tr><td>receive</td><td><code>sum(rate(node_network_receive_bytes_total{device!~"lo &#124; veth.&ast; &#124; docker.&ast; &#124; flannel.&ast; &#124; cali.&ast; &#124; cbr.&ast;",instance=~"$instance"}[5m]))</code></td></tr><tr><td>transmit</td><td><code>sum(rate(node_network_transmit_bytes_total{device!~"lo &#124; veth.&ast; &#124; docker.&ast; &#124; flannel.&ast; &#124; cali.&ast; &#124; cbr.&ast;",instance=~"$instance"}[5m]))</code></td></tr></table> |
# Etcd Metrics
## Etcd Metrics
### Etcd Has a Leader
@@ -181,7 +180,7 @@ For more information about querying Prometheus, refer to the official [Prometheu
| Detail | <table><tr><td>wal</td><td>`histogram_quantile(0.99, sum(rate(etcd_disk_wal_fsync_duration_seconds_bucket[5m])) by (instance, le))`</td></tr><tr><td>db</td><td>`histogram_quantile(0.99, sum(rate(etcd_disk_backend_commit_duration_seconds_bucket[5m])) by (instance, le))`</td></tr></table> |
| Summary | <table><tr><td>wal</td><td>`sum(histogram_quantile(0.99, sum(rate(etcd_disk_wal_fsync_duration_seconds_bucket[5m])) by (instance, le)))`</td></tr><tr><td>db</td><td>`sum(histogram_quantile(0.99, sum(rate(etcd_disk_backend_commit_duration_seconds_bucket[5m])) by (instance, le)))`</td></tr></table> |
# Kubernetes Components Metrics
## Kubernetes Components Metrics
### API Server Request Latency
@@ -239,8 +238,7 @@ For more information about querying Prometheus, refer to the official [Prometheu
| Detail | `topk(10, histogram_quantile(0.95,sum by (le, host, path)(rate(nginx_ingress_controller_request_duration_seconds_bucket{host!="_"}[5m]))))` |
| Summary | `topk(10, histogram_quantile(0.95,sum by (le, host)(rate(nginx_ingress_controller_request_duration_seconds_bucket{host!="_"}[5m]))))` |
# Rancher Logging Metrics
## Rancher Logging Metrics
### Fluentd Buffer Queue Rate
@@ -270,7 +268,7 @@ For more information about querying Prometheus, refer to the official [Prometheu
| Detail | `sum(rate(fluentd_output_status_num_records_total[5m])) by (instance)` |
| Summary | `sum(rate(fluentd_output_status_num_records_total[5m]))` |
# Workload Metrics
## Workload Metrics
### Workload CPU Utilization
@@ -307,7 +305,7 @@ For more information about querying Prometheus, refer to the official [Prometheu
| Detail | <table><tr><td>read</td><td>`sum(rate(container_fs_reads_bytes_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m])) by (pod_name)`</td></tr><tr><td>write</td><td>`sum(rate(container_fs_writes_bytes_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m])) by (pod_name)`</td></tr></table> |
| Summary | <table><tr><td>read</td><td>`sum(rate(container_fs_reads_bytes_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m]))`</td></tr><tr><td>write</td><td>`sum(rate(container_fs_writes_bytes_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m]))`</td></tr></table> |
# Pod Metrics
## Pod Metrics
### Pod CPU Utilization
@@ -344,7 +342,7 @@ For more information about querying Prometheus, refer to the official [Prometheu
| Detail | <table><tr><td>read</td><td>`sum(rate(container_fs_reads_bytes_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m])) by (container_name)`</td></tr><tr><td>write</td><td>`sum(rate(container_fs_writes_bytes_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m])) by (container_name)`</td></tr></table> |
| Summary | <table><tr><td>read</td><td>`sum(rate(container_fs_reads_bytes_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`</td></tr><tr><td>write</td><td>`sum(rate(container_fs_writes_bytes_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`</td></tr></table> |
# Container Metrics
## Container Metrics
### Container CPU Utilization
@@ -12,7 +12,7 @@ While configuring monitoring at either the [cluster level](../../../pages-for-su
- [Persistent Storage](#persistent-storage)
- [Remote Storage](#remote-storage)
# Basic Configuration
## Basic Configuration
Option | Description
-------|-------------
@@ -27,7 +27,7 @@ Prometheus [Memory Limit](https://kubernetes.io/docs/concepts/configuration/mana
Prometheus [Memory Reservation](https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/#meaning-of-memory) | Memory resource requests for the Prometheus pod.
Selector | Ability to select the nodes in which Prometheus and Grafana pods are deployed to. To use this option, the nodes must have labels.
# Advanced Options
## Advanced Options
Since monitoring is an [application](https://github.com/rancher/system-charts/tree/dev/charts/rancher-monitoring) from the [Rancher catalog](../../../pages-for-subheaders/helm-charts-in-rancher.md), it can be configured like any other catalog application, by passing in values to Helm.
@@ -64,7 +64,7 @@ Some example key-value pairs are:
| `prometheus.livenessProbe.timeoutSeconds` | 60 |
| `prometheus.readinessProbe.timeoutSeconds` | 60 |
# Node Exporter
## Node Exporter
The [node exporter](https://github.com/prometheus/node_exporter/blob/master/README.md) is a popular open source exporter, which exposes the metrics for hardware and \*NIX kernels OS. It is designed to monitor the host system. However, there are still issues with namespaces when running it in a container, mostly around filesystem mount spaces. In order to monitor actual network metrics for the container network, the node exporter must be deployed with the `hostNetwork` mode.
@@ -72,7 +72,7 @@ When configuring Prometheus and enabling the node exporter, enter a host port in
>**Warning:** In order for Prometheus to collect the metrics of the node exporter, after enabling cluster monitoring, you must open the <b>Node Exporter Host Port</b> in the host firewall rules to allow intranet access. By default, `9796` is used as that host port.
# Persistent Storage
## Persistent Storage
>**Prerequisite:** Configure one or more StorageClasses to use as [persistent storage](../../../pages-for-subheaders/create-kubernetes-persistent-storage.md) for your Prometheus or Grafana pod.
@@ -80,7 +80,7 @@ By default, when you enable Prometheus for either a cluster or project, all moni
When enabling persistent storage for Prometheus or Grafana, specify the size of the persistent volume and select the StorageClass.
# Remote Storage
## Remote Storage
>**Prerequisite:** Need a remote storage endpoint to be available. The possible list of integrations is available [here](https://prometheus.io/docs/operating/integrations/)
@@ -4,7 +4,7 @@ title: Disabling Istio
This section describes how to disable Istio in a cluster, namespace, or workload.
# Disable Istio in a Cluster
## Disable Istio in a Cluster
To disable Istio,
@@ -14,13 +14,13 @@ To disable Istio,
**Result:** The `cluster-istio` application in the cluster's `system` project gets removed. The Istio sidecar cannot be deployed on any workloads in the cluster.
# Disable Istio in a Namespace
## Disable Istio in a Namespace
1. In the Rancher UI, go to the project that has the namespace where you want to disable Istio.
1. On the **Workloads** tab, you will see a list of namespaces and the workloads deployed in them. Go to the namespace where you want to disable and click the **&#8942; > Disable Istio Auto Injection.**
**Result:** When workloads are deployed in this namespace, they will not have the Istio sidecar.
# Remove the Istio Sidecar from a Workload
## Remove the Istio Sidecar from a Workload
Disable Istio in the namespace, then redeploy the workloads with in it. They will be deployed without the Istio sidecar.
@@ -4,7 +4,7 @@ title: Role-based Access Control
This section describes the permissions required to access Istio features and how to configure access to the Kiali and Jaeger visualizations.
# Cluster-level Access
## Cluster-level Access
By default, only cluster administrators can:
@@ -12,7 +12,7 @@ By default, only cluster administrators can:
- Configure resource allocations for Istio
- View each UI for Prometheus, Grafana, Kiali, and Jaeger
# Project-level Access
## Project-level Access
After Istio is enabled in a cluster, project owners and members have permission to:
@@ -22,7 +22,7 @@ After Istio is enabled in a cluster, project owners and members have permission
- View the Kiali and Jaeger visualizations if cluster administrators give access to project members
- Configure Istio's resources (such as the gateway, destination rules, or virtual services) with `kubectl` (This does not apply to read-only project members)
# Access to Visualizations
## Access to Visualizations
By default, the Kiali and Jaeger visualizations are restricted to the cluster owner because the information in them could be sensitive.
@@ -40,7 +40,7 @@ To configure who has permission to access the Kiali and Jaeger UI,
**Result:** The access levels for Kiali and Jaeger have been updated.
# Summary of Default Permissions for Istio Users
## Summary of Default Permissions for Istio Users
| Permission | Cluster Administrators | Project Owners | Project Members | Read-only Project Members |
|------------------------------------------|----------------|----------------|-----------------|---------------------------|
@@ -3,7 +3,7 @@ title: Release Notes
---
# Istio 1.5.8
## Istio 1.5.8
### Important note on 1.5.x versions
@@ -15,11 +15,11 @@ Rancher integrates with a variety of popular IT services, including:
- **Microsoft Teams**: (Available as of v2.4.6) Send alert notifications to Teams using a webhook.
# Roles-based Access Control for Notifiers
## Roles-based Access Control for Notifiers
Notifiers are configured at the cluster level. This model ensures that only cluster owners need to configure notifiers, leaving project owners to simply configure alerts in the scope of their projects. You don't need to dispense privileges like SMTP server access or cloud account access.
# Adding Notifiers
## Adding Notifiers
Set up a notifier so that you can begin configuring and sending alerts.
@@ -31,7 +31,7 @@ Set up a notifier so that you can begin configuring and sending alerts.
**Result:** Your notifier is added to Rancher.
# Configuration
## Configuration
- [Slack](#slack)
- [Email](#email)
@@ -140,7 +140,7 @@ _Available as of v2.4.6_
**Validation:** Click **Test.** If the test is successful, the Teams notifier output is **MicrosoftTeams setting validated.**
# Managing Notifiers
## Managing Notifiers
After you set up notifiers, you can manage them. From the **Global** view, open the cluster that you want to manage your notifiers. Select **Tools > Notifiers**. You can:
@@ -148,7 +148,7 @@ After you set up notifiers, you can manage them. From the **Global** view, open
- **Clone** them, to quickly setup slightly different notifiers.
- **Delete** them when they're no longer necessary.
# Example Payload for a Webhook Alert Notifier
## Example Payload for a Webhook Alert Notifier
```json
{
@@ -184,7 +184,8 @@ After you set up notifiers, you can manage them. From the **Global** view, open
}
}
```
# What's Next?
## What's Next?
After creating a notifier, set up alerts to receive notifications of Rancher system events.