mirror of
https://github.com/rancher/rancher-docs.git
synced 2026-09-29 22:49:17 +00:00
Add version 2.9 preview
This commit is contained in:
Billy Tat
+121
@@ -0,0 +1,121 @@
|
||||
---
|
||||
title: Built-in Dashboards
|
||||
---
|
||||
|
||||
<head>
|
||||
<link rel="canonical" href="https://ranchermanager.docs.rancher.com/integrations-in-rancher/monitoring-and-alerting/built-in-dashboards"/>
|
||||
</head>
|
||||
|
||||
## Grafana UI
|
||||
|
||||
[Grafana](https://grafana.com/grafana/) allows you to query, visualize, alert on and understand your metrics no matter where they are stored. Create, explore, and share dashboards with your team and foster a data driven culture.
|
||||
|
||||
To see the default dashboards for time series data visualization, go to the Grafana UI.
|
||||
|
||||
### Customizing Grafana
|
||||
|
||||
To view and customize the PromQL queries powering the Grafana dashboard, see [this page.](../../how-to-guides/advanced-user-guides/monitoring-alerting-guides/customize-grafana-dashboard.md)
|
||||
|
||||
### Persistent Grafana Dashboards
|
||||
|
||||
To create a persistent Grafana dashboard, see [this page.](../../how-to-guides/advanced-user-guides/monitoring-alerting-guides/create-persistent-grafana-dashboard.md)
|
||||
|
||||
### Access to Grafana
|
||||
|
||||
For information about role-based access control for Grafana, see [this section.](rbac-for-monitoring.md#role-based-access-control-for-grafana)
|
||||
|
||||
|
||||
## Alertmanager UI
|
||||
|
||||
When `rancher-monitoring` is installed, the Prometheus Alertmanager UI is deployed, allowing you to view your alerts and the current Alertmanager configuration.
|
||||
|
||||
:::note
|
||||
|
||||
This section assumes familiarity with how monitoring components work together. For more information about Alertmanager, see [How Alertmanager Works.](how-monitoring-works.md#3-how-alertmanager-works)
|
||||
|
||||
:::
|
||||
|
||||
### Accessing the Alertmanager UI
|
||||
|
||||
The Alertmanager UI lets you see the most recently fired alerts.
|
||||
|
||||
:::note Prerequisite:
|
||||
|
||||
The `rancher-monitoring` application must be installed.
|
||||
|
||||
:::
|
||||
|
||||
To see the Alertmanager UI,
|
||||
|
||||
1. In the upper left corner, click **☰ > Cluster Management**.
|
||||
1. On the **Clusters** page, go to the cluster where you want to see the Alertmanager UI, click **Explore**.
|
||||
1. In the left navigation bar, click **Monitoring**.
|
||||
1. Click **Alertmanager**.
|
||||
|
||||
**Result:** The Alertmanager UI opens in a new tab. For help with configuration, refer to the [official Alertmanager documentation.](https://prometheus.io/docs/alerting/latest/alertmanager/)
|
||||
|
||||
For more information on configuring Alertmanager in Rancher, see [this page.](../../how-to-guides/advanced-user-guides/monitoring-v2-configuration-guides/advanced-configuration/alertmanager.md)
|
||||
|
||||
<figcaption>The Alertmanager UI</figcaption>
|
||||
|
||||
|
||||
|
||||
|
||||
### Viewing Default Alerts
|
||||
|
||||
To see alerts that are fired by default, go to the Alertmanager UI and click **Expand all groups**.
|
||||
|
||||
|
||||
## Prometheus UI
|
||||
|
||||
By default, the [kube-state-metrics service](https://github.com/kubernetes/kube-state-metrics) provides a wealth of information about CPU and memory utilization to the monitoring application. These metrics cover Kubernetes resources across namespaces. This means that in order to see resource metrics for a service, you don't need to create a new ServiceMonitor for it. Because the data is already in the time series database, you can go to the Prometheus UI and run a PromQL query to get the information. The same query can be used to configure a Grafana dashboard to show a graph of those metrics over time.
|
||||
|
||||
To see the Prometheus UI, install `rancher-monitoring`. Then:
|
||||
|
||||
1. In the upper left corner, click **☰ > Cluster Management**.
|
||||
1. On the **Clusters** page, go to the cluster where you want to see the Prometheus UI and click **Explore**.
|
||||
1. In the left navigation bar, click **Monitoring**.
|
||||
1. Click **Prometheus Graph**.
|
||||
|
||||
<figcaption>Prometheus Graph UI</figcaption>
|
||||
|
||||
|
||||
|
||||
### Viewing the Prometheus Targets
|
||||
|
||||
To see what services you are monitoring, you will need to see your targets. Targets are set up by ServiceMonitors and PodMonitors as sources to scrape metrics from. You won't need to directly edit targets, but the Prometheus UI can be useful for giving you an overview of all of the sources of metrics that are being scraped.
|
||||
|
||||
To see the Prometheus Targets, install `rancher-monitoring`. Then:
|
||||
|
||||
|
||||
1. In the upper left corner, click **☰ > Cluster Management**.
|
||||
1. On the **Clusters** page, go to the cluster where you want to see the Prometheus targets and click **Explore**.
|
||||
1. In the left navigation bar, click **Monitoring**.
|
||||
1. Click **Prometheus Targets**.
|
||||
|
||||
<figcaption>Targets in the Prometheus UI</figcaption>
|
||||
|
||||
|
||||
|
||||
### Viewing the PrometheusRules
|
||||
|
||||
When you define a Rule (which is declared within a RuleGroup in a PrometheusRule resource), the [spec of the Rule itself](https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/api.md#rule) contains labels that are used by Alertmanager to figure out which Route should receive a certain Alert.
|
||||
|
||||
To see the PrometheusRules, install `rancher-monitoring`. Then:
|
||||
|
||||
1. In the upper left corner, click **☰ > Cluster Management**.
|
||||
1. On the **Clusters** page, go to the cluster where you want to see the visualizations and click **Explore**.
|
||||
1. In the left navigation bar, click **Monitoring**.
|
||||
1. Click **Prometheus Rules**.
|
||||
|
||||
You can also see the rules in the Prometheus UI:
|
||||
|
||||
<figcaption>Rules in the Prometheus UI</figcaption>
|
||||
|
||||
|
||||
|
||||
For more information on configuring PrometheusRules in Rancher, see [this page.](../../how-to-guides/advanced-user-guides/monitoring-v2-configuration-guides/advanced-configuration/prometheusrules.md)
|
||||
|
||||
## Legacy UI
|
||||
|
||||
For information on the dashboards available in v2.2 to v2.4 of Rancher, before the introduction of the `rancher-monitoring` application, see the [Rancher v2.0—v2.4 docs](/versioned_docs/version-2.0-2.4/explanations/integrations-in-rancher/cluster-monitoring/viewing-metrics.md).
|
||||
+253
@@ -0,0 +1,253 @@
|
||||
---
|
||||
title: How Monitoring Works
|
||||
---
|
||||
|
||||
<head>
|
||||
<link rel="canonical" href="https://ranchermanager.docs.rancher.com/integrations-in-rancher/monitoring-and-alerting/how-monitoring-works"/>
|
||||
</head>
|
||||
|
||||
## 1. Architecture Overview
|
||||
|
||||
_**The following sections describe how data flows through the Monitoring V2 application:**_
|
||||
|
||||
### Prometheus Operator
|
||||
|
||||
Prometheus Operator observes ServiceMonitors, PodMonitors, and PrometheusRules being created. When the Prometheus configuration resources are created, Prometheus Operator calls the Prometheus API to sync the new configuration. As the diagram at the end of this section shows, the Prometheus Operator acts as the intermediary between Prometheus and Kubernetes, calling the Prometheus API to synchronize Prometheus with the monitoring-related resources in Kubernetes.
|
||||
|
||||
### ServiceMonitors and PodMonitors
|
||||
|
||||
ServiceMonitors and PodMonitors declaratively specify targets, such as Services and Pods, that need to be monitored.
|
||||
|
||||
- Targets are scraped on a recurring schedule based on the configured Prometheus scrape interval, and the metrics that are scraped are stored into the Prometheus Time Series Database (TSDB).
|
||||
|
||||
- In order to perform the scrape, ServiceMonitors and PodMonitors are defined with label selectors that determine which Services or Pods should be scraped and endpoints that determine how the scrape should happen on the given target, e.g., scrape/metrics in TCP 10252, proxying through IP addr x.x.x.x.
|
||||
|
||||
- Out of the box, Monitoring V2 comes with certain pre-configured exporters that are deployed based on the type of Kubernetes cluster that it is deployed on. For more information, see [Scraping and Exposing Metrics](#5-scraping-and-exposing-metrics).
|
||||
|
||||
### How PushProx Works
|
||||
|
||||
- Certain internal Kubernetes components are scraped via a proxy deployed as part of Monitoring V2 called **PushProx**. The Kubernetes components that expose metrics to Prometheus through PushProx are the following:
|
||||
`kube-controller-manager`, `kube-scheduler`, `etcd`, and `kube-proxy`.
|
||||
|
||||
- For each PushProx exporter, we deploy one PushProx client onto all target nodes. For example, a PushProx client is deployed onto all controlplane nodes for kube-controller-manager, all etcd nodes for kube-etcd, and all nodes for kubelet.
|
||||
|
||||
- We deploy exactly one PushProx proxy per exporter. The process for exporting metrics is as follows:
|
||||
|
||||
1. The PushProx Client establishes an outbound connection with the PushProx Proxy.
|
||||
1. The client then polls the proxy for scrape requests that have come into the proxy.
|
||||
1. When the proxy receives a scrape request from Prometheus, the client sees it as a result of the poll.
|
||||
1. The client scrapes the internal component.
|
||||
1. The internal component responds by pushing metrics back to the proxy.
|
||||
|
||||
|
||||
<figcaption><br/>Process for Exporting Metrics with PushProx:<br/></figcaption>
|
||||
|
||||
|
||||
|
||||
### PrometheusRules
|
||||
|
||||
PrometheusRules allow users to define rules for what metrics or time series database queries should result in alerts being fired. Rules are evaluated on an interval.
|
||||
|
||||
- **Recording rules** create a new time series based on existing series that have been collected. They are frequently used to precompute complex queries.
|
||||
- **Alerting rules** run a particular query and fire an alert from Prometheus if the query evaluates to a non-zero value.
|
||||
|
||||
### Alert Routing
|
||||
|
||||
Once Prometheus determines that an alert needs to be fired, alerts are forwarded to **Alertmanager**.
|
||||
|
||||
- Alerts contain labels that come from the PromQL query itself and additional labels and annotations that can be provided as part of specifying the initial PrometheusRule.
|
||||
|
||||
- Before receiving any alerts, Alertmanager will use the **routes** and **receivers** specified in its configuration to form a routing tree on which all incoming alerts are evaluated. Each node of the routing tree can specify additional grouping, labeling, and filtering that needs to happen based on the labels attached to the Prometheus alert. A node on the routing tree (usually a leaf node) can also specify that an alert that reaches it needs to be sent out to a configured Receiver, e.g., Slack, PagerDuty, SMS, etc. Note that Alertmanager will send an alert first to **alertingDriver**, then alertingDriver will send or forward alert to the proper destination.
|
||||
|
||||
- Routes and receivers are also stored in the Kubernetes API via the Alertmanager Secret. When the Secret is updated, Alertmanager is also updated automatically. Note that routing occurs via labels only (not via annotations, etc.).
|
||||
|
||||
<figcaption>How data flows through the monitoring application:</figcaption>
|
||||
|
||||
|
||||
## 2. How Prometheus Works
|
||||
|
||||
### Storing Time Series Data
|
||||
|
||||
After collecting metrics from exporters, Prometheus stores the time series in a local on-disk time series database. Prometheus optionally integrates with remote systems, but `rancher-monitoring` uses local storage for the time series database.
|
||||
|
||||
Once stored, users can query this TSDB using PromQL, the query language for Prometheus.
|
||||
|
||||
PromQL queries can be visualized in one of two ways:
|
||||
|
||||
1. By supplying the query in Prometheus's Graph UI, which will show a simple graphical view of the data.
|
||||
1. By creating a Grafana Dashboard that contains the PromQL query and additional formatting directives that label axes, add units, change colors, use alternative visualizations, etc.
|
||||
|
||||
### Defining Rules for Prometheus
|
||||
|
||||
Rules define queries that Prometheus needs to execute on a regular `evaluationInterval` to perform certain actions, such as firing an alert (alerting rules) or precomputing a query based on others existing in its TSDB (recording rules). These rules are encoded in PrometheusRules custom resources. When PrometheusRule custom resources are created or updated, the Prometheus Operator observes the change and calls the Prometheus API to synchronize the set of rules that Prometheus is currently evaluating on a regular interval.
|
||||
|
||||
A PrometheusRule allows you to define one or more RuleGroups. Each RuleGroup consists of a set of Rule objects that can each represent either an alerting or a recording rule with the following fields:
|
||||
|
||||
- The name of the new alert or record
|
||||
- A PromQL expression for the new alert or record
|
||||
- Labels that should be attached to the alert or record that identify it (e.g. cluster name or severity)
|
||||
- Annotations that encode any additional important pieces of information that need to be displayed on the notification for an alert (e.g. summary, description, message, runbook URL, etc.). This field is not required for recording rules.
|
||||
|
||||
Upon evaluating a [rule](https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/api.md#rule), Prometheus runs the provided PromQL query, adds the provided labels, and runs the appropriate action for the rule. If the rule triggers an alert, Prometheus also adds the provided annotations. For example, an Alerting Rule that adds `team: front-end` as a label to the provided PromQL query will append that label to the fired alert, which will allow Alertmanager to forward the alert to the correct Receiver.
|
||||
|
||||
### Alerting and Recording Rules
|
||||
|
||||
Prometheus doesn't maintain the state of whether alerts are active. It fires alerts repetitively at every evaluation interval, relying on Alertmanager to group and filter the alerts into meaningful notifications.
|
||||
|
||||
The `evaluation_interval` constant defines how often Prometheus evaluates its alerting rules against the time series database. Similar to the `scrape_interval`, the `evaluation_interval` also defaults to one minute.
|
||||
|
||||
The rules are contained in a set of rule files. Rule files include both alerting rules and recording rules, but only alerting rules result in alerts being fired after their evaluation.
|
||||
|
||||
For recording rules, Prometheus runs a query, then stores it as a time series. This synthetic time series is useful for storing the results of an expensive or time-consuming query so that it can be queried more quickly in the future.
|
||||
|
||||
Alerting rules are more commonly used. Whenever an alerting rule evaluates to a positive number, Prometheus fires an alert.
|
||||
|
||||
The Rule file adds labels and annotations to alerts before firing them, depending on the use case:
|
||||
|
||||
- Labels indicate information that identifies the alert and could affect the routing of the alert. For example, if when sending an alert about a certain container, the container ID could be used as a label.
|
||||
|
||||
- Annotations denote information that doesn't affect where an alert is routed, for example, a runbook or an error message.
|
||||
|
||||
## 3. How Alertmanager Works
|
||||
|
||||
The Alertmanager handles alerts sent by client applications such as the Prometheus server. It takes care of the following tasks:
|
||||
|
||||
- Deduplicating, grouping, and routing alerts to the correct receiver integration such as email, PagerDuty, or OpsGenie
|
||||
|
||||
- Silencing and inhibition of alerts
|
||||
|
||||
- Tracking alerts that fire over time
|
||||
|
||||
- Sending out the status of whether an alert is currently firing, or if it is resolved
|
||||
|
||||
### Alerts Forwarded by alertingDrivers
|
||||
|
||||
When alertingDrivers are installed, this creates a `Service` that can be used as the receiver's URL for Teams or SMS, based on the alertingDriver's configuration. The URL in the Receiver points to the alertingDrivers; so the Alertmanager sends alert first to alertingDriver, then alertingDriver forwards or sends alert to the proper destination.
|
||||
|
||||
### Routing Alerts to Receivers
|
||||
|
||||
Alertmanager coordinates where alerts are sent. It allows you to group alerts based on labels and fire them based on whether certain labels are matched. One top-level route accepts all alerts. From there, Alertmanager continues routing alerts to receivers based on whether they match the conditions of the next route.
|
||||
|
||||
While the Rancher UI forms only allow editing a routing tree that is two levels deep, you can configure more deeply nested routing structures by editing the Alertmanager Secret.
|
||||
|
||||
### Configuring Multiple Receivers
|
||||
|
||||
By editing the forms in the Rancher UI, you can set up a Receiver resource with all the information Alertmanager needs to send alerts to your notification system.
|
||||
|
||||
By editing custom YAML in the Alertmanager or Receiver configuration, you can also send alerts to multiple notification systems. For more information, see the section on configuring [Receivers.](../../reference-guides/monitoring-v2-configuration/receivers.md#configuring-multiple-receivers)
|
||||
|
||||
## 4. Monitoring V2 Specific Components
|
||||
|
||||
Prometheus Operator introduces a set of [Custom Resource Definitions](https://github.com/prometheus-operator/prometheus-operator#customresourcedefinitions) that allow users to deploy and manage Prometheus and Alertmanager instances by creating and modifying those custom resources on a cluster.
|
||||
|
||||
Prometheus Operator will automatically update your Prometheus configuration based on the live state of the resources and configuration options that are edited in the Rancher UI.
|
||||
|
||||
### Resources Deployed by Default
|
||||
|
||||
By default, a set of resources curated by the [kube-prometheus](https://github.com/prometheus-operator/kube-prometheus) project are deployed onto your cluster as part of installing the Rancher Monitoring Application to set up a basic Monitoring/Alerting stack.
|
||||
|
||||
The resources that get deployed onto your cluster to support this solution can be found in the [`rancher-monitoring`](https://github.com/rancher/charts/tree/main/charts/rancher-monitoring) Helm chart, which closely tracks the upstream [kube-prometheus-stack](https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack) Helm chart maintained by the Prometheus community with certain changes tracked in the [CHANGELOG.md](https://github.com/rancher/charts/blob/main/charts/rancher-monitoring/CHANGELOG.md).
|
||||
|
||||
### Default Exporters
|
||||
|
||||
Monitoring V2 deploys three default exporters that provide additional metrics for Prometheus to store:
|
||||
|
||||
1. `node-exporter`: exposes hardware and OS metrics for Linux hosts. For more information on `node-exporter`, refer to the [upstream documentation](https://prometheus.io/docs/guides/node-exporter/).
|
||||
|
||||
1. `windows-exporter`: exposes hardware and OS metrics for Windows hosts (only deployed on Windows clusters). For more information on `windows-exporter`, refer to the [upstream documentation](https://github.com/prometheus-community/windows_exporter).
|
||||
|
||||
1. `kube-state-metrics`: expose additional metrics that track the state of resources contained in the Kubernetes API (e.g., pods, workloads, etc.). For more information on `kube-state-metrics`, refer to the [upstream documentation](https://github.com/kubernetes/kube-state-metrics/tree/master/docs).
|
||||
|
||||
ServiceMonitors and PodMonitors will scrape these exporters, as defined [here](#defining-what-metrics-are-scraped). Prometheus stores these metrics, and you can query the results via either Prometheus's UI or Grafana.
|
||||
|
||||
See the [architecture](#1-architecture-overview) section for more information on recording rules, alerting rules, and Alertmanager.
|
||||
|
||||
### Components Exposed in the Rancher UI
|
||||
|
||||
When the monitoring application is installed, you will be able to edit the following components in the Rancher UI:
|
||||
|
||||
| Component | Type of Component | Purpose and Common Use Cases for Editing |
|
||||
|--------------|------------------------|---------------------------|
|
||||
| ServiceMonitor | Custom resource | Sets up Kubernetes Services to scrape custom metrics from. Automatically updates the scrape configuration in the Prometheus custom resource. |
|
||||
| PodMonitor | Custom resource | Sets up Kubernetes Pods to scrape custom metrics from. Automatically updates the scrape configuration in the Prometheus custom resource. |
|
||||
| Receiver | Configuration block (part of Alertmanager) | Modifies information on where to send an alert (e.g., Slack, PagerDuty, etc.) and any necessary information to send the alert (e.g., TLS certs, proxy URLs, etc.). Automatically updates the Alertmanager custom resource. |
|
||||
| Route | Configuration block (part of Alertmanager) | Modifies the routing tree that is used to filter, label, and group alerts based on labels and send them to the appropriate Receiver. Automatically updates the Alertmanager custom resource. |
|
||||
| PrometheusRule | Custom resource | Defines additional queries that need to trigger alerts or define materialized views of existing series that are within Prometheus's TSDB. Automatically updates the Prometheus custom resource. |
|
||||
|
||||
### PushProx
|
||||
|
||||
PushProx allows Prometheus to scrape metrics across a network boundary, which prevents users from having to expose metrics ports for internal Kubernetes components on each node in a Kubernetes cluster.
|
||||
|
||||
Since the metrics for Kubernetes components are generally exposed on the host network of nodes in the cluster, PushProx deploys a DaemonSet of clients that sit on the hostNetwork of each node and make an outbound connection to a single proxy that is sitting on the Kubernetes API. Prometheus can then be configured to proxy scrape requests through the proxy to each client, which allows it to scrape metrics from the internal Kubernetes components without requiring any inbound node ports to be open.
|
||||
|
||||
Refer to [Scraping Metrics with PushProx](#scraping-metrics-with-pushprox) for more.
|
||||
|
||||
## 5. Scraping and Exposing Metrics
|
||||
|
||||
### Defining what Metrics are Scraped
|
||||
|
||||
ServiceMonitors and PodMonitors define targets that are intended for Prometheus to scrape. The [Prometheus custom resource](https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/design.md#prometheus) tells Prometheus which ServiceMonitors or PodMonitors it should use to find out where to scrape metrics from.
|
||||
|
||||
The Prometheus Operator observes the ServiceMonitors and PodMonitors. When it observes that they are created or updated, it calls the Prometheus API to update the scrape configuration in the Prometheus custom resource and keep it in sync with the scrape configuration in the ServiceMonitors or PodMonitors. This scrape configuration tells Prometheus which endpoints to scrape metrics from and how it will label the metrics from those endpoints.
|
||||
|
||||
Prometheus scrapes all of the metrics defined in its scrape configuration at every `scrape_interval`, which is one minute by default.
|
||||
|
||||
The scrape configuration can be viewed as part of the Prometheus custom resource that is exposed in the Rancher UI.
|
||||
|
||||
### How the Prometheus Operator Sets up Metrics Scraping
|
||||
|
||||
The Prometheus Deployment or StatefulSet scrapes metrics, and the configuration of Prometheus is controlled by the Prometheus custom resources. The Prometheus Operator watches for Prometheus and Alertmanager resources, and when they are created, the Prometheus Operator creates a Deployment or StatefulSet for Prometheus or Alertmanager with the user-defined configuration.
|
||||
|
||||
When the Prometheus Operator observes ServiceMonitors, PodMonitors, and PrometheusRules being created, it knows that the scrape configuration needs to be updated in Prometheus. It updates Prometheus by first updating the configuration and rules files in the volumes of Prometheus's Deployment or StatefulSet. Then it calls the Prometheus API to sync the new configuration, resulting in the Prometheus Deployment or StatefulSet to be modified in place.
|
||||
|
||||
### How Kubernetes Component Metrics are Exposed
|
||||
|
||||
Prometheus scrapes metrics from deployments known as [exporters,](https://prometheus.io/docs/instrumenting/exporters/) which export the time series data in a format that Prometheus can ingest. In Prometheus, time series consist of streams of timestamped values belonging to the same metric and the same set of labeled dimensions.
|
||||
|
||||
### Scraping Metrics with PushProx
|
||||
|
||||
Certain internal Kubernetes components are scraped via a proxy deployed as part of Monitoring V2 called PushProx. For detailed information on PushProx, refer [here](#how-pushprox-works) and to the above [architecture](#1-architecture-overview) section.
|
||||
|
||||
### Scraping Metrics
|
||||
|
||||
The following Kubernetes components are directly scraped by Prometheus:
|
||||
|
||||
- kubelet*
|
||||
- ingress-nginx**
|
||||
- coreDns/kubeDns
|
||||
- kube-api-server
|
||||
|
||||
\* You can optionally use `hardenedKubelet.enabled` to use a PushProx, but that is not the default.
|
||||
|
||||
** For RKE and RKE2 clusters, ingress-nginx is deployed by default and treated as an internal Kubernetes component.
|
||||
|
||||
|
||||
### Scraping Metrics Based on Kubernetes Distribution
|
||||
|
||||
Metrics are scraped differently based on the Kubernetes distribution. For help with terminology, refer [here](#terminology). For details, see the table below:
|
||||
|
||||
<figcaption>How Metrics are Exposed to Prometheus</figcaption>
|
||||
|
||||
| Kubernetes Component | RKE | RKE2 | KubeADM | K3s |
|
||||
|-----|-----|-----|-----|-----|
|
||||
| kube-controller-manager | rkeControllerManager.enabled |rke2ControllerManager.enabled | kubeAdmControllerManager.enabled | k3sServer.enabled |
|
||||
| kube-scheduler | rkeScheduler.enabled | rke2Scheduler.enabled |kubeAdmScheduler.enabled | k3sServer.enabled |
|
||||
| etcd | rkeEtcd.enabled | rke2Etcd.enabled | kubeAdmEtcd.enabled | Not available |
|
||||
| kube-proxy | rkeProxy.enabled | rke2Proxy.enabled | kubeAdmProxy.enabled | k3sServer.enabled |
|
||||
| kubelet | Collects metrics directly exposed by kubelet | Collects metrics directly exposed by kubelet | Collects metrics directly exposed by kubelet | Collects metrics directly exposed by kubelet |
|
||||
| ingress-nginx* | Collects metrics directly exposed by kubelet, exposed by rkeIngressNginx.enabled | Collects metrics directly exposed by kubelet, Exposed by rke2IngressNginx.enabled | Not available | Not available |
|
||||
| coreDns/kubeDns | Collects metrics directly exposed by coreDns/kubeDns | Collects metrics directly exposed by coreDns/kubeDns | Collects metrics directly exposed by coreDns/kubeDns | Collects metrics directly exposed by coreDns/kubeDns |
|
||||
| kube-api-server | Collects metrics directly exposed by kube-api-server |Collects metrics directly exposed by kube-api-server | Collects metrics directly exposed by kube-appi-server | Collects metrics directly exposed by kube-api-server |
|
||||
|
||||
\* For RKE and RKE2 clusters, ingress-nginx is deployed by default and treated as an internal Kubernetes component.
|
||||
|
||||
### Terminology
|
||||
|
||||
- **kube-scheduler:** The internal Kubernetes component that uses information in the pod spec to decide on which node to run a pod.
|
||||
- **kube-controller-manager:** The internal Kubernetes component that is responsible for node management (detecting if a node fails), pod replication and endpoint creation.
|
||||
- **etcd:** The internal Kubernetes component that is the distributed key/value store which Kubernetes uses for persistent storage of all cluster information.
|
||||
- **kube-proxy:** The internal Kubernetes component that watches the API server for pods/services changes in order to maintain the network up to date.
|
||||
- **kubelet:** The internal Kubernetes component that watches the API server for pods on a node and makes sure they are running.
|
||||
- **ingress-nginx:** An Ingress controller for Kubernetes using NGINX as a reverse proxy and load balancer.
|
||||
- **coreDns/kubeDns:** The internal Kubernetes component responsible for DNS.
|
||||
- **kube-api-server:** The main internal Kubernetes component that is responsible for exposing APIs for the other master components.
|
||||
+105
@@ -0,0 +1,105 @@
|
||||
---
|
||||
title: Monitoring and Alerting
|
||||
description: Prometheus lets you view metrics from your different Rancher and Kubernetes objects. Learn about the scope of monitoring and how to enable cluster monitoring
|
||||
---
|
||||
|
||||
<head>
|
||||
<link rel="canonical" href="https://ranchermanager.docs.rancher.com/integrations-in-rancher/monitoring-and-alerting"/>
|
||||
</head>
|
||||
|
||||
The `rancher-monitoring` application can quickly deploy leading open-source monitoring and alerting solutions onto your cluster.
|
||||
|
||||
Introduced in Rancher v2.5, the application is powered by [Prometheus](https://prometheus.io/), [Grafana](https://grafana.com/grafana/), [Alertmanager](https://prometheus.io/docs/alerting/latest/alertmanager/), the [Prometheus Operator](https://github.com/prometheus-operator/prometheus-operator), and the [Prometheus adapter.](https://github.com/DirectXMan12/k8s-prometheus-adapter)
|
||||
|
||||
For information on V1 monitoring and alerting, available in Rancher v2.2 up to v2.4, please see the Rancher v2.0—v2.4 docs on [cluster monitoring](/versioned_docs/version-2.0-2.4/explanations/integrations-in-rancher/cluster-monitoring/cluster-monitoring.md), [alerting](/versioned_docs/version-2.0-2.4/explanations/integrations-in-rancher/cluster-alerts/cluster-alerts.md), [notifiers](/versioned_docs/version-2.0-2.4/explanations/integrations-in-rancher/notifiers.md) and other [tools](/versioned_docs/version-2.0-2.4/reference-guides/rancher-project-tools/rancher-project-tools.md).
|
||||
|
||||
Using the `rancher-monitoring` application, you can quickly deploy leading open-source monitoring and alerting solutions onto your cluster.
|
||||
|
||||
### Features
|
||||
|
||||
Prometheus lets you view metrics from your Rancher and Kubernetes objects. Using timestamps, Prometheus lets you query and view these metrics in easy-to-read graphs and visuals, either through the Rancher UI or Grafana, which is an analytics viewing platform deployed along with Prometheus.
|
||||
|
||||
By viewing data that Prometheus scrapes from your cluster control plane, nodes, and deployments, you can stay on top of everything happening in your cluster. You can then use these analytics to better run your organization: stop system emergencies before they start, develop maintenance strategies, or restore crashed servers.
|
||||
|
||||
The monitoring application:
|
||||
|
||||
- Monitors the state and processes of your cluster nodes, Kubernetes components, and software deployments.
|
||||
- Defines alerts based on metrics collected via Prometheus.
|
||||
- Creates custom Grafana dashboards.
|
||||
- Configures alert-based notifications via email, Slack, PagerDuty, etc. using Prometheus Alertmanager.
|
||||
- Defines precomputed, frequently needed or computationally expensive expressions as new time series based on metrics collected via Prometheus.
|
||||
- Exposes collected metrics from Prometheus to the Kubernetes Custom Metrics API via Prometheus Adapter for use in HPA.
|
||||
|
||||
See [How Monitoring Works](how-monitoring-works.md) for an explanation of how the monitoring components work together.
|
||||
|
||||
## Default Components and Deployments
|
||||
|
||||
### Built-in Dashboards
|
||||
|
||||
By default, the monitoring application deploys Grafana dashboards (curated by the [kube-prometheus](https://github.com/prometheus-operator/kube-prometheus) project) onto a cluster.
|
||||
|
||||
It also deploys an Alertmanager UI and a Prometheus UI. For more information about these tools, see [Built-in Dashboards.](built-in-dashboards.md)
|
||||
### Default Metrics Exporters
|
||||
|
||||
By default, Rancher Monitoring deploys exporters (such as [node-exporter](https://github.com/prometheus/node_exporter) and [kube-state-metrics](https://github.com/kubernetes/kube-state-metrics)).
|
||||
|
||||
These default exporters automatically scrape metrics for CPU and memory from all components of your Kubernetes cluster, including your workloads.
|
||||
|
||||
### Default Alerts
|
||||
|
||||
The monitoring application deploys some alerts by default. To see the default alerts, go to the [Alertmanager UI](built-in-dashboards.md#alertmanager-ui) and click **Expand all groups.**
|
||||
|
||||
### Components Exposed in the Rancher UI
|
||||
|
||||
For a list of monitoring components exposed in the Rancher UI, along with common use cases for editing them, see [this section.](how-monitoring-works.md#components-exposed-in-the-rancher-ui)
|
||||
|
||||
## Role-based Access Control
|
||||
|
||||
For more information on configuring access to monitoring, see [this page.](rbac-for-monitoring.md)
|
||||
|
||||
:::note
|
||||
|
||||
Rancher and Project read permissions don't necessarily apply to monitoring resources. See [monitoring-ui-view](rbac-for-monitoring.md#additional-monitoring-clusterroles) for more details.
|
||||
|
||||
:::
|
||||
|
||||
## Guides
|
||||
|
||||
- [Enable monitoring](../../how-to-guides/advanced-user-guides/monitoring-alerting-guides/enable-monitoring.md)
|
||||
- [Uninstall monitoring](../../how-to-guides/advanced-user-guides/monitoring-alerting-guides/uninstall-monitoring.md)
|
||||
- [Monitoring workloads](../../how-to-guides/advanced-user-guides/monitoring-alerting-guides/set-up-monitoring-for-workloads.md)
|
||||
- [Customizing Grafana dashboards](../../how-to-guides/advanced-user-guides/monitoring-alerting-guides/customize-grafana-dashboard.md)
|
||||
- [Persistent Grafana dashboards](../../how-to-guides/advanced-user-guides/monitoring-alerting-guides/create-persistent-grafana-dashboard.md)
|
||||
- [Debugging high memory usage](../../how-to-guides/advanced-user-guides/monitoring-alerting-guides/debug-high-memory-usage.md)
|
||||
|
||||
## Configuration
|
||||
|
||||
### Configuring Monitoring Resources in Rancher
|
||||
|
||||
The configuration reference assumes familiarity with how monitoring components work together. For more information, see [How Monitoring Works.](how-monitoring-works.md)
|
||||
|
||||
- [ServiceMonitor and PodMonitor](../../reference-guides/monitoring-v2-configuration/servicemonitors-and-podmonitors.md)
|
||||
- [Receiver](../../reference-guides/monitoring-v2-configuration/receivers.md)
|
||||
- [Route](../../reference-guides/monitoring-v2-configuration/routes.md)
|
||||
- [PrometheusRule](../../how-to-guides/advanced-user-guides/monitoring-v2-configuration-guides/advanced-configuration/prometheusrules.md)
|
||||
- [Prometheus](../../how-to-guides/advanced-user-guides/monitoring-v2-configuration-guides/advanced-configuration/prometheus.md)
|
||||
- [Alertmanager](../../how-to-guides/advanced-user-guides/monitoring-v2-configuration-guides/advanced-configuration/alertmanager.md)
|
||||
|
||||
### Configuring Helm Chart Options
|
||||
|
||||
For more information on `rancher-monitoring` chart options, including options to set resource limits and requests, see [Helm Chart Options](../../reference-guides/monitoring-v2-configuration/helm-chart-options.md).
|
||||
|
||||
## Windows Cluster Support
|
||||
|
||||
When deployed onto an RKE1 Windows cluster, Monitoring V2 will now automatically deploy a [windows-exporter](https://github.com/prometheus-community/windows_exporter) DaemonSet and set up a ServiceMonitor to collect metrics from each of the deployed Pods. This will populate Prometheus with `windows_` metrics that are akin to the `node_` metrics exported by [node_exporter](https://github.com/prometheus/node_exporter) for Linux hosts.
|
||||
|
||||
To be able to fully deploy Monitoring V2 for Windows, all of your Windows hosts must have a minimum [wins](https://github.com/rancher/wins) version of v0.1.0.
|
||||
|
||||
For more details on how to upgrade wins on existing Windows hosts, see [Windows cluster support for Monitoring V2.](windows-support.md).
|
||||
|
||||
|
||||
## Known Issues
|
||||
|
||||
There is a [known issue](https://github.com/rancher/rancher/issues/28787#issuecomment-693611821) that K3s clusters require more than the allotted default memory. If you enable monitoring on a K3s cluster, set `prometheus.prometheusSpec.resources.memory.limit` to 2500 Mi and `prometheus.prometheusSpec.resources.memory.request` to 1750 Mi.
|
||||
|
||||
See [Debugging High Memory Usage](../../how-to-guides/advanced-user-guides/monitoring-alerting-guides/debug-high-memory-usage.md) for advice and recommendations.
|
||||
+369
@@ -0,0 +1,369 @@
|
||||
---
|
||||
title: PromQL Expression Reference
|
||||
---
|
||||
|
||||
<head>
|
||||
<link rel="canonical" href="https://ranchermanager.docs.rancher.com/integrations-in-rancher/monitoring-and-alerting/promql-expressions"/>
|
||||
</head>
|
||||
|
||||
The PromQL expressions in this doc can be used to configure alerts.
|
||||
|
||||
For more information about querying the Prometheus time series database, refer to the official [Prometheus documentation.](https://prometheus.io/docs/prometheus/latest/querying/basics/)
|
||||
|
||||
|
||||
## Cluster Metrics
|
||||
|
||||
### Cluster CPU Utilization
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | `1 - (avg(irate(node_cpu_seconds_total{mode="idle"}[5m])) by (instance))` |
|
||||
| Summary | `1 - (avg(irate(node_cpu_seconds_total{mode="idle"}[5m])))` |
|
||||
|
||||
### Cluster Load Average
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | <table><tr><td>load1</td><td>`sum(node_load1) by (instance) / count(node_cpu_seconds_total{mode="system"}) by (instance)`</td></tr><tr><td>load5</td><td>`sum(node_load5) by (instance) / count(node_cpu_seconds_total{mode="system"}) by (instance)`</td></tr><tr><td>load15</td><td>`sum(node_load15) by (instance) / count(node_cpu_seconds_total{mode="system"}) by (instance)`</td></tr></table> |
|
||||
| Summary | <table><tr><td>load1</td><td>`sum(node_load1) by (instance) / count(node_cpu_seconds_total{mode="system"})`</td></tr><tr><td>load5</td><td>`sum(node_load5) by (instance) / count(node_cpu_seconds_total{mode="system"})`</td></tr><tr><td>load15</td><td>`sum(node_load15) by (instance) / count(node_cpu_seconds_total{mode="system"})`</td></tr></table> |
|
||||
|
||||
### Cluster Memory Utilization
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | `1 - sum(node_memory_MemAvailable_bytes) by (instance) / sum(node_memory_MemTotal_bytes) by (instance)` |
|
||||
| Summary | `1 - sum(node_memory_MemAvailable_bytes) / sum(node_memory_MemTotal_bytes)` |
|
||||
|
||||
### Cluster Disk Utilization
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | `(sum(node_filesystem_size_bytes{device!="rootfs"}) by (instance) - sum(node_filesystem_free_bytes{device!="rootfs"}) by (instance)) / sum(node_filesystem_size_bytes{device!="rootfs"}) by (instance)` |
|
||||
| Summary | `(sum(node_filesystem_size_bytes{device!="rootfs"}) - sum(node_filesystem_free_bytes{device!="rootfs"})) / sum(node_filesystem_size_bytes{device!="rootfs"})` |
|
||||
|
||||
### Cluster Disk I/O
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | <table><tr><td>read</td><td>`sum(rate(node_disk_read_bytes_total[5m])) by (instance)`</td></tr><tr><td>written</td><td>`sum(rate(node_disk_written_bytes_total[5m])) by (instance)`</td></tr></table> |
|
||||
| Summary | <table><tr><td>read</td><td>`sum(rate(node_disk_read_bytes_total[5m]))`</td></tr><tr><td>written</td><td>`sum(rate(node_disk_written_bytes_total[5m]))`</td></tr></table> |
|
||||
|
||||
### Cluster Network Packets
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | <table><tr><td>receive-dropped</td><td><code>sum(rate(node_network_receive_drop_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*"}[5m])) by (instance)</code></td></tr><tr><td>receive-errs</td><td><code>sum(rate(node_network_receive_errs_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*"}[5m])) by (instance)</code></td></tr><tr><td>receive-packets</td><td><code>sum(rate(node_network_receive_packets_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*"}[5m])) by (instance)</code></td></tr><tr><td>transmit-dropped</td><td><code>sum(rate(node_network_transmit_drop_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*"}[5m])) by (instance)</code></td></tr><tr><td>transmit-errs</td><td><code>sum(rate(node_network_transmit_errs_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*"}[5m])) by (instance)</code></td></tr><tr><td>transmit-packets</td><td><code>sum(rate(node_network_transmit_packets_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*"}[5m])) by (instance)</code></td></tr></table> |
|
||||
| Summary | <table><tr><td>receive-dropped</td><td><code>sum(rate(node_network_receive_drop_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*"}[5m]))</code></td></tr><tr><td>receive-errs</td><td><code>sum(rate(node_network_receive_errs_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*"}[5m]))</code></td></tr><tr><td>receive-packets</td><td><code>sum(rate(node_network_receive_packets_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*"}[5m]))</code></td></tr><tr><td>transmit-dropped</td><td><code>sum(rate(node_network_transmit_drop_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*"}[5m]))</code></td></tr><tr><td>transmit-errs</td><td><code>sum(rate(node_network_transmit_errs_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*"}[5m]))</code></td></tr><tr><td>transmit-packets</td><td><code>sum(rate(node_network_transmit_packets_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*"}[5m]))</code></td></tr></table> |
|
||||
|
||||
### Cluster Network I/O
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | <table><tr><td>receive</td><td><code>sum(rate(node_network_receive_bytes_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*"}[5m])) by (instance)</code></td></tr><tr><td>transmit</td><td><code>sum(rate(node_network_transmit_bytes_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*"}[5m])) by (instance)</code></td></tr></table> |
|
||||
| Summary | <table><tr><td>receive</td><td><code>sum(rate(node_network_receive_bytes_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*"}[5m]))</code></td></tr><tr><td>transmit</td><td><code>sum(rate(node_network_transmit_bytes_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*"}[5m]))</code></td></tr></table> |
|
||||
|
||||
## Node Metrics
|
||||
|
||||
### Node CPU Utilization
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | `avg(irate(node_cpu_seconds_total{mode!="idle", instance=~"$instance"}[5m])) by (mode)` |
|
||||
| Summary | `1 - (avg(irate(node_cpu_seconds_total{mode="idle", instance=~"$instance"}[5m])))` |
|
||||
|
||||
### Node Load Average
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | <table><tr><td>load1</td><td>`sum(node_load1{instance=~"$instance"}) / count(node_cpu_seconds_total{mode="system",instance=~"$instance"})`</td></tr><tr><td>load5</td><td>`sum(node_load5{instance=~"$instance"}) / count(node_cpu_seconds_total{mode="system",instance=~"$instance"})`</td></tr><tr><td>load15</td><td>`sum(node_load15{instance=~"$instance"}) / count(node_cpu_seconds_total{mode="system",instance=~"$instance"})`</td></tr></table> |
|
||||
| Summary | <table><tr><td>load1</td><td>`sum(node_load1{instance=~"$instance"}) / count(node_cpu_seconds_total{mode="system",instance=~"$instance"})`</td></tr><tr><td>load5</td><td>`sum(node_load5{instance=~"$instance"}) / count(node_cpu_seconds_total{mode="system",instance=~"$instance"})`</td></tr><tr><td>load15</td><td>`sum(node_load15{instance=~"$instance"}) / count(node_cpu_seconds_total{mode="system",instance=~"$instance"})`</td></tr></table> |
|
||||
|
||||
### Node Memory Utilization
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | `1 - sum(node_memory_MemAvailable_bytes{instance=~"$instance"}) / sum(node_memory_MemTotal_bytes{instance=~"$instance"})` |
|
||||
| Summary | `1 - sum(node_memory_MemAvailable_bytes{instance=~"$instance"}) / sum(node_memory_MemTotal_bytes{instance=~"$instance"}) ` |
|
||||
|
||||
### Node Disk Utilization
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | `(sum(node_filesystem_size_bytes{device!="rootfs",instance=~"$instance"}) by (device) - sum(node_filesystem_free_bytes{device!="rootfs",instance=~"$instance"}) by (device)) / sum(node_filesystem_size_bytes{device!="rootfs",instance=~"$instance"}) by (device)` |
|
||||
| Summary | `(sum(node_filesystem_size_bytes{device!="rootfs",instance=~"$instance"}) - sum(node_filesystem_free_bytes{device!="rootfs",instance=~"$instance"})) / sum(node_filesystem_size_bytes{device!="rootfs",instance=~"$instance"})` |
|
||||
|
||||
### Node Disk I/O
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | <table><tr><td>read</td><td>`sum(rate(node_disk_read_bytes_total{instance=~"$instance"}[5m]))`</td></tr><tr><td>written</td><td>`sum(rate(node_disk_written_bytes_total{instance=~"$instance"}[5m]))`</td></tr></table> |
|
||||
| Summary | <table><tr><td>read</td><td>`sum(rate(node_disk_read_bytes_total{instance=~"$instance"}[5m]))`</td></tr><tr><td>written</td><td>`sum(rate(node_disk_written_bytes_total{instance=~"$instance"}[5m]))`</td></tr></table> |
|
||||
|
||||
### Node Network Packets
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | <table><tr><td>receive-dropped</td><td><code>sum(rate(node_network_receive_drop_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*",instance=~"$instance"}[5m])) by (device)</code></td></tr><tr><td>receive-errs</td><td><code>sum(rate(node_network_receive_errs_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*",instance=~"$instance"}[5m])) by (device)</code></td></tr><tr><td>receive-packets</td><td><code>sum(rate(node_network_receive_packets_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*",instance=~"$instance"}[5m])) by (device)</code></td></tr><tr><td>transmit-dropped</td><td><code>sum(rate(node_network_transmit_drop_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*",instance=~"$instance"}[5m])) by (device)</code></td></tr><tr><td>transmit-errs</td><td><code>sum(rate(node_network_transmit_errs_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*",instance=~"$instance"}[5m])) by (device)</code></td></tr><tr><td>transmit-packets</td><td><code>sum(rate(node_network_transmit_packets_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*",instance=~"$instance"}[5m])) by (device)</code></td></tr></table> |
|
||||
| Summary | <table><tr><td>receive-dropped</td><td><code>sum(rate(node_network_receive_drop_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*",instance=~"$instance"}[5m]))</code></td></tr><tr><td>receive-errs</td><td><code>sum(rate(node_network_receive_errs_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*",instance=~"$instance"}[5m]))</code></td></tr><tr><td>receive-packets</td><td><code>sum(rate(node_network_receive_packets_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*",instance=~"$instance"}[5m]))</code></td></tr><tr><td>transmit-dropped</td><td><code>sum(rate(node_network_transmit_drop_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*",instance=~"$instance"}[5m]))</code></td></tr><tr><td>transmit-errs</td><td><code>sum(rate(node_network_transmit_errs_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*",instance=~"$instance"}[5m]))</code></td></tr><tr><td>transmit-packets</td><td><code>sum(rate(node_network_transmit_packets_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*",instance=~"$instance"}[5m]))</code></td></tr></table> |
|
||||
|
||||
### Node Network I/O
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | <table><tr><td>receive</td><td><code>sum(rate(node_network_receive_bytes_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*",instance=~"$instance"}[5m])) by (device)</code></td></tr><tr><td>transmit</td><td><code>sum(rate(node_network_transmit_bytes_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*",instance=~"$instance"}[5m])) by (device)</code></td></tr></table> |
|
||||
| Summary | <table><tr><td>receive</td><td><code>sum(rate(node_network_receive_bytes_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*",instance=~"$instance"}[5m]))</code></td></tr><tr><td>transmit</td><td><code>sum(rate(node_network_transmit_bytes_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*",instance=~"$instance"}[5m]))</code></td></tr></table> |
|
||||
|
||||
## Etcd Metrics
|
||||
|
||||
### Etcd Has a Leader
|
||||
|
||||
`max(etcd_server_has_leader)`
|
||||
|
||||
### Number of Times the Leader Changes
|
||||
|
||||
`max(etcd_server_leader_changes_seen_total)`
|
||||
|
||||
### Number of Failed Proposals
|
||||
|
||||
`sum(etcd_server_proposals_failed_total)`
|
||||
|
||||
### GRPC Client Traffic
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | <table><tr><td>in</td><td>`sum(rate(etcd_network_client_grpc_received_bytes_total[5m])) by (instance)`</td></tr><tr><td>out</td><td>`sum(rate(etcd_network_client_grpc_sent_bytes_total[5m])) by (instance)`</td></tr></table> |
|
||||
| Summary | <table><tr><td>in</td><td>`sum(rate(etcd_network_client_grpc_received_bytes_total[5m]))`</td></tr><tr><td>out</td><td>`sum(rate(etcd_network_client_grpc_sent_bytes_total[5m]))`</td></tr></table> |
|
||||
|
||||
### Peer Traffic
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | <table><tr><td>in</td><td>`sum(rate(etcd_network_peer_received_bytes_total[5m])) by (instance)`</td></tr><tr><td>out</td><td>`sum(rate(etcd_network_peer_sent_bytes_total[5m])) by (instance)`</td></tr></table> |
|
||||
| Summary | <table><tr><td>in</td><td>`sum(rate(etcd_network_peer_received_bytes_total[5m]))`</td></tr><tr><td>out</td><td>`sum(rate(etcd_network_peer_sent_bytes_total[5m]))`</td></tr></table> |
|
||||
|
||||
### DB Size
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | `sum(etcd_debugging_mvcc_db_total_size_in_bytes) by (instance)` |
|
||||
| Summary | `sum(etcd_debugging_mvcc_db_total_size_in_bytes)` |
|
||||
|
||||
### Active Streams
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | <table><tr><td>lease-watch</td><td>`sum(grpc_server_started_total{grpc_service="etcdserverpb.Lease",grpc_type="bidi_stream"}) by (instance) - sum(grpc_server_handled_total{grpc_service="etcdserverpb.Lease",grpc_type="bidi_stream"}) by (instance)`</td></tr><tr><td>watch</td><td>`sum(grpc_server_started_total{grpc_service="etcdserverpb.Watch",grpc_type="bidi_stream"}) by (instance) - sum(grpc_server_handled_total{grpc_service="etcdserverpb.Watch",grpc_type="bidi_stream"}) by (instance)`</td></tr></table> |
|
||||
| Summary | <table><tr><td>lease-watch</td><td>`sum(grpc_server_started_total{grpc_service="etcdserverpb.Lease",grpc_type="bidi_stream"}) - sum(grpc_server_handled_total{grpc_service="etcdserverpb.Lease",grpc_type="bidi_stream"})`</td></tr><tr><td>watch</td><td>`sum(grpc_server_started_total{grpc_service="etcdserverpb.Watch",grpc_type="bidi_stream"}) - sum(grpc_server_handled_total{grpc_service="etcdserverpb.Watch",grpc_type="bidi_stream"})`</td></tr></table> |
|
||||
|
||||
### Raft Proposals
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | <table><tr><td>applied</td><td>`sum(increase(etcd_server_proposals_applied_total[5m])) by (instance)`</td></tr><tr><td>committed</td><td>`sum(increase(etcd_server_proposals_committed_total[5m])) by (instance)`</td></tr><tr><td>pending</td><td>`sum(increase(etcd_server_proposals_pending[5m])) by (instance)`</td></tr><tr><td>failed</td><td>`sum(increase(etcd_server_proposals_failed_total[5m])) by (instance)`</td></tr></table> |
|
||||
| Summary | <table><tr><td>applied</td><td>`sum(increase(etcd_server_proposals_applied_total[5m]))`</td></tr><tr><td>committed</td><td>`sum(increase(etcd_server_proposals_committed_total[5m]))`</td></tr><tr><td>pending</td><td>`sum(increase(etcd_server_proposals_pending[5m]))`</td></tr><tr><td>failed</td><td>`sum(increase(etcd_server_proposals_failed_total[5m]))`</td></tr></table> |
|
||||
|
||||
### RPC Rate
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | <table><tr><td>total</td><td>`sum(rate(grpc_server_started_total{grpc_type="unary"}[5m])) by (instance)`</td></tr><tr><td>fail</td><td>`sum(rate(grpc_server_handled_total{grpc_type="unary",grpc_code!="OK"}[5m])) by (instance)`</td></tr></table> |
|
||||
| Summary | <table><tr><td>total</td><td>`sum(rate(grpc_server_started_total{grpc_type="unary"}[5m]))`</td></tr><tr><td>fail</td><td>`sum(rate(grpc_server_handled_total{grpc_type="unary",grpc_code!="OK"}[5m]))`</td></tr></table> |
|
||||
|
||||
### Disk Operations
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | <table><tr><td>commit-called-by-backend</td><td>`sum(rate(etcd_disk_backend_commit_duration_seconds_sum[1m])) by (instance)`</td></tr><tr><td>fsync-called-by-wal</td><td>`sum(rate(etcd_disk_wal_fsync_duration_seconds_sum[1m])) by (instance)`</td></tr></table> |
|
||||
| Summary | <table><tr><td>commit-called-by-backend</td><td>`sum(rate(etcd_disk_backend_commit_duration_seconds_sum[1m]))`</td></tr><tr><td>fsync-called-by-wal</td><td>`sum(rate(etcd_disk_wal_fsync_duration_seconds_sum[1m]))`</td></tr></table> |
|
||||
|
||||
### Disk Sync Duration
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | <table><tr><td>wal</td><td>`histogram_quantile(0.99, sum(rate(etcd_disk_wal_fsync_duration_seconds_bucket[5m])) by (instance, le))`</td></tr><tr><td>db</td><td>`histogram_quantile(0.99, sum(rate(etcd_disk_backend_commit_duration_seconds_bucket[5m])) by (instance, le))`</td></tr></table> |
|
||||
| Summary | <table><tr><td>wal</td><td>`sum(histogram_quantile(0.99, sum(rate(etcd_disk_wal_fsync_duration_seconds_bucket[5m])) by (instance, le)))`</td></tr><tr><td>db</td><td>`sum(histogram_quantile(0.99, sum(rate(etcd_disk_backend_commit_duration_seconds_bucket[5m])) by (instance, le)))`</td></tr></table> |
|
||||
|
||||
## Kubernetes Components Metrics
|
||||
|
||||
### API Server Request Latency
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | `avg(apiserver_request_latencies_sum / apiserver_request_latencies_count) by (instance, verb) /1e+06` |
|
||||
| Summary | `avg(apiserver_request_latencies_sum / apiserver_request_latencies_count) by (instance) /1e+06` |
|
||||
|
||||
### API Server Request Rate
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | `sum(rate(apiserver_request_count[5m])) by (instance, code)` |
|
||||
| Summary | `sum(rate(apiserver_request_count[5m])) by (instance)` |
|
||||
|
||||
### Scheduling Failed Pods
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | `sum(kube_pod_status_scheduled{condition="false"})` |
|
||||
| Summary | `sum(kube_pod_status_scheduled{condition="false"})` |
|
||||
|
||||
### Controller Manager Queue Depth
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | <table><tr><td>volumes</td><td>`sum(volumes_depth) by instance`</td></tr><tr><td>deployment</td><td>`sum(deployment_depth) by instance`</td></tr><tr><td>replicaset</td><td>`sum(replicaset_depth) by instance`</td></tr><tr><td>service</td><td>`sum(service_depth) by instance`</td></tr><tr><td>serviceaccount</td><td>`sum(serviceaccount_depth) by instance`</td></tr><tr><td>endpoint</td><td>`sum(endpoint_depth) by instance`</td></tr><tr><td>daemonset</td><td>`sum(daemonset_depth) by instance`</td></tr><tr><td>statefulset</td><td>`sum(statefulset_depth) by instance`</td></tr><tr><td>replicationmanager</td><td>`sum(replicationmanager_depth) by instance`</td></tr></table> |
|
||||
| Summary | <table><tr><td>volumes</td><td>`sum(volumes_depth)`</td></tr><tr><td>deployment</td><td>`sum(deployment_depth)`</td></tr><tr><td>replicaset</td><td>`sum(replicaset_depth)`</td></tr><tr><td>service</td><td>`sum(service_depth)`</td></tr><tr><td>serviceaccount</td><td>`sum(serviceaccount_depth)`</td></tr><tr><td>endpoint</td><td>`sum(endpoint_depth)`</td></tr><tr><td>daemonset</td><td>`sum(daemonset_depth)`</td></tr><tr><td>statefulset</td><td>`sum(statefulset_depth)`</td></tr><tr><td>replicationmanager</td><td>`sum(replicationmanager_depth)`</td></tr></table> |
|
||||
|
||||
### Scheduler E2E Scheduling Latency
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | `histogram_quantile(0.99, sum(scheduler_e2e_scheduling_latency_microseconds_bucket) by (le, instance)) / 1e+06` |
|
||||
| Summary | `sum(histogram_quantile(0.99, sum(scheduler_e2e_scheduling_latency_microseconds_bucket) by (le, instance)) / 1e+06)` |
|
||||
|
||||
### Scheduler Preemption Attempts
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | `sum(rate(scheduler_total_preemption_attempts[5m])) by (instance)` |
|
||||
| Summary | `sum(rate(scheduler_total_preemption_attempts[5m]))` |
|
||||
|
||||
### Ingress Controller Connections
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | <table><tr><td>reading</td><td>`sum(nginx_ingress_controller_nginx_process_connections{state="reading"}) by (instance)`</td></tr><tr><td>waiting</td><td>`sum(nginx_ingress_controller_nginx_process_connections{state="waiting"}) by (instance)`</td></tr><tr><td>writing</td><td>`sum(nginx_ingress_controller_nginx_process_connections{state="writing"}) by (instance)`</td></tr><tr><td>accepted</td><td>`sum(ceil(increase(nginx_ingress_controller_nginx_process_connections_total{state="accepted"}[5m]))) by (instance)`</td></tr><tr><td>active</td><td>`sum(ceil(increase(nginx_ingress_controller_nginx_process_connections_total{state="active"}[5m]))) by (instance)`</td></tr><tr><td>handled</td><td>`sum(ceil(increase(nginx_ingress_controller_nginx_process_connections_total{state="handled"}[5m]))) by (instance)`</td></tr></table> |
|
||||
| Summary | <table><tr><td>reading</td><td>`sum(nginx_ingress_controller_nginx_process_connections{state="reading"})`</td></tr><tr><td>waiting</td><td>`sum(nginx_ingress_controller_nginx_process_connections{state="waiting"})`</td></tr><tr><td>writing</td><td>`sum(nginx_ingress_controller_nginx_process_connections{state="writing"})`</td></tr><tr><td>accepted</td><td>`sum(ceil(increase(nginx_ingress_controller_nginx_process_connections_total{state="accepted"}[5m])))`</td></tr><tr><td>active</td><td>`sum(ceil(increase(nginx_ingress_controller_nginx_process_connections_total{state="active"}[5m])))`</td></tr><tr><td>handled</td><td>`sum(ceil(increase(nginx_ingress_controller_nginx_process_connections_total{state="handled"}[5m])))`</td></tr></table> |
|
||||
|
||||
### Ingress Controller Request Process Time
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | `topk(10, histogram_quantile(0.95,sum by (le, host, path)(rate(nginx_ingress_controller_request_duration_seconds_bucket{host!="_"}[5m]))))` |
|
||||
| Summary | `topk(10, histogram_quantile(0.95,sum by (le, host)(rate(nginx_ingress_controller_request_duration_seconds_bucket{host!="_"}[5m]))))` |
|
||||
|
||||
## Rancher Logging Metrics
|
||||
|
||||
|
||||
### Fluentd Buffer Queue Rate
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | `sum(rate(fluentd_output_status_buffer_queue_length[5m])) by (instance)` |
|
||||
| Summary | `sum(rate(fluentd_output_status_buffer_queue_length[5m]))` |
|
||||
|
||||
### Fluentd Input Rate
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | `sum(rate(fluentd_input_status_num_records_total[5m])) by (instance)` |
|
||||
| Summary | `sum(rate(fluentd_input_status_num_records_total[5m]))` |
|
||||
|
||||
### Fluentd Output Errors Rate
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | `sum(rate(fluentd_output_status_num_errors[5m])) by (type)` |
|
||||
| Summary | `sum(rate(fluentd_output_status_num_errors[5m]))` |
|
||||
|
||||
### Fluentd Output Rate
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | `sum(rate(fluentd_output_status_num_records_total[5m])) by (instance)` |
|
||||
| Summary | `sum(rate(fluentd_output_status_num_records_total[5m]))` |
|
||||
|
||||
## Workload Metrics
|
||||
|
||||
### Workload CPU Utilization
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | <table><tr><td>cfs throttled seconds</td><td>`sum(rate(container_cpu_cfs_throttled_seconds_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m])) by (pod_name)`</td></tr><tr><td>user seconds</td><td>`sum(rate(container_cpu_user_seconds_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m])) by (pod_name)`</td></tr><tr><td>system seconds</td><td>`sum(rate(container_cpu_system_seconds_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m])) by (pod_name)`</td></tr><tr><td>usage seconds</td><td>`sum(rate(container_cpu_usage_seconds_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m])) by (pod_name)`</td></tr></table> |
|
||||
| Summary | <table><tr><td>cfs throttled seconds</td><td>`sum(rate(container_cpu_cfs_throttled_seconds_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m]))`</td></tr><tr><td>user seconds</td><td>`sum(rate(container_cpu_user_seconds_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m]))`</td></tr><tr><td>system seconds</td><td>`sum(rate(container_cpu_system_seconds_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m]))`</td></tr><tr><td>usage seconds</td><td>`sum(rate(container_cpu_usage_seconds_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m]))`</td></tr></table> |
|
||||
|
||||
### Workload Memory Utilization
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | `sum(container_memory_working_set_bytes{namespace="$namespace",pod_name=~"$podName", container_name!=""}) by (pod_name)` |
|
||||
| Summary | `sum(container_memory_working_set_bytes{namespace="$namespace",pod_name=~"$podName", container_name!=""})` |
|
||||
|
||||
### Workload Network Packets
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | <table><tr><td>receive-packets</td><td>`sum(rate(container_network_receive_packets_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m])) by (pod_name)`</td></tr><tr><td>receive-dropped</td><td>`sum(rate(container_network_receive_packets_dropped_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m])) by (pod_name)`</td></tr><tr><td>receive-errors</td><td>`sum(rate(container_network_receive_errors_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m])) by (pod_name)`</td></tr><tr><td>transmit-packets</td><td>`sum(rate(container_network_transmit_packets_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m])) by (pod_name)`</td></tr><tr><td>transmit-dropped</td><td>`sum(rate(container_network_transmit_packets_dropped_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m])) by (pod_name)`</td></tr><tr><td>transmit-errors</td><td>`sum(rate(container_network_transmit_errors_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m])) by (pod_name)`</td></tr></table> |
|
||||
| Summary | <table><tr><td>receive-packets</td><td>`sum(rate(container_network_receive_packets_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m]))`</td></tr><tr><td>receive-dropped</td><td>`sum(rate(container_network_receive_packets_dropped_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m]))`</td></tr><tr><td>receive-errors</td><td>`sum(rate(container_network_receive_errors_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m]))`</td></tr><tr><td>transmit-packets</td><td>`sum(rate(container_network_transmit_packets_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m]))`</td></tr><tr><td>transmit-dropped</td><td>`sum(rate(container_network_transmit_packets_dropped_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m]))`</td></tr><tr><td>transmit-errors</td><td>`sum(rate(container_network_transmit_errors_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m]))`</td></tr></table> |
|
||||
|
||||
### Workload Network I/O
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | <table><tr><td>receive</td><td>`sum(rate(container_network_receive_bytes_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m])) by (pod_name)`</td></tr><tr><td>transmit</td><td>`sum(rate(container_network_transmit_bytes_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m])) by (pod_name)`</td></tr></table> |
|
||||
| Summary | <table><tr><td>receive</td><td>`sum(rate(container_network_receive_bytes_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m]))`</td></tr><tr><td>transmit</td><td>`sum(rate(container_network_transmit_bytes_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m]))`</td></tr></table> |
|
||||
|
||||
### Workload Disk I/O
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | <table><tr><td>read</td><td>`sum(rate(container_fs_reads_bytes_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m])) by (pod_name)`</td></tr><tr><td>write</td><td>`sum(rate(container_fs_writes_bytes_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m])) by (pod_name)`</td></tr></table> |
|
||||
| Summary | <table><tr><td>read</td><td>`sum(rate(container_fs_reads_bytes_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m]))`</td></tr><tr><td>write</td><td>`sum(rate(container_fs_writes_bytes_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m]))`</td></tr></table> |
|
||||
|
||||
## Pod Metrics
|
||||
|
||||
### Pod CPU Utilization
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | <table><tr><td>cfs throttled seconds</td><td>`sum(rate(container_cpu_cfs_throttled_seconds_total{container_name!="POD",namespace="$namespace",pod_name="$podName", container_name!=""}[5m])) by (container_name)`</td></tr><tr><td>usage seconds</td><td>`sum(rate(container_cpu_usage_seconds_total{container_name!="POD",namespace="$namespace",pod_name="$podName", container_name!=""}[5m])) by (container_name)`</td></tr><tr><td>system seconds</td><td>`sum(rate(container_cpu_system_seconds_total{container_name!="POD",namespace="$namespace",pod_name="$podName", container_name!=""}[5m])) by (container_name)`</td></tr><tr><td>user seconds</td><td>`sum(rate(container_cpu_user_seconds_total{container_name!="POD",namespace="$namespace",pod_name="$podName", container_name!=""}[5m])) by (container_name)`</td></tr></table> |
|
||||
| Summary | <table><tr><td>cfs throttled seconds</td><td>`sum(rate(container_cpu_cfs_throttled_seconds_total{container_name!="POD",namespace="$namespace",pod_name="$podName", container_name!=""}[5m]))`</td></tr><tr><td>usage seconds</td><td>`sum(rate(container_cpu_usage_seconds_total{container_name!="POD",namespace="$namespace",pod_name="$podName", container_name!=""}[5m]))`</td></tr><tr><td>system seconds</td><td>`sum(rate(container_cpu_system_seconds_total{container_name!="POD",namespace="$namespace",pod_name="$podName", container_name!=""}[5m]))`</td></tr><tr><td>user seconds</td><td>`sum(rate(container_cpu_user_seconds_total{container_name!="POD",namespace="$namespace",pod_name="$podName", container_name!=""}[5m]))`</td></tr></table> |
|
||||
|
||||
### Pod Memory Utilization
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | `sum(container_memory_working_set_bytes{container_name!="POD",namespace="$namespace",pod_name="$podName",container_name!=""}) by (container_name)` |
|
||||
| Summary | `sum(container_memory_working_set_bytes{container_name!="POD",namespace="$namespace",pod_name="$podName",container_name!=""})` |
|
||||
|
||||
### Pod Network Packets
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | <table><tr><td>receive-packets</td><td>`sum(rate(container_network_receive_packets_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`</td></tr><tr><td>receive-dropped</td><td>`sum(rate(container_network_receive_packets_dropped_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`</td></tr><tr><td>receive-errors</td><td>`sum(rate(container_network_receive_errors_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`</td></tr><tr><td>transmit-packets</td><td>`sum(rate(container_network_transmit_packets_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`</td></tr><tr><td>transmit-dropped</td><td>`sum(rate(container_network_transmit_packets_dropped_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`</td></tr><tr><td>transmit-errors</td><td>`sum(rate(container_network_transmit_errors_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`</td></tr></table> |
|
||||
| Summary | <table><tr><td>receive-packets</td><td>`sum(rate(container_network_receive_packets_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`</td></tr><tr><td>receive-dropped</td><td>`sum(rate(container_network_receive_packets_dropped_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`</td></tr><tr><td>receive-errors</td><td>`sum(rate(container_network_receive_errors_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`</td></tr><tr><td>transmit-packets</td><td>`sum(rate(container_network_transmit_packets_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`</td></tr><tr><td>transmit-dropped</td><td>`sum(rate(container_network_transmit_packets_dropped_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`</td></tr><tr><td>transmit-errors</td><td>`sum(rate(container_network_transmit_errors_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`</td></tr></table> |
|
||||
|
||||
### Pod Network I/O
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | <table><tr><td>receive</td><td>`sum(rate(container_network_receive_bytes_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`</td></tr><tr><td>transmit</td><td>`sum(rate(container_network_transmit_bytes_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`</td></tr></table> |
|
||||
| Summary | <table><tr><td>receive</td><td>`sum(rate(container_network_receive_bytes_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`</td></tr><tr><td>transmit</td><td>`sum(rate(container_network_transmit_bytes_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`</td></tr></table> |
|
||||
|
||||
### Pod Disk I/O
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | <table><tr><td>read</td><td>`sum(rate(container_fs_reads_bytes_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m])) by (container_name)`</td></tr><tr><td>write</td><td>`sum(rate(container_fs_writes_bytes_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m])) by (container_name)`</td></tr></table> |
|
||||
| Summary | <table><tr><td>read</td><td>`sum(rate(container_fs_reads_bytes_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`</td></tr><tr><td>write</td><td>`sum(rate(container_fs_writes_bytes_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`</td></tr></table> |
|
||||
|
||||
## Container Metrics
|
||||
|
||||
### Container CPU Utilization
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| cfs throttled seconds | `sum(rate(container_cpu_cfs_throttled_seconds_total{namespace="$namespace",pod_name="$podName",container_name="$containerName"}[5m]))` |
|
||||
| usage seconds | `sum(rate(container_cpu_usage_seconds_total{namespace="$namespace",pod_name="$podName",container_name="$containerName"}[5m]))` |
|
||||
| system seconds | `sum(rate(container_cpu_system_seconds_total{namespace="$namespace",pod_name="$podName",container_name="$containerName"}[5m]))` |
|
||||
| user seconds | `sum(rate(container_cpu_user_seconds_total{namespace="$namespace",pod_name="$podName",container_name="$containerName"}[5m]))` |
|
||||
|
||||
### Container Memory Utilization
|
||||
|
||||
`sum(container_memory_working_set_bytes{namespace="$namespace",pod_name="$podName",container_name="$containerName"})`
|
||||
|
||||
### Container Disk I/O
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| read | `sum(rate(container_fs_reads_bytes_total{namespace="$namespace",pod_name="$podName",container_name="$containerName"}[5m]))` |
|
||||
| write | `sum(rate(container_fs_writes_bytes_total{namespace="$namespace",pod_name="$podName",container_name="$containerName"}[5m]))` |
|
||||
+266
@@ -0,0 +1,266 @@
|
||||
---
|
||||
title: Role-based Access Control
|
||||
---
|
||||
|
||||
<head>
|
||||
<link rel="canonical" href="https://ranchermanager.docs.rancher.com/integrations-in-rancher/monitoring-and-alerting/rbac-for-monitoring"/>
|
||||
</head>
|
||||
|
||||
This section describes the expectations for RBAC for Rancher Monitoring.
|
||||
|
||||
|
||||
## Cluster Admins
|
||||
|
||||
By default, only those with the cluster-admin `ClusterRole` should be able to:
|
||||
|
||||
- Install the `rancher-monitoring` App onto a cluster and all other relevant configuration performed on the chart deploy
|
||||
- e.g. whether default dashboards are created, what exporters are deployed onto the cluster to collect metrics, etc.
|
||||
- Create / modify / delete Prometheus deployments in the cluster via Prometheus CRs
|
||||
- Create / modify / delete Alertmanager deployments in the cluster via Alertmanager CRs
|
||||
- Persist new Grafana dashboards or datasources via creating ConfigMaps in the appropriate namespace
|
||||
- Expose certain Prometheus metrics to the k8s Custom Metrics API for HPA via a Secret in the `cattle-monitoring-system` namespace
|
||||
|
||||
## Users with Kubernetes ClusterRole-based Permissions
|
||||
|
||||
The `rancher-monitoring` chart installs the following three `ClusterRoles`. By default, they aggregate into the corresponding k8s `ClusterRoles`:
|
||||
|
||||
| ClusterRole | Aggregates To Default K8s ClusterRole |
|
||||
| ------------------------------| ---------------------------|
|
||||
| `monitoring-admin` | `admin`|
|
||||
| `monitoring-edit` | `edit` |
|
||||
| `monitoring-view` | `view ` |
|
||||
|
||||
These `ClusterRoles` provide different levels of access to the Monitoring CRDs based on the actions that can be performed:
|
||||
|
||||
| CRDs (monitoring.coreos.com) | Admin | Edit | View |
|
||||
| ------------------------------| ---------------------------| ---------------------------| ---------------------------|
|
||||
| <ul><li>`prometheuses`</li><li>`alertmanagers`</li></ul>| Get, List, Watch | Get, List, Watch | Get, List, Watch |
|
||||
| <ul><li>`servicemonitors`</li><li>`podmonitors`</li><li>`prometheusrules`</li></ul>| * | * | Get, List, Watch |
|
||||
|
||||
On a high level, the following permissions are assigned by default as a result.
|
||||
|
||||
### Users with Kubernetes Admin/Edit Permissions
|
||||
|
||||
Only those with the the cluster-admin, admin or edit `ClusterRole` should be able to:
|
||||
|
||||
- Modify the scrape configuration of Prometheus deployments via ServiceMonitor and PodMonitor CRs
|
||||
- Modify the alerting / recording rules of a Prometheus deployment via PrometheusRules CRs
|
||||
|
||||
### Users with Kubernetes View Permissions
|
||||
|
||||
Only those with who have some Kubernetes `ClusterRole` should be able to:
|
||||
|
||||
- View the configuration of Prometheuses that are deployed within the cluster
|
||||
- View the configuration of Alertmanagers that are deployed within the cluster
|
||||
- View the scrape configuration of Prometheus deployments via ServiceMonitor and PodMonitor CRs
|
||||
- View the alerting/recording rules of a Prometheus deployment via PrometheusRules CRs
|
||||
|
||||
### Additional Monitoring Roles
|
||||
|
||||
Monitoring also creates additional `Roles` that are not assigned to users by default but are created within the cluster. They can be bound to a namespace by deploying a `RoleBinding` that references it. To define a `RoleBinding` with `kubectl` instead of through Rancher, click [here](#assigning-roles-and-clusterroles-with-kubectl).
|
||||
|
||||
Admins should use these roles to provide more fine-grained access to users:
|
||||
|
||||
| Role | Purpose |
|
||||
| ------------------------------| ---------------------------|
|
||||
| monitoring-config-admin | Allow admins to assign roles to users to be able to view / modify Secrets and ConfigMaps within the cattle-monitoring-system namespace. Modifying Secrets / ConfigMaps in this namespace could allow users to alter the cluster's Alertmanager configuration, Prometheus Adapter configuration, additional Grafana datasources, TLS secrets, etc. |
|
||||
| monitoring-config-edit | Allow admins to assign roles to users to be able to view / modify Secrets and ConfigMaps within the cattle-monitoring-system namespace. Modifying Secrets / ConfigMaps in this namespace could allow users to alter the cluster's Alertmanager configuration, Prometheus Adapter configuration, additional Grafana datasources, TLS secrets, etc. |
|
||||
| monitoring-config-view | Allow admins to assign roles to users to be able to view Secrets and ConfigMaps within the cattle-monitoring-system namespace. Viewing Secrets / ConfigMaps in this namespace could allow users to observe the cluster's Alertmanager configuration, Prometheus Adapter configuration, additional Grafana datasources, TLS secrets, etc. |
|
||||
| monitoring-dashboard-admin | Allow admins to assign roles to users to be able to edit / view ConfigMaps within the cattle-dashboards namespace. ConfigMaps in this namespace will correspond to Grafana Dashboards that are persisted onto the cluster. |
|
||||
| monitoring-dashboard-edit | Allow admins to assign roles to users to be able to edit / view ConfigMaps within the cattle-dashboards namespace. ConfigMaps in this namespace will correspond to Grafana Dashboards that are persisted onto the cluster. |
|
||||
| monitoring-dashboard-view | Allow admins to assign roles to users to be able to view ConfigMaps within the cattle-dashboards namespace. ConfigMaps in this namespace will correspond to Grafana Dashboards that are persisted onto the cluster. |
|
||||
|
||||
|
||||
### Assigning Monitoring Roles through Custom Roles
|
||||
|
||||
Admins may assign custom roles in the Rancher UI for admin, editing, and viewing monitoring. These "roles" are created by default when the monitoring app is installed. Additionally, these roles are also deployed to the corresponding Kubernetes roles: admin, edit, and view `ClusterRoles`.
|
||||
|
||||
:::note Important
|
||||
|
||||
The UI won't offer `monitoring-admin`, `monitoring-edit`, and `monitoring-view` options when users are being added to a cluster. These monitoring roles can only be assigned by manually creating a custom role that inherits from Project Owner and Project Monitoring View roles.
|
||||
|
||||
:::
|
||||
|
||||
1. Create the custom role:
|
||||
|
||||
1.1 Click **☰ > Users & Authentication > Roles**.
|
||||
|
||||
1.2 Select the appropriate tab, e.g., **Cluster** role. Then click **Create Cluster Role**.
|
||||
|
||||
1.3 In the **Name** field, create a custom role such as `View Monitoring`, `Edit Monitoring`, or `Admin Monitoring`.
|
||||
|
||||
1.4 Click **Inherit From > Add Resource**, then select the Kubernetes role, as applicable, from the dropdown.
|
||||
|
||||
1.5 Click **Create**.
|
||||
|
||||
|
||||
2. Assign the custom role to a new user:
|
||||
|
||||
2.1 Click **☰ > Cluster Management > Cluster Explore > Cluster > Cluster Members > Add**.
|
||||
|
||||
2.2 Search for your new user name from **Select Member** options displayed.
|
||||
|
||||
2.3 Assign the new custom role from **Cluster Permissions** to the new user.
|
||||
|
||||
2.4 Click **Create**.
|
||||
|
||||
**Result:** The new user should now be able to see the monitoring tools.
|
||||
|
||||
### Additional Monitoring ClusterRoles
|
||||
|
||||
Monitoring also creates additional `ClusterRoles` that aren't assigned to users by default but are created within the cluster. They aren't aggregated by default but can be bound to a namespace by deploying a `RoleBinding` or `ClusterRoleBinding` that references it. To define a `RoleBinding` with `kubectl` instead of through Rancher, click [here](#assigning-roles-and-clusterroles-with-kubectl).
|
||||
|
||||
| Role | Purpose |
|
||||
| ------------------------------| ---------------------------|
|
||||
| monitoring-ui-view | _Available as of Monitoring v2 14.5.100+_ This ClusterRole allows users with write access to the project to view metrics graphs for the specified cluster in the Rancher UI. This is done by granting Read-only access to external Monitoring UIs. Users with this role have permission to list the Prometheus, Alertmanager, and Grafana endpoints and make GET requests to Prometheus, Alertmanager, and Grafana UIs through the Rancher proxy. <br/> <br/> This role doesn't grant access to monitoring endpoints. As a result, users with this role won't be able to view cluster monitoring graphs and dashboards in the Rancher UI; however, they are able to access the monitoring Grafana, Prometheus, and Alertmanager UIs if provided those links. |
|
||||
|
||||
:::note
|
||||
|
||||
A user bound to the **View Monitoring** Rancher role and read-only project permissions can't view links in the Monitoring UI. They can still access external monitoring UIs if provided links to those UIs. If you wish to grant access to users with the **View Monitoring** role and read-only project permissions, move the `cattle-monitoring-system` namespace into the project.
|
||||
|
||||
:::
|
||||
|
||||
### Assigning Roles and ClusterRoles with kubectl
|
||||
|
||||
#### Using `kubectl create`
|
||||
|
||||
One method is to use either `kubectl create clusterrolebinding` or `kubectl create rolebinding` to assign a `Role` or `ClusterRole`. This is shown in the following examples:
|
||||
|
||||
- Assign to a specific user:
|
||||
<Tabs groupId="role-type">
|
||||
<TabItem value="clusterrolebinding">
|
||||
|
||||
```plain
|
||||
kubectl create clusterrolebinding my-binding --clusterrole=monitoring-ui-view --user=u-l4npx
|
||||
```
|
||||
|
||||
</TabItem>
|
||||
<TabItem value="rolebinding">
|
||||
|
||||
```plain
|
||||
kubectl create rolebinding my-binding --clusterrole=monitoring-ui-view --user=u-l4npx --namespace=my-namespace
|
||||
```
|
||||
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
- Assign to all authenticated users:
|
||||
<Tabs groupId="role-type">
|
||||
<TabItem value="clusterrolebinding">
|
||||
|
||||
```plain
|
||||
kubectl create clusterrolebinding my-binding --clusterrole=monitoring-ui-view --group=system:authenticated
|
||||
```
|
||||
|
||||
</TabItem>
|
||||
<TabItem value="rolebinding">
|
||||
|
||||
```plain
|
||||
kubectl create rolebinding my-binding --clusterrole=monitoring-ui-view --group=system:authenticated --namespace=my-namespace
|
||||
```
|
||||
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
|
||||
#### Using YAML Files
|
||||
|
||||
Another method is to define bindings in YAML files that you create. You must first configure the `RoleBinding` or `ClusterRoleBinding` with a YAML file. Then, apply the configuration changes by running the `kubectl apply` command.
|
||||
|
||||
- **Roles**: Below is an example YAML file to help you configure `RoleBindings` in Kubernetes. You'll need to fill in the name below.
|
||||
|
||||
:::note
|
||||
|
||||
Names are case-sensitive.
|
||||
|
||||
:::
|
||||
|
||||
```yaml
|
||||
# monitoring-config-view-role-binding.yaml
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: RoleBinding
|
||||
metadata:
|
||||
name: monitoring-config-view
|
||||
namespace: cattle-monitoring-system
|
||||
roleRef:
|
||||
kind: Role
|
||||
name: monitoring-config-view
|
||||
apiGroup: rbac.authorization.k8s.io
|
||||
subjects:
|
||||
- kind: User
|
||||
name: u-b4qkhsnliz # this can be found via `kubectl get users -A`
|
||||
apiGroup: rbac.authorization.k8s.io
|
||||
```
|
||||
|
||||
- **kubectl**: Below is an example of a `kubectl` command used to apply the binding you've created in the YAML file. Remember to fill in your YAML filename accordingly.
|
||||
```plain
|
||||
kubectl apply -f monitoring-config-view-role-binding.yaml
|
||||
```
|
||||
|
||||
## Users with Rancher Based Permissions
|
||||
|
||||
The relationship between the default roles deployed by Rancher (i.e. cluster-owner, cluster-member, project-owner, project-member), the default Kubernetes roles, and the roles deployed by the rancher-monitoring chart are detailed in the table below:
|
||||
|
||||
<figcaption>Default Rancher Permissions and Corresponding Kubernetes ClusterRoles</figcaption>
|
||||
|
||||
| Rancher Role | Kubernetes Role | Monitoring ClusterRole / Role | ClusterRoleBinding or RoleBinding? |
|
||||
| --------- | --------- | --------- | --------- |
|
||||
| cluster-owner | cluster-admin | N/A | ClusterRoleBinding |
|
||||
| cluster-member | admin | monitoring-admin | ClusterRoleBinding |
|
||||
| project-owner | admin | monitoring-admin | RoleBinding within Project namespace |
|
||||
| project-member | edit | monitoring-edit | RoleBinding within Project namespace |
|
||||
|
||||
In addition to these default roles, the following Rancher project roles can be applied to members of your cluster to provide access to monitoring. These Rancher roles are tied to ClusterRoles deployed by the monitoring chart:
|
||||
|
||||
<figcaption>Non-default Rancher Permissions and Corresponding Kubernetes ClusterRoles</figcaption>
|
||||
|
||||
| Rancher Role | Kubernetes ClusterRole | Available In Rancher From | Available in Monitoring v2 From |
|
||||
|--------------------------|-------------------------------|-------|------|
|
||||
| View Monitoring* | [monitoring-ui-view](#additional-monitoring-clusterroles) | 2.4.8+ | 9.4.204+ |
|
||||
|
||||
:::note
|
||||
|
||||
A user bound to the **View Monitoring** Rancher role and read-only project permissions can't view links in the Monitoring UI. They can still access external monitoring UIs if provided links to those UIs. If you wish to grant access to users with the **View Monitoring** role and read-only project permissions, move the `cattle-monitoring-system` namespace into the project.
|
||||
|
||||
:::
|
||||
|
||||
### Differences in 2.5.x
|
||||
|
||||
Users with the project-member or project-owners roles assigned will not be given access to either Prometheus or Grafana in Rancher 2.5.x since we only create Grafana or Prometheus on a cluster-level.
|
||||
|
||||
In addition, while project owners will still be only able to add ServiceMonitors / PodMonitors that scrape resources within their project's namespace by default, PrometheusRules are not scoped to a single namespace / project. Therefore, any alert rules or recording rules created by project-owners within their project namespace will be applied across the entire cluster, although they will be unable to view / edit / delete any rules that were created outside the project's namespace.
|
||||
|
||||
### Assigning Additional Access
|
||||
|
||||
If cluster-admins would like to provide additional admin/edit access to users outside of the roles offered by the rancher-monitoring chart, the following table identifies the potential impact:
|
||||
|
||||
|CRDs (monitoring.coreos.com) | Can it cause impact outside of a namespace / project? | Impact |
|
||||
|----------------------------| ------| ----------------------------|
|
||||
| `prometheuses`| Yes, this resource can scrape metrics from any targets across the entire cluster (unless the Operator itself is otherwise configured). | User will be able to define the configuration of new cluster-level Prometheus deployments that should be created in the cluster. |
|
||||
| `alertmanagers`| No | User will be able to define the configuration of new cluster-level Alertmanager deployments that should be created in the cluster. Note: if you just want to allow users to configure settings like Routes and Receivers, you should just provide access to the Alertmanager Config Secret instead. |
|
||||
| <ul><li>`servicemonitors`</li><li>`podmonitors`</li></ul>| No, not by default; this is configurable via `ignoreNamespaceSelectors` on the Prometheus CR. | User will be able to set up scrapes by Prometheus on endpoints exposed by Services / Pods within the namespace they are given this permission in. |
|
||||
| `prometheusrules`| Yes, PrometheusRules are cluster-scoped. | User will be able to define alert or recording rules on Prometheus based on any series collected across the entire cluster. |
|
||||
|
||||
| k8s Resources | Namespace | Can it cause impact outside of a namespace / project? | Impact |
|
||||
|----------------------------| ------| ------| ----------------------------|
|
||||
| <ul><li>`secrets`</li><li>`configmaps`</li></ul>| `cattle-monitoring-system` | Yes, Configs and Secrets in this namespace can impact the entire monitoring / alerting pipeline. | User will be able to create or edit Secrets / ConfigMaps such as the Alertmanager Config, Prometheus Adapter Config, TLS secrets, additional Grafana datasources, etc. This can have broad impact on all cluster monitoring / alerting. |
|
||||
| <ul><li>`secrets`</li><li>`configmaps`</li></ul>| `cattle-dashboards` | Yes, Configs and Secrets in this namespace can create dashboards that make queries on all metrics collected at a cluster-level. | User will be able to create Secrets / ConfigMaps that persist new Grafana Dashboards only. |
|
||||
|
||||
## Role-based Access Control for Grafana
|
||||
|
||||
Rancher allows any users who are authenticated by Kubernetes and have access the Grafana service deployed by the Rancher Monitoring chart to access Grafana via the Rancher Dashboard UI. By default, all users who are able to access Grafana are given the [Viewer](https://grafana.com/docs/grafana/latest/permissions/organization_roles/#viewer-role) role, which allows them to view any of the default dashboards deployed by Rancher.
|
||||
|
||||
However, users can choose to log in to Grafana as an [Admin](https://grafana.com/docs/grafana/latest/permissions/organization_roles/#admin-role) if necessary. The default Admin username and password for the Grafana instance will be `admin`/`prom-operator`, but alternative credentials can also be supplied on deploying or upgrading the chart.
|
||||
|
||||
To see the Grafana UI, install `rancher-monitoring`. Then:
|
||||
|
||||
1. In the upper left corner, click **☰ > Cluster Management**.
|
||||
1. On the **Clusters** page, go to the cluster where you want to see the visualizations and click **Explore**.
|
||||
1. In the left navigation bar, click **Monitoring**.
|
||||
1. Click **Grafana**.
|
||||
|
||||
<figcaption>Cluster Compute Resources Dashboard in Grafana</figcaption>
|
||||
|
||||
|
||||
|
||||
<figcaption>Default Dashboards in Grafana</figcaption>
|
||||
|
||||
|
||||
+52
@@ -0,0 +1,52 @@
|
||||
---
|
||||
title: Windows Cluster Support for Monitoring V2
|
||||
---
|
||||
|
||||
<head>
|
||||
<link rel="canonical" href="https://ranchermanager.docs.rancher.com/integrations-in-rancher/monitoring-and-alerting/windows-support"/>
|
||||
</head>
|
||||
|
||||
_Available as of v2.5.8_
|
||||
|
||||
Starting at Monitoring V2 14.5.100 (used by default in Rancher 2.5.8), Monitoring V2 can now be deployed on a Windows cluster and will scrape metrics from Windows nodes using [prometheus-community/windows_exporter](https://github.com/prometheus-community/windows_exporter) (previously named `wmi_exporter`).
|
||||
|
||||
## Cluster Requirements
|
||||
|
||||
Monitoring V2 for Windows can only scrape metrics from Windows hosts that have a minimum `wins` version of v0.1.0. To be able to fully deploy Monitoring V2 for Windows, all of your hosts must meet this requirement.
|
||||
|
||||
If you provision a fresh RKE1 cluster in Rancher 2.5.8, your cluster should already meet this requirement.
|
||||
|
||||
### Upgrading Existing Clusters to wins v0.1.0
|
||||
|
||||
If the cluster was provisioned before Rancher 2.5.8 (even if the current Rancher version is 2.5.8), you will not be able to successfully deploy Monitoring V2 for Windows until you upgrade the wins version on each host to at least v0.1.0.
|
||||
|
||||
To facilitate this upgrade, Rancher 2.5.8 has released a brand new Helm chart called `rancher-wins-upgrader`.
|
||||
|
||||
1. Deploy `rancher-wins-upgrader` with the following override:
|
||||
```yaml
|
||||
# Masquerading bootstraps the wins-upgrader installation via
|
||||
# a previously whitelisted process path since the normal install path,
|
||||
# c:\etc\rancher\wins\wins-upgrade.exe is not normally whitelisted.
|
||||
# In this case, we are using the previously whitelisted process
|
||||
# path used by Monitoring V1.
|
||||
masquerade:
|
||||
enabled: true
|
||||
as: c:\\etc\wmi-exporter\wmi-exporter.exe
|
||||
```
|
||||
:::note Note for Non-Default Windows Prefix Path:
|
||||
|
||||
- If you set up the RKE cluster with a `cluster.yml` that has a non-default `win_prefix_path`, you will need to update the `masquerade.as` field with your prefix path in place of `c:\\`.
|
||||
|
||||
- For example, if you have `win_prefix_path: 'c:\host\opt\'`, then you will need to set `as: c:\host\opt\etc\wmi-exporter\wmi-exporter.exe`.
|
||||
|
||||
:::
|
||||
|
||||
2. Once all your hosts have been successfully upgraded, please ensure that you deploy the Helm chart once again with default values to avoid conflicts with the following settings:
|
||||
```yaml
|
||||
masquerade:
|
||||
enabled: false
|
||||
```
|
||||
|
||||
**Result:** The hosts are ready for Monitoring V2 to be installed. You may choose to uninstall the `rancher-wins-upgrader` chart or keep it in your cluster to facilitate future upgrades.
|
||||
|
||||
For more information on how it can be used, please see the [README.md](https://github.com/rancher/wins/blob/master/charts/rancher-wins-upgrader/README.md) of the chart.
|
||||
Reference in New Issue
Block a user