mirror of
https://github.com/rancher/rancher-docs.git
synced 2026-09-25 12:38:05 +00:00
Reorganize monitoring docs
This commit is contained in:
Catherine Luse
@@ -13,17 +13,13 @@ Using the `rancher-monitoring` application, you can quickly deploy leading open-
|
||||
|
||||
- [Features](#features)
|
||||
- [How Monitoring Works](#how-monitoring-works)
|
||||
- [Enable Monitoring](#enable-monitoring)
|
||||
- [Default Alerts, Targets, and Grafana Dashboards](#default-alerts-targets-and-grafana-dashboards)
|
||||
- [Uninstall Monitoring](#uninstall-monitoring)
|
||||
- [Monitoring Workloads](#monitoring-workloads)
|
||||
- [Windows Cluster Support](#windows-cluster-support)
|
||||
- [Setting up Metrics for HPA](#setting-up-metrics-for-hpa)
|
||||
- [Default Components and Deployments](#default-components-and-deployments)
|
||||
- [Role-based Access Control](#role-based-access-control)
|
||||
- [Upgrading from Monitoring V1 to V2](#upgrading-from-monitoring-v1-to-v2)
|
||||
- [Guides](#guides)
|
||||
- [Windows Cluster Support](#windows-cluster-support)
|
||||
- [Known Issues](#known-issues)
|
||||
|
||||
# Features
|
||||
### Features
|
||||
|
||||
Prometheus lets you view metrics from your Rancher and Kubernetes objects. Using timestamps, Prometheus lets you query and view these metrics in easy-to-read graphs and visuals, either through the Rancher UI or Grafana, which is an analytics viewing platform deployed along with Prometheus.
|
||||
|
||||
@@ -44,41 +40,49 @@ The monitoring application allows you to:
|
||||
|
||||
For an explanation of how the monitoring components work together, see [this page.](./how-monitoring-works)
|
||||
|
||||
# Enable Monitoring
|
||||
# Default Components and Deployments
|
||||
|
||||
To enable monitoring, follow the steps on [this page.](./enable-monitoring)
|
||||
### Built-in Dashboards
|
||||
|
||||
# Default Alerts, Targets, and Grafana Dashboards
|
||||
By default, the monitoring application deploys Grafana dashboards (curated by the [kube-prometheus](https://github.com/prometheus-operator/kube-prometheus) project) onto a cluster.
|
||||
|
||||
By default, Rancher Monitoring deploys exporters (such as [node-exporter](https://github.com/prometheus/node_exporter) and [kube-state-metrics](https://github.com/kubernetes/kube-state-metrics)) as well as default Prometheus alerts and Grafana dashboards (curated by the [kube-prometheus](https://github.com/prometheus-operator/kube-prometheus) project) onto a cluster.
|
||||
It also deploys an Alertmanager UI and a Prometheus UI. For more information about these tools, see [Built-in Dashboards.](./dashboards)
|
||||
### Default Metrics Exporters
|
||||
|
||||
### Grafana UI
|
||||
By default, Rancher Monitoring deploys exporters (such as [node-exporter](https://github.com/prometheus/node_exporter) and [kube-state-metrics](https://github.com/kubernetes/kube-state-metrics)).
|
||||
|
||||
[Grafana](https://grafana.com/grafana/) allows you to query, visualize, alert on and understand your metrics no matter where they are stored. Create, explore, and share dashboards with your team and foster a data driven culture.
|
||||
These default exporters automatically scrape metrics for CPU and memory from all components of your Kubernetes cluster, including your workloads.
|
||||
|
||||
To see the default dashboards for time series data visualization, go to the Grafana UI. In the left navigation bar, click the icon with four boxes and click **Manage.**
|
||||
### Default Alerts
|
||||
|
||||
To view and customize the PromQL queries powering the Grafana dashboard, see [this page.](./customize-grafana)
|
||||
The monitoring application deploys some alerts by default. To see the default alerts, go to the [Alertmanager UI](./dashboard/accessing-the-alertmanager-ui) and click **Expand all groups.**
|
||||
|
||||
To create a persistent Grafana dashboard, see [this page.](./persist-grafana)
|
||||
### Components Exposed in the Rancher UI
|
||||
|
||||
For information about role-based access control for Grafana, see [this section.](./rbac/#role-based-access-control-for-grafana)
|
||||
For a list of monitoring components exposed in the Rancher UI, along with common use cases for editing them, see [this section.](./how-monitoring-works/#components-exposed-in-the-rancher-ui)
|
||||
|
||||
### Alertmanager UI
|
||||
# Role-based Access Control
|
||||
|
||||
To see the default alerts, go to the [Alertmanager UI](./alertmanager-ui) and click **Expand all groups.**
|
||||
For information on configuring access to monitoring, see [this page.](./rbac)
|
||||
|
||||
### Prometheus UI
|
||||
|
||||
To see what services you are monitoring, you will need to see your targets. To view the default targets, refer to [Viewing the Prometheus Targets.](./prometheus-ui/#viewing-the-prometheus-targets)
|
||||
# Guides
|
||||
|
||||
- [Enable monitoring](./guides/enable-monitoring)
|
||||
- [Uninstall monitoring](./guides/uninstall)
|
||||
- [Monitoring Rancher apps](./guides/monitoring-rancher-apps)
|
||||
- [Monitoring workloads](./guides/monitoring-workloads)
|
||||
- [Customizing Grafana dashboards](./guides/customize-grafana)
|
||||
- [Persistent Grafana dashboards](./guides/persist-grafana)
|
||||
- [Setting up metrics for horizontal pod autoscaling](./guides/hpa)
|
||||
- [Debugging high memory usage](./guides/memory-usage)
|
||||
- [Migrating from Monitoring V1 to V2](./guides/migrating)
|
||||
|
||||
# Configuration
|
||||
|
||||
> The configuration reference assumes familiarity with how monitoring components work together. For more information, see [How Monitoring Works.](./how-monitoring-works)
|
||||
|
||||
### Configuring Monitoring Resources in Rancher
|
||||
|
||||
> The configuration reference assumes familiarity with how monitoring components work together. For more information, see [How Monitoring Works.](./how-monitoring-works)
|
||||
|
||||
- [ServiceMonitor and PodMonitor](./configuration/servicemonitor-podmonitor)
|
||||
- [Receiver](./configuration/receiver)
|
||||
- [Route](./configuration/route)
|
||||
@@ -90,28 +94,6 @@ To see what services you are monitoring, you will need to see your targets. To v
|
||||
|
||||
For more information on `rancher-monitoring` chart options, including options to set resource limits and requests, see [this page.](./configuration/helm-chart-options)
|
||||
|
||||
# Uninstall Monitoring
|
||||
|
||||
1. From the **Cluster Explorer,** click Apps & Marketplace.
|
||||
1. Click **Installed Apps.**
|
||||
1. Go to the `cattle-monitoring-system` namespace and check the boxes for `rancher-monitoring-crd` and `rancher-monitoring`.
|
||||
1. Click **Delete.**
|
||||
1. Confirm **Delete.**
|
||||
|
||||
**Result:** `rancher-monitoring` is uninstalled.
|
||||
|
||||
> **Note on Persistent Grafana Dashboards:** For users who are using Monitoring V2 v9.4.203 or below, uninstalling the Monitoring chart will delete the cattle-dashboards namespace, which will delete all persisted dashboards, unless the namespace is marked with the annotation `helm.sh/resource-policy: "keep"`. This annotation is added by default in Monitoring V2 v14.5.100+ but can be manually applied on the cattle-dashboards namespace before an uninstall if an older version of the Monitoring chart is currently installed onto your cluster.
|
||||
|
||||
# Monitoring Workloads
|
||||
|
||||
The steps for setting up monitoring for workloads depends on whether you want basic metrics such as CPU and memory for the workload, or whether you want to scrape custom metrics from the workload.
|
||||
|
||||
If you only need CPU and memory time series for the workload, you don't need to deploy a ServiceMonitor or PodMonitor because the monitoring application already collects metrics data on resource usage by default. The resource usage time series data is in Prometheus's local time series database. Grafana shows the data in aggregate, but you can see the data for the individual workload by using a PromQL query that extracts the data for that workload. Once you have the PromQL query, you can execute the query individually in the Prometheus UI and see the time series visualized there, or you can use the query to customize a Grafana dashboard to display the workload metrics. For examples of PromQL queries for workload metrics, see [this section.](https://rancher.com/docs/rancher/v2.5/en/monitoring-alerting/configuration/expression/#workload-metrics)
|
||||
|
||||
To set up custom metrics for your workload, you will need to set up an exporter and create a new ServiceMonitor custom resource to configure Prometheus to scrape metrics from your exporter.
|
||||
|
||||
For more information, see [this section.](./monitoring-workloads)
|
||||
|
||||
# Windows Cluster Support
|
||||
|
||||
_Available as of v2.5.8_
|
||||
@@ -122,27 +104,7 @@ To be able to fully deploy Monitoring V2 for Windows, all of your Windows hosts
|
||||
|
||||
For more details on how to upgrade wins on existing Windows hosts, refer to the section on [Windows cluster support for Monitoring V2.](./windows-clusters)
|
||||
|
||||
# Setting up Metrics for HPA
|
||||
|
||||
The monitoring app installs a Prometheus adapter that can be used for making the metrics from monitoring available from the Kubernetes API. This is useful for horizontal pod autoscaling based on custom metrics.
|
||||
|
||||
For details, see [this section.](./hpa)
|
||||
|
||||
# Monitoring Rancher Apps
|
||||
|
||||
A common pattern for Rancher apps is to package a ServiceMonitor in the Helm chart for the application. The ServiceMonitor contains a preconfigured Prometheus target for monitoring.
|
||||
|
||||
When the ServiceMonitor is enabled and monitoring is also enabled, Prometheus will be able to scrape metrics from the Rancher application.
|
||||
|
||||
For details, see [this page.](./monitoring-rancher-apps)
|
||||
|
||||
# Role-based Access Control
|
||||
|
||||
For information on configuring access to monitoring, see [this page.](./rbac)
|
||||
|
||||
# Upgrading from Monitoring V1 to V2
|
||||
|
||||
For more information about upgrading the Monitoring app in Rancher 2.5, please refer to the [migration docs](./migrating).
|
||||
|
||||
# Known Issues
|
||||
|
||||
|
||||
@@ -1,21 +0,0 @@
|
||||
---
|
||||
title: Alertmanager UI
|
||||
weight: 8
|
||||
---
|
||||
|
||||
When `rancher-monitoring` is installed, the Prometheus Alertmanager UI is deployed, allowing you to view your alerts and the current Alertmanager configuration.
|
||||
|
||||
> This section assumes familiarity with how monitoring components work together. For more information about Alertmanager, see [this section.](../how-monitoring-works/#how-alertmanager-works)
|
||||
|
||||
# Accessing the Alertmanager UI
|
||||
|
||||
> **Prerequisite:** The `rancher-monitoring` application must be installed.
|
||||
|
||||
To see the Alertmanager UI, go to the **Cluster Explorer.** In the top left corner, click **Cluster Explorer > Monitoring.** Then click **Alertmanager.**
|
||||
|
||||
**Result:** The Alertmanager UI opens in a new tab. For help with configuration, refer to the [official Alertmanager documentation.](https://prometheus.io/docs/alerting/latest/alertmanager/)
|
||||
|
||||
For more information on configuring Alertmanager in Rancher, see [this page.](./configuration/alertmanager)
|
||||
|
||||
<figcaption>The Alertmanager UI</figcaption>
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
---
|
||||
title: Configuration
|
||||
weight: 3
|
||||
weight: 5
|
||||
aliases:
|
||||
- /rancher/v2.5/en/monitoring-alerting/v2.5/configuration
|
||||
---
|
||||
|
||||
@@ -1,19 +0,0 @@
|
||||
---
|
||||
title: Customizing Grafana Dashboards
|
||||
weight: 9
|
||||
---
|
||||
|
||||
there’s already a wealth of information provided by kube-state-metrics. Cpu utilization, memory utilization for different things across namespaces. If you just want resource metrics for prod, you don’t need to create a new ServiceMonitor for it. All you need to do is go to the prometheus UI and do a PromQL query to get the information.
|
||||
|
||||
- let’s say you want metrics that apply only for the container alertmanager.
|
||||
- then you can do graphs of that over time.
|
||||
- link to the promql queries used to make grafana dashboards. To get those queries,
|
||||
- go to grafana
|
||||
- right click on a graphic and click explore
|
||||
- it shows you the PromQL queries that are embedded in it
|
||||
- show screenshot of how to get to explore
|
||||
- show screenshot of how to modify the query
|
||||
- can modify it
|
||||
- grafana shows you updated based on your modifications to the query
|
||||
- customizing grafana dashboards section
|
||||
- also link to persisting grafana dashboards section
|
||||
@@ -0,0 +1,78 @@
|
||||
---
|
||||
title: Built-in Dashboards
|
||||
weight: 3
|
||||
---
|
||||
|
||||
- [Grafana UI](#grafana-ui)
|
||||
- [Alertmanager UI](#alertmanager-ui)
|
||||
- [Prometheus UI](#prometheus-ui)
|
||||
|
||||
# Grafana UI
|
||||
|
||||
[Grafana](https://grafana.com/grafana/) allows you to query, visualize, alert on and understand your metrics no matter where they are stored. Create, explore, and share dashboards with your team and foster a data driven culture.
|
||||
|
||||
To see the default dashboards for time series data visualization, go to the Grafana UI. In the left navigation bar, click the icon with four boxes and click **Manage.**
|
||||
|
||||
### Customizing Grafana
|
||||
|
||||
To view and customize the PromQL queries powering the Grafana dashboard, see [this page.](./customize-grafana)
|
||||
|
||||
### Persistent Grafana Dashboards
|
||||
|
||||
To create a persistent Grafana dashboard, see [this page.](./persist-grafana)
|
||||
|
||||
### Access to Grafana
|
||||
|
||||
For information about role-based access control for Grafana, see [this section.](./rbac/#role-based-access-control-for-grafana)
|
||||
|
||||
|
||||
# Alertmanager UI
|
||||
|
||||
When `rancher-monitoring` is installed, the Prometheus Alertmanager UI is deployed, allowing you to view your alerts and the current Alertmanager configuration.
|
||||
|
||||
> This section assumes familiarity with how monitoring components work together. For more information about Alertmanager, see [this section.](../how-monitoring-works/#how-alertmanager-works)
|
||||
|
||||
|
||||
### Accessing the Alertmanager UI
|
||||
|
||||
> **Prerequisite:** The `rancher-monitoring` application must be installed.
|
||||
|
||||
To see the Alertmanager UI, go to the **Cluster Explorer.** In the top left corner, click **Cluster Explorer > Monitoring.** Then click **Alertmanager.**
|
||||
|
||||
**Result:** The Alertmanager UI opens in a new tab. For help with configuration, refer to the [official Alertmanager documentation.](https://prometheus.io/docs/alerting/latest/alertmanager/)
|
||||
|
||||
For more information on configuring Alertmanager in Rancher, see [this page.](./configuration/alertmanager)
|
||||
|
||||
<figcaption>The Alertmanager UI</figcaption>
|
||||
|
||||
|
||||
|
||||
### Viewing Default Alerts
|
||||
|
||||
To see the default alerts, go to the [Alertmanager UI](./alertmanager-ui) and click **Expand all groups.**
|
||||
|
||||
|
||||
# Prometheus UI
|
||||
|
||||
To see the Prometheus UI, install `rancher-monitoring`. Then go to the **Cluster Explorer.** In the top left corner, click **Cluster Explorer > Monitoring.** Then click **Prometheus Graph.**
|
||||
|
||||
<figcaption>Prometheus Graph UI</figcaption>
|
||||
|
||||
|
||||
### Viewing the Prometheus Targets
|
||||
|
||||
To see what services you are monitoring, you will need to see your targets.
|
||||
|
||||
To see the Prometheus Targets, install `rancher-monitoring`. Then go to the **Cluster Explorer.** In the top left corner, click **Cluster Explorer > Monitoring.** Then click **Prometheus Targets.**
|
||||
|
||||
<figcaption>Targets in the Prometheus UI</figcaption>
|
||||
|
||||
|
||||
### Viewing the PrometheusRules
|
||||
|
||||
To see the PrometheusRules, install `rancher-monitoring`. Then go to the **Cluster Explorer.** In the top left corner, click **Cluster Explorer > Monitoring.** Then click **Prometheus Rules.**
|
||||
|
||||
<figcaption>Rules in the Prometheus UI</figcaption>
|
||||
|
||||
|
||||
For more information on PrometheusRules in Rancher, see [this page.](./configuration/prometheusrules)
|
||||
+3
-2
@@ -1,11 +1,12 @@
|
||||
---
|
||||
title: Prometheus Expressions
|
||||
weight: 4
|
||||
title: Prometheus Expression Reference
|
||||
weight: 6
|
||||
aliases:
|
||||
- /rancher/v2.5/en/project-admin/tools/monitoring/expression
|
||||
- /rancher/v2.5/en/cluster-admin/tools/monitoring/expression
|
||||
- /rancher/v2.5/en/monitoring-alerting/legacy/monitoring/cluster-monitoring/expression
|
||||
- /rancher/v2.5/en/monitoring-alerting/v2.5/configuration/expression
|
||||
- /rancher/v2.5/en/monitoring/alerting/configuration/expression
|
||||
---
|
||||
|
||||
The PromQL expressions in this doc can be used to configure alerts.
|
||||
@@ -0,0 +1,4 @@
|
||||
---
|
||||
title: Guides
|
||||
weight: 4
|
||||
---
|
||||
@@ -0,0 +1,19 @@
|
||||
---
|
||||
title: Customizing Grafana Dashboards
|
||||
weight: 5
|
||||
---
|
||||
|
||||
there’s already a wealth of information provided by kube-state-metrics. Cpu utilization, memory utilization for different things across namespaces. If you just want resource metrics for prod, you don’t need to create a new ServiceMonitor for it. All you need to do is go to the prometheus UI and do a PromQL query to get the information.
|
||||
|
||||
- let’s say you want metrics that apply only for the container alertmanager.
|
||||
- then you can do graphs of that over time.
|
||||
- link to the promql queries used to make grafana dashboards. To get those queries,
|
||||
- go to grafana
|
||||
- right click on a graphic and click explore
|
||||
- it shows you the PromQL queries that are embedded in it
|
||||
- show screenshot of how to get to explore
|
||||
- show screenshot of how to modify the query
|
||||
- can modify it
|
||||
- grafana shows you updated based on your modifications to the query
|
||||
- customizing grafana dashboards section
|
||||
- also link to persisting grafana dashboards section
|
||||
+1
-1
@@ -1,6 +1,6 @@
|
||||
---
|
||||
title: Enable Monitoring
|
||||
weight: 14
|
||||
weight: 1
|
||||
---
|
||||
|
||||
As an [administrator]({{<baseurl>}}/rancher/v2.5/en/admin-settings/rbac/global-permissions/) or [cluster owner]({{<baseurl>}}/rancher/v2.5/en/admin-settings/rbac/cluster-project-roles/#cluster-roles), you can configure Rancher to deploy Prometheus to monitor your Kubernetes cluster.
|
||||
+7
-1
@@ -1,4 +1,10 @@
|
||||
# Setting up Metrics for HPA
|
||||
---
|
||||
title: Setting up Metrics for HPA
|
||||
weight: 7
|
||||
---
|
||||
|
||||
The monitoring app installs a Prometheus adapter that can be used for making the metrics from monitoring available from the Kubernetes API. This is useful for horizontal pod autoscaling based on custom metrics.
|
||||
|
||||
|
||||
- kube-state-metrics: monitors internal K8s components
|
||||
-
|
||||
+5
-5
@@ -1,6 +1,6 @@
|
||||
---
|
||||
title: Debugging High Memory Usage
|
||||
weight: 12
|
||||
weight: 8
|
||||
---
|
||||
|
||||
Every time series in Prometheus is uniquely identified by its [metric name](https://prometheus.io/docs/practices/naming/#metric-names) and optional key-value pairs called [labels.](https://prometheus.io/docs/practices/naming/#labels)
|
||||
@@ -9,12 +9,12 @@ The labels allow the ability to filter and aggregate the time series data, but t
|
||||
|
||||
Each time series has a defined set of labels, and Prometheus generates a new time series for all unique combinations of labels. If a metric has two labels attached, two time series are generated for that metric. Changing any label value, including adding or removing a label, will create a new time series.
|
||||
|
||||
Prometheus is optimized to store data that is index-based on series.It is designed for a relatively consistent number of time series and a relatively large number of samples that need to be collected from the exporters over time.
|
||||
Prometheus is optimized to store data that is index-based on series. It is designed for a relatively consistent number of time series and a relatively large number of samples that need to be collected from the exporters over time.
|
||||
|
||||
Inversely, Prometheus is not optimized to accommodate a rapidly changing number of time series. For that reason, large bursts of memory usage can occur when monitoring is installed on clusters where many resources are being created and destroyed, especially on multi-tenant clusters.
|
||||
|
||||
# Reducing Memory Bursts
|
||||
### Reducing Memory Bursts
|
||||
|
||||
To reduce memory consumption, Prometheus can be configured to store fewer time series, by scraping fewer metrics or by attaching fewer labels to the time series. To see which series use the most memory, you can check the TSDB status page in the Prometheus UI.
|
||||
To reduce memory consumption, Prometheus can be configured to store fewer time series, by scraping fewer metrics or by attaching fewer labels to the time series. To see which series use the most memory, you can check the TSDB (time series database) status page in the Prometheus UI.
|
||||
|
||||
Distributed Prometheus solutions, such as [Thanos](https://thanos.io/) and [Cortex](https://cortexmetrics.io/) use an alternate architecture in which multiple small Prometheus instances are deployed. In the case of Thanos, the metrics from each Prometheus are aggregated into the common Thanos deployment, and then those metrics are exported to a persistent store, such as S3. This more robust architecture avoids burdening any single Prometheus instance with too many time series, while also preserving the ability to query metrics on a global level.
|
||||
Distributed Prometheus solutions such as [Thanos](https://thanos.io/) and [Cortex](https://cortexmetrics.io/) use an alternate architecture in which multiple small Prometheus instances are deployed. In the case of Thanos, the metrics from each Prometheus are aggregated into the common Thanos deployment, and then those metrics are exported to a persistent store, such as S3. This more robust architecture avoids burdening any single Prometheus instance with too many time series, while also preserving the ability to query metrics on a global level.
|
||||
+17
-3
@@ -1,12 +1,21 @@
|
||||
---
|
||||
title: Migrating to Rancher v2.5 Monitoring
|
||||
weight: 5
|
||||
weight: 9
|
||||
aliases:
|
||||
- /rancher/v2.5/en/monitoring-alerting/v2.5/migrating
|
||||
---
|
||||
|
||||
If you previously enabled Monitoring, Alerting, or Notifiers in Rancher before v2.5, there is no automatic upgrade path for switching to the new monitoring/alerting solution. Before deploying the new monitoring solution via Cluster Explore, you will need to disable and remove all existing custom alerts, notifiers and monitoring installations for the whole cluster and in all projects.
|
||||
|
||||
- [Monitoring Before Rancher v2.5](#monitoring-before-rancher-v2-5)
|
||||
- [Monitoring and Alerting via Cluster Explorer in Rancher v2.5](#monitoring-and-alerting-via-cluster-explorer-in-rancher-v2-5)
|
||||
- [Changes to Role-based Access Control](#changes-to-role-based-access-control)
|
||||
- [Migrating from Monitoring V1 to Monitoring V2](#migrating-from-monitoring-v1-to-monitoring-v2)
|
||||
- [Migrating Grafana Dashboards](#migrating-grafana-dashboards)
|
||||
- [Migrating Alerts](#migrating-alerts)
|
||||
- [Migrating Notifiers](#migrating-notifiers)
|
||||
- [Migrating for RKE Template Users](#migrating-for-rke-template-users)
|
||||
|
||||
# Monitoring Before Rancher v2.5
|
||||
|
||||
As of v2.2.0, Rancher's Cluster Manager allowed users to enable Monitoring & Alerting V1 (both powered by [Prometheus Operator](https://github.com/prometheus-operator/prometheus-operator)) independently within a cluster.
|
||||
@@ -17,7 +26,7 @@ Monitoring V1 could be configured on both a cluster-level and on a project-level
|
||||
|
||||
When Alerts or Notifiers are enabled, Alerting V1 deploys [Prometheus Alertmanager](https://prometheus.io/docs/alerting/latest/alertmanager/) and a set of Rancher controllers onto a cluster that allows users to define alerts and configure alert-based notifications via Email, Slack, PagerDuty, etc. Users can choose to create different types of alerts depending on what needs to be monitored (e.g. System Services, Resources, CIS Scans, etc.); however, PromQL Expression-based alerts can only be created if Monitoring V1 is enabled.
|
||||
|
||||
# Monitoring/Alerting via Cluster Explorer in Rancher 2.5
|
||||
# Monitoring and Alerting via Cluster Explorer in Rancher 2.5
|
||||
|
||||
As of v2.5.0, Rancher's Cluster Explorer now allows users to enable Monitoring & Alerting V2 (both powered by [Prometheus Operator](https://github.com/prometheus-operator/prometheus-operator)) together within a cluster.
|
||||
|
||||
@@ -121,6 +130,11 @@ or add the Prometheus Rule through the Cluster Explorer
|
||||
|
||||
For more details on how to configure PrometheusRules in Monitoring V2 see [Monitoring Configuration]({{<baseurl>}}/rancher/v2.5/en/monitoring-alerting/v2.5/configuration#prometheusrules).
|
||||
|
||||
### Migrating notifiers
|
||||
### Migrating Notifiers
|
||||
|
||||
There is no direct equivalent for how notifiers work in Monitoring V1. Instead you have to replicate the desired setup with [Routes and Receivers]({{<baseurl>}}/rancher/v2.5/en/monitoring-alerting/v2.5/configuration#alertmanager-config) in Monitoring V2.
|
||||
|
||||
|
||||
### Migrating for RKE Template Users
|
||||
|
||||
If the cluster is managed using an RKE template, you will need to disable monitoring in future RKE template revisions to prevent legacy monitoring from being re-enabled.
|
||||
+8
-1
@@ -1,8 +1,15 @@
|
||||
---
|
||||
title: Monitoring Rancher Apps
|
||||
weight: 11
|
||||
weight: 3
|
||||
---
|
||||
|
||||
|
||||
A common pattern for Rancher apps is to package a ServiceMonitor in the Helm chart for the application. The ServiceMonitor contains a preconfigured Prometheus target for monitoring.
|
||||
|
||||
When the ServiceMonitor is enabled and monitoring is also enabled, Prometheus will be able to scrape metrics from the Rancher application.
|
||||
|
||||
|
||||
|
||||
CIS application has a flag that lets you deploy a service monitor in it. As a general practice we expose charts for prometheus metrics to have that service monitor definition. The moment it’s deployed into the cluster, the prometheus scrape configuration will automatically be updated to reflect the service monitors that it has access to.
|
||||
|
||||
In logging v2 they will deploy a service monitor and we will just absorb it.
|
||||
+25
-3
@@ -1,13 +1,34 @@
|
||||
---
|
||||
title: Setting up Monitoring for a Workload
|
||||
weight: 10
|
||||
weight: 4
|
||||
---
|
||||
|
||||
- [Display CPU and Memory Metrics for a Workload](#display-cpu-and-memory-metrics-for-a-workload)
|
||||
- [Setting up Metrics Beyond CPU and Memory](#setting-up-metrics-beyond-cpu-and-memory)
|
||||
|
||||
If you only need CPU and memory time series for the workload, you don't need to deploy a ServiceMonitor or PodMonitor because the monitoring application already collects metrics data on resource usage by default.
|
||||
|
||||
|
||||
|
||||
|
||||
The steps for setting up monitoring for workloads depends on whether you want basic metrics such as CPU and memory for the workload, or whether you want to scrape custom metrics from the workload.
|
||||
|
||||
If you only need CPU and memory time series for the workload, you don't need to deploy a ServiceMonitor or PodMonitor because the monitoring application already collects metrics data on resource usage by default. The resource usage time series data is in Prometheus's local time series database. Grafana shows the data in aggregate, but you can see the data for the individual workload by using a PromQL query that extracts the data for that workload. Once you have the PromQL query, you can execute the query individually in the Prometheus UI and see the time series visualized there, or you can use the query to customize a Grafana dashboard to display the workload metrics. For examples of PromQL queries for workload metrics, see [this section.](https://rancher.com/docs/rancher/v2.5/en/monitoring-alerting/configuration/expression/#workload-metrics)
|
||||
|
||||
To set up custom metrics for your workload, you will need to set up an exporter and create a new ServiceMonitor custom resource to configure Prometheus to scrape metrics from your exporter.
|
||||
|
||||
For more information, see [this section.](./monitoring-workloads)
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
explain how some applications come with a servicemonitor packaged within them
|
||||
|
||||
for example, some rancher applications come with servicemonitors (link to section)
|
||||
|
||||
# Setting up Metrics for CPU and Memory
|
||||
### Display CPU and Memory Metrics for a Workload
|
||||
|
||||
By default, the monitoring application already scrapes CPU and memory.
|
||||
|
||||
To get some fine-grained detail for a particular workload, you can customize a Grafana dashboard to display the metrics for a particular workload.
|
||||
@@ -17,7 +38,8 @@ To get some fine-grained detail for a particular workload, you can customize a G
|
||||
For more information on customizing Grafana to show the workload metrics, see this section. (Link)
|
||||
|
||||
|
||||
# Setting up Metrics Beyond CPU and Memory
|
||||
### Setting up Metrics Beyond CPU and Memory
|
||||
|
||||
For custom metrics, you will need to expose the metrics on your application in a format supported by Prometheus.
|
||||
|
||||
Then we recommend that you should create a new ServiceMonitor custom resource. When this resource is created, the Prometheus custom resource will be automatically updated so that its scrape configuration includes the new custom metrics endpoint. Then Prometheus will begin scraping metrics from the endpoint.
|
||||
+1
-1
@@ -1,6 +1,6 @@
|
||||
---
|
||||
title: Persistent Grafana Dashboards
|
||||
weight: 4
|
||||
weight: 6
|
||||
aliases:
|
||||
- /rancher/v2.5/en/monitoring-alerting/v2.5/persist-grafana
|
||||
---
|
||||
@@ -0,0 +1,14 @@
|
||||
---
|
||||
title: Uninstall Monitoring
|
||||
weight: 2
|
||||
---
|
||||
|
||||
1. From the **Cluster Explorer,** click Apps & Marketplace.
|
||||
1. Click **Installed Apps.**
|
||||
1. Go to the `cattle-monitoring-system` namespace and check the boxes for `rancher-monitoring-crd` and `rancher-monitoring`.
|
||||
1. Click **Delete.**
|
||||
1. Confirm **Delete.**
|
||||
|
||||
**Result:** `rancher-monitoring` is uninstalled.
|
||||
|
||||
> **Note on Persistent Grafana Dashboards:** For users who are using Monitoring V2 v9.4.203 or below, uninstalling the Monitoring chart will delete the cattle-dashboards namespace, which will delete all persisted dashboards, unless the namespace is marked with the annotation `helm.sh/resource-policy: "keep"`. This annotation is added by default in Monitoring V2 v14.5.100+ but can be manually applied on the cattle-dashboards namespace before an uninstall if an older version of the Monitoring chart is currently installed onto your cluster.
|
||||
@@ -3,7 +3,7 @@ title: How Monitoring Works
|
||||
weight: 1
|
||||
---
|
||||
|
||||
- [1. How Monitoring Components Work Together](#1-how-monitoring-components-work-together)
|
||||
- [1. How Data Flows through the Monitoring Application](#1-how-data-flows-through-the-monitoring-application)
|
||||
- [2. How Prometheus Works](#2-how-prometheus-works)
|
||||
- [2.1. Defining what Metrics are Scraped](#2-1-defining-what-metrics-are-scraped)
|
||||
- [2.2. Scraping Metrics from Exporters](#2-2-scraping-metrics-from-exporters)
|
||||
@@ -20,13 +20,12 @@ weight: 1
|
||||
- [4.3. Default Exporters](#4-3-default-exporters)
|
||||
- [5. Components Exposed in the Rancher UI](#5-components-exposed-in-the-rancher-ui)
|
||||
|
||||
# 1. How Monitoring Components Work Together
|
||||
# 1. How Data Flows through the Monitoring Application
|
||||
|
||||
The below diagram shows the linear flow of data through the monitoring application in chronological order:
|
||||
|
||||
<figcaption>Monitoring Components</figcaption>
|
||||
|
||||
|
||||
|
||||
|
||||
# 2. How Prometheus Works
|
||||
|
||||
@@ -61,7 +60,7 @@ The database can then be queried using PromQL, the query language for Prometheus
|
||||
|
||||
The PromQL query language is the primary tool to query Prometheus for time series data.
|
||||
|
||||
In Grafana, you can right-click a CPU utilization and click Inspect. This opens a panel that shows the [raw query results.](https://grafana.com/docs/grafana/latest/panels/inspect-panel/)The raw results demonstrate how each dashboard is powered by PromQL queries.
|
||||
In Grafana, you can right-click a CPU utilization and click Inspect. This opens a panel that shows the [raw query results.](https://grafana.com/docs/grafana/latest/panels/inspect-panel/#inspect-raw-query-results)The raw results demonstrate how each dashboard is powered by PromQL queries.
|
||||
|
||||
### 2.5. Defining Rules for when Alerts Should be Fired
|
||||
|
||||
@@ -136,13 +135,15 @@ There are also certain special types of ConfigMaps and Secrets such as those cor
|
||||
|
||||
PushProx enhances the security of the monitoring application, allowing it to be installed on hardened Kubernetes clusters.
|
||||
|
||||
To expose Kubernetes metrics, PushProxes use a client proxy model to expose specific ports within default Kubernetes components. The proxy allows `rancher-monitoring` to scrape metrics from processes on the hostNetwork, such as the `kube-api-server`, without opening up node ports to inbound connections.
|
||||
To expose Kubernetes metrics, PushProxes use a client proxy model to expose specific ports within default Kubernetes components. Node exporters expose metrics to PushProx through an outbound connection.
|
||||
|
||||
The proxy allows `rancher-monitoring` to scrape metrics from processes on the hostNetwork, such as the `kube-api-server`, without opening up node ports to inbound connections.
|
||||
|
||||
PushProx is a DaemonSet that listens for clients that seek to register. Once registered, it proxies scrape requests through the established connection. Then the client executes the request to etcd.
|
||||
|
||||
All of the default ServiceMonitors, such as `rancher-monitoring-kube-controller-manager`, are configured to hit the metrics endpoint of the client using this proxy.
|
||||
|
||||
### Default Exporters
|
||||
### 4.3. Default Exporters
|
||||
|
||||
`rancher-monitoring` deploys two exporters to expose metrics to prometheus: `node-exporter` and `windows-exporter`. Both are deployed as DaemonSets.
|
||||
|
||||
@@ -152,16 +153,16 @@ For more information on `node-exporter`, refer to the [upstream documentation.](
|
||||
|
||||
[kube-state-metrics](https://github.com/kubernetes/kube-state-metrics) is also useful because it exports metrics for Kubernetes components.
|
||||
|
||||
# Components Exposed in the Rancher UI
|
||||
# 5. Components Exposed in the Rancher UI
|
||||
|
||||
When the monitoring application is installed, you will be able to edit the following components in the Rancher UI:
|
||||
|
||||
| Component | Type of Component | Common Use Case for Editing |
|
||||
| Component | Type of Component | Purpose and Common Use Cases for Editing |
|
||||
|--------------|------------------------|---------------------------|
|
||||
| ServiceMonitor | Custom resource | Set up targets to scrape custom metrics from. Automatically updates the scrape configuration in the Prometheus custom resource. |
|
||||
| PodMonitor | Custom resource | Set up targets to scrape custom metrics from. Automatically updates the scrape configuration in the Prometheus custom resource. |
|
||||
| Receiver | Configuration block | Set up a notification system to receive alerts. Automatically updates the Alertmanager custom resource. |
|
||||
| Route | Configuration block | Add identifying information to make alerts more meaningful and direct them to individual teams. Automatically updates the Alertmanager custom resource. |
|
||||
| Receiver | Configuration block (part of Alertmanager) | Set up a notification system to receive alerts. Automatically updates the Alertmanager custom resource. |
|
||||
| Route | Configuration block (part of Alertmanager) | Add identifying information to make alerts more meaningful and direct them to individual teams. Automatically updates the Alertmanager custom resource. |
|
||||
| PrometheusRule | Custom resource | For more advanced use cases, you may want to define what Prometheus metrics or time series database queries should result in alerts being fired. Automatically updates the Prometheus custom resource. |
|
||||
| Alertmanager | Custom resource | Edit this custom resource only if you need more advanced configuration options beyond what the Rancher UI exposes in the Routes and Receivers sections. For example, you might want to edit this resource to add a routing tree with more than two levels. |
|
||||
| Prometheus | Custom resource | Edit this custom resource only if you need more advanced configuration beyond what can be configured using ServiceMonitors, PodMonitors, or [Rancher monitoring Helm chart options.](./configuration/helm-chart-options) |
|
||||
@@ -1,25 +0,0 @@
|
||||
---
|
||||
title: Prometheus UI
|
||||
weight: 7
|
||||
---
|
||||
|
||||
To see the Prometheus UI, install `rancher-monitoring`. Then go to the **Cluster Explorer.** In the top left corner, click **Cluster Explorer > Monitoring.** Then click **Prometheus Graph.**
|
||||
|
||||
<figcaption>Prometheus Graph UI</figcaption>
|
||||
|
||||
|
||||
### Viewing the Prometheus Targets
|
||||
|
||||
To see the Prometheus Targets, install `rancher-monitoring`. Then go to the **Cluster Explorer.** In the top left corner, click **Cluster Explorer > Monitoring.** Then click **Prometheus Targets.**
|
||||
|
||||
<figcaption>Targets in the Prometheus UI</figcaption>
|
||||
|
||||
|
||||
### Viewing the PrometheusRules
|
||||
|
||||
To see the PrometheusRules, install `rancher-monitoring`. Then go to the **Cluster Explorer.** In the top left corner, click **Cluster Explorer > Monitoring.** Then click **Prometheus Rules.**
|
||||
|
||||
<figcaption>Rules in the Prometheus UI</figcaption>
|
||||
|
||||
|
||||
For more information on PrometheusRules in Rancher, see [this page.](./configuration/prometheusrules)
|
||||
@@ -1,6 +1,7 @@
|
||||
---
|
||||
title: RBAC
|
||||
weight: 3
|
||||
title: Role-based Access Control
|
||||
shortTitle: RBAC
|
||||
weight: 2
|
||||
aliases:
|
||||
- /rancher/v2.5/en/cluster-admin/tools/monitoring/rbac
|
||||
- /rancher/v2.5/en/monitoring-alerting/v2.5/rbac
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
---
|
||||
title: Windows Cluster Support for Monitoring V2
|
||||
shortTitle: Windows Clusters
|
||||
shortTitle: Windows Support
|
||||
weight: 5
|
||||
---
|
||||
|
||||
|
||||
File diff suppressed because one or more lines are too long
|
After Width: | Height: | Size: 28 KiB |
Reference in New Issue
Block a user