mirror of
https://github.com/rancher/rancher-docs.git
synced 2026-09-25 12:38:05 +00:00
Clean up monitoring docs, minor corrections
This commit is contained in:
Catherine Luse
@@ -69,11 +69,9 @@ For information on configuring access to monitoring, see [this page.](./rbac)
|
||||
|
||||
- [Enable monitoring](./guides/enable-monitoring)
|
||||
- [Uninstall monitoring](./guides/uninstall)
|
||||
- [Monitoring Rancher apps](./guides/monitoring-rancher-apps)
|
||||
- [Monitoring workloads](./guides/monitoring-workloads)
|
||||
- [Customizing Grafana dashboards](./guides/customize-grafana)
|
||||
- [Persistent Grafana dashboards](./guides/persist-grafana)
|
||||
- [Setting up metrics for horizontal pod autoscaling](./guides/hpa)
|
||||
- [Debugging high memory usage](./guides/memory-usage)
|
||||
- [Migrating from Monitoring V1 to V2](./guides/migrating)
|
||||
|
||||
|
||||
@@ -9,11 +9,9 @@ This page captures some of the most important options for configuring Monitoring
|
||||
|
||||
For information on configuring custom scrape targets and rules for Prometheus, please refer to the upstream documentation for the [Prometheus Operator.](https://github.com/prometheus-operator/prometheus-operator) Some of the most important custom resources are explained in the Prometheus Operator [design documentation.](https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/design.md) The Prometheus Operator documentation can help also you set up RBAC, Thanos, or custom configuration.
|
||||
|
||||
This section assumes that you understand how the Prometheus Operator’s custom resources work together. For more information, see [this section.]
|
||||
|
||||
# Setting Resource Limits and Requests
|
||||
|
||||
The resource requests and limits for the monitoring application can be configured when installing `rancher-monitoring`. For more information about the default limits, see [this page.](./resource-limits)
|
||||
The resource requests and limits for the monitoring application can be configured when installing `rancher-monitoring`. For more information about the default limits, see [this page.](./helm-chart-options/#configuring-resource-limits-and-requests)
|
||||
|
||||
# Prometheus Configuration
|
||||
|
||||
@@ -24,38 +22,27 @@ Instead, to configure Prometheus to scrape custom metrics, you will only need to
|
||||
|
||||
### ServiceMonitor and PodMonitor Configuration
|
||||
|
||||
For details, see [this page.](./)
|
||||
For details, see [this page.](./servicemonitor-podmonitor)
|
||||
|
||||
### Advanced Prometheus Configuration
|
||||
|
||||
Link to ‘how monitoring works’ for the section about the Prometheus CR.
|
||||
|
||||
For more information about directly editing the Prometheus custom resource, which may be helpful in advanced use cases, see [this page.](./advanced/prometheus)
|
||||
|
||||
# Alertmanager Configuration
|
||||
|
||||
The Alertmanager custom resource usually doesn't need to be edited directly. For most common use cases, you can manage alerts by updating Routes and Receivers.
|
||||
|
||||
Routes and receivers are part of the configuration of the alertmanager custom resource. In the Rancher UI, Routes and Receivers are not true custom resources, but pseudo-custom resources that are mapped to sections within the Alertmanager custom resource.
|
||||
|
||||
When routes and receivers are updated, the monitoring application will automatically update Alertmanager to reflect those changes.
|
||||
Routes and receivers are part of the configuration of the alertmanager custom resource. In the Rancher UI, Routes and Receivers are not true custom resources, but pseudo-custom resources that the Prometheus Operator uses to synchronize your configuration with the Alertmanager custom resource. When routes and receivers are updated, the monitoring application will automatically update Alertmanager to reflect those changes.
|
||||
|
||||
For some advanced use cases, you may want to configure alertmanager directly. For more information, refer to [this page.](./advanced/alertmanager)
|
||||
|
||||
|
||||
|
||||
### Receivers
|
||||
|
||||
[link to section of how monitoring works that explains receivers]
|
||||
|
||||
For details on how to configure receivers, see [this page.](./receiver)
|
||||
Receivers are used to set up notifications. For details on how to configure receivers, see [this page.](./receiver)
|
||||
### Routes
|
||||
[link to section of how monitoring works that explains routes]
|
||||
|
||||
The route needs to refer to a receiver that has already been configured.
|
||||
Routes filter notifications before they reach receivers. Each route needs to refer to a receiver that has already been configured. For details on how to configure routes, see [this page.](./route)
|
||||
|
||||
### Advanced
|
||||
|
||||
Link to ‘how monitoring works’ for the section about the alertmanager CR.
|
||||
|
||||
For more information about directly editing the Alertmanager custom resource, which may be helpful in advanced use cases, see [this page.](./advanced/alertmanager)
|
||||
@@ -1,4 +1,16 @@
|
||||
---
|
||||
title: Advanced Configuration
|
||||
weight: 5
|
||||
---
|
||||
weight: 500
|
||||
---
|
||||
|
||||
### Alertmanager
|
||||
|
||||
For information on configuring the Alertmanager custom resource, see [this page.](./alertmanager)
|
||||
|
||||
### Prometheus
|
||||
|
||||
For information on configuring the Prometheus custom resource, see [this page.](./prometheus)
|
||||
|
||||
### PrometheusRules
|
||||
|
||||
For information on configuring the Prometheus custom resource, see [this page.](./prometheusrules)
|
||||
+1
-1
@@ -7,7 +7,7 @@ It is usually not necessary to directly edit the Alertmanager custom resource. F
|
||||
|
||||
When Receivers and Routes are updated, the monitoring application will automatically update the Alertmanager custom resource to be consistent with those changes.
|
||||
|
||||
> This section assumes familiarity with how monitoring components work together. For more information about Alertmanager, see [this section.](../how-monitoring-works/#how-alertmanager-works)
|
||||
> This section assumes familiarity with how monitoring components work together. For more information about Alertmanager, see [this section.](../../../how-monitoring-works/#how-alertmanager-works)
|
||||
|
||||
# About the Alertmanager Custom Resource
|
||||
|
||||
|
||||
+7
-9
@@ -8,14 +8,12 @@ aliases:
|
||||
---
|
||||
|
||||
It is usually not necessary to directly edit the Prometheus custom resource because the monitoring application automatically updates it based on changes to ServiceMonitors and PodMonitors.
|
||||
|
||||
> This section assumes familiarity with how monitoring components work together. For more information about Alertmanager, see [this section.](../how-monitoring-works/#how-alertmanager-works)
|
||||
|
||||
|
||||
|
||||
> This section assumes familiarity with how monitoring components work together. For more information, see [this section.](../../../how-monitoring-works/)
|
||||
|
||||
# About the Prometheus Custom Resource
|
||||
- when the Prometheus operator observes it, it creates prometheus-rancher-monitoring-prometheus, which is the prometheus deployment that is created based on the configuration in the Prometheus CR.
|
||||
- This is where we configure details like what Alertmanagers are connected to Prometheus, what are the external URLs, and other details that prometheus needs. Rancher builds this CR for you. It has fields for pod monitor and service monitor selectors - technically you can filter that to include only the ones in a certain namespace.
|
||||
- monitoring v2 only supports one prometheus per cluster because we haven’t supported project level monitoring. But you might want to edit prometheus Cr if you want to limit the namespaces.
|
||||
- prometheus also has the rules and routes in it.
|
||||
|
||||
The Prometheus CR defines a desired Prometheus deployment. The Prometheus Operator observes the Prometheus CR. When the CR changes, the Prometheus Operator creates `prometheus-rancher-monitoring-prometheus`, a Prometheus deployment based on the CR configuration.
|
||||
|
||||
The Prometheus CR specifies details such as rules and what Alertmanagers are connected to Prometheus. Rancher builds this CR for you.
|
||||
|
||||
Monitoring V2 only supports one Prometheus per cluster. However, you might want to edit the Prometheus CR if you want to limit monitoring to certain namespaces.
|
||||
@@ -1,9 +1,8 @@
|
||||
---
|
||||
title: Examples
|
||||
weight: 5
|
||||
weight: 400
|
||||
---
|
||||
|
||||
|
||||
### ServiceMonitor
|
||||
|
||||
An example ServiceMonitor custom resource can be found [here.](https://github.com/prometheus-operator/prometheus-operator/blob/master/example/prometheus-operator-crd/monitoring.coreos.com_servicemonitors.yaml)
|
||||
|
||||
@@ -12,7 +12,7 @@ aliases:
|
||||
|
||||
The [Alertmanager Config](https://prometheus.io/docs/alerting/latest/configuration/#configuration-file) Secret contains the configuration of an Alertmanager instance that sends out notifications based on alerts it receives from Prometheus.
|
||||
|
||||
> This section assumes familiarity with how monitoring components work together. For more information about Alertmanager, see [this section.](../how-monitoring-works/#how-alertmanager-works)
|
||||
> This section assumes familiarity with how monitoring components work together. For more information about Alertmanager, see [this section.](../../how-monitoring-works/#3-how-alertmanager-works)
|
||||
|
||||
- [Creating Receivers in the Rancher UI](#creating-receivers-in-the-rancher-ui)
|
||||
- [Receiver Configuration](#receiver-configuration)
|
||||
|
||||
@@ -10,7 +10,7 @@ When a Route is changed, the Prometheus Operator regenerates the Alertmanager cu
|
||||
|
||||
For more information about configuring routes, refer to the [official Alertmanager documentation.](https://www.prometheus.io/docs/alerting/latest/configuration/#route)
|
||||
|
||||
> This section assumes familiarity with how monitoring components work together. For more information about Alertmanager, see [this section.](../how-monitoring-works/#how-alertmanager-works)
|
||||
> This section assumes familiarity with how monitoring components work together. For more information about Alertmanager, see [this section.](../../how-monitoring-works/#3-how-alertmanager-works)
|
||||
|
||||
- [Route Restrictions](#route-restrictions)
|
||||
- [Route Configuration](#route-configuration)
|
||||
@@ -20,13 +20,13 @@ For more information about configuring routes, refer to the [official Alertmanag
|
||||
|
||||
# Route Restrictions
|
||||
|
||||
- Alertmanager proxies alerts for Prometheus based on a configuration. It has receivers and a routing tree.
|
||||
- Receivers: One or more notification providers (Slack, PagerDuty, etc.) to send alerts to.
|
||||
- Routing tree: A set of routes that filter alerts to certain receivers based on labels.
|
||||
- Alerting drivers proxy alerts for Alertmanager to non-native receivers, such as Microsoft Teams and SMS.
|
||||
- can configure a routing tree to send and then continue. We only support routing trees with one root and then a depth of one more, for a depth two tree. But technically a ‘continue’ route lets you make the tree deeper.
|
||||
- the receiver is for one or more notification providers. So if you know every alert for slack should also go to pager duty, you can put both configs in the same receiver.
|
||||
- we now support broad SMS, not just Aliyun.
|
||||
Alertmanager proxies alerts for Prometheus based on its receivers and a routing tree that filters alerts to certain receivers based on labels.
|
||||
|
||||
Alerting drivers proxy alerts for Alertmanager to non-native receivers, such as Microsoft Teams and SMS.
|
||||
|
||||
In the Rancher UI for configuring routes and receivers, you can configure routing trees with one root and then a depth of one more level, for a tree with a depth of two. But if you use a `continue` route when configuring Alertmanager directly, you can make the tree deeper.
|
||||
|
||||
Each receiver is for one or more notification providers. So if you know that every alert for Slack should also go to PagerDuty, you can configure both in the same receiver.
|
||||
|
||||
# Route Configuration
|
||||
|
||||
|
||||
+1
-1
@@ -10,7 +10,7 @@ These configuration objects declaratively specify the endpoints that Prometheus
|
||||
|
||||
ServiceMonitors are more commonly used than PodMonitors, and we recommend them for most use cases.
|
||||
|
||||
> This section assumes familiarity with how monitoring components work together. For more information about Alertmanager, see [this section.](../how-monitoring-works/#how-alertmanager-works)
|
||||
> This section assumes familiarity with how monitoring components work together. For more information, see [this section.](../../how-monitoring-works/)
|
||||
|
||||
### ServiceMonitors
|
||||
|
||||
|
||||
@@ -6,10 +6,8 @@ weight: 4
|
||||
|
||||
- [Enable monitoring](./enable-monitoring)
|
||||
- [Uninstall monitoring](./uninstall)
|
||||
- [Monitoring Rancher apps](./monitoring-rancher-apps)
|
||||
- [Monitoring workloads](./monitoring-workloads)
|
||||
- [Customizing Grafana dashboards](./customize-grafana)
|
||||
- [Persistent Grafana dashboards](./persist-grafana)
|
||||
- [Setting up metrics for horizontal pod autoscaling](./hpa)
|
||||
- [Debugging high memory usage](./memory-usage)
|
||||
- [Migrating from Monitoring V1 to V2](./migrating)
|
||||
@@ -25,28 +25,17 @@ To see the links to the external monitoring UIs, including Grafana dashboards, y
|
||||
For any panel, you can click the title and click **Explore** to get the PromQL queries powering the graphic.
|
||||
|
||||
For this example, we would like to get the CPU usage for the Alertmanager container, so we click **CPU Utilization > Inspect.**
|
||||
1. The **Data** tab shows the underlying data as a time series, with the time in first column and the PromQL query result in the second column. Copy the PromQL query.
|
||||
|
||||
The **Data** tab shows the underlying data as a time series, with the time in first column and the PromQL query result in the second column. Copy the PromQL query.
|
||||
|
||||
```
|
||||
(1 - (avg(irate({__name__=~"node_cpu_seconds_total|windows_cpu_time_total",mode="idle"}[5m])))) * 100
|
||||
|
||||
```
|
||||
|
||||
### Modifying an Existing Grafana Panel
|
||||
You can then modify the query in the Grafana panel or create a new Grafana panel using the query.
|
||||
|
||||
1. Open the Grafana dashboard.
|
||||
See also:
|
||||
|
||||
|
||||
### Creating a New Grafana Panel in a Dashboard
|
||||
|
||||
|
||||
- let’s say you want metrics that apply only for the container alertmanager.
|
||||
- link to the promql queries used to make grafana dashboards. To get those queries,
|
||||
- go to grafana
|
||||
- right click on a graphic and click explore
|
||||
- it shows you the PromQL queries that are embedded in it
|
||||
|
||||
- can modify it
|
||||
- grafana shows you updated based on your modifications to the query
|
||||
|
||||
- also link to persisting grafana dashboards section
|
||||
- [Grafana docs on editing a panel](https://grafana.com/docs/grafana/latest/panels/panel-editor/)
|
||||
- [Grafana docs on adding a panel to a dashboard](https://grafana.com/docs/grafana/latest/panels/add-a-panel/)
|
||||
@@ -1,31 +0,0 @@
|
||||
---
|
||||
title: Setting up Metrics for HPA
|
||||
weight: 7
|
||||
---
|
||||
|
||||
The monitoring app installs a Prometheus adapter that can be used for making the metrics from monitoring available from the Kubernetes API. This is useful for horizontal pod autoscaling based on custom metrics.
|
||||
|
||||
|
||||
- kube-state-metrics: monitors internal K8s components
|
||||
-
|
||||
For HPA it’s important to talk about kubernetes metrics APIs. For every rke cluster, metrics server is added on. HPA can hit that, can scale up or down based on pod or node usage.
|
||||
|
||||
We package Prometheus Adapter. It implements a k8s metrics api, says I want to expose these metrics in the k8s api so it can be used for HPA.
|
||||
|
||||
|
||||
- kubernetes metrics APIs are implemented as adapters.
|
||||
- the default adapter that has been implemented for a long time is the resource metrics API. This is why when you deploy RKE, the default API that is added on is metrics server.
|
||||
- Metrics server is a kubernetes project that is an adapter that implements the resource metrics API. It collects different node metrics and stores it in a way that is accessible by HPA.
|
||||
- If you want prometheus metrics to be stored on the Kubernetes API for you to be able to do HPA on, then the relevant way to configure that is by using Prometheus Adapter. It is packaged by default in monitoring v2, but not v1.
|
||||
- if you want to do the custom metrics API, there is a secret for Prometheus Adapter that you can modify that will start exposing selected metrics from Prometheus onto those APIs, which can then be consumed by HPA.
|
||||
- resource metrics: implemented by metrics-server, deployed as an RKE add-on
|
||||
- custom metrics Api: implemented by Prometheus Adapter, exposed for use within the cluster (e.g. HPA)
|
||||
- External Metrics API: implemented by Prometheus Adapter, exposed for use outside the cluster.
|
||||
|
||||
|
||||
Kubernetes metrics API
|
||||
- for HPA, how do I query prometheus to use that?
|
||||
- prometheus stores data within its own time series database
|
||||
- there are times when you also want to expose that within kubernetes itself, so that things like HPA can use it.
|
||||
- k8s has metrics apis that are implemented as adapters
|
||||
- big one is metrics API
|
||||
@@ -1,20 +0,0 @@
|
||||
---
|
||||
title: Monitoring Rancher Apps
|
||||
weight: 3
|
||||
---
|
||||
|
||||
|
||||
A common pattern for Rancher apps is to package a ServiceMonitor in the Helm chart for the application. The ServiceMonitor contains a preconfigured Prometheus target for monitoring.
|
||||
|
||||
When the ServiceMonitor is enabled and monitoring is also enabled, Prometheus will be able to scrape metrics from the Rancher application.
|
||||
|
||||
|
||||
|
||||
CIS application has a flag that lets you deploy a service monitor in it. As a general practice we expose charts for prometheus metrics to have that service monitor definition. The moment it’s deployed into the cluster, the prometheus scrape configuration will automatically be updated to reflect the service monitors that it has access to.
|
||||
|
||||
In logging v2 they will deploy a service monitor and we will just absorb it.
|
||||
|
||||
|
||||
question: someone found out from looking through rancher helm charts that some of them already have a service monitor defined that you might have to turn on, and if you do, those metrics are prepackaged for Prometheus in the right format.
|
||||
|
||||
It's a common pattern to have service monitor packaged inside. That’s how we do it for cis scans.
|
||||
@@ -8,36 +8,20 @@ weight: 4
|
||||
|
||||
If you only need CPU and memory time series for the workload, you don't need to deploy a ServiceMonitor or PodMonitor because the monitoring application already collects metrics data on resource usage by default.
|
||||
|
||||
The steps for setting up monitoring for workloads depend on whether you want basic metrics such as CPU and memory for the workload, or whether you want to scrape custom metrics from the workload.
|
||||
|
||||
If you only need CPU and memory time series for the workload, you don't need to deploy a ServiceMonitor or PodMonitor because the monitoring application already collects metrics data on resource usage by default. The resource usage time series data is in Prometheus's local time series database.
|
||||
|
||||
|
||||
The steps for setting up monitoring for workloads depends on whether you want basic metrics such as CPU and memory for the workload, or whether you want to scrape custom metrics from the workload.
|
||||
|
||||
If you only need CPU and memory time series for the workload, you don't need to deploy a ServiceMonitor or PodMonitor because the monitoring application already collects metrics data on resource usage by default. The resource usage time series data is in Prometheus's local time series database. Grafana shows the data in aggregate, but you can see the data for the individual workload by using a PromQL query that extracts the data for that workload. Once you have the PromQL query, you can execute the query individually in the Prometheus UI and see the time series visualized there, or you can use the query to customize a Grafana dashboard to display the workload metrics. For examples of PromQL queries for workload metrics, see [this section.](https://rancher.com/docs/rancher/v2.5/en/monitoring-alerting/configuration/expression/#workload-metrics)
|
||||
Grafana shows the data in aggregate, but you can see the data for the individual workload by using a PromQL query that extracts the data for that workload. Once you have the PromQL query, you can execute the query individually in the Prometheus UI and see the time series visualized there, or you can use the query to customize a Grafana dashboard to display the workload metrics. For examples of PromQL queries for workload metrics, see [this section.](https://rancher.com/docs/rancher/v2.5/en/monitoring-alerting/configuration/expression/#workload-metrics)
|
||||
|
||||
To set up custom metrics for your workload, you will need to set up an exporter and create a new ServiceMonitor custom resource to configure Prometheus to scrape metrics from your exporter.
|
||||
|
||||
For more information, see [this section.](./monitoring-workloads)
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
explain how some applications come with a servicemonitor packaged within them
|
||||
|
||||
for example, some rancher applications come with servicemonitors (link to section)
|
||||
|
||||
### Display CPU and Memory Metrics for a Workload
|
||||
|
||||
By default, the monitoring application already scrapes CPU and memory.
|
||||
|
||||
To get some fine-grained detail for a particular workload, you can customize a Grafana dashboard to display the metrics for a particular workload.
|
||||
|
||||
- there’s already a wealth of information provided by kube-state-metrics. Cpu utilization, memory utilization for different things across namespaces. If you just want resource metrics for prod, you don’t need to create a new ServiceMonitor for it. All you need to do is go to the prometheus UI and do a PromQL query to get the information.
|
||||
|
||||
For more information on customizing Grafana to show the workload metrics, see this section. (Link)
|
||||
|
||||
|
||||
### Setting up Metrics Beyond CPU and Memory
|
||||
|
||||
For custom metrics, you will need to expose the metrics on your application in a format supported by Prometheus.
|
||||
@@ -45,12 +29,3 @@ For custom metrics, you will need to expose the metrics on your application in a
|
||||
Then we recommend that you should create a new ServiceMonitor custom resource. When this resource is created, the Prometheus custom resource will be automatically updated so that its scrape configuration includes the new custom metrics endpoint. Then Prometheus will begin scraping metrics from the endpoint.
|
||||
|
||||
You can also create a PodMonitor to expose the custom metrics endpoint, but ServiceMonitors are more appropriate for the majority of use cases.
|
||||
|
||||
- let’s say we expose metrics at a particular endpoint. Let’s take rancher-monitoring-kube-state-metrics. For example they have a container port where they expose metrics from.
|
||||
- the approach I would take - although we don’t have a clean UI from it - is to create it from YAML.
|
||||
- for something like for grafana we’d create it like this - like for rancher-monitoring-grafana - where the basic details we need to provide are:
|
||||
- what is the actual endpoint that you want to hit (spec.endpoints, path and port) - what’s the HTTP path that you want to hit and what’s the port.
|
||||
- namespaceSelector: what namespaces does that particular deployment exist in within Kubernetes, and use matchNames to select them.
|
||||
- you can also use selector.matchLabels.
|
||||
- That’s what it takes to add monitoring if a serviceMonitor is not already defined.
|
||||
- example: use the rancher-monitoring-grafana YAML
|
||||
@@ -19,7 +19,7 @@ After configuring Rancher and GitHub, you can deploy containers running Jenkins
|
||||
- Run unit tests.
|
||||
- Run regression tests.
|
||||
|
||||
>**Note: Rancher's pipeline provides a simple CI/CD experience, but it does not offer the full power and flexibility of and is not a replacement of enterprise-grade Jenkins or other CI tools your team uses.
|
||||
>**Note:** Rancher's pipeline provides a simple CI/CD experience, but it does not offer the full power and flexibility of and is not a replacement of enterprise-grade Jenkins or other CI tools your team uses.
|
||||
|
||||
This section covers the following topics:
|
||||
|
||||
|
||||
@@ -17,7 +17,7 @@ The charts page contains all Rancher, Partner, and Custom Charts.
|
||||
|
||||
All three types are deployed and managed in the same way.
|
||||
|
||||
> Apps managed by the Cluster Manager (the global view in the legacy Rancher UI) should continue to be managed only by the Cluster Manager, and apps managed with **Apps & Marketplace** in the new UI must be managed only by **Apps & Marketplace**.
|
||||
> Apps managed by the Cluster Manager (the global view in the legacy Rancher UI) should continue to be managed only by the Cluster Manager, and apps managed with <b>Apps & Marketplace</b> in the new UI must be managed only by <b>Apps & Marketplace</b>.
|
||||
|
||||
### Repositories
|
||||
|
||||
|
||||
@@ -65,11 +65,9 @@ For information on configuring access to monitoring, see [this page.](./rbac)
|
||||
|
||||
- [Enable monitoring](./guides/enable-monitoring)
|
||||
- [Uninstall monitoring](./guides/uninstall)
|
||||
- [Monitoring Rancher apps](./guides/monitoring-rancher-apps)
|
||||
- [Monitoring workloads](./guides/monitoring-workloads)
|
||||
- [Customizing Grafana dashboards](./guides/customize-grafana)
|
||||
- [Persistent Grafana dashboards](./guides/persist-grafana)
|
||||
- [Setting up metrics for horizontal pod autoscaling](./guides/hpa)
|
||||
- [Debugging high memory usage](./guides/memory-usage)
|
||||
- [Migrating from Monitoring V1 to V2](./guides/migrating)
|
||||
|
||||
|
||||
@@ -7,11 +7,9 @@ This page captures some of the most important options for configuring Monitoring
|
||||
|
||||
For information on configuring custom scrape targets and rules for Prometheus, please refer to the upstream documentation for the [Prometheus Operator.](https://github.com/prometheus-operator/prometheus-operator) Some of the most important custom resources are explained in the Prometheus Operator [design documentation.](https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/design.md) The Prometheus Operator documentation can help also you set up RBAC, Thanos, or custom configuration.
|
||||
|
||||
This section assumes that you understand how the Prometheus Operator’s custom resources work together. For more information, see [this section.]
|
||||
|
||||
# Setting Resource Limits and Requests
|
||||
|
||||
The resource requests and limits for the monitoring application can be configured when installing `rancher-monitoring`. For more information about the default limits, see [this page.](./resource-limits)
|
||||
The resource requests and limits for the monitoring application can be configured when installing `rancher-monitoring`. For more information about the default limits, see [this page.](./helm-chart-options/#configuring-resource-limits-and-requests)
|
||||
|
||||
# Prometheus Configuration
|
||||
|
||||
@@ -22,38 +20,27 @@ Instead, to configure Prometheus to scrape custom metrics, you will only need to
|
||||
|
||||
### ServiceMonitor and PodMonitor Configuration
|
||||
|
||||
For details, see [this page.](./)
|
||||
For details, see [this page.](./servicemonitor-podmonitor)
|
||||
|
||||
### Advanced Prometheus Configuration
|
||||
|
||||
Link to ‘how monitoring works’ for the section about the Prometheus CR.
|
||||
|
||||
For more information about directly editing the Prometheus custom resource, which may be helpful in advanced use cases, see [this page.](./advanced/prometheus)
|
||||
|
||||
# Alertmanager Configuration
|
||||
|
||||
The Alertmanager custom resource usually doesn't need to be edited directly. For most common use cases, you can manage alerts by updating Routes and Receivers.
|
||||
|
||||
Routes and receivers are part of the configuration of the alertmanager custom resource. In the Rancher UI, Routes and Receivers are not true custom resources, but pseudo-custom resources that are mapped to sections within the Alertmanager custom resource.
|
||||
|
||||
When routes and receivers are updated, the monitoring application will automatically update Alertmanager to reflect those changes.
|
||||
Routes and receivers are part of the configuration of the alertmanager custom resource. In the Rancher UI, Routes and Receivers are not true custom resources, but pseudo-custom resources that the Prometheus Operator uses to synchronize your configuration with the Alertmanager custom resource. When routes and receivers are updated, the monitoring application will automatically update Alertmanager to reflect those changes.
|
||||
|
||||
For some advanced use cases, you may want to configure alertmanager directly. For more information, refer to [this page.](./advanced/alertmanager)
|
||||
|
||||
|
||||
|
||||
### Receivers
|
||||
|
||||
[link to section of how monitoring works that explains receivers]
|
||||
|
||||
For details on how to configure receivers, see [this page.](./receiver)
|
||||
Receivers are used to set up notifications. For details on how to configure receivers, see [this page.](./receiver)
|
||||
### Routes
|
||||
[link to section of how monitoring works that explains routes]
|
||||
|
||||
The route needs to refer to a receiver that has already been configured.
|
||||
Routes filter notifications before they reach receivers. Each route needs to refer to a receiver that has already been configured. For details on how to configure routes, see [this page.](./route)
|
||||
|
||||
### Advanced
|
||||
|
||||
Link to ‘how monitoring works’ for the section about the alertmanager CR.
|
||||
|
||||
For more information about directly editing the Alertmanager custom resource, which may be helpful in advanced use cases, see [this page.](./advanced/alertmanager)
|
||||
@@ -1,4 +1,16 @@
|
||||
---
|
||||
title: Advanced Configuration
|
||||
weight: 5
|
||||
---
|
||||
weight: 500
|
||||
---
|
||||
|
||||
### Alertmanager
|
||||
|
||||
For information on configuring the Alertmanager custom resource, see [this page.](./alertmanager)
|
||||
|
||||
### Prometheus
|
||||
|
||||
For information on configuring the Prometheus custom resource, see [this page.](./prometheus)
|
||||
|
||||
### PrometheusRules
|
||||
|
||||
For information on configuring the Prometheus custom resource, see [this page.](./prometheusrules)
|
||||
+1
-1
@@ -7,7 +7,7 @@ It is usually not necessary to directly edit the Alertmanager custom resource. F
|
||||
|
||||
When Receivers and Routes are updated, the monitoring application will automatically update the Alertmanager custom resource to be consistent with those changes.
|
||||
|
||||
> This section assumes familiarity with how monitoring components work together. For more information about Alertmanager, see [this section.](../how-monitoring-works/#how-alertmanager-works)
|
||||
> This section assumes familiarity with how monitoring components work together. For more information about Alertmanager, see [this section.](../../../how-monitoring-works/#3-how-alertmanager-works)
|
||||
|
||||
# About the Alertmanager Custom Resource
|
||||
|
||||
|
||||
+7
-8
@@ -5,13 +5,12 @@ weight: 1
|
||||
|
||||
It is usually not necessary to directly edit the Prometheus custom resource because the monitoring application automatically updates it based on changes to ServiceMonitors and PodMonitors.
|
||||
|
||||
> This section assumes familiarity with how monitoring components work together. For more information about Alertmanager, see [this section.](../how-monitoring-works/#how-alertmanager-works)
|
||||
|
||||
|
||||
|
||||
> This section assumes familiarity with how monitoring components work together. For more information, see [this section.](../../../how-monitoring-works/)
|
||||
|
||||
# About the Prometheus Custom Resource
|
||||
- when the Prometheus operator observes it, it creates prometheus-rancher-monitoring-prometheus, which is the prometheus deployment that is created based on the configuration in the Prometheus CR.
|
||||
- This is where we configure details like what Alertmanagers are connected to Prometheus, what are the external URLs, and other details that prometheus needs. Rancher builds this CR for you. It has fields for pod monitor and service monitor selectors - technically you can filter that to include only the ones in a certain namespace.
|
||||
- monitoring v2 only supports one prometheus per cluster because we haven’t supported project level monitoring. But you might want to edit prometheus Cr if you want to limit the namespaces.
|
||||
- prometheus also has the rules and routes in it.
|
||||
|
||||
The Prometheus CR defines a desired Prometheus deployment. The Prometheus Operator observes the Prometheus CR. When the CR changes, the Prometheus Operator creates `prometheus-rancher-monitoring-prometheus`, a Prometheus deployment based on the CR configuration.
|
||||
|
||||
The Prometheus CR specifies details such as rules and what Alertmanagers are connected to Prometheus. Rancher builds this CR for you.
|
||||
|
||||
Monitoring V2 only supports one Prometheus per cluster. However, you might want to edit the Prometheus CR if you want to limit monitoring to certain namespaces.
|
||||
+2
-15
@@ -5,13 +5,11 @@ weight: 3
|
||||
|
||||
A PrometheusRule defines a group of Prometheus alerting and/or recording rules.
|
||||
|
||||
> This section assumes familiarity with how monitoring components work together. For more information about Alertmanager, see [this section.](../how-monitoring-works/#how-alertmanager-works)
|
||||
> This section assumes familiarity with how monitoring components work together. For more information, see [this section.](../../../how-monitoring-works)
|
||||
|
||||
|
||||
### Creating PrometheusRules in the Rancher UI
|
||||
|
||||
_Available as of v2.5.4_
|
||||
|
||||
> **Prerequisite:** The monitoring application needs to be installed.
|
||||
|
||||
To create rule groups in the Rancher UI,
|
||||
@@ -43,11 +41,6 @@ For examples, refer to the Prometheus documentation on [recording rules](https:/
|
||||
|
||||
# Configuration
|
||||
|
||||
{{% tabs %}}
|
||||
{{% tab "Rancher v2.5.4" %}}
|
||||
Rancher v2.5.4 introduced the capability to configure PrometheusRules by filling out forms in the Rancher UI.
|
||||
|
||||
|
||||
### Rule Group
|
||||
|
||||
| Field | Description |
|
||||
@@ -78,10 +71,4 @@ Rancher v2.5.4 introduced the capability to configure PrometheusRules by filling
|
||||
|-------|----------------|
|
||||
| Time Series Name | The name of the time series to output to. Must be a valid metric name. |
|
||||
| PromQL Expression | The PromQL expression to evaluate. Prometheus will evaluate the current value of this PromQL expression on every evaluation cycle and the result will be recorded as a new set of time series with the metric name as given by 'record'. For more information about expressions, refer to the [Prometheus documentation](https://prometheus.io/docs/prometheus/latest/querying/basics/) or our [example PromQL expressions.](../expression) |
|
||||
| Labels | Labels to add or overwrite before storing the result. |
|
||||
|
||||
{{% /tab %}}
|
||||
{{% tab "Rancher v2.5.0-v2.5.3" %}}
|
||||
For Rancher v2.5.0-v2.5.3, PrometheusRules must be configured in YAML. For examples, refer to the Prometheus documentation on [recording rules](https://prometheus.io/docs/prometheus/latest/configuration/recording_rules/) and [alerting rules.](https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/)
|
||||
{{% /tab %}}
|
||||
{{% /tabs %}}
|
||||
| Labels | Labels to add or overwrite before storing the result. |
|
||||
@@ -1,6 +1,6 @@
|
||||
---
|
||||
title: Examples
|
||||
weight: 5
|
||||
weight: 400
|
||||
---
|
||||
|
||||
|
||||
|
||||
@@ -6,7 +6,7 @@ weight: 1
|
||||
|
||||
The [Alertmanager Config](https://prometheus.io/docs/alerting/latest/configuration/#configuration-file) Secret contains the configuration of an Alertmanager instance that sends out notifications based on alerts it receives from Prometheus.
|
||||
|
||||
> This section assumes familiarity with how monitoring components work together. For more information about Alertmanager, see [this section.](../how-monitoring-works/#how-alertmanager-works)
|
||||
> This section assumes familiarity with how monitoring components work together. For more information about Alertmanager, see [this section.](../../how-monitoring-works/#3-how-alertmanager-works)
|
||||
|
||||
- [Creating Receivers in the Rancher UI](#creating-receivers-in-the-rancher-ui)
|
||||
- [Receiver Configuration](#receiver-configuration)
|
||||
@@ -18,10 +18,6 @@ The [Alertmanager Config](https://prometheus.io/docs/alerting/latest/configurati
|
||||
- [Custom](#custom)
|
||||
- [Teams](#teams)
|
||||
- [SMS](#sms)
|
||||
- [Route Configuration](#route-configuration)
|
||||
- [Receiver](#receiver)
|
||||
- [Grouping](#grouping)
|
||||
- [Matching](#matching)
|
||||
- [Configuring Multiple Receivers](#configuring-multiple-receivers)
|
||||
- [Example Alertmanager Config](../examples/#example-alertmanager-config)
|
||||
- [Example Route Config for CIS Scan Alerts](#example-route-config-for-cis-scan-alerts)
|
||||
@@ -58,17 +54,6 @@ Currently the Rancher Alerting Drivers app provides access to the following inte
|
||||
- Microsoft Teams, based on the [prom2teams](https://github.com/idealista/prom2teams) driver
|
||||
- SMS, based on the [Sachet](https://github.com/messagebird/sachet) driver
|
||||
|
||||
### Changes in Rancher v2.5.8
|
||||
|
||||
Rancher v2.5.8 added Microsoft Teams and SMS as configurable receivers in the Rancher UI.
|
||||
|
||||
### Changes in Rancher v2.5.4
|
||||
|
||||
Rancher v2.5.4 introduced the capability to configure receivers by filling out forms in the Rancher UI.
|
||||
|
||||
{{% tabs %}}
|
||||
{{% tab "Rancher v2.5.8+" %}}
|
||||
|
||||
The following types of receivers can be configured in the Rancher UI:
|
||||
|
||||
- <a href="#slack">Slack</a>
|
||||
@@ -223,87 +208,6 @@ url http://rancher-alerting-drivers-sachet.ns-1.svc:9876/alert
|
||||
|
||||
<!-- https://github.com/messagebird/sachet -->
|
||||
|
||||
{{% /tab %}}
|
||||
{{% tab "Rancher v2.5.4-2.5.7" %}}
|
||||
|
||||
The following types of receivers can be configured in the Rancher UI:
|
||||
|
||||
- <a href="#slack-254-257">Slack</a>
|
||||
- <a href="#email-254-257">Email</a>
|
||||
- <a href="#pagerduty-254-257">PagerDuty</a>
|
||||
- <a href="#opsgenie-254-257">Opsgenie</a>
|
||||
- <a href="#webhook-254-257">Webhook</a>
|
||||
- <a href="#custom-254-257">Custom</a>
|
||||
|
||||
The custom receiver option can be used to configure any receiver in YAML that cannot be configured by filling out the other forms in the Rancher UI.
|
||||
|
||||
### Slack {#slack-254-257}
|
||||
|
||||
| Field | Type | Description |
|
||||
|------|--------------|------|
|
||||
| URL | String | Enter your Slack webhook URL. For instructions to create a Slack webhook, see the [Slack documentation.](https://get.slack.help/hc/en-us/articles/115005265063-Incoming-WebHooks-for-Slack) |
|
||||
| Default Channel | String | Enter the name of the channel that you want to send alert notifications in the following format: `#<channelname>`. |
|
||||
| Proxy URL | String | Proxy for the webhook notifications. |
|
||||
| Enable Send Resolved Alerts | Bool | Whether to send a follow-up notification if an alert has been resolved (e.g. [Resolved] High CPU Usage). |
|
||||
|
||||
### Email {#email-254-257}
|
||||
|
||||
| Field | Type | Description |
|
||||
|------|--------------|------|
|
||||
| Default Recipient Address | String | The email address that will receive notifications. |
|
||||
| Enable Send Resolved Alerts | Bool | Whether to send a follow-up notification if an alert has been resolved (e.g. [Resolved] High CPU Usage). |
|
||||
|
||||
SMTP options:
|
||||
|
||||
| Field | Type | Description |
|
||||
|------|--------------|------|
|
||||
| Sender | String | Enter an email address available on your SMTP mail server that you want to send the notification from. |
|
||||
| Host | String | Enter the IP address or hostname for your SMTP server. Example: `smtp.email.com`. |
|
||||
| Use TLS | Bool | Use TLS for encryption. |
|
||||
| Username | String | Enter a username to authenticate with the SMTP server. |
|
||||
| Password | String | Enter a password to authenticate with the SMTP server. |
|
||||
|
||||
### PagerDuty {#pagerduty-254-257}
|
||||
|
||||
| Field | Type | Description |
|
||||
|------|------|-------|
|
||||
| Integration Type | String | `Events API v2` or `Prometheus`. |
|
||||
| Default Integration Key | String | For instructions to get an integration key, see the [PagerDuty documentation.](https://www.pagerduty.com/docs/guides/prometheus-integration-guide/) |
|
||||
| Proxy URL | String | Proxy for the PagerDuty notifications. |
|
||||
| Enable Send Resolved Alerts | Bool | Whether to send a follow-up notification if an alert has been resolved (e.g. [Resolved] High CPU Usage). |
|
||||
|
||||
### Opsgenie {#opsgenie-254-257}
|
||||
|
||||
| Field | Description |
|
||||
|------|-------------|
|
||||
| API Key | For instructions to get an API key, refer to the [Opsgenie documentation.](https://docs.opsgenie.com/docs/api-key-management) |
|
||||
| Proxy URL | Proxy for the Opsgenie notifications. |
|
||||
| Enable Send Resolved Alerts | Whether to send a follow-up notification if an alert has been resolved (e.g. [Resolved] High CPU Usage). |
|
||||
|
||||
Opsgenie Responders:
|
||||
|
||||
| Field | Type | Description |
|
||||
|-------|------|--------|
|
||||
| Type | String | Schedule, Team, User, or Escalation. For more information on alert responders, refer to the [Opsgenie documentation.](https://docs.opsgenie.com/docs/alert-recipients-and-teams) |
|
||||
| Send To | String | Id, Name, or Username of the Opsgenie recipient. |
|
||||
|
||||
### Webhook {#webhook-1}
|
||||
|
||||
| Field | Description |
|
||||
|-------|--------------|
|
||||
| URL | Webhook URL for the app of your choice. |
|
||||
| Proxy URL | Proxy for the webhook notification. |
|
||||
| Enable Send Resolved Alerts | Whether to send a follow-up notification if an alert has been resolved (e.g. [Resolved] High CPU Usage). |
|
||||
|
||||
### Custom {#custom-254-257}
|
||||
|
||||
The YAML provided here will be directly appended to your receiver within the Alertmanager Config Secret.
|
||||
|
||||
{{% /tab %}}
|
||||
{{% tab "Rancher v2.5.0-2.5.3" %}}
|
||||
The Alertmanager must be configured in YAML, as shown in these [examples.](#example-alertmanager-configs)
|
||||
{{% /tab %}}
|
||||
{{% /tabs %}}
|
||||
|
||||
# Configuring Multiple Receivers
|
||||
|
||||
|
||||
@@ -10,7 +10,7 @@ When a Route is changed, the Prometheus Operator regenerates the Alertmanager cu
|
||||
|
||||
For more information about configuring routes, refer to the [official Alertmanager documentation.](https://www.prometheus.io/docs/alerting/latest/configuration/#route)
|
||||
|
||||
> This section assumes familiarity with how monitoring components work together. For more information about Alertmanager, see [this section.](../how-monitoring-works/#how-alertmanager-works)
|
||||
> This section assumes familiarity with how monitoring components work together. For more information about Alertmanager, see [this section.](../../how-monitoring-works/#3-how-alertmanager-works)
|
||||
|
||||
- [Route Restrictions](#route-restrictions)
|
||||
- [Route Configuration](#route-configuration)
|
||||
@@ -20,13 +20,13 @@ For more information about configuring routes, refer to the [official Alertmanag
|
||||
|
||||
# Route Restrictions
|
||||
|
||||
- Alertmanager proxies alerts for Prometheus based on a configuration. It has receivers and a routing tree.
|
||||
- Receivers: One or more notification providers (Slack, PagerDuty, etc.) to send alerts to.
|
||||
- Routing tree: A set of routes that filter alerts to certain receivers based on labels.
|
||||
- Alerting drivers proxy alerts for Alertmanager to non-native receivers, such as Microsoft Teams and SMS.
|
||||
- can configure a routing tree to send and then continue. We only support routing trees with one root and then a depth of one more, for a depth two tree. But technically a ‘continue’ route lets you make the tree deeper.
|
||||
- the receiver is for one or more notification providers. So if you know every alert for slack should also go to pager duty, you can put both configs in the same receiver.
|
||||
- we now support broad SMS, not just Aliyun.
|
||||
Alertmanager proxies alerts for Prometheus based on its receivers and a routing tree that filters alerts to certain receivers based on labels.
|
||||
|
||||
Alerting drivers proxy alerts for Alertmanager to non-native receivers, such as Microsoft Teams and SMS.
|
||||
|
||||
In the Rancher UI for configuring routes and receivers, you can configure routing trees with one root and then a depth of one more level, for a tree with a depth of two. But if you use a `continue` route when configuring Alertmanager directly, you can make the tree deeper.
|
||||
|
||||
Each receiver is for one or more notification providers. So if you know that every alert for Slack should also go to PagerDuty, you can configure both in the same receiver.
|
||||
|
||||
# Route Configuration
|
||||
|
||||
@@ -36,8 +36,6 @@ Labels should be used for identifying information that can affect the routing of
|
||||
|
||||
Annotations should be used for information that does not affect who receives the alert, such as a runbook url or error message.
|
||||
|
||||
{{% tabs %}}
|
||||
{{% tab "Rancher v2.5.4+" %}}
|
||||
|
||||
### Receiver
|
||||
The route needs to refer to a [receiver](#receiver-configuration) that has already been configured.
|
||||
@@ -66,9 +64,3 @@ The **Match Regex** field refers to a set of regex-matchers used to identify whi
|
||||
match_re:
|
||||
[ <labelname>: <regex>, ... ]
|
||||
```
|
||||
|
||||
{{% /tab %}}
|
||||
{{% tab "Rancher v2.5.0-2.5.3" %}}
|
||||
The Alertmanager must be configured in YAML, as shown in this [example.](./examples/#alertmanager-config)
|
||||
{{% /tab %}}
|
||||
{{% /tabs %}}
|
||||
+1
-1
@@ -10,7 +10,7 @@ These configuration objects declaratively specify the endpoints that Prometheus
|
||||
|
||||
ServiceMonitors are more commonly used than PodMonitors, and we recommend them for most use cases.
|
||||
|
||||
> This section assumes familiarity with how monitoring components work together. For more information about Alertmanager, see [this section.](../how-monitoring-works/#how-alertmanager-works)
|
||||
> This section assumes familiarity with how monitoring components work together. For more information about Alertmanager, see [this section.](../../how-monitoring-works/)
|
||||
|
||||
### ServiceMonitors
|
||||
|
||||
|
||||
@@ -6,10 +6,8 @@ weight: 4
|
||||
|
||||
- [Enable monitoring](./enable-monitoring)
|
||||
- [Uninstall monitoring](./uninstall)
|
||||
- [Monitoring Rancher apps](./monitoring-rancher-apps)
|
||||
- [Monitoring workloads](./monitoring-workloads)
|
||||
- [Customizing Grafana dashboards](./customize-grafana)
|
||||
- [Persistent Grafana dashboards](./persist-grafana)
|
||||
- [Setting up metrics for horizontal pod autoscaling](./hpa)
|
||||
- [Debugging high memory usage](./memory-usage)
|
||||
- [Migrating from Monitoring V1 to V2](./migrating)
|
||||
@@ -14,9 +14,9 @@ To see the links to the external monitoring UIs, including Grafana dashboards, y
|
||||
### Signing in to Grafana
|
||||
|
||||
1. In the Rancher UI, go to the cluster that has the dashboard you want to customize.
|
||||
1. In the left navigation menu, click **Monitoring**.
|
||||
1. Click **Grafana**. The Grafana dashboard should open in a new tab.
|
||||
1. Go to the log in icon in the lower left corner and click **Sign In**.
|
||||
1. In the left navigation menu, click **Monitoring.**
|
||||
1. Click **Grafana.** The Grafana dashboard should open in a new tab.
|
||||
1. Go to the log in icon in the lower left corner and click **Sign In.**
|
||||
1. Log in to Grafana. The default Admin username and password for the Grafana instance is `admin/prom-operator`. (Regardless of who has the password, cluster administrator permission in Rancher is still required access the Grafana instance.) Alternative credentials can also be supplied on deploying or upgrading the chart.
|
||||
|
||||
|
||||
@@ -24,29 +24,18 @@ To see the links to the external monitoring UIs, including Grafana dashboards, y
|
||||
|
||||
For any panel, you can click the title and click **Explore** to get the PromQL queries powering the graphic.
|
||||
|
||||
For this example, we would like to get the CPU usage for the Alertmanager container, so we click **CPU Utilization > Inspect**.
|
||||
1. The **Data** tab shows the underlying data as a time series, with the time in first column and the PromQL query result in the second column. Copy the PromQL query.
|
||||
For this example, we would like to get the CPU usage for the Alertmanager container, so we click **CPU Utilization > Inspect.**
|
||||
|
||||
The **Data** tab shows the underlying data as a time series, with the time in first column and the PromQL query result in the second column. Copy the PromQL query.
|
||||
|
||||
```
|
||||
(1 - (avg(irate({__name__=~"node_cpu_seconds_total|windows_cpu_time_total",mode="idle"}[5m])))) * 100
|
||||
|
||||
```
|
||||
|
||||
### Modifying an Existing Grafana Panel
|
||||
You can then modify the query in the Grafana panel or create a new Grafana panel using the query.
|
||||
|
||||
1. Open the Grafana dashboard.
|
||||
See also:
|
||||
|
||||
|
||||
### Creating a New Grafana Panel in a Dashboard
|
||||
|
||||
|
||||
- let’s say you want metrics that apply only for the container alertmanager.
|
||||
- link to the promql queries used to make grafana dashboards. To get those queries,
|
||||
- go to grafana
|
||||
- right click on a graphic and click explore
|
||||
- it shows you the PromQL queries that are embedded in it
|
||||
|
||||
- can modify it
|
||||
- grafana shows you updated based on your modifications to the query
|
||||
|
||||
- also link to persisting grafana dashboards section
|
||||
- [Grafana docs on editing a panel](https://grafana.com/docs/grafana/latest/panels/panel-editor/)
|
||||
- [Grafana docs on adding a panel to a dashboard](https://grafana.com/docs/grafana/latest/panels/add-a-panel/)
|
||||
@@ -1,31 +0,0 @@
|
||||
---
|
||||
title: Setting up Metrics for HPA
|
||||
weight: 7
|
||||
---
|
||||
|
||||
The monitoring app installs a Prometheus adapter that can be used for making the metrics from monitoring available from the Kubernetes API. This is useful for horizontal pod autoscaling based on custom metrics.
|
||||
|
||||
|
||||
- kube-state-metrics: monitors internal K8s components
|
||||
-
|
||||
For HPA it’s important to talk about kubernetes metrics APIs. For every rke cluster, metrics server is added on. HPA can hit that, can scale up or down based on pod or node usage.
|
||||
|
||||
We package Prometheus Adapter. It implements a k8s metrics api, says I want to expose these metrics in the k8s api so it can be used for HPA.
|
||||
|
||||
|
||||
- kubernetes metrics APIs are implemented as adapters.
|
||||
- the default adapter that has been implemented for a long time is the resource metrics API. This is why when you deploy RKE, the default API that is added on is metrics server.
|
||||
- Metrics server is a kubernetes project that is an adapter that implements the resource metrics API. It collects different node metrics and stores it in a way that is accessible by HPA.
|
||||
- If you want prometheus metrics to be stored on the Kubernetes API for you to be able to do HPA on, then the relevant way to configure that is by using Prometheus Adapter. It is packaged by default in monitoring v2, but not v1.
|
||||
- if you want to do the custom metrics API, there is a secret for Prometheus Adapter that you can modify that will start exposing selected metrics from Prometheus onto those APIs, which can then be consumed by HPA.
|
||||
- resource metrics: implemented by metrics-server, deployed as an RKE add-on
|
||||
- custom metrics Api: implemented by Prometheus Adapter, exposed for use within the cluster (e.g. HPA)
|
||||
- External Metrics API: implemented by Prometheus Adapter, exposed for use outside the cluster.
|
||||
|
||||
|
||||
Kubernetes metrics API
|
||||
- for HPA, how do I query prometheus to use that?
|
||||
- prometheus stores data within its own time series database
|
||||
- there are times when you also want to expose that within kubernetes itself, so that things like HPA can use it.
|
||||
- k8s has metrics apis that are implemented as adapters
|
||||
- big one is metrics API
|
||||
@@ -1,20 +0,0 @@
|
||||
---
|
||||
title: Monitoring Rancher Apps
|
||||
weight: 3
|
||||
---
|
||||
|
||||
|
||||
A common pattern for Rancher apps is to package a ServiceMonitor in the Helm chart for the application. The ServiceMonitor contains a preconfigured Prometheus target for monitoring.
|
||||
|
||||
When the ServiceMonitor is enabled and monitoring is also enabled, Prometheus will be able to scrape metrics from the Rancher application.
|
||||
|
||||
|
||||
|
||||
CIS application has a flag that lets you deploy a service monitor in it. As a general practice we expose charts for prometheus metrics to have that service monitor definition. The moment it’s deployed into the cluster, the prometheus scrape configuration will automatically be updated to reflect the service monitors that it has access to.
|
||||
|
||||
In logging v2 they will deploy a service monitor and we will just absorb it.
|
||||
|
||||
|
||||
question: someone found out from looking through rancher helm charts that some of them already have a service monitor defined that you might have to turn on, and if you do, those metrics are prepackaged for Prometheus in the right format.
|
||||
|
||||
It's a common pattern to have service monitor packaged inside. That’s how we do it for cis scans.
|
||||
@@ -8,36 +8,20 @@ weight: 4
|
||||
|
||||
If you only need CPU and memory time series for the workload, you don't need to deploy a ServiceMonitor or PodMonitor because the monitoring application already collects metrics data on resource usage by default.
|
||||
|
||||
The steps for setting up monitoring for workloads depend on whether you want basic metrics such as CPU and memory for the workload, or whether you want to scrape custom metrics from the workload.
|
||||
|
||||
If you only need CPU and memory time series for the workload, you don't need to deploy a ServiceMonitor or PodMonitor because the monitoring application already collects metrics data on resource usage by default. The resource usage time series data is in Prometheus's local time series database.
|
||||
|
||||
|
||||
The steps for setting up monitoring for workloads depends on whether you want basic metrics such as CPU and memory for the workload, or whether you want to scrape custom metrics from the workload.
|
||||
|
||||
If you only need CPU and memory time series for the workload, you don't need to deploy a ServiceMonitor or PodMonitor because the monitoring application already collects metrics data on resource usage by default. The resource usage time series data is in Prometheus's local time series database. Grafana shows the data in aggregate, but you can see the data for the individual workload by using a PromQL query that extracts the data for that workload. Once you have the PromQL query, you can execute the query individually in the Prometheus UI and see the time series visualized there, or you can use the query to customize a Grafana dashboard to display the workload metrics. For examples of PromQL queries for workload metrics, see [this section.](https://rancher.com/docs/rancher/v2.5/en/monitoring-alerting/configuration/expression/#workload-metrics)
|
||||
Grafana shows the data in aggregate, but you can see the data for the individual workload by using a PromQL query that extracts the data for that workload. Once you have the PromQL query, you can execute the query individually in the Prometheus UI and see the time series visualized there, or you can use the query to customize a Grafana dashboard to display the workload metrics. For examples of PromQL queries for workload metrics, see [this section.](https://rancher.com/docs/rancher/v2.5/en/monitoring-alerting/configuration/expression/#workload-metrics)
|
||||
|
||||
To set up custom metrics for your workload, you will need to set up an exporter and create a new ServiceMonitor custom resource to configure Prometheus to scrape metrics from your exporter.
|
||||
|
||||
For more information, see [this section.](./monitoring-workloads)
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
explain how some applications come with a servicemonitor packaged within them
|
||||
|
||||
for example, some rancher applications come with servicemonitors (link to section)
|
||||
|
||||
### Display CPU and Memory Metrics for a Workload
|
||||
|
||||
By default, the monitoring application already scrapes CPU and memory.
|
||||
|
||||
To get some fine-grained detail for a particular workload, you can customize a Grafana dashboard to display the metrics for a particular workload.
|
||||
|
||||
- there’s already a wealth of information provided by kube-state-metrics. Cpu utilization, memory utilization for different things across namespaces. If you just want resource metrics for prod, you don’t need to create a new ServiceMonitor for it. All you need to do is go to the prometheus UI and do a PromQL query to get the information.
|
||||
|
||||
For more information on customizing Grafana to show the workload metrics, see this section. (Link)
|
||||
|
||||
|
||||
### Setting up Metrics Beyond CPU and Memory
|
||||
|
||||
For custom metrics, you will need to expose the metrics on your application in a format supported by Prometheus.
|
||||
@@ -45,12 +29,3 @@ For custom metrics, you will need to expose the metrics on your application in a
|
||||
Then we recommend that you should create a new ServiceMonitor custom resource. When this resource is created, the Prometheus custom resource will be automatically updated so that its scrape configuration includes the new custom metrics endpoint. Then Prometheus will begin scraping metrics from the endpoint.
|
||||
|
||||
You can also create a PodMonitor to expose the custom metrics endpoint, but ServiceMonitors are more appropriate for the majority of use cases.
|
||||
|
||||
- let’s say we expose metrics at a particular endpoint. Let’s take rancher-monitoring-kube-state-metrics. For example they have a container port where they expose metrics from.
|
||||
- the approach I would take - although we don’t have a clean UI from it - is to create it from YAML.
|
||||
- for something like for grafana we’d create it like this - like for rancher-monitoring-grafana - where the basic details we need to provide are:
|
||||
- what is the actual endpoint that you want to hit (spec.endpoints, path and port) - what’s the HTTP path that you want to hit and what’s the port.
|
||||
- namespaceSelector: what namespaces does that particular deployment exist in within Kubernetes, and use matchNames to select them.
|
||||
- you can also use selector.matchLabels.
|
||||
- That’s what it takes to add monitoring if a serviceMonitor is not already defined.
|
||||
- example: use the rancher-monitoring-grafana YAML
|
||||
@@ -3,7 +3,7 @@ title: Pipelines
|
||||
weight: 10
|
||||
---
|
||||
|
||||
> As of Rancher v2.5, Git-based deployment pipelines are now deprecated. We recommend handling pipelines with Rancher Continuous Delivery powered by [Fleet.](https://fleet.rancher.io/) To get to Fleet in Rancher, click **☰ > Continuous Delivery**. Note that pipelines in Kubernetes 1.21+ are no longer supported.
|
||||
> As of Rancher v2.5, Git-based deployment pipelines are now deprecated. We recommend handling pipelines with Rancher Continuous Delivery powered by [Fleet.](https://fleet.rancher.io/) To get to Fleet in Rancher, click <b>☰ > Continuous Delivery</b>. Note that pipelines in Kubernetes 1.21+ are no longer supported.
|
||||
|
||||
Rancher's pipeline provides a simple CI/CD experience. Use it to automatically checkout code, run builds or scripts, publish Docker images or catalog applications, and deploy the updated software to users.
|
||||
|
||||
@@ -17,7 +17,7 @@ After configuring Rancher and GitHub, you can deploy containers running Jenkins
|
||||
- Run unit tests.
|
||||
- Run regression tests.
|
||||
|
||||
>**Note: Rancher's pipeline provides a simple CI/CD experience, but it does not offer the full power and flexibility of and is not a replacement of enterprise-grade Jenkins or other CI tools your team uses.
|
||||
>**Note:** Rancher's pipeline provides a simple CI/CD experience, but it does not offer the full power and flexibility of and is not a replacement of enterprise-grade Jenkins or other CI tools your team uses.
|
||||
|
||||
This section covers the following topics:
|
||||
|
||||
|
||||
@@ -32,12 +32,12 @@
|
||||
{{end}}
|
||||
{{ if in .Dir "/pipelines" }}
|
||||
<div class="alert alert-notice">
|
||||
<strong>As of Rancher v2.5, Git-based deployment pipelines are now recommended to be handled with Rancher Continuous Delivery powered by <a href="https://rancher.com/docs/rancher/v2.5/en/deploy-across-clusters/fleet/">Fleet</a>, available in Cluster Explorer.</strong>
|
||||
<strong>As of Rancher v2.5, Git-based deployment pipelines are now recommended to be handled with Rancher Continuous Delivery powered by <a href="https://rancher.com/docs/rancher/v2.5/en/deploy-across-clusters/fleet/">Fleet</a>, available in the Rancher UI by clicking <b>☰ > Continuous Delivery</b>.</strong>
|
||||
</div>
|
||||
{{end}}
|
||||
{{ if in .Dir "/multi-cluster-apps" }}
|
||||
<div class="alert alert-notice">
|
||||
<strong>As of Rancher v2.5, multi-cluster apps are deprecated.</strong> We now recommend using <a href="https://rancher.com/docs/rancher/v2.5/en/deploy-across-clusters/fleet/">Fleet</a> for deploying apps across clusters. It is available in Cluster Explorer.
|
||||
<strong>As of Rancher v2.5, multi-cluster apps are deprecated.</strong> We now recommend using <a href="https://rancher.com/docs/rancher/v2.5/en/deploy-across-clusters/fleet/">Fleet</a> for deploying apps across clusters. It is available in the Rancher UI by clicking <b>☰ > Continuous Delivery</b>.
|
||||
</div>
|
||||
{{end}}
|
||||
{{ if in .Dir "os/v1.x" }}
|
||||
|
||||
File diff suppressed because one or more lines are too long
|
Before Width: | Height: | Size: 42 KiB After Width: | Height: | Size: 42 KiB |
Reference in New Issue
Block a user