mirror of
https://github.com/rancher/rancher-docs.git
synced 2026-09-26 04:58:07 +00:00
Resolve merge conflict
This commit is contained in:
Catherine Luse
@@ -11,84 +11,174 @@ aliases:
|
||||
|
||||
Using Rancher, you can monitor the state and processes of your cluster nodes, Kubernetes components, and software deployments through integration with [Prometheus](https://prometheus.io/), a leading open-source monitoring solution.
|
||||
|
||||
This page describes how to enable monitoring for a cluster.
|
||||
[use Arvind's wording for explaining kube-prometheus stack]
|
||||
|
||||
This section covers the following topics:
|
||||
Prometheus collects metrics from the cluster components, which you can view in graphs and charts.
|
||||
|
||||
- [Changes in Rancher v2.5](#changes-in-rancher-v2-5)
|
||||
- [About Prometheus](#about-prometheus)
|
||||
- [Monitoring scope](#monitoring-scope)
|
||||
- [Enabling cluster monitoring](#enabling-cluster-monitoring)
|
||||
- [Configuration](#configuration)
|
||||
- [Examples](#examples)
|
||||
- [Create ServiceMonitor Custom Resource](#create-servicemonitor-custom-resource)
|
||||
- [PodMonitor](#podmonitor)
|
||||
- [PrometheusRule](#prometheusrule)
|
||||
- [Alertmanager Config](#alertmanager-config)
|
||||
- [Configuring a Persistent Grafana Dashboard](#configuring-a-persistent-grafana-dashboard)
|
||||
- [Configuring Grafana to Use Multiple Data Sources](#configuring-grafana-to-use-multiple-data-sources)
|
||||
This page describes how to enable monitoring for a cluster using Rancher's new monitoring application, which was introduced in Rancher v2.5. For the legacy docs about monitoring, refer to [this section.](../legacy)
|
||||
|
||||
> Before enabling monitoring, be sure to review the [resource requirements.](#resource-requirements)
|
||||
|
||||
# Changes in Rancher v2.5
|
||||
- [Changes in Rancher v2.5 and Migrating to Monitoring v2](#changes-in-rancher-v2-5-and-migrating-to-monitoring-v2)
|
||||
- [Differences between `rancher-monitoring` and Upstream Prometheus](#differences-between-rancher-monitoring-and-upstream-prometheus)
|
||||
- [Changes to Role-based Access Control](#changes-to-role-based-access-control)
|
||||
- [Default Alerts, Targets and Grafana Dashboards](#default-alerts-targets-and-grafana-dashboards)
|
||||
- [Enable Monitoring](#enable-monitoring)
|
||||
- [Uninstall Monitoring](#uninstall-monitoring)
|
||||
- [Resource Requirements](#resource-requirements)
|
||||
- [Configuration Reference](#configuration-reference)
|
||||
- [Dashboards](#dashboards)
|
||||
- [Grafana UI](#grafana-ui)
|
||||
- [Prometheus UI](#prometheus-ui)
|
||||
- [Viewing the Prometheus Targets](#viewing-the-prometheus-targets)
|
||||
- [Viewing the Prometheus Rules](#viewing-the-prometheus-rules)
|
||||
- [Viewing Active Alerts in Alertmanager](#viewing-active-alerts-in-alertmanager)
|
||||
- [Prometheus Adapter](#prometheus-adapter)
|
||||
|
||||
Rancher's monitoring application is powered by the Prometheus operator, and it now relies less on Rancher's in-house monitoring tools.
|
||||
### Changes in Rancher v2.5 and Migrating to Monitoring v2
|
||||
|
||||
This change allows Rancher to automatically support new features of the Prometheus operator API. Now all of the features exposed by the upstream Prometheus operator are available in the monitoring application, and you have more flexibility to configure monitoring.
|
||||
If you previously enabled monitoring in Rancher prior to v2.5, there is no upgrade path for the monitoring application. You will need to disable monitoring and re-enable monitoring in Rancher.
|
||||
|
||||
Previously, you would use the Rancher UI to configure monitoring. The Rancher UI created CRDs that were maintained by Rancher and updated the Prometheus state. In Rancher v2.5, you directly create CRDs for the monitoring application, and those CRDs are exposed in the Rancher UI.
|
||||
For a list of changes in the new monitoring application, refer to this [page.](../migrating)
|
||||
|
||||
The differences between Rancher's monitoring feature and the upstream Prometheus operator can be found in the [changelog.](https://github.com/rancher/charts/blob/dev-v2.5/packages/rancher-monitoring/overlay/CHANGELOG.md)
|
||||
### Differences between `rancher-monitoring` and Upstream Prometheus
|
||||
|
||||
# About Prometheus
|
||||
In general, any feature supported by the upstream [Prometheus Operator Helm chart ](https://github.com/helm/charts/tree/master/stable/prometheus-operator) should be supported in the corresponding version Helm chart for Rancher's monitoring application.
|
||||
|
||||
Prometheus provides a _time series_ of your data, which is, according to [Prometheus documentation](https://prometheus.io/docs/concepts/data_model/):
|
||||
The deviations from the upstream Prometheus Operator Helm chart are recorded in the [CHANGELOG.md file](https://github.com/rancher/charts/blob/dev-v2.5/charts/rancher-monitoring/CHANGELOG.md) in the Helm chart of Rancher's monitoring application.
|
||||
|
||||
>A stream of timestamped values belonging to the same metric and the same set of labeled dimensions, along with comprehensive statistics and metrics of the monitored cluster.
|
||||
### Changes to Role-based Access Control
|
||||
|
||||
In other words, Prometheus lets you view metrics from your different Rancher and Kubernetes objects. Using timestamps, Prometheus lets you query and view these metrics in easy-to-read graphs and visuals, either through the Rancher UI or [Grafana](https://grafana.com/), which is an analytics viewing platform deployed along with Prometheus.
|
||||
Project owners and members no longer get access to Grafana or Prometheus by default. If view-only users had access to Grafana, they would be able to see data from any namespace. For Kiali, any user can edit things they don’t own in any namespace.
|
||||
|
||||
By viewing data that Prometheus scrapes from your cluster control plane, nodes, and deployments, you can stay on top of everything happening in your cluster. You can then use these analytics to better run your organization: stop system emergencies before they start, develop maintenance strategies, restore crashed servers, etc.
|
||||
For more information about role-based access control in `rancher-monitoring`, refer to [this page.](./rbac)
|
||||
|
||||
# Enabling Cluster Monitoring
|
||||
|
||||
As an [administrator]({{<baseurl>}}/rancher/v2.x/en/admin-settings/rbac/global-permissions/) or [cluster owner]({{<baseurl>}}/rancher/v2.x/en/admin-settings/rbac/cluster-project-roles/#cluster-roles), you can configure Rancher to deploy Prometheus to monitor your Kubernetes cluster.
|
||||
|
||||
> **Prerequisite:** Make sure that you are allowing traffic on port 9796 for each of your nodes because Prometheus will scrape metrics from here.
|
||||
> If you want to set up Alertmanager, Grafana or Ingress, it has to be done with the settings on the Helm chart deployment. It's problematic to create Ingress outside the deployment.
|
||||
|
||||
> The default username and password for the Grafana instance will be `admin/admin`. However, Grafana dashboards are served via the Rancher authentication proxy, so only users who are currently authenticated into the Rancher server have access to the Grafana dashboard.
|
||||
> **Prerequisites:**
|
||||
>
|
||||
> - Make sure that you are allowing traffic on port 9796 for each of your nodes because Prometheus will scrape metrics from here.
|
||||
> - Make sure your cluster fulfills the resource requirements. The cluster should have at least 1950Mi memory available, 2700m CPU, and 50Gi storage. A breakdown of the resource limits and requests is [here.](#resource-requirements)
|
||||
|
||||
# Configuration
|
||||
1. In the Rancher UI, go to the cluster where you want to install monitoring and click **Cluster Explorer.**
|
||||
1. Click **Apps.**
|
||||
1. Click the `rancher-monitoring` app.
|
||||
1. Optional: Click **Chart Options** and configure alerting, Prometheus and Grafana. For help, refer to the [configuration reference.](#../configuration)
|
||||
1. Scroll to the bottom of the Helm chart README and click **Install.**
|
||||
|
||||
For information on configuring custom Prometheus metrics and alerting rules, refer to the upstream documentation for the [Prometheus operator.](https://github.com/prometheus-operator/prometheus-operator) This documentation can help you set up RBAC, Thanos, or custom configuration.
|
||||
**Result:** The monitoring app is deployed in the `cattle-monitoring-system` namespace.
|
||||
|
||||
To create an additional scrape configuration, refer to [this page.](https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/additional-scrape-config.md)
|
||||
### Next Steps
|
||||
|
||||
# Examples
|
||||
To configure Prometheus resources from the Rancher UI, click **Apps & Marketplace > Monitoring** in the upper left corner.
|
||||
|
||||
### Create ServiceMonitor Custom Resource
|
||||
# Uninstall Monitoring
|
||||
|
||||
An example ServiceMonitor custom resource can be found [here.](https://github.com/prometheus-operator/prometheus-operator/blob/master/example/prometheus-operator-crd/monitoring.coreos.com_servicemonitors.yaml)
|
||||
1. From the **Cluster Explorer,** click Apps & Marketplace.
|
||||
1. Click **Installed Apps.**
|
||||
1. Go to the `cattle-monitoring-system` namespace and check the boxes for `rancher-monitoring-crd` and `rancher-monitoring`.
|
||||
1. Click **Delete.**
|
||||
1. Confirm **Delete.**
|
||||
|
||||
### PodMonitor
|
||||
**Result:** `rancher-monitoring` is uninstalled.
|
||||
|
||||
An example PodMonitor can be found [here.](https://github.com/prometheus-operator/prometheus-operator/blob/master/example/user-guides/getting-started/example-app-pod-monitor.yaml) and an example Prometheus resource that refers to it can be found [here.](https://github.com/prometheus-operator/prometheus-operator/blob/master/example/user-guides/getting-started/prometheus-pod-monitor.yaml)
|
||||
# Resource Requirements
|
||||
|
||||
### PrometheusRule
|
||||
The resource requests and limits can be configured when installing `rancher-monitoring`.
|
||||
|
||||
Prometheus rule files are held in PrometheusRule custom resources. Use the label selector field ruleSelector in the Prometheus object to define the rule files that you want to be mounted into Prometheus. An example PrometheusRule is on [this page.](https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/user-guides/alerting.md)
|
||||
The default values are in the `values.yaml` in the `rancher-monitoring` Helm chart.
|
||||
|
||||
### Alertmanager Config
|
||||
There is a [known issue](https://github.com/rancher/rancher/issues/28787#issuecomment-693611821) that K3s clusters require more default memory. If you are enabling monitoring on a K3s cluster, we recommend to setting `prometheus.prometheusSpec.resources.memory.limit` to 2500Mi` and `prometheus.prometheusSpec.resources.memory.request` to 1750Mi.
|
||||
|
||||
The Prometheus Operator introduces an Alertmanager resource, which allows users to declaratively describe an Alertmanager cluster.
|
||||
| Resource Name | Memory Limit | CPU Limit | Memory Request | CPU Request |
|
||||
| ------------- | ------------ | ----------- | ---------------- | ------------------ |
|
||||
| alertmanager | 500Mi | 1000m | 100Mi | 100m |
|
||||
| grafana | 200Mi | 200m | 100Mi | 100m |
|
||||
| kube-state-metrics subchart | 200Mi | 100m | 130Mi | 100m |
|
||||
| prometheus-node-exporter subchart | 50Mi | 200m | 30Mi | 100m |
|
||||
| prometheusOperator | 500Mi | 200m | 100Mi | 100m |
|
||||
| prometheus | 500Mi | 1000m | 100Mi | 100m |
|
||||
| **Total** | **1950Mi** | **2700m** | **560Mi** | **600m** |
|
||||
|
||||
The upstream Prometheus documentation includes information on how to [set up](https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/user-guides/alerting.md) and [configure](https://prometheus.io/docs/alerting/latest/configuration/) Alertmanager.
|
||||
At least 50Gi storage is recommended.
|
||||
|
||||
### Configuring a Persistent Grafana Dashboard
|
||||
# Configuration Reference
|
||||
|
||||
To allow the Grafana dashboard to persist after it restarts, you will need to add the configuration JSON into a ConfigMap.
|
||||
For the configuration reference and examples, refer to [this page.](../configuration)
|
||||
|
||||
You can add this configuration to the ConfigMap using the Rancher UI.
|
||||
# Dashboards
|
||||
|
||||
### Configuring Grafana to Use Multiple Data Sources
|
||||
Installing `rancher-monitoring` makes the following dashboards available from the Rancher UI.
|
||||
|
||||
### Grafana UI
|
||||
|
||||
The default username and password for the Grafana instance will be `admin`/`prom-operator`. However, Grafana dashboards are served via the Rancher authentication proxy, so only users who are currently authenticated into the Rancher server have access to the Grafana dashboard. For information about the default permissions, refer to the [roles-based access control section.](./rbac)
|
||||
|
||||
To see the Grafana UI, install `rancher-monitoring`. Then go to the **Cluster Explorer.** In the top left corner, click **Cluster Explorer > Monitoring.** Then click **Grafana.
|
||||
|
||||
- How to view the currently deployed dashboard
|
||||
- how to create a custom dashboard on the Grafana UI
|
||||
- Configuring a persistent Grafana dashboard - will be built on the second, once you have a JSON dashboard, how do you copy it to a configmap
|
||||
- configuring Grafana to use multiple data sources - might not need this.
|
||||
|
||||
<figcaption>Cluster Compute Resources Dashboard in Grafana</figcaption>
|
||||
|
||||
|
||||
<figcaption>Default Dashboards in Grafana</figcaption>
|
||||
|
||||
|
||||
To allow the Grafana dashboard to persist after it restarts, you will need to add the configuration JSON into a ConfigMap. You can add this configuration to the ConfigMap using the Rancher UI.
|
||||
|
||||
### Prometheus UI
|
||||
|
||||
To see the Prometheus UI, install `rancher-monitoring`. Then go to the **Cluster Explorer.** In the top left corner, click **Cluster Explorer > Monitoring.** Then click **Prometheus Graph.**
|
||||
|
||||
<figcaption>Prometheus Graph UI</figcaption>
|
||||
|
||||
|
||||
### Viewing the Prometheus Targets
|
||||
|
||||
To see the Prometheus Targets, install `rancher-monitoring`. Then go to the **Cluster Explorer.** In the top left corner, click **Cluster Explorer > Monitoring.** Then click **Prometheus Targets.**
|
||||
|
||||
<figcaption>Targets in the Prometheus UI</figcaption>
|
||||
|
||||
|
||||
### Viewing the Prometheus Rules
|
||||
|
||||
To see the Prometheus Rules, install `rancher-monitoring`. Then go to the **Cluster Explorer.** In the top left corner, click **Cluster Explorer > Monitoring.** Then click **Prometheus Rules.**
|
||||
|
||||
<figcaption>Rules in the Prometheus UI</figcaption>
|
||||
|
||||
|
||||
### Viewing Active Alerts in Alertmanager
|
||||
|
||||
When `rancher-monitoring` is installed, the Prometheus Alertmanager UI is deployed.
|
||||
|
||||
The Alertmanager handles alerts sent by client applications such as the Prometheus server. It takes care of deduplicating, grouping, and routing them to the correct receiver integration such as email, PagerDuty, or OpsGenie. It also takes care of silencing and inhibition of alerts.
|
||||
|
||||
In the Alertmanager UI, you can view your alerts and the current Alertmanager configuration.
|
||||
|
||||
To see the Prometheus Rules, install `rancher-monitoring`. Then go to the **Cluster Explorer.** In the top left corner, click **Cluster Explorer > Monitoring.** Then click **Alertmanager.**
|
||||
|
||||
**Result:** The Alertmanager UI opens in a new tab. For help with configuration, refer to the [official Alertmanager documentation.](https://prometheus.io/docs/alerting/latest/alertmanager/)
|
||||
|
||||
<figcaption>The Alertmanager UI</figcaption>
|
||||
|
||||
|
||||
# Prometheus Adapter
|
||||
|
||||
The [Prometheus adapter](https://github.com/prometheus-community/helm-charts/tree/main/charts/prometheus-adapter) is useful if you want to expose the metrics collected by `rancher-monitoring` to the custom metrics API for Kubernetes. A common use case is to use the custom metrics for the horizontal pod autoscaler.
|
||||
|
||||
The Prometheus adapter allows you to provide a secret to determine what metrics from Prometheus get sent to the custom metrics API.
|
||||
|
||||
You can provide configuration via Helm for the Prometheus adapter.
|
||||
|
||||
<<<<<<< HEAD
|
||||
The data from Prometheus is used as the data source for the Grafana dashboard. Multiple data sources can be configured for Grafana.
|
||||
=======
|
||||
For more information about using the Promethus adapter, refer to this [documentation.](https://github.com/DirectXMan12/k8s-prometheus-adapter/blob/master/docs/config-walkthrough.md)
|
||||
>>>>>>> Update monitoring docs
|
||||
|
||||
@@ -0,0 +1,192 @@
|
||||
---
|
||||
title: Configuration
|
||||
weight: 3
|
||||
---
|
||||
|
||||
This page captures some of the most important options for configuring the custom resources for monitoring.
|
||||
|
||||
For information on configuring custom Prometheus metrics and alerting rules, refer to the upstream documentation for the [Prometheus operator.](https://github.com/prometheus-operator/prometheus-operator) Some of the most important custom resources are explained in the Prometheus [design documentation.](https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/design.md) The Prometheus documentation can help also you set up RBAC, Thanos, or custom configuration.
|
||||
|
||||
- [About Prometheus Configuration](#about-prometheus-configuration)
|
||||
- [Configuring Targets with ServiceMonitors and PodMonitors](#configuring-targets-with-servicemonitors-and-podmonitors)
|
||||
- [ServiceMonitors](#servicemonitors)
|
||||
- [PodMonitors](#podmonitors)
|
||||
- [Configuring Alerts or Recording Rules with PrometheusRules](#configuring-alerts-or-recording-rules-with-prometheusrules)
|
||||
- [PrometheusRules](#prometheusrules)
|
||||
- [Recording Rules](#recording-rules)
|
||||
- [Alertmanager Config](#alertmanager-config)
|
||||
- [Trusted CA for Notifiers](#trusted-ca-for-notifiers)
|
||||
- [Additional Scrape Configurations](#additional-scrape-configurations)
|
||||
- [Examples](#examples)
|
||||
|
||||
# About Prometheus Configuration
|
||||
|
||||
The primary way that users will be able to customize this feature for specific Monitoring and Alerting use cases is by creating and/or modifying ConfigMaps, Secrets, and Custom Resources pertaining to this deployment.
|
||||
|
||||
Prometheus Operator introduces a set of [Custom Resource Definitions](https://github.com/prometheus-operator/prometheus-operator#customresourcedefinitions) that allow users to deploy and manage Prometheus and Alertmanager instances by creating and modifying those custom resources on a cluster.
|
||||
|
||||
Prometheus Operator will automatically update your Prometheus configuration based on the live state of these custom resources.
|
||||
|
||||
There are also certain special types of ConfigMaps/Secrets such as those corresponding to Grafana Dashboards, Grafana Datasources, and Alertmanager Configs that will automatically update your Prometheus configuration via sidecar proxies that observe the live state of those resources within your cluster.
|
||||
|
||||
By default, a set of these resources is created within your cluster as part of deploying this chart to set up a basic Monitoring/Alerting stack. We use the ones generated by [kube-prometheus.](https://github.com/prometheus-operator/kube-prometheus) For more information how to configure custom targets, alerts, notifiers, and dashboards after deploying the chart, see below.
|
||||
|
||||
# Configuring Targets with ServiceMonitors and PodMonitors
|
||||
|
||||
Customizing the scrape configuration used by Prometheus to determine which resources to scrape metrics from will primarily involve creating / modifying the following resources within your cluster:
|
||||
|
||||
### ServiceMonitors
|
||||
|
||||
This CRD declaratively specifies how groups of Kubernetes services should be monitored. Any Services in your cluster that match the labels located within the ServiceMonitor selector field will be monitored based on the endpoints specified on the ServiceMonitor. For more information on what fields can be specified, please look at the [spec](https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/api.md#servicemonitor) provided by Prometheus Operator.
|
||||
|
||||
For more information about how ServiceMonitors work, refer to the [Prometheus documentation.](https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/user-guides/running-exporters.md)
|
||||
|
||||
### PodMonitors
|
||||
|
||||
This CRD declaratively specifies how group of pods should be monitored. Any Pods in your cluster that match the labels located within the PodMonitor selector field will be monitored based on the podMetricsEndpoints specified on the PodMonitor. For more information on what fields can be specified, please look at the [spec](https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/api.md#podmonitorspec) provided by Prometheus Operator.
|
||||
|
||||
# Configuring Alerts or Recording Rules with PrometheusRules
|
||||
|
||||
Customizing the alerts recorded by Prometheus and the configuration of notifiers that should notify users when alerts are triggered will primarily involve creating/modifying PrometheusRules and the Alertmanager config.
|
||||
|
||||
The Alertmanager needs to be a Secret that exists in the `cattle-monitoring` system namespace in order for that to be used.
|
||||
|
||||
### PrometheusRules
|
||||
|
||||
This CRD defines a desired set of Prometheus alerting and/or recording rules.
|
||||
|
||||
Creating the desired alerting rule will involve creating or updating a RuleGroup with your desired alerting rule. Each alerting rule needs to:
|
||||
|
||||
- Name the alert
|
||||
- Provide a PromQL expression for the alert
|
||||
- Provide any labels or annotations that need to be sent to Alertmanager on the alert
|
||||
|
||||
Labels are concise values that identify the alert (e.g. cluster name or severity), while Annotations contain other important pieces of information that need to be displayed when notifying users about an alert (e.g. summary, description, message, runbook URL, etc.).
|
||||
|
||||
For more information on what fields can be specified, please look at the [spec.](https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/api.md#prometheusrulespec)
|
||||
|
||||
### Recording Rules
|
||||
|
||||
If you would like to add [recording rules](https://prometheus.io/docs/prometheus/latest/configuration/recording_rules/) to define a frequently needed or computationally intensive PromQL query that should be stored as a new set of time series, you should create a PrometheusRule.
|
||||
|
||||
Creating the desired recording rule will involve creating or updating a RuleGroup with your desired recording rule.
|
||||
|
||||
Each recording rule needs to:
|
||||
|
||||
- Name the new time series
|
||||
- Provide a PromQL expression for the series
|
||||
- Provide any labels that need to be added to this series
|
||||
|
||||
For more information on what fields can be specified, please look at [this page.](https://prometheus.io/docs/prometheus/latest/configuration/recording_rules/#rule)
|
||||
|
||||
### Alertmanager Config
|
||||
|
||||
The [Alertmanager Config](https://prometheus.io/docs/alerting/latest/configuration/#configuration-file) Secret contains the configuration of an Alertmanager instance that sends out notifications based on alerts it receives from Prometheus.
|
||||
|
||||
There only needs to be one Alertmanager in the cluster, and when `rancher-monitor` is installed, a default Alertmanager config is created.
|
||||
|
||||
This Secret should be updated or modified any time you want to:
|
||||
|
||||
- Add in new notifiers or receivers
|
||||
- Change the alerts that should be sent to specific notifiers or receivers
|
||||
- Change the group of alerts that are sent out
|
||||
|
||||
> Normally, the Alertmanager config secret is configured as a resource that's not managed by Helm. For the `rancher-monitoring` chart, the default case will leave that secret behind. You should save the Secret so that if you uninstall and reinstall `rancher-monitoring`, you wouldn't lose your alerting configuration.
|
||||
|
||||
For more information on what fields can be specified in this secret, please look at the [Prometheus Alertmanager docs](https://prometheus.io/docs/alerting/latest/alertmanager/)
|
||||
|
||||
The full spec for the Alertmanager configuration file and what it takes in can be found [here.](https://prometheus.io/docs/alerting/latest/configuration/#configuration-file)
|
||||
|
||||
It is recommended to use an Existing Secret within the namespace `cattle-monitoring-system` for the Alertmanager Config in order to avoid having to to manage the Secret via Helm.
|
||||
|
||||
Using the default Helm managed Secret will require a user to redeploy the chart in order to persist any changes to the Alertmanager configuration on a `helm upgrade`. Using an existing Secret will allow the secret to be directly modified to perform live updates on the Alertmanager configuration if necessary.
|
||||
|
||||
If using an Existing Secret, please add the label `alertmanager_config` to the Secret to enable modifying the secret using the Rancher Dashboard UI.
|
||||
|
||||
Each [Receiver](https://prometheus.io/docs/alerting/latest/configuration/#receiver) (an Alertmanager entity that describes the notification configuration of one or more providers) must define the notification configuration of exactly one provider (i.e. Slack, PagerDuty, OpsGenie, etc.).
|
||||
|
||||
In the Alertmanager documentation about routes, it is said that there must be one "configured top-level route, which must match all alerts (i.e. not have any configured matchers)." On this page, we refer to the notifier using that route as the master notifier. The other notifiers that are referenced in child routes (usually a list provided in `.routes`) we refer to as notifiers.
|
||||
|
||||
For more information, refer to the [official Prometheus documentation about configuring routes.](https://www.prometheus.io/docs/alerting/latest/configuration/#route)
|
||||
|
||||
# Trusted CA for Notifiers
|
||||
|
||||
If you need to add a trusted CA to your notifier, follow these steps:
|
||||
|
||||
1. Create the `cattle-monitoring-system` namespace.
|
||||
1. Add your trusted CA secret to the `cattle-monitoring-system` namespace.
|
||||
1. Deploy or upgrade the `rancher-monitoring` Helm chart. In the chart options, reference the secret in **Alerting > Additional Secrets.**
|
||||
|
||||
**Result:** The default Alertmanager custom resource will have access to your trusted CA.
|
||||
|
||||
# Additional Scrape Configurations
|
||||
|
||||
If the scrape configuration you want cannot be specified via a ServiceMonitor or PodMonitor at the moment, you can provide an additionalScrapeConfigSecret on deploying or upgrading `rancher-monitoring`.
|
||||
|
||||
A [scrape_config section](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config) specifies a set of targets and parameters describing how to scrape them. In the general case, one scrape configuration specifies a single job.
|
||||
|
||||
If you want to enable Istio in the cluster, you might need to use an additional scrape configuration to ensure you can view traffic, metrics and graphs for resources deployed in other namespaces than only the `istio-system`.
|
||||
|
||||
To create an additional scrape configuration, refer to [this page.](https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/additional-scrape-config.md)
|
||||
|
||||
|
||||
# Examples
|
||||
|
||||
### ServiceMonitor
|
||||
|
||||
An example ServiceMonitor custom resource can be found [here.](https://github.com/prometheus-operator/prometheus-operator/blob/master/example/prometheus-operator-crd/monitoring.coreos.com_servicemonitors.yaml)
|
||||
|
||||
### PodMonitor
|
||||
|
||||
An example PodMonitor can be found [here.](https://github.com/prometheus-operator/prometheus-operator/blob/master/example/user-guides/getting-started/example-app-pod-monitor.yaml) An example Prometheus resource that refers to it can be found [here.](https://github.com/prometheus-operator/prometheus-operator/blob/master/example/user-guides/getting-started/prometheus-pod-monitor.yaml)
|
||||
|
||||
### PrometheusRule
|
||||
|
||||
Prometheus rule files are held in PrometheusRule custom resources. Use the label selector field ruleSelector in the Prometheus object to define the rule files that you want to be mounted into Prometheus. An example PrometheusRule is on [this page.](https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/user-guides/alerting.md)
|
||||
|
||||
### Alertmanager Config
|
||||
|
||||
The following values should generate the Alertmanager Config YAML that will be stored under the key `alertmanager.yaml`. This is an example of a Config contained in this key that represents one master Slack notifier with one master Route and no other notifiers configured.
|
||||
|
||||
```yaml
|
||||
route:
|
||||
group_by: ['job']
|
||||
group_wait: 30s
|
||||
group_interval: 5m
|
||||
repeat_interval: 3h
|
||||
receiver: 'slack-notifications'
|
||||
receivers:
|
||||
- name: 'slack-notifications'
|
||||
slack_configs:
|
||||
- send_resolved: true
|
||||
text: '{{ template "slack.rancher.text" . }}'
|
||||
api_url: <user-provided slack webhook url here>
|
||||
templates:
|
||||
- /etc/alertmanager/config/*.tmpl
|
||||
```
|
||||
|
||||
The following is the default Alertmanager config secret deployed by the `rancher-monitoring chart`. This example shows one master Custom Receiver with one Master Route, one other Route, and no other notifiers:
|
||||
|
||||
```yaml
|
||||
global:
|
||||
resolve_timeout: 5m
|
||||
route:
|
||||
group_by: ['job']
|
||||
group_wait: 30s
|
||||
group_interval: 5m
|
||||
repeat_interval: 12h
|
||||
receiver: 'null'
|
||||
routes:
|
||||
- group_by: ['job']
|
||||
group_wait: 30s
|
||||
group_interval: 5m
|
||||
repeat_interval: 12h
|
||||
match:
|
||||
alertname: Watchdog
|
||||
receiver: 'slack'
|
||||
receivers:
|
||||
- name: 'null'
|
||||
- name: 'slack'
|
||||
templates:
|
||||
- /etc/alertmanager/config/*.tmpl
|
||||
```
|
||||
@@ -0,0 +1,34 @@
|
||||
---
|
||||
title: Migrating to Rancher v2.5 Monitoring
|
||||
weight: 5
|
||||
---
|
||||
|
||||
If you previously enabled monitoring in Rancher prior to v2.5, there is no upgrade path for the monitoring application. You will need to disable monitoring and re-enable monitoring in Rancher.
|
||||
|
||||
### Major Changes
|
||||
|
||||
The new version of Rancher's monitoring application is powered by the Prometheus operator, and it now relies less on Rancher's in-house monitoring tools.
|
||||
|
||||
This change allows Rancher to automatically support new features of the Prometheus operator API. Now all of the features exposed by the upstream Prometheus operator are available in the monitoring application, and you have more flexibility to configure monitoring.
|
||||
|
||||
Previously, you would use the Rancher UI to configure monitoring. The Rancher UI created CRDs that were maintained by Rancher and updated the Prometheus state. In Rancher v2.5, you directly create CRDs for the monitoring application, and those CRDs are exposed in the Rancher UI.
|
||||
|
||||
Other important changes include:
|
||||
|
||||
- In the older version of monitoring, an Ingress was created outside of the Helm chart deployment, whereas in the new version, an Ingress is created as part of the Helm chart deployment.
|
||||
- The monitoring application has the ability to create Prometheus ServiceMonitors and PodMonitors.
|
||||
- We exposed a [PushProx](https://github.com/prometheus-community/PushProx) exporter, based on the Prometheus project called PushProx. This [`rancher-pushprox` chart](https://github.com/rancher/dev-charts/tree/master/packages/rancher-pushprox/charts) sets up a Deployment of a PushProx proxy and a DaemonSet of PushProx clients. It monitors internal Kubernetes components for K3s, RKE, and kubeAdm clusters.
|
||||
- One aspect of alerting moved into the monitoring application, specifically the ability to create Prometheus rules. Now alerts are created in monitoring, and you configure how you get notified about the alerts.
|
||||
- In Rancher v2.4, alerts were configured separately from notifiers. Both alerts and notifiers are configured in the alert configuration.
|
||||
- To create alerts, you will create Prometheus alert custom resources.
|
||||
- Instead of having a notifier UI where you can add notifiers, notifications should be configured within the Alertmanager secret exposed in the rancher UI. This will let you configure routes, which is something that couldn't be done with Rancher v2.4.
|
||||
- Ingresses need to be set up during the chart deployment or upgrade. You can configure an Ingress for Alertmanager, Prometheus and Grafana. To access each of those services from outside the cluster, they each would need separate Ingresses. Each Ingress would take in the same fields. If you want to use your own custom Ingress, we recommend setting the fields during the `rancher-monitoring` Helm chart deployment because it ensures the fields are as expected.
|
||||
- A [PushProx exporter](https://github.com/prometheus-community/PushProx) was exposed. PushProx exporters are created on the cluster where `rancher-monitoring` is installed. etcd, controlplane, proxy, and controller-manager metrics will all use the PushProx exporter.
|
||||
|
||||
A list of differences between Rancher's monitoring feature and the upstream Prometheus operator can be found in the [changelog.](https://github.com/rancher/charts/blob/dev-v2.5/packages/rancher-monitoring/overlay/CHANGELOG.md)
|
||||
|
||||
### Prometheus Operator Helm Chart Changes
|
||||
|
||||
Rancher used to use the [Prometheus operator.](https://github.com/prometheus-operator/prometheus-operator) Some parts of the Prometheus operator were included in the [kube-prometheus-stack.](https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack)
|
||||
|
||||
The operator that manages Prometheus is still called the Prometheus operator, but it is now deployed as part of the `kube-prometheus-stack` Helm chart.
|
||||
@@ -9,6 +9,11 @@ This section describes the permissions required to access Monitoring features.
|
||||
|
||||
The `rancher-monitoring` chart installs three `ClusterRoles`.
|
||||
|
||||
- [Cluster-Admin Access](#cluster-admin-access)
|
||||
- [Admin and Edit Access](#admin-and-edit-access)
|
||||
- [Summary of Default Monitoring Permissions for Kubernetes Default Roles](#summary-of-default-monitoring-permissions-for-kubernetes-default-roles)
|
||||
- [Additional Monitoring Roles](#additional-monitoring-roles)
|
||||
|
||||
# Cluster-Admin Access
|
||||
|
||||
By default, only those with the cluster-admin `ClusterRole` can:
|
||||
@@ -20,7 +25,7 @@ By default, only those with the cluster-admin `ClusterRole` can:
|
||||
- Persist new Grafana dashboards or datasources via creating ConfigMaps in the appropriate namespace
|
||||
- Expose certain Prometheus metrics to the k8s Custom Metrics API for HPA via a Secret in the `cattle-monitoring-system` namespace
|
||||
|
||||
## Admin and Edit access
|
||||
### Admin and Edit access
|
||||
|
||||
By default, only Admin and Edit roles can:
|
||||
|
||||
@@ -29,7 +34,7 @@ By default, only Admin and Edit roles can:
|
||||
- Modify the scrape configuration of Prometheus deployments via ServiceMonitor and PodMonitor CRs
|
||||
- Modify the alerting / recording rules of a Prometheus deployment via PrometheusRules CRs
|
||||
|
||||
# Summary of Default Permissions for Kubernetes Default Roles
|
||||
# Summary of Default Monitoring Permissions for Kubernetes Default Roles
|
||||
|
||||
Monitoring creates three `ClusterRoles` and adds Monitoring CRD access to the following default K8s `ClusterRoles`:
|
||||
|
||||
@@ -39,7 +44,7 @@ Monitoring creates three `ClusterRoles` and adds Monitoring CRD access to the fo
|
||||
| `monitoring-edit`| `edit` |
|
||||
| `monitoring-view` | `view `|
|
||||
|
||||
Rancher will continue to use cluster-owner, cluster-member, project-owner, project-member, etc as role names, but will utilize default roles to determine access. For each default K8s `ClusterRole` there are different Istio CRD permissions and K8s actions (Create (C), Get (G), List (L), Update (U), Patch (P), Delete(D), All (*)) that can be performed.
|
||||
Rancher will continue to use cluster-owner, cluster-member, project-owner, project-member, etc. as role names, but will utilize default roles to determine access. For each default K8s `ClusterRole` there are different Istio CRD permissions and Kubernetes actions (Create (C), Get (G), List (L), Update (U), Patch (P), Watch (W), Delete(D), All (*)) that can be performed.
|
||||
|
||||
|
||||
|CRDs | Admin | Edit | View |
|
||||
@@ -47,17 +52,17 @@ Rancher will continue to use cluster-owner, cluster-member, project-owner, proje
|
||||
| <ul><li>`monitoring.coreos.com`</li><ul><li>`prometheuses`</li><li>`alertmanagers`</li></ul></ul>| GLW | GLW | GLW|
|
||||
| <ul><li>`monitoring.coreos.com`</li><ul><li>`servicemonitors`</li><li>`podmonitors`</li><li>`prometheusrules`</li></ul></ul>| * | * | GLW|
|
||||
|
||||
# Additional Roles
|
||||
# Additional Monitoring Roles
|
||||
|
||||
Monitoring also creates six `Roles` to enable admins to assign more fine-grained access to monitoring within a cluster:
|
||||
|
||||
| Role created by chart | Purpose |
|
||||
| ------------------------------| ---------------------------|
|
||||
monitoring-config-admin | Allow admins to assign roles to users to be able to view / modify Secrets and ConfigMaps within the cattle-monitoring-system namespace. Modifying Secrets / ConfigMaps in this namespace could allow users to alter the cluster's Alertmanager configuration, Prometheus Adapter configuration, additional Grafana datasources, TLS secrets, etc. |
|
||||
monitoring-config-edit | Allow admins to assign roles to users to be able to view / modify Secrets and ConfigMaps within the cattle-monitoring-system namespace. Modifying Secrets / ConfigMaps in this namespace could allow users to alter the cluster's Alertmanager configuration, Prometheus Adapter configuration, additional Grafana datasources, TLS secrets, etc. |
|
||||
monitoring-config-view | Allow admins to assign roles to users to be able to view Secrets and ConfigMaps within the cattle-monitoring-system namespace. Viewing Secrets / ConfigMaps in this namespace could allow users to observe the cluster's Alertmanager configuration, Prometheus Adapter configuration, additional Grafana datasources, TLS secrets, etc. |
|
||||
monitoring-dashboard-admin | Allow admins to assign roles to users to be able to edit / view ConfigMaps within the cattle-dashboards namespace. ConfigMaps in this namespace will correspond to Grafana Dashboards that are persisted onto the cluster. |
|
||||
monitoring-dashboard-edit | Allow admins to assign roles to users to be able to edit / view ConfigMaps within the cattle-dashboards namespace. ConfigMaps in this namespace will correspond to Grafana Dashboards that are persisted onto the cluster. |
|
||||
monitoring-dashboard-view | Allow admins to assign roles to users to be able to view ConfigMaps within the cattle-dashboards namespace. ConfigMaps in this namespace will correspond to Grafana Dashboards that are persisted onto the cluster. |
|
||||
`monitoring-config-admin` | Allow admins to assign roles to users to be able to view/modify Secrets and ConfigMaps within the `cattle-monitoring-system` namespace. Modifying Secrets/ConfigMaps in this namespace could allow users to alter the cluster's Alertmanager configuration, Prometheus Adapter configuration, additional Grafana datasources, TLS secrets, etc. |
|
||||
`monitoring-config-edit` | Allow admins to assign roles to users to be able to view / modify Secrets and ConfigMaps within the cattle-monitoring-system namespace. Modifying Secrets/ConfigMaps in this namespace could allow users to alter the cluster's Alertmanager configuration, Prometheus Adapter configuration, additional Grafana datasources, TLS secrets, etc. |
|
||||
`monitoring-config-view` | Allow admins to assign roles to users to be able to view Secrets and ConfigMaps within the cattle-monitoring-system namespace. Viewing Secrets/ConfigMaps in this namespace could allow users to observe the cluster's Alertmanager configuration, Prometheus Adapter configuration, additional Grafana datasources, TLS secrets, etc. |
|
||||
`monitoring-dashboard-admin` | Allow admins to assign roles to users to be able to edit/view ConfigMaps within the cattle-dashboards namespace. ConfigMaps in this namespace will correspond to Grafana Dashboards that are persisted onto the cluster. |
|
||||
`monitoring-dashboard-edit` | Allow admins to assign roles to users to be able to edit / view ConfigMaps within the cattle-dashboards namespace. ConfigMaps in this namespace will correspond to Grafana Dashboards that are persisted onto the cluster. |
|
||||
`monitoring-dashboard-view` | Allow admins to assign roles to users to be able to view ConfigMaps within the cattle-dashboards namespace. ConfigMaps in this namespace will correspond to Grafana Dashboards that are persisted onto the cluster. |
|
||||
|
||||
These Roles are not assigned by default but will be created in the cluster.
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 108 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 310 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 222 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 128 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 292 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 355 KiB |
Reference in New Issue
Block a user