diff --git a/content/rancher/v2.x/en/monitoring-alerting/_index.md b/content/rancher/v2.x/en/monitoring-alerting/_index.md index 4a779f60c7d..20636221efb 100644 --- a/content/rancher/v2.x/en/monitoring-alerting/_index.md +++ b/content/rancher/v2.x/en/monitoring-alerting/_index.md @@ -4,11 +4,8 @@ shortTitle: Monitoring/Alerting description: Prometheus lets you view metrics from your different Rancher and Kubernetes objects. Learn about the scope of monitoring and how to enable cluster monitoring weight: 14 aliases: -<<<<<<< HEAD - /rancher/v2.x/en/dashboard/monitoring-alerting - /rancher/v2.x/en/dashboard/notifiers -======= ->>>>>>> Revise monitoring docs - /rancher/v2.x/en/cluster-admin/tools/monitoring/ --- @@ -34,7 +31,7 @@ For more information about upgrading the Monitoring app in Rancher 2.5, please r For the docs about monitoring for earlier Rancher versions, refer to [this section.](../legacy) -> Before enabling monitoring, be sure to review the [resource requirements.](#resource-requirements) +> Before enabling monitoring, be sure to review the resource requirements. The default values in [this section](#setting-resource-limits-and-requests) are the minimum required resource limits and requests. - [Monitoring Components](#monitoring-components) - [Prometheus](#about-prometheus) @@ -44,15 +41,15 @@ For the docs about monitoring for earlier Rancher versions, refer to [this secti - [Prometheus Adapter](#about-prometheus-adapter) - [Enable Monitoring](#enable-monitoring) - [Default Alerts, Targets and Grafana Dashboards](#default-alerts-targets-and-grafana-dashboards) -- [Uninstall Monitoring](#uninstall-monitoring) -- [Resource Requirements](#resource-requirements) -- [Configuration Reference](#configuration-reference) -- [Dashboards](#dashboards) +- [Using Monitoring](#using-monitoring) - [Grafana UI](#grafana-ui) - [Prometheus UI](#prometheus-ui) - [Viewing the Prometheus Targets](#viewing-the-prometheus-targets) - [Viewing the Prometheus Rules](#viewing-the-prometheus-rules) - [Viewing Active Alerts in Alertmanager](#viewing-active-alerts-in-alertmanager) +- [Uninstall Monitoring](#uninstall-monitoring) +- [Setting Resource Limits and Requests](#setting-resource-limits-and-requests) +- [Known Issues](#known-issues) # Monitoring Components @@ -83,17 +80,32 @@ As an [administrator]({{}}/rancher/v2.x/en/admin-settings/rbac/global-p > - Make sure that you are allowing traffic on port 9796 for each of your nodes because Prometheus will scrape metrics from here. > - Make sure your cluster fulfills the resource requirements. The cluster should have at least 1950Mi memory available, 2700m CPU, and 50Gi storage. A breakdown of the resource limits and requests is [here.](#resource-requirements) +Monitoring can be enabled through the Rancher UI or with the Helm CLI. + +### Enabling Monitoring with the Rancher UI + 1. In the Rancher UI, go to the cluster where you want to install monitoring and click **Cluster Explorer.** 1. Click **Apps.** 1. Click the `rancher-monitoring` app. -1. Optional: Click **Chart Options** and configure alerting, Prometheus and Grafana. For help, refer to the [configuration reference.](#../configuration) +1. Optional: Click **Chart Options** and configure alerting, Prometheus and Grafana. For help, refer to the [configuration reference.](../configuration) 1. Scroll to the bottom of the Helm chart README and click **Install.** **Result:** The monitoring app is deployed in the `cattle-monitoring-system` namespace. +### Enabling Monitoring with the Helm CLI + +Helm CLI users can install the Rancher Monitoring chart directly from the `rancher/charts` GitHub repository using the following commands: + +``` +helm install rancher-monitoring-crd rancher/stable +helm install rancher-monitoring rancher/stable +``` + +However, it is advised to use the Cluster Explorer UI for most use cases. + ### Default Alerts, Targets and Grafana Dashboards -When `rancher-monitoring` is installed, some alerts, targets, and Grafana dashboards will be set up by default. +By default, Rancher Monitoring deploys exporters (such as [node-exporter](https://github.com/prometheus/node_exporter) and [kube-state-metrics](https://github.com/kubernetes/kube-state-metrics)) as well as default Prometheus alerts and Grafana dashboards (curated by the [kube-prometheus](https://github.com/prometheus-operator/kube-prometheus) project) onto a cluster. To see the default alerts, go to the [Alertmanager UI](#alertmanager-ui) and click **Expand all groups.** @@ -105,55 +117,18 @@ To see the default dashboards, go to the [Grafana UI.](#grafana-ui) In the left To configure Prometheus resources from the Rancher UI, click **Apps & Marketplace > Monitoring** in the upper left corner. -# Uninstall Monitoring - -1. From the **Cluster Explorer,** click Apps & Marketplace. -1. Click **Installed Apps.** -1. Go to the `cattle-monitoring-system` namespace and check the boxes for `rancher-monitoring-crd` and `rancher-monitoring`. -1. Click **Delete.** -1. Confirm **Delete.** - -**Result:** `rancher-monitoring` is uninstalled. - -# Resource Requirements - -The resource requests and limits can be configured when installing `rancher-monitoring`. - -The default values are in the `values.yaml` in the `rancher-monitoring` Helm chart. - -There is a [known issue](https://github.com/rancher/rancher/issues/28787#issuecomment-693611821) that K3s clusters require more default memory. If you are enabling monitoring on a K3s cluster, we recommend to setting `prometheus.prometheusSpec.resources.memory.limit` to 2500Mi` and `prometheus.prometheusSpec.resources.memory.request` to 1750Mi. - -| Resource Name | Memory Limit | CPU Limit | Memory Request | CPU Request | -| ------------- | ------------ | ----------- | ---------------- | ------------------ | -| alertmanager | 500Mi | 1000m | 100Mi | 100m | -| grafana | 200Mi | 200m | 100Mi | 100m | -| kube-state-metrics subchart | 200Mi | 100m | 130Mi | 100m | -| prometheus-node-exporter subchart | 50Mi | 200m | 30Mi | 100m | -| prometheusOperator | 500Mi | 200m | 100Mi | 100m | -| prometheus | 500Mi | 1000m | 100Mi | 100m | -| **Total** | **1950Mi** | **2700m** | **560Mi** | **600m** | - -At least 50Gi storage is recommended. - -# Configuration Reference - -For the configuration reference and examples, refer to [this page.](../configuration) - -# Dashboards +# Using Monitoring Installing `rancher-monitoring` makes the following dashboards available from the Rancher UI. ### Grafana UI -The default username and password for the Grafana instance will be `admin`/`prom-operator`. However, Grafana dashboards are served via the Rancher authentication proxy, so only users who are currently authenticated into the Rancher server have access to the Grafana dashboard. For information about the default permissions, refer to the [roles-based access control section.](./rbac) +Rancher allows any users who are authenticated by Kubernetes and have access the Grafana service deployed by the Rancher Monitoring chart to access Grafana via the Rancher Dashboard UI. By default, all users who are able to access Grafana are given the [Viewer](https://grafana.com/docs/grafana/latest/permissions/organization_roles/#viewer-role) role, which allows them to view any of the default dashboards deployed by Rancher. + +However, users can choose to log in to Grafana as an [Admin](https://grafana.com/docs/grafana/latest/permissions/organization_roles/#admin-role) if necessary. The default Admin username and password for the Grafana instance will be `admin`/`prom-operator`, but alternative credentials can also be supplied on deploying or upgrading the chart. To see the Grafana UI, install `rancher-monitoring`. Then go to the **Cluster Explorer.** In the top left corner, click **Cluster Explorer > Monitoring.** Then click **Grafana. -- How to view the currently deployed dashboard -- how to create a custom dashboard on the Grafana UI -- Configuring a persistent Grafana dashboard - will be built on the second, once you have a JSON dashboard, how do you copy it to a configmap -- configuring Grafana to use multiple data sources - might not need this. -
Cluster Compute Resources Dashboard in Grafana
![Cluster Compute Resources Dashboard in Grafana]({{}}/img/rancher/cluster-compute-resources-dashboard.png) @@ -198,7 +173,38 @@ To see the Prometheus Rules, install `rancher-monitoring`. Then go to the **Clus
The Alertmanager UI
![Alertmanager UI]({{}}/img/rancher/alertmanager-ui.png) +# Uninstall Monitoring +1. From the **Cluster Explorer,** click Apps & Marketplace. +1. Click **Installed Apps.** +1. Go to the `cattle-monitoring-system` namespace and check the boxes for `rancher-monitoring-crd` and `rancher-monitoring`. +1. Click **Delete.** +1. Confirm **Delete.** + +**Result:** `rancher-monitoring` is uninstalled. + +# Setting Resource Limits and Requests + +The resource requests and limits can be configured when installing `rancher-monitoring`. + +The default values are in the [values.yaml](https://github.com/rancher/charts/tree/master/charts/prometheus/v9.1.0) in the `rancher-monitoring` Helm chart. + +The default values in the table below are the minimum required resource limits and requests. + +| Resource Name | Memory Limit | CPU Limit | Memory Request | CPU Request | +| ------------- | ------------ | ----------- | ---------------- | ------------------ | +| alertmanager | 500Mi | 1000m | 100Mi | 100m | +| grafana | 200Mi | 200m | 100Mi | 100m | +| kube-state-metrics subchart | 200Mi | 100m | 130Mi | 100m | +| prometheus-node-exporter subchart | 50Mi | 200m | 30Mi | 100m | +| prometheusOperator | 500Mi | 200m | 100Mi | 100m | +| prometheus | 2500Mi | 1000m | 1750Mi | 750m | +| **Total** | **3950Mi** | **2700m** | **2210Mi** | **1250m** | + +At least 50Gi storage is recommended. + + +<<<<<<< HEAD <<<<<<< HEAD <<<<<<< HEAD The data from Prometheus is used as the data source for the Grafana dashboard. Multiple data sources can be configured for Grafana. @@ -207,3 +213,8 @@ For more information about using the Promethus adapter, refer to this [documenta >>>>>>> Update monitoring docs ======= >>>>>>> Revise monitoring docs +======= +# Known Issues + +There is a [known issue](https://github.com/rancher/rancher/issues/28787#issuecomment-693611821) that K3s clusters require more default memory. If you are enabling monitoring on a K3s cluster, we recommend to setting `prometheus.prometheusSpec.resources.memory.limit` to 2500Mi` and `prometheus.prometheusSpec.resources.memory.request` to 1750Mi. +>>>>>>> Add Arvind's changes to monitoring docs diff --git a/content/rancher/v2.x/en/monitoring-alerting/configuration/_index.md b/content/rancher/v2.x/en/monitoring-alerting/configuration/_index.md index 8d6268f9640..640a6d5ab58 100644 --- a/content/rancher/v2.x/en/monitoring-alerting/configuration/_index.md +++ b/content/rancher/v2.x/en/monitoring-alerting/configuration/_index.md @@ -37,53 +37,38 @@ Customizing the scrape configuration used by Prometheus to determine which resou ### ServiceMonitors -This CRD declaratively specifies how groups of Kubernetes services should be monitored. Any Services in your cluster that match the labels located within the ServiceMonitor selector field will be monitored based on the endpoints specified on the ServiceMonitor. For more information on what fields can be specified, please look at the [spec](https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/api.md#servicemonitor) provided by Prometheus Operator. +This CRD declaratively specifies how groups of Kubernetes services should be monitored. Any Services in your cluster that match the labels located within the ServiceMonitor `selector` field will be monitored based on the `endpoints` specified on the ServiceMonitor. For more information on what fields can be specified, please look at the [spec](https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/api.md#servicemonitor) provided by Prometheus Operator. For more information about how ServiceMonitors work, refer to the [Prometheus documentation.](https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/user-guides/running-exporters.md) ### PodMonitors -This CRD declaratively specifies how group of pods should be monitored. Any Pods in your cluster that match the labels located within the PodMonitor selector field will be monitored based on the podMetricsEndpoints specified on the PodMonitor. For more information on what fields can be specified, please look at the [spec](https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/api.md#podmonitorspec) provided by Prometheus Operator. +This CRD declaratively specifies how group of pods should be monitored. Any Pods in your cluster that match the labels located within the PodMonitor `selector` field will be monitored based on the `podMetricsEndpoints` specified on the PodMonitor. For more information on what fields can be specified, please look at the [spec](https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/api.md#podmonitorspec) provided by Prometheus Operator. # Configuring Alerts or Recording Rules with PrometheusRules -Customizing the alerts recorded by Prometheus and the configuration of notifiers that should notify users when alerts are triggered will primarily involve creating/modifying PrometheusRules and the Alertmanager config. +Customizing the alerts recorded by Prometheus and the configuration of notifications based on those alerts will primarily involve creating/modifying PrometheusRules and the Alertmanager Config Secret. -The Alertmanager needs to be a Secret that exists in the `cattle-monitoring` system namespace in order for that to be used. +The Alertmanager Config Secret needs to be a Secret that exists in the `cattle-monitoring` system namespace in order for that to be used. On deploying the Rancher Monitoring application, a default Secret is automatically created if an existing Secret is not specified as part of the `Chart Options`; however, the Secret created on installing the chart will never be modified / removed on an upgrade / uninstall. ### PrometheusRules -This CRD defines a desired set of Prometheus alerting and/or recording rules. +This CRD defines a group of Prometheus alerting and/or recording rules. -Creating the desired alerting rule will involve creating or updating a RuleGroup with your desired alerting rule. Each alerting rule needs to: +To add a group of alerting / recording rules, you should create a PrometheusRule CR the defines a RuleGroup with your desired rules, each specifying: -- Name the alert -- Provide a PromQL expression for the alert -- Provide any labels or annotations that need to be sent to Alertmanager on the alert - -Labels are concise values that identify the alert (e.g. cluster name or severity), while Annotations contain other important pieces of information that need to be displayed when notifying users about an alert (e.g. summary, description, message, runbook URL, etc.). +- The name of the new alert / record +- A PromQL expression for the new alert / record (`expr`) +- Labels that should be attached to the alert / record that identify it (e.g. cluster name or severity) +- Annotations that encode any additional important pieces of information that need to be displayed on the notification for an alert (e.g. summary, description, message, runbook URL, etc.). This field is not required for recording rules. For more information on what fields can be specified, please look at the [spec.](https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/api.md#prometheusrulespec) -### Recording Rules - -If you would like to add [recording rules](https://prometheus.io/docs/prometheus/latest/configuration/recording_rules/) to define a frequently needed or computationally intensive PromQL query that should be stored as a new set of time series, you should create a PrometheusRule. - -Creating the desired recording rule will involve creating or updating a RuleGroup with your desired recording rule. - -Each recording rule needs to: - -- Name the new time series -- Provide a PromQL expression for the series -- Provide any labels that need to be added to this series - -For more information on what fields can be specified, please look at [this page.](https://prometheus.io/docs/prometheus/latest/configuration/recording_rules/#rule) - ### Alertmanager Config The [Alertmanager Config](https://prometheus.io/docs/alerting/latest/configuration/#configuration-file) Secret contains the configuration of an Alertmanager instance that sends out notifications based on alerts it receives from Prometheus. -There only needs to be one Alertmanager in the cluster, and when `rancher-monitor` is installed, a default Alertmanager config is created. +By default, Rancher Monitoring deploys a single Alertmanager onto a cluster that uses a default Alertmanager Config Secret. As part of the chart deployment options, you can opt to increase the number of replicas of the Alertmanager deployed onto your cluster that can all be managed using the same underlying Alertmanager Config Secret. This Secret should be updated or modified any time you want to: @@ -91,22 +76,12 @@ This Secret should be updated or modified any time you want to: - Change the alerts that should be sent to specific notifiers or receivers - Change the group of alerts that are sent out -> Normally, the Alertmanager config secret is configured as a resource that's not managed by Helm. For the `rancher-monitoring` chart, the default case will leave that secret behind. You should save the Secret so that if you uninstall and reinstall `rancher-monitoring`, you wouldn't lose your alerting configuration. +> By default, you can either choose to supply an existing Alertmanager Config Secret (i.e. any Secret in the `cattle-monitoring-system` namespace) or allow Rancher Monitoring to deploy a default Alertmanager Config Secret onto your cluster. By default, the Alertmanager Config Secret created by Rancher will never be modified / deleted on an upgrade / uninstall of the `rancher-monitoring` chart to prevent users from losing or overwriting their alerting configuration when executing operations on the chart. For more information on what fields can be specified in this secret, please look at the [Prometheus Alertmanager docs](https://prometheus.io/docs/alerting/latest/alertmanager/) The full spec for the Alertmanager configuration file and what it takes in can be found [here.](https://prometheus.io/docs/alerting/latest/configuration/#configuration-file) -It is recommended to use an Existing Secret within the namespace `cattle-monitoring-system` for the Alertmanager Config in order to avoid having to to manage the Secret via Helm. - -Using the default Helm managed Secret will require a user to redeploy the chart in order to persist any changes to the Alertmanager configuration on a `helm upgrade`. Using an existing Secret will allow the secret to be directly modified to perform live updates on the Alertmanager configuration if necessary. - -If using an Existing Secret, please add the label `alertmanager_config` to the Secret to enable modifying the secret using the Rancher Dashboard UI. - -Each [Receiver](https://prometheus.io/docs/alerting/latest/configuration/#receiver) (an Alertmanager entity that describes the notification configuration of one or more providers) must define the notification configuration of exactly one provider (i.e. Slack, PagerDuty, OpsGenie, etc.). - -In the Alertmanager documentation about routes, it is said that there must be one "configured top-level route, which must match all alerts (i.e. not have any configured matchers)." On this page, we refer to the notifier using that route as the master notifier. The other notifiers that are referenced in child routes (usually a list provided in `.routes`) we refer to as notifiers. - For more information, refer to the [official Prometheus documentation about configuring routes.](https://www.prometheus.io/docs/alerting/latest/configuration/#route) # Trusted CA for Notifiers @@ -121,7 +96,7 @@ If you need to add a trusted CA to your notifier, follow these steps: # Additional Scrape Configurations -If the scrape configuration you want cannot be specified via a ServiceMonitor or PodMonitor at the moment, you can provide an additionalScrapeConfigSecret on deploying or upgrading `rancher-monitoring`. +If the scrape configuration you want cannot be specified via a ServiceMonitor or PodMonitor at the moment, you can provide an `additionalScrapeConfigSecret` on deploying or upgrading `rancher-monitoring`. A [scrape_config section](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config) specifies a set of targets and parameters describing how to scrape them. In the general case, one scrape configuration specifies a single job. @@ -146,7 +121,7 @@ Prometheus rule files are held in PrometheusRule custom resources. Use the label ### Alertmanager Config -The following values should generate the Alertmanager Config YAML that will be stored under the key `alertmanager.yaml`. This is an example of a Config contained in this key that represents one master Slack notifier with one master Route and no other notifiers configured. +To set up notifications via Slack, the following Alertmanager Config YAML should be placed into the `alertmanager.yaml` key of the Alertmanager Config Secret, where the `api_url` should be updated to use your Webhook URL from Slack: ```yaml route: @@ -163,30 +138,4 @@ receivers: api_url: templates: - /etc/alertmanager/config/*.tmpl -``` - -The following is the default Alertmanager config secret deployed by the `rancher-monitoring chart`. This example shows one master Custom Receiver with one Master Route, one other Route, and no other notifiers: - -```yaml -global: - resolve_timeout: 5m -route: - group_by: ['job'] - group_wait: 30s - group_interval: 5m - repeat_interval: 12h - receiver: 'null' - routes: - - group_by: ['job'] - group_wait: 30s - group_interval: 5m - repeat_interval: 12h - match: - alertname: Watchdog - receiver: 'slack' -receivers: -- name: 'null' -- name: 'slack' -templates: -- /etc/alertmanager/config/*.tmpl ``` \ No newline at end of file diff --git a/content/rancher/v2.x/en/monitoring-alerting/migrating/_index.md b/content/rancher/v2.x/en/monitoring-alerting/migrating/_index.md index 37369281b65..36f668624f3 100644 --- a/content/rancher/v2.x/en/monitoring-alerting/migrating/_index.md +++ b/content/rancher/v2.x/en/monitoring-alerting/migrating/_index.md @@ -3,35 +3,25 @@ title: Migrating to Rancher v2.5 Monitoring weight: 5 --- -If you previously enabled monitoring in Rancher prior to v2.5, there is no upgrade path for the monitoring application. You will need to disable monitoring and re-enable monitoring in Rancher. +If you previously enabled Monitoring, Alerting, or Notifiers in Rancher prior to v2.5, there is no upgrade path for switching to the new monitoring/alerting solution. You will need to disable monitoring/alerting/notifiers in Cluster Manager before deploying the new monitoring solution via Cluster Explorer. -### Major Changes +### Legacy Monitoring/Alerting via Cluster Manager -The new version of Rancher's monitoring application is powered by the Prometheus operator, and it now relies less on Rancher's in-house monitoring tools. +As of v2.2.0, Rancher's Cluster Manager allowed users to enable Monitoring & Alerting V1 (both powered by [Prometheus Operator](https://github.com/prometheus-operator/prometheus-operator)) independently within a cluster. For more information on how to configure Monitoring & Alerting V1, see the [Legacy docs](/rancher/v2.x/en/monitoring-alerting/legacy). -This change allows Rancher to automatically support new features of the Prometheus operator API. Now all of the features exposed by the upstream Prometheus operator are available in the monitoring application, and you have more flexibility to configure monitoring. +When Monitoring is enabled, Monitoring V1 deploys [Prometheus](https://prometheus.io/) and [Grafana](https://grafana.com/docs/grafana/latest/getting-started/what-is-grafana/) onto a cluster to monitor the state of processes of your cluster nodes, Kubernetes components, and software deployments and create custom dashboards to make it easy to visualize collected metrics. -Previously, you would use the Rancher UI to configure monitoring. The Rancher UI created CRDs that were maintained by Rancher and updated the Prometheus state. In Rancher v2.5, you directly create CRDs for the monitoring application, and those CRDs are exposed in the Rancher UI. +Monitoring V1 could be configured on both a cluster-level and on a project-level and would automatically scrape certain workloads deployed as Apps on the Rancher cluster. -Other important changes include: +When Alerts or Notifiers are enabled, Alerting V1 deploys [Prometheus Alertmanager](https://prometheus.io/docs/alerting/latest/alertmanager/) and a set of Rancher controllers onto a cluster that allows users to define alerts and configure alert-based notifications via Email, Slack, PagerDuty, etc. Users can choose to create different types of alerts depending on what needs to be monitored (e.g. System Services, Resources, CIS Scans, etc.); however, PromQL Expression-based alerts can only be created if Monitoring V1 is enabled. -- In the older version of monitoring, an Ingress was created outside of the Helm chart deployment, whereas in the new version, an Ingress is created as part of the Helm chart deployment. -- The monitoring application has the ability to create Prometheus ServiceMonitors and PodMonitors. -- We exposed a [PushProx](https://github.com/prometheus-community/PushProx) exporter, based on the Prometheus project called PushProx. This [`rancher-pushprox` chart](https://github.com/rancher/dev-charts/tree/master/packages/rancher-pushprox/charts) sets up a Deployment of a PushProx proxy and a DaemonSet of PushProx clients. It monitors internal Kubernetes components for K3s, RKE, and kubeAdm clusters. -- One aspect of alerting moved into the monitoring application, specifically the ability to create Prometheus rules. Now alerts are created in monitoring, and you configure how you get notified about the alerts. -- In Rancher v2.4, alerts were configured separately from notifiers. Both alerts and notifiers are configured in the alert configuration. -- To create alerts, you will create Prometheus alert custom resources. -- Instead of having a notifier UI where you can add notifiers, notifications should be configured within the Alertmanager secret exposed in the rancher UI. This will let you configure routes, which is something that couldn't be done with Rancher v2.4. -- Ingresses need to be set up during the chart deployment or upgrade. You can configure an Ingress for Alertmanager, Prometheus and Grafana. To access each of those services from outside the cluster, they each would need separate Ingresses. Each Ingress would take in the same fields. If you want to use your own custom Ingress, we recommend setting the fields during the `rancher-monitoring` Helm chart deployment because it ensures the fields are as expected. -- A [PushProx exporter](https://github.com/prometheus-community/PushProx) was exposed. PushProx exporters are created on the cluster where `rancher-monitoring` is installed. etcd, controlplane, proxy, and controller-manager metrics will all use the PushProx exporter. +### Monitoring/Alerting via Cluster Explorer in Rancher 2.5 -A list of differences between Rancher's monitoring feature and the upstream Prometheus operator can be found in the [changelog.](https://github.com/rancher/charts/blob/dev-v2.5/packages/rancher-monitoring/overlay/CHANGELOG.md) +As of v2.5.0, Rancher's Cluster Explorer now allows users to enable Monitoring & Alerting V2 (both powered by [Prometheus Operator](https://github.com/prometheus-operator/prometheus-operator)) together within a cluster. -### Prometheus Operator Helm Chart Changes +Unlike in Monitoring & Alerting V1, both features are packaged in a single Helm chart found [here](https://github.com/rancher/charts/tree/dev-v2.5/charts/rancher-monitoring). The behavior of this chart and configurable fields closely matches [kube-prometheus-stack](https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack), a Prometheus Community Helm chart, and any deviations from the upstream chart can be found in the [CHANGELOG.md](https://github.com/rancher/charts/blob/dev-v2.5/charts/rancher-monitoring/CHANGELOG.md) maintained with the chart. -Rancher used to use the [Prometheus operator.](https://github.com/prometheus-operator/prometheus-operator) Some parts of the Prometheus operator were included in the [kube-prometheus-stack.](https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack) - -The operator that manages Prometheus is still called the Prometheus operator, but it is now deployed as part of the `kube-prometheus-stack` Helm chart. +If you currently use Monitoring or Alerting V1 and are interested in upgrading to Monitoring & Alerting V2 in Rancher 2.5, please see [Migrating to 2.5.x](/rancher/v2.x/en/monitoring-alerting/2.5.x/migrating/). For more information on how to configure Monitoring & Alerting V2, see the [2.5.x docs](/rancher/v2.x/en/monitoring-alerting/2.5.x). ### Changes to Role-based Access Control diff --git a/content/rancher/v2.x/en/monitoring-alerting/rbac/_index.md b/content/rancher/v2.x/en/monitoring-alerting/rbac/_index.md index ee68b323195..cb39c6c0c6c 100644 --- a/content/rancher/v2.x/en/monitoring-alerting/rbac/_index.md +++ b/content/rancher/v2.x/en/monitoring-alerting/rbac/_index.md @@ -4,19 +4,11 @@ weight: 3 aliases: - /rancher/v2.x/en/cluster-admin/tools/monitoring/rbac --- +This section describes the expectations for RBAC for Rancher Monitoring. -This section describes the permissions required to access Monitoring features. +## Cluster Admins -The `rancher-monitoring` chart installs three `ClusterRoles`. - -- [Cluster-Admin Access](#cluster-admin-access) - - [Admin and Edit Access](#admin-and-edit-access) -- [Summary of Default Monitoring Permissions for Kubernetes Default Roles](#summary-of-default-monitoring-permissions-for-kubernetes-default-roles) -- [Additional Monitoring Roles](#additional-monitoring-roles) - -# Cluster-Admin Access - -By default, only those with the cluster-admin `ClusterRole` can: +By default, only those with the cluster-admin `ClusterRole` should be able to: - Install the `rancher-monitoring` App onto a cluster and all other relevant configuration performed on the chart deploy - e.g. whether default dashboards are created, what exporters are deployed onto the cluster to collect metrics, etc. @@ -25,44 +17,83 @@ By default, only those with the cluster-admin `ClusterRole` can: - Persist new Grafana dashboards or datasources via creating ConfigMaps in the appropriate namespace - Expose certain Prometheus metrics to the k8s Custom Metrics API for HPA via a Secret in the `cattle-monitoring-system` namespace -### Admin and Edit access +## Users with k8s ClusterRole-based Permissions -By default, only Admin and Edit roles can: +The `rancher-monitoring` chart installs the following three `ClusterRoles`. By default, they aggregate into the corresponding k8s `ClusterRoles`: + +| ClusterRole | Aggregates To Default K8s ClusterRole | +| ------------------------------| ---------------------------| +| `monitoring-admin` | `admin`| +| `monitoring-edit` | `edit` | +| `monitoring-view` | `view ` | + +These `ClusterRoles` provide different levels of access to the Monitoring CRDs based on the actions that can be performed: + +| CRDs (monitoring.coreos.com) | Admin | Edit | View | +| ------------------------------| ---------------------------| ---------------------------| ---------------------------| +|
  • `prometheuses`
  • `alertmanagers`
| Get, List, Watch | Get, List, Watch | Get, List, Watch | +|
  • `servicemonitors`
  • `podmonitors`
  • `prometheusrules`
| * | * | Get, List, Watch | + +On a high level, the following permissions are assigned by default as a result. + +### Users with k8s Admin / Edit Permissions + +Only those with the the cluster-admin / admin / edit `ClusterRole` should be able to: -- View the configuration of Prometheuses that are deployed within the cluster -- View the configuraiton of Alertmanagers that are deployed within the cluster - Modify the scrape configuration of Prometheus deployments via ServiceMonitor and PodMonitor CRs - Modify the alerting / recording rules of a Prometheus deployment via PrometheusRules CRs -# Summary of Default Monitoring Permissions for Kubernetes Default Roles +### Users with k8s View Permissions -Monitoring creates three `ClusterRoles` and adds Monitoring CRD access to the following default K8s `ClusterRoles`: +Only those with who have some k8s `ClusterRole` should be able to: -| ClusterRole created by chart | Default K8s ClusterRole | +- View the configuration of Prometheuses that are deployed within the cluster +- View the configuraiton of Alertmanagers that are deployed within the cluster +- View the scrape configuration of Prometheus deployments via ServiceMonitor and PodMonitor CRs +- View the alerting / recording rules of a Prometheus deployment via PrometheusRules CRs + +## Additional Monitoring Roles + +Monitoring also creates six additional `Roles` that are not assigned to users by default but are created within the cluster. Admins should use these roles to provide more fine-grained access to users: + +| Role | Purpose | | ------------------------------| ---------------------------| -| `monitoring-admin` | `admin`| -| `monitoring-edit`| `edit` | -| `monitoring-view` | `view `| +| monitoring-config-admin | Allow admins to assign roles to users to be able to view / modify Secrets and ConfigMaps within the cattle-monitoring-system namespace. Modifying Secrets / ConfigMaps in this namespace could allow users to alter the cluster's Alertmanager configuration, Prometheus Adapter configuration, additional Grafana datasources, TLS secrets, etc. | +| monitoring-config-edit | Allow admins to assign roles to users to be able to view / modify Secrets and ConfigMaps within the cattle-monitoring-system namespace. Modifying Secrets / ConfigMaps in this namespace could allow users to alter the cluster's Alertmanager configuration, Prometheus Adapter configuration, additional Grafana datasources, TLS secrets, etc. | +| monitoring-config-view | Allow admins to assign roles to users to be able to view Secrets and ConfigMaps within the cattle-monitoring-system namespace. Viewing Secrets / ConfigMaps in this namespace could allow users to observe the cluster's Alertmanager configuration, Prometheus Adapter configuration, additional Grafana datasources, TLS secrets, etc. | +| monitoring-dashboard-admin | Allow admins to assign roles to users to be able to edit / view ConfigMaps within the cattle-dashboards namespace. ConfigMaps in this namespace will correspond to Grafana Dashboards that are persisted onto the cluster. | +| monitoring-dashboard-edit | Allow admins to assign roles to users to be able to edit / view ConfigMaps within the cattle-dashboards namespace. ConfigMaps in this namespace will correspond to Grafana Dashboards that are persisted onto the cluster. | +| monitoring-dashboard-view | Allow admins to assign roles to users to be able to view ConfigMaps within the cattle-dashboards namespace. ConfigMaps in this namespace will correspond to Grafana Dashboards that are persisted onto the cluster. | -Rancher will continue to use cluster-owner, cluster-member, project-owner, project-member, etc. as role names, but will utilize default roles to determine access. For each default K8s `ClusterRole` there are different Istio CRD permissions and Kubernetes actions (Create (C), Get (G), List (L), Update (U), Patch (P), Watch (W), Delete(D), All (*)) that can be performed. +## Users with Rancher Cluster Manager Based Permissions +The relationship between the default roles deployed by Rancher Cluster Manager (i.e. cluster-owner, cluster-member, project-owner, project-member), the default k8s roles, and the roles deployed by the rancher-monitoring chart are detailed in the table below: -|CRDs | Admin | Edit | View | -|----------------------------| ------| -----| -----| -|
  • `monitoring.coreos.com`
    • `prometheuses`
    • `alertmanagers`
| GLW | GLW | GLW| -|
  • `monitoring.coreos.com`
    • `servicemonitors`
    • `podmonitors`
    • `prometheusrules`
| * | * | GLW| +| Cluster Manager Role | k8s Role | Monitoring ClusterRole / Role | ClusterRoleBinding or RoleBinding? | +| --------- | --------- | --------- | --------- | +| cluster-owner | cluster-admin | N/A | ClusterRoleBinding | +| cluster-member | admin | monitoring-admin | ClusterRoleBinding | +| project-owner | edit | monitoring-admin | RoleBinding within Project namespace | +| project-member | view | monitoring-edit | RoleBinding within Project namespace | -# Additional Monitoring Roles +### Differences in 2.5.x -Monitoring also creates six `Roles` to enable admins to assign more fine-grained access to monitoring within a cluster: +Users with the project-member or project-owners roles assigned will not be given access to either Prometheus or Grafana in Rancher 2.5.x since we only create Grafana or Prometheus on a cluster-level. -| Role created by chart | Purpose | -| ------------------------------| ---------------------------| -`monitoring-config-admin` | Allow admins to assign roles to users to be able to view/modify Secrets and ConfigMaps within the `cattle-monitoring-system` namespace. Modifying Secrets/ConfigMaps in this namespace could allow users to alter the cluster's Alertmanager configuration, Prometheus Adapter configuration, additional Grafana datasources, TLS secrets, etc. | -`monitoring-config-edit` | Allow admins to assign roles to users to be able to view / modify Secrets and ConfigMaps within the cattle-monitoring-system namespace. Modifying Secrets/ConfigMaps in this namespace could allow users to alter the cluster's Alertmanager configuration, Prometheus Adapter configuration, additional Grafana datasources, TLS secrets, etc. | -`monitoring-config-view` | Allow admins to assign roles to users to be able to view Secrets and ConfigMaps within the cattle-monitoring-system namespace. Viewing Secrets/ConfigMaps in this namespace could allow users to observe the cluster's Alertmanager configuration, Prometheus Adapter configuration, additional Grafana datasources, TLS secrets, etc. | -`monitoring-dashboard-admin` | Allow admins to assign roles to users to be able to edit/view ConfigMaps within the cattle-dashboards namespace. ConfigMaps in this namespace will correspond to Grafana Dashboards that are persisted onto the cluster. | -`monitoring-dashboard-edit` | Allow admins to assign roles to users to be able to edit / view ConfigMaps within the cattle-dashboards namespace. ConfigMaps in this namespace will correspond to Grafana Dashboards that are persisted onto the cluster. | -`monitoring-dashboard-view` | Allow admins to assign roles to users to be able to view ConfigMaps within the cattle-dashboards namespace. ConfigMaps in this namespace will correspond to Grafana Dashboards that are persisted onto the cluster. | +In addition, while project owners will still be only able to add ServiceMonitors / PodMonitors that scrape resources within their project's namespace by default, PrometheusRules are not scoped to a single namespace / project. Therefore, any alert rules or recording rules created by project-owners within their project namespace will be applied across the entire cluster, although they will be unable to view / edit / delete any rules that were created outside the project's namespace. -These Roles are not assigned by default but will be created in the cluster. \ No newline at end of file +### Assigning Additional Access + +If cluster-admins would like to provide additional admin/edit access to users outside of the roles offered by the rancher-monitoring chart, the following table identifies the potential impact: + +|CRDs (monitoring.coreos.com) | Can it cause impact outside of a namespace / project? | Impact | +|----------------------------| ------| ----------------------------| +| `prometheuses`| Yes, this resource can scrape metrics from any targets across the entire cluster (unless the Operator itself is otherwise configured). | User will be able to define the configuration of new cluster-level Prometheus deployments that should be created in the cluster. | +| `alertmanagers`| No | User will be able to define the configuration of new cluster-level Alertmanager deployments that should be created in the cluster. Note: if you just want to allow users to configure settings like Routes and Receivers, you should just provide access to the Alertmanager Config Secret instead. | +|
  • `servicemonitors`
  • `podmonitors`
| No, not by default; this is configurable via `ignoreNamespaceSelectors` on the Prometheus CR. | User will be able to set up scrapes by Prometheus on endpoints exposed by Services / Pods within the namespace they are given this permission in. | +| `prometheusrules`| Yes, PrometheusRules are cluster-scoped. | User will be able to define alert or recording rules on Prometheus based on any series collected across the entire cluster. | + +| k8s Resources | Namespace | Can it cause impact outside of a namespace / project? | Impact | +|----------------------------| ------| ------| ----------------------------| +|
  • `secrets`
  • `configmaps`
| `cattle-monitoring-system` | Yes, Configs and Secrets in this namespace can impact the entire monitoring / alerting pipeline. | User will be able to create or edit Secrets / ConfigMaps such as the Alertmanager Config, Prometheus Adapter Config, TLS secrets, additional Grafana datasoruces, etc. This can have broad impact on all cluster monitoring / alerting. | +|
  • `secrets`
  • `configmaps`
| `cattle-dashboards` | Yes, Configs and Secrets in this namespace can create dashboards that make queries on all metrics collected at a cluster-level. | User will be able to create Secrets / ConfigMaps that persist new Grafana Dashboards only. |