mirror of
https://github.com/rancher/rancher-docs.git
synced 2026-09-29 14:38:50 +00:00
Add v2.14 preview docs (#2212)
This commit is contained in:
Lucas Saintarbor
+159
@@ -0,0 +1,159 @@
|
||||
---
|
||||
title: Persistent Grafana Dashboards
|
||||
---
|
||||
|
||||
<head>
|
||||
<link rel="canonical" href="https://ranchermanager.docs.rancher.com/how-to-guides/advanced-user-guides/monitoring-alerting-guides/create-persistent-grafana-dashboard"/>
|
||||
</head>
|
||||
|
||||
To allow the Grafana dashboard to persist after the Grafana instance restarts, add the dashboard configuration JSON into a ConfigMap. ConfigMaps also allow the dashboards to be deployed with a GitOps or CD based approach. This allows the dashboard to be put under version control.
|
||||
|
||||
- [Creating a Persistent Grafana Dashboard](#creating-a-persistent-grafana-dashboard)
|
||||
- [Known Issues](#known-issues)
|
||||
|
||||
## Creating a Persistent Grafana Dashboard
|
||||
|
||||
<Tabs>
|
||||
<TabItem value="Rancher v2.5.8+">
|
||||
|
||||
:::note Prerequisites:
|
||||
|
||||
- The monitoring application needs to be installed.
|
||||
- To create the persistent dashboard, you must have at least the **Manage Config Maps** Rancher RBAC permissions assigned to you in the project or namespace that contains the Grafana Dashboards. This correlates to the `monitoring-dashboard-edit` or `monitoring-dashboard-admin` Kubernetes native RBAC Roles exposed by the Monitoring chart.
|
||||
- To see the links to the external monitoring UIs, including Grafana dashboards, you will need at least a [project-member role.](../../../integrations-in-rancher/monitoring-and-alerting/rbac-for-monitoring.md#users-with-rancher-based-permissions)
|
||||
|
||||
:::
|
||||
|
||||
### 1. Get the JSON model of the dashboard that you want to persist
|
||||
|
||||
To create a persistent dashboard, you will need to get the JSON model of the dashboard you want to persist. You can use a premade dashboard or build your own.
|
||||
|
||||
To use a premade dashboard, go to [https://grafana.com/grafana/dashboards](https://grafana.com/grafana/dashboards), open up its detail page, and click on the **Download JSON** button to get the JSON model for the next step.
|
||||
|
||||
To use your own dashboard:
|
||||
|
||||
1. Click on the link to open Grafana. On the cluster detail page, click **Monitoring**.
|
||||
1. Log in to Grafana. Note: The default Admin username and password for the Grafana instance is `admin/prom-operator`. Alternative credentials can also be supplied on deploying or upgrading the chart.
|
||||
|
||||
:::note
|
||||
|
||||
Regardless of who has the password, in order to access the Grafana instance, you still need at least the <b>Manage Services</b> or <b>View Monitoring</b> permissions in the project that Rancher Monitoring is deployed into. Alternative credentials can also be supplied on deploying or upgrading the chart.
|
||||
|
||||
:::
|
||||
|
||||
1. Create a dashboard using Grafana's UI. Once complete, go to the dashboard's settings by clicking on the gear icon in the top navigation menu. In the left navigation menu, click **JSON Model**.
|
||||
1. Copy the JSON data structure that appears.
|
||||
|
||||
### 2. Create a ConfigMap using the Grafana JSON model
|
||||
|
||||
Create a ConfigMap in the namespace that contains your Grafana Dashboards (e.g. `cattle-dashboards` by default).
|
||||
|
||||
The ConfigMap should look like this:
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
labels:
|
||||
grafana_dashboard: "1"
|
||||
name: <dashboard-name>
|
||||
namespace: cattle-dashboards # Change if using a non-default namespace
|
||||
data:
|
||||
<dashboard-name>.json: |-
|
||||
<copied-json>
|
||||
```
|
||||
|
||||
By default, Grafana is configured to watch all ConfigMaps with the `grafana_dashboard` label within the `cattle-dashboards` namespace.
|
||||
|
||||
To specify that you would like Grafana to watch for ConfigMaps across all namespaces, refer to [this section](#configuring-namespaces-for-the-grafana-dashboard-configmap).
|
||||
|
||||
To create the ConfigMap through the Rancher UI, first make sure that you are currently logged in to the Grafana UI, to ensure that dashboards import without encountering permissions issues. Then, return to the Rancher UI and perform the following steps:
|
||||
|
||||
1. In the upper left corner, click **☰ > Cluster Management**.
|
||||
1. On the **Clusters** page, go to the cluster where you want to see the visualizations and click **Explore**.
|
||||
1. Click **More Resources > Core > ConfigMaps**.
|
||||
1. Click **Create**.
|
||||
1. On the **Data** tab, set up the key-value pairs similar to the example above. When entering the value for `<dashboard-name>.json`, click **Read from File** to upload the JSON data model as the value.
|
||||
1. On the **Labels & Annotations** tab, click **Add Label** and enter `grafana_dashboard` as the key, and `1` as the value.
|
||||
1. Click **Create**.
|
||||
|
||||
**Result:** After the ConfigMap is created, it should show up on the Grafana UI and be persisted even if the Grafana pod is restarted.
|
||||
|
||||
:::note
|
||||
|
||||
The actual key-value pair may differ if you have modified the Helm chart to watch a different dashboard label and value.
|
||||
|
||||
:::
|
||||
|
||||
Dashboards that are persisted using ConfigMaps cannot be deleted or edited from the Grafana UI.
|
||||
|
||||
If you attempt to delete the dashboard in the Grafana UI, you will see the error message "Dashboard cannot be deleted because it was provisioned." To delete the dashboard, you will need to delete the ConfigMap.
|
||||
|
||||
### Configuring Namespaces for the Grafana Dashboard ConfigMap
|
||||
|
||||
To specify that you would like Grafana to watch for ConfigMaps across all namespaces, set this value in the `rancher-monitoring` Helm chart:
|
||||
|
||||
```
|
||||
grafana.sidecar.dashboards.searchNamespace=ALL
|
||||
```
|
||||
|
||||
Note that the RBAC roles exposed by the Monitoring chart to add Grafana Dashboards are still restricted to giving permissions for users to add dashboards in the namespace defined in `grafana.dashboards.namespace`, which defaults to `cattle-dashboards`.
|
||||
|
||||
</TabItem>
|
||||
<TabItem value="Rancher before v2.5.8">
|
||||
|
||||
:::note Prerequisites:
|
||||
|
||||
- The monitoring application needs to be installed.
|
||||
- You must have the cluster-admin ClusterRole permission.
|
||||
|
||||
:::
|
||||
|
||||
1. In the upper left corner, click **☰ > Cluster Management**.
|
||||
1. On the **Clusters** page, go to the cluster where you want to configure the Grafana namespace and click **Explore**.
|
||||
1. In the left navigation bar, click **Monitoring**.
|
||||
1. Click **Grafana**.
|
||||
1. Log in to Grafana. Note: The default Admin username and password for the Grafana instance is `admin/prom-operator`. Alternative credentials can also be supplied on deploying or upgrading the chart.
|
||||
|
||||
:::note
|
||||
|
||||
Regardless of who has the password, cluster administrator permission in Rancher is still required to access the Grafana instance.
|
||||
|
||||
:::
|
||||
|
||||
1. Go to the dashboard that you want to persist. In the top navigation menu, go to the dashboard settings by clicking the gear icon.
|
||||
1. In the left navigation menu, click **JSON Model**.
|
||||
1. Copy the JSON data structure that appears.
|
||||
1. Create a ConfigMap in the `cattle-dashboards` namespace. The ConfigMap needs to have the label `grafana_dashboard: "1"`. Paste the JSON into the ConfigMap in the format shown in the example below:
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
labels:
|
||||
grafana_dashboard: "1"
|
||||
name: <dashboard-name>
|
||||
namespace: cattle-dashboards
|
||||
data:
|
||||
<dashboard-name>.json: |-
|
||||
<copied-json>
|
||||
```
|
||||
|
||||
**Result:** After the ConfigMap is created, it should show up on the Grafana UI and be persisted even if the Grafana pod is restarted.
|
||||
|
||||
Dashboards that are persisted using ConfigMaps cannot be deleted from the Grafana UI. If you attempt to delete the dashboard in the Grafana UI, you will see the error message "Dashboard cannot be deleted because it was provisioned." To delete the dashboard, you will need to delete the ConfigMap.
|
||||
|
||||
To prevent the persistent dashboard from being deleted when Monitoring v2 is uninstalled, add the following annotation to the `cattle-dashboards` namespace:
|
||||
|
||||
```
|
||||
helm.sh/resource-policy: "keep"
|
||||
```
|
||||
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
|
||||
## Known Issues
|
||||
|
||||
For users who are using Monitoring V2 v9.4.203 or below, uninstalling the Monitoring chart will delete the `cattle-dashboards` namespace, which will delete all persisted dashboards, unless the namespace is marked with the annotation `helm.sh/resource-policy: "keep"`.
|
||||
|
||||
This annotation will be added by default in the new monitoring chart released by Rancher v2.5.8, but it still needs to be manually applied for users of earlier Rancher versions.
|
||||
+43
@@ -0,0 +1,43 @@
|
||||
---
|
||||
title: Customizing Grafana Dashboards
|
||||
---
|
||||
|
||||
<head>
|
||||
<link rel="canonical" href="https://ranchermanager.docs.rancher.com/how-to-guides/advanced-user-guides/monitoring-alerting-guides/customize-grafana-dashboard"/>
|
||||
</head>
|
||||
|
||||
In this section, you'll learn how to customize the Grafana dashboard to show metrics that apply to a certain container.
|
||||
|
||||
### Prerequisites
|
||||
|
||||
Before you can customize a Grafana dashboard, the `rancher-monitoring` application must be installed.
|
||||
|
||||
To see the links to the external monitoring UIs, including Grafana dashboards, you will need at least a [project-member role.](../../../integrations-in-rancher/monitoring-and-alerting/rbac-for-monitoring.md#users-with-rancher-based-permissions)
|
||||
|
||||
### Signing in to Grafana
|
||||
|
||||
1. In the Rancher UI, go to the cluster that has the dashboard you want to customize.
|
||||
1. In the left navigation menu, click **Monitoring.**
|
||||
1. Click **Grafana.** The Grafana dashboard should open in a new tab.
|
||||
1. Go to the log in icon in the lower left corner and click **Sign In.**
|
||||
1. Log in to Grafana. The default Admin username and password for the Grafana instance is `admin/prom-operator`. (Regardless of who has the password, cluster administrator permission in Rancher is still required access the Grafana instance.) Alternative credentials can also be supplied on deploying or upgrading the chart.
|
||||
|
||||
|
||||
### Getting the PromQL Query Powering a Grafana Panel
|
||||
|
||||
For any panel, you can click the title and click **Explore** to get the PromQL queries powering the graphic.
|
||||
|
||||
For this example, we would like to get the CPU usage for the Alertmanager container, so we click **CPU Utilization > Inspect.**
|
||||
|
||||
The **Data** tab shows the underlying data as a time series, with the time in first column and the PromQL query result in the second column. Copy the PromQL query.
|
||||
|
||||
```
|
||||
(1 - (avg(irate({__name__=~"node_cpu_seconds_total|windows_cpu_time_total",mode="idle"}[5m])))) * 100
|
||||
```
|
||||
|
||||
You can then modify the query in the Grafana panel or create a new Grafana panel using the query.
|
||||
|
||||
See also:
|
||||
|
||||
- [Grafana docs on editing a panel](https://grafana.com/docs/grafana/latest/panels-visualizations/configure-panel-options/#edit-a-panel)
|
||||
- [Grafana docs on adding a panel to a dashboard](https://grafana.com/docs/grafana/latest/panels-visualizations/panel-editor-overview)
|
||||
+23
@@ -0,0 +1,23 @@
|
||||
---
|
||||
title: Debugging High Memory Usage
|
||||
---
|
||||
|
||||
<head>
|
||||
<link rel="canonical" href="https://ranchermanager.docs.rancher.com/how-to-guides/advanced-user-guides/monitoring-alerting-guides/debug-high-memory-usage"/>
|
||||
</head>
|
||||
|
||||
Every time series in Prometheus is uniquely identified by its [metric name](https://prometheus.io/docs/practices/naming/#metric-names) and optional key-value pairs called [labels.](https://prometheus.io/docs/practices/naming/#labels)
|
||||
|
||||
The labels allow the ability to filter and aggregate the time series data, but they also multiply the amount of data that Prometheus collects.
|
||||
|
||||
Each time series has a defined set of labels, and Prometheus generates a new time series for all unique combinations of labels. If a metric has two labels attached, two time series are generated for that metric. Changing any label value, including adding or removing a label, will create a new time series.
|
||||
|
||||
Prometheus is optimized to store data that is index-based on series. It is designed for a relatively consistent number of time series and a relatively large number of samples that need to be collected from the exporters over time.
|
||||
|
||||
Inversely, Prometheus is not optimized to accommodate a rapidly changing number of time series. For that reason, large bursts of memory usage can occur when monitoring is installed on clusters where many resources are being created and destroyed, especially on multi-tenant clusters.
|
||||
|
||||
### Reducing Memory Bursts
|
||||
|
||||
To reduce memory consumption, Prometheus can be configured to store fewer time series, by scraping fewer metrics or by attaching fewer labels to the time series. To see which series use the most memory, you can check the TSDB (time series database) status page in the Prometheus UI.
|
||||
|
||||
Distributed Prometheus solutions such as [Thanos](https://thanos.io/) and [Cortex](https://cortexmetrics.io/) use an alternate architecture in which multiple small Prometheus instances are deployed. In the case of Thanos, the metrics from each Prometheus are aggregated into the common Thanos deployment, and then those metrics are exported to a persistent store, such as S3. This more robust architecture avoids burdening any single Prometheus instance with too many time series, while also preserving the ability to query metrics on a global level.
|
||||
+147
@@ -0,0 +1,147 @@
|
||||
---
|
||||
title: Enable Monitoring
|
||||
---
|
||||
|
||||
<head>
|
||||
<link rel="canonical" href="https://ranchermanager.docs.rancher.com/how-to-guides/advanced-user-guides/monitoring-alerting-guides/enable-monitoring"/>
|
||||
</head>
|
||||
|
||||
As an [administrator](../../new-user-guides/authentication-permissions-and-global-configuration/manage-role-based-access-control-rbac/global-permissions.md) or [cluster owner](../../new-user-guides/authentication-permissions-and-global-configuration/manage-role-based-access-control-rbac/cluster-and-project-roles.md#cluster-roles), you can configure Rancher to deploy Prometheus to monitor your Kubernetes cluster.
|
||||
|
||||
This page describes how to enable monitoring and alerting within a cluster using the new monitoring application.
|
||||
|
||||
You can enable monitoring with or without SSL.
|
||||
|
||||
## Requirements
|
||||
|
||||
- Allow traffic on port 9796 for each of your nodes. Prometheus scrapes metrics from these ports.
|
||||
- You may also need to allow traffic on port 10254 for each of your nodes, if [PushProx](../../../integrations-in-rancher/monitoring-and-alerting/how-monitoring-works.md#pushprox) is disabled (`ingressNginx.enabled` set to `false`), or you've upgraded from a previous Rancher version that had v1 monitoring already installed.
|
||||
- Make sure that your cluster fulfills the resource requirements. The cluster should have at least 1950Mi memory available, 2700m CPU, and 50Gi storage. See [Configuring Resource Limits and Requests](../../../reference-guides/monitoring-v2-configuration/helm-chart-options.md#configuring-resource-limits-and-requests) for a breakdown of the resource limits and requests.
|
||||
|
||||
:::note
|
||||
|
||||
If you want to set up Alertmanager, Grafana or Ingress, it has to be done with the settings on the Helm chart deployment. It's problematic to create Ingress outside the deployment.
|
||||
|
||||
:::
|
||||
|
||||
## Setting Resource Limits and Requests
|
||||
|
||||
The resource requests and limits can be configured when installing `rancher-monitoring`. To configure Prometheus resources from the Rancher UI, click **Apps > Monitoring** in the upper left corner.
|
||||
|
||||
For more information about the default limits, see [this page.](../../../reference-guides/monitoring-v2-configuration/helm-chart-options.md#configuring-resource-limits-and-requests)
|
||||
|
||||
## Install the Monitoring Application
|
||||
|
||||
### Enable Monitoring for use without SSL
|
||||
|
||||
1. Click **☰ > Cluster Management**.
|
||||
1. Go to the cluster that you created and click **Explore**.
|
||||
1. Click **Cluster Tools** (bottom left corner).
|
||||
1. Click **Install** by Monitoring.
|
||||
1. Optional: Customize requests, limits and more for Alerting, Prometheus, and Grafana in the Values step. For help, refer to the [configuration reference.](../../../reference-guides/monitoring-v2-configuration/helm-chart-options.md)
|
||||
|
||||
**Result:** The monitoring app is deployed in the `cattle-monitoring-system` namespace.
|
||||
|
||||
### Enable Monitoring for use with SSL
|
||||
|
||||
1. Follow the steps on [this page](../../new-user-guides/kubernetes-resources-setup/secrets.md) to create a secret in order for SSL to be used for alerts.
|
||||
- The secret should be created in the `cattle-monitoring-system` namespace. If it doesn't exist, create it first.
|
||||
- Add the `ca`, `cert`, and `key` files to the secret.
|
||||
1. In the upper left corner, click **☰ > Cluster Management**.
|
||||
1. On the **Clusters** page, go to the cluster where you want to enable monitoring for use with SSL and click **Explore**.
|
||||
1. Click **Apps > Charts**.
|
||||
1. Click **Monitoring**.
|
||||
1. Click **Install** or **Update**, depending on whether you have already installed Monitoring.
|
||||
1. Check the box for **Customize Helm options before install** and click **Next**.
|
||||
1. Click **Alerting**.
|
||||
1. In the **Additional Secrets** field, add the secrets created earlier.
|
||||
|
||||
**Result:** The monitoring app is deployed in the `cattle-monitoring-system` namespace.
|
||||
|
||||
When [creating a receiver,](../../../reference-guides/monitoring-v2-configuration/receivers.md#creating-receivers-in-the-rancher-ui) SSL-enabled receivers such as email or webhook will have a **SSL** section with fields for **CA File Path**, **Cert File Path**, and **Key File Path**. Fill in these fields with the paths to each of `ca`, `cert`, and `key`. The path will be of the form `/etc/alertmanager/secrets/name-of-file-in-secret`.
|
||||
|
||||
For example, if you created a secret with these key-value pairs:
|
||||
|
||||
```yaml
|
||||
ca.crt=`base64-content`
|
||||
cert.pem=`base64-content`
|
||||
key.pfx=`base64-content`
|
||||
```
|
||||
|
||||
Then **Cert File Path** would be set to `/etc/alertmanager/secrets/cert.pem`.
|
||||
|
||||
## Rancher Performance Dashboard
|
||||
|
||||
When monitoring is installed on the upstream (local) cluster, you are given basic health metrics about the Rancher pods, such as CPU and memory data. To get advanced metrics for your local Rancher server, you must additionally enable the Rancher Performance Dashboard for Grafana.
|
||||
|
||||
This dashboard provides access to the following advanced metrics:
|
||||
|
||||
- Handler Average Execution Times Over Last 5 Minutes
|
||||
- Rancher API Average Request Times Over Last 5 Minutes
|
||||
- Subscribe Average Request Times Over Last 5 Minutes
|
||||
- Lasso Controller Work Queue Depth (Top 20)
|
||||
- Number of Rancher Requests (Top 20)
|
||||
- Number of Failed Rancher API Requests (Top 20)
|
||||
- K8s Proxy Store Average Request Times Over Last 5 Minutes (Top 20)
|
||||
- K8s Proxy Client Average Request Times Over Last 5 Minutes (Top 20)
|
||||
- Cached Objects by GroupVersionKind (Top 20)
|
||||
- Lasso Handler Executions (Top 20)
|
||||
- Handler Executions Over Last 2 Minutes (Top 20)
|
||||
- Total Handler Executions with Error (Top 20)
|
||||
- Data Transmitted by Remote Dialer Sessions (Top 20)
|
||||
- Errors for Remote Dialer Sessions (Top 20)
|
||||
- Remote Dialer Connections Removed (Top 20)
|
||||
- Remote Dialer Connections Added by Client (Top 20)
|
||||
|
||||
:::note
|
||||
|
||||
Profiling data (such as advanced memory or CPU analysis) is not present as it is a very context-dependent technique that's meant for debugging and not intended for normal observation.
|
||||
|
||||
:::
|
||||
|
||||
### Enabling the Rancher Performance Dashboard
|
||||
|
||||
To enable the Rancher Performance Dashboard:
|
||||
|
||||
<Tabs groupId="UIorCLI">
|
||||
<TabItem value="Helm">
|
||||
|
||||
Use the following options with the Helm CLI:
|
||||
|
||||
```bash
|
||||
--set extraEnv\[0\].name="CATTLE_PROMETHEUS_METRICS" --set-string extraEnv\[0\].value=true
|
||||
```
|
||||
|
||||
You can also include the following snippet in your Rancher Helm chart's values.yaml file:
|
||||
|
||||
```yaml
|
||||
extraEnv:
|
||||
- name: "CATTLE_PROMETHEUS_METRICS"
|
||||
value: "true"
|
||||
```
|
||||
|
||||
</TabItem>
|
||||
<TabItem value="UI">
|
||||
|
||||
1. Click **☰ > Cluster Management**.
|
||||
1. Go to the row of the `local` cluster and click **Explore**.
|
||||
1. Click **Workloads > Deployments**.
|
||||
1. Use the dropdown menu at the top to filter for **All Namespaces**.
|
||||
1. Under the `cattle-system` namespace, go to the `rancher` row and click **⋮ > Edit Config**
|
||||
1. Under **Environment Variables**, click **Add Variable**.
|
||||
1. For **Type**, select `Key/Value Pair`.
|
||||
1. For **Variable Name**, enter `CATTLE_PROMETHEUS_METRICS`.
|
||||
1. For **Value**, enter `true`.
|
||||
1. Click **Save** to apply the change.
|
||||
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
|
||||
### Accessing the Rancher Performance Dashboard
|
||||
|
||||
1. Click **☰ > Cluster Management**.
|
||||
1. Go to the row of the `local` cluster and click **Explore**.
|
||||
1. Click **Monitoring**
|
||||
1. Select the **Grafana** dashboard.
|
||||
1. From the sidebar, click **Search dashboards**.
|
||||
1. Enter `Rancher Performance Debugging` and select it.
|
||||
+14
@@ -0,0 +1,14 @@
|
||||
---
|
||||
title: Monitoring/Alerting Guides
|
||||
---
|
||||
|
||||
<head>
|
||||
<link rel="canonical" href="https://ranchermanager.docs.rancher.com/how-to-guides/advanced-user-guides/monitoring-alerting-guides"/>
|
||||
</head>
|
||||
|
||||
- [Enable monitoring](enable-monitoring.md)
|
||||
- [Uninstall monitoring](uninstall-monitoring.md)
|
||||
- [Monitoring workloads](set-up-monitoring-for-workloads.md)
|
||||
- [Customizing Grafana dashboards](customize-grafana-dashboard.md)
|
||||
- [Persistent Grafana dashboards](create-persistent-grafana-dashboard.md)
|
||||
- [Debugging high memory usage](debug-high-memory-usage.md)
|
||||
+11
@@ -0,0 +1,11 @@
|
||||
---
|
||||
title: Customizing Grafana Dashboards
|
||||
---
|
||||
|
||||
<head>
|
||||
<link rel="canonical" href="https://ranchermanager.docs.rancher.com/how-to-guides/advanced-user-guides/monitoring-alerting-guides/prometheus-federator-guides/customize-grafana-dashboards"/>
|
||||
</head>
|
||||
|
||||
Grafana dashboards are customized the same way whether it's for rancher-monitoring or for Prometheus Federator.
|
||||
|
||||
For instructions, refer to [this page](../customize-grafana-dashboard.md).
|
||||
+90
@@ -0,0 +1,90 @@
|
||||
---
|
||||
title: Enable Prometheus Federator
|
||||
---
|
||||
|
||||
<head>
|
||||
<link rel="canonical" href="https://ranchermanager.docs.rancher.com/how-to-guides/advanced-user-guides/monitoring-alerting-guides/prometheus-federator-guides/enable-prometheus-federator"/>
|
||||
</head>
|
||||
|
||||
## Requirements
|
||||
|
||||
By default, Prometheus Federator is configured and intended to be deployed alongside [rancher-monitoring](../../../../integrations-in-rancher/monitoring-and-alerting/monitoring-and-alerting.md), which deploys Prometheus Operator alongside a Cluster Prometheus that each Project Monitoring Stack is configured to federate namespace-scoped metrics from by default.
|
||||
|
||||
For instructions on installing rancher-monitoring, refer to [this page](../enable-monitoring.md).
|
||||
|
||||
The default configuration should already be compatible with your rancher-monitoring stack. However, to optimize the security and usability of Prometheus Federator in your cluster, we recommend making these additional configurations to rancher-monitoring:
|
||||
|
||||
- [Ensure the cattle-monitoring-system namespace is placed into the System Project](#ensure-the-cattle-monitoring-system-namespace-is-placed-into-the-system-project-or-a-similarly-locked-down-project-that-has-access-to-other-projects-in-the-cluster).
|
||||
- [Configure rancher-monitoring to only watch for resources created by the Helm chart itself](#configure-rancher-monitoring-to-only-watch-for-resources-created-by-the-helm-chart-itself).
|
||||
- [Increase the CPU / memory limits of the Cluster Prometheus](#increase-the-cpu--memory-limits-of-the-cluster-prometheus).
|
||||
|
||||
### Ensure the cattle-monitoring-system namespace is placed into the System Project (or a similarly locked down Project that has access to other Projects in the cluster)
|
||||
|
||||
|
||||
|
||||
Prometheus Operator's security model expects that the namespace it is deployed into (e.g., `cattle-monitoring-system`) has limited access for anyone except Cluster Admins to avoid privilege escalation via execing into Pods (such as the Jobs executing Helm operations). In addition, deploying Prometheus Federator and all Project Prometheus stacks into the System Project ensures that each Project Prometheus is able to reach out to scrape workloads across all Projects, even if Network Policies are defined via Project Network Isolation. It also provides limited access for Project Owners, Project Members, and other users so that they're unable to access data that they shouldn't have access to (i.e., being allowed to exec into pods, set up the ability to scrape namespaces outside of a given Project, etc.).
|
||||
|
||||
1. Open the `System` project to check your namespaces:
|
||||
|
||||
Click **Cluster > Projects/Namespaces** in the Rancher UI. This will display all of the namespaces in the `System` project:
|
||||
|
||||
|
||||
|
||||
1. If you have an existing Monitoring V2 installation within the `cattle-monitoring-system` namespace, but that namespace is not in the `System` project, you may move the `cattle-monitoring-system` namespace into the `System` project or into another project of limited access. To do so, you may either:
|
||||
|
||||
- Drag and drop the namespace into the `System` project or
|
||||
- Select **⋮** to the right of the namespace, click **Move**, then choose `System` from the **Target Project** dropdown
|
||||
|
||||
|
||||
|
||||
### Configure rancher-monitoring to only watch for resources created by the Helm chart itself
|
||||
|
||||
Since each Project Monitoring Stack will watch the other namespaces and collect additional custom workload metrics or dashboards already, it's recommended to configure the following settings on all selectors to ensure that the Cluster Prometheus Stack only monitors resources created by the Helm Chart itself:
|
||||
|
||||
```
|
||||
matchLabels:
|
||||
release: "rancher-monitoring"
|
||||
```
|
||||
|
||||
The following selector fields are recommended to have this value:
|
||||
- `.Values.alertmanager.alertmanagerSpec.alertmanagerConfigSelector`
|
||||
- `.Values.prometheus.prometheusSpec.serviceMonitorSelector`
|
||||
- `.Values.prometheus.prometheusSpec.podMonitorSelector`
|
||||
- `.Values.prometheus.prometheusSpec.ruleSelector`
|
||||
- `.Values.prometheus.prometheusSpec.probeSelector`
|
||||
|
||||
Once this setting is turned on, you can always create ServiceMonitors or PodMonitors that are picked up by the Cluster Prometheus by adding the label `release: "rancher-monitoring"` to them, in which case they will be ignored by Project Monitoring Stacks automatically by default, even if the namespace in which those ServiceMonitors or PodMonitors reside in are not system namespaces.
|
||||
|
||||
:::note
|
||||
|
||||
If you don't want to allow users to be able to create ServiceMonitors and PodMonitors that aggregate into the Cluster Prometheus in Project namespaces, you can additionally set the namespaceSelectors on the chart to only target system namespaces (which must contain `cattle-monitoring-system` and `cattle-dashboards`, where resources are deployed into by default by rancher-monitoring; you will also need to monitor the `default` namespace to get apiserver metrics or create a custom ServiceMonitor to scrape apiserver metrics from the Service residing in the default namespace) to limit your Cluster Prometheus from picking up other Prometheus Operator CRs. In that case, it would be recommended to turn `.Values.prometheus.prometheusSpec.ignoreNamespaceSelectors=true` to allow you to define ServiceMonitors that can monitor non-system namespaces from within a system namespace.
|
||||
|
||||
:::
|
||||
|
||||
### Increase the CPU / memory limits of the Cluster Prometheus
|
||||
|
||||
Depending on a cluster's setup, it's generally recommended to give a large amount of dedicated memory to the Cluster Prometheus to avoid restarts due to out-of-memory errors (OOMKilled) usually caused by churn created in the cluster that causes a large number of high cardinality metrics to be generated and ingested by Prometheus within one block of time. This is one of the reasons why the default Rancher Monitoring stack expects around 4GB of RAM to be able to operate in a normal-sized cluster. However, when introducing Project Monitoring Stacks that are all sending `/federate` requests to the same Cluster Prometheus and are reliant on the Cluster Prometheus being "up" to federate that system data on their namespaces, it's even more important that the Cluster Prometheus has an ample amount of CPU / memory assigned to it to prevent an outage that can cause data gaps across all Project Prometheis in the cluster.
|
||||
|
||||
:::note
|
||||
|
||||
There are no specific recommendations on how much memory the Cluster Prometheus should be configured with since it depends entirely on the user's setup (namely the likelihood of encountering a high churn rate and the scale of metrics that could be generated at that time); it generally varies per setup.
|
||||
|
||||
:::
|
||||
|
||||
## Install the Prometheus Federator Application
|
||||
|
||||
1. Click **☰ > Cluster Management**.
|
||||
1. Go to the cluster that you want to install Prometheus Federator and click **Explore**.
|
||||
1. Click **Apps -> Charts**.
|
||||
1. Click the **Prometheus Federator** chart.
|
||||
1. Click **Install**.
|
||||
1. On the **Metadata** page, click **Next**.
|
||||
1. In the **Namespaces** > **Project Release Namespace Project ID** field, the `System Project` is used as the default but can be overridden with another project with similarly [limited access](#ensure-the-cattle-monitoring-system-namespace-is-placed-into-the-system-project-or-a-similarly-locked-down-project-that-has-access-to-other-projects-in-the-cluster). Project IDs can be found with the following command run in the local upstream cluster:
|
||||
|
||||
```plain
|
||||
kubectl get projects -A -o custom-columns="NAMESPACE":.metadata.namespace,"ID":.metadata.name,"NAME":.spec.displayName
|
||||
```
|
||||
|
||||
1. Click **Install**.
|
||||
|
||||
**Result:** The Prometheus Federator app is deployed in the `cattle-monitoring-system` namespace.
|
||||
+21
@@ -0,0 +1,21 @@
|
||||
---
|
||||
title: Installing Project Monitors
|
||||
---
|
||||
|
||||
<head>
|
||||
<link rel="canonical" href="https://ranchermanager.docs.rancher.com/how-to-guides/advanced-user-guides/monitoring-alerting-guides/prometheus-federator-guides/project-monitors"/>
|
||||
</head>
|
||||
|
||||
Install **Project Monitors** in each project where you want to enable project monitoring.
|
||||
|
||||
1. Click **☰ > Cluster Management**.
|
||||
|
||||
1. On the **Clusters** page, go to the cluster where you want to enable monitoring and click **Explore**.
|
||||
|
||||
1. Click **Monitoring > Project Monitors** on the left nav bar. Then click **Create** in the upper right.
|
||||
|
||||
|
||||
|
||||
1. Select your project from the drop-down menu, then click **Create** again.
|
||||
|
||||
|
||||
+12
@@ -0,0 +1,12 @@
|
||||
---
|
||||
title: Prometheus Federator Guides
|
||||
---
|
||||
|
||||
<head>
|
||||
<link rel="canonical" href="https://ranchermanager.docs.rancher.com/how-to-guides/advanced-user-guides/monitoring-alerting-guides/prometheus-federator-guides"/>
|
||||
</head>
|
||||
|
||||
- [Enable Prometheus Operator](enable-prometheus-federator.md)
|
||||
- [Uninstall Prometheus Operator](uninstall-prometheus-federator.md)
|
||||
- [Customize Grafana Dashboards](customize-grafana-dashboards.md)
|
||||
- [Set Up Workloads](set-up-workloads.md)
|
||||
+17
@@ -0,0 +1,17 @@
|
||||
---
|
||||
title: Setting up Prometheus Federator for a Workload
|
||||
---
|
||||
|
||||
<head>
|
||||
<link rel="canonical" href="https://ranchermanager.docs.rancher.com/how-to-guides/advanced-user-guides/monitoring-alerting-guides/prometheus-federator-guides/set-up-workloads"/>
|
||||
</head>
|
||||
|
||||
### Display CPU and Memory Metrics for a Workload
|
||||
|
||||
Displaying CPU and memory metrics with Prometheus Federator is done the same way as with rancher-monitoring. For instructions, refer [here](../set-up-monitoring-for-workloads.md#display-cpu-and-memory-metrics-for-a-workload).
|
||||
|
||||
### Setting up Metrics Beyond CPU and Memory
|
||||
|
||||
Setting up metrics beyond CPU and memory with Prometheus Federator is done the same way as with rancher-monitoring. For instructions, refer [here](../set-up-monitoring-for-workloads.md#setting-up-metrics-beyond-cpu-and-memory).
|
||||
|
||||
<!-- ### Custom Metrics -->
|
||||
+17
@@ -0,0 +1,17 @@
|
||||
---
|
||||
title: Uninstall Prometheus Federator
|
||||
---
|
||||
|
||||
<head>
|
||||
<link rel="canonical" href="https://ranchermanager.docs.rancher.com/how-to-guides/advanced-user-guides/monitoring-alerting-guides/prometheus-federator-guides/uninstall-prometheus-federator"/>
|
||||
</head>
|
||||
|
||||
1. Click **☰ > Cluster Management**.
|
||||
1. Go to the cluster that you created and click **Explore**.
|
||||
1. In the left navigation bar, click **Apps**.
|
||||
1. Click **Installed Apps**.
|
||||
1. Go to the `cattle-monitoring-system` namespace and check the boxes for `rancher-monitoring-crd` and `rancher-monitoring`.
|
||||
1. Click **Delete**.
|
||||
1. Confirm **Delete**.
|
||||
|
||||
**Result:** `prometheus-federator` is uninstalled.
|
||||
+31
@@ -0,0 +1,31 @@
|
||||
---
|
||||
title: Setting up Monitoring for a Workload
|
||||
---
|
||||
|
||||
<head>
|
||||
<link rel="canonical" href="https://ranchermanager.docs.rancher.com/how-to-guides/advanced-user-guides/monitoring-alerting-guides/set-up-monitoring-for-workloads"/>
|
||||
</head>
|
||||
|
||||
If you only need CPU and memory time series for the workload, you don't need to deploy a ServiceMonitor or PodMonitor because the monitoring application already collects metrics data on resource usage by default.
|
||||
|
||||
The steps for setting up monitoring for workloads depend on whether you want basic metrics such as CPU and memory for the workload, or whether you want to scrape custom metrics from the workload.
|
||||
|
||||
If you only need CPU and memory time series for the workload, you don't need to deploy a ServiceMonitor or PodMonitor because the monitoring application already collects metrics data on resource usage by default. The resource usage time series data is in Prometheus's local time series database.
|
||||
|
||||
Grafana shows the data in aggregate, but you can see the data for the individual workload by using a PromQL query that extracts the data for that workload. Once you have the PromQL query, you can execute the query individually in the Prometheus UI and see the time series visualized there, or you can use the query to customize a Grafana dashboard to display the workload metrics. For examples of PromQL queries for workload metrics, see [this section](../../../integrations-in-rancher/monitoring-and-alerting/promql-expressions.md#workload-metrics).
|
||||
|
||||
To set up custom metrics for your workload, you will need to set up an exporter and create a new ServiceMonitor custom resource to configure Prometheus to scrape metrics from your exporter.
|
||||
|
||||
### Display CPU and Memory Metrics for a Workload
|
||||
|
||||
By default, the monitoring application already scrapes CPU and memory.
|
||||
|
||||
To get some fine-grained detail for a particular workload, you can customize a Grafana dashboard to display the metrics for a particular workload.
|
||||
|
||||
### Setting up Metrics Beyond CPU and Memory
|
||||
|
||||
For custom metrics, you will need to expose the metrics on your application in a format supported by Prometheus.
|
||||
|
||||
Then we recommend that you should create a new ServiceMonitor custom resource. When this resource is created, the Prometheus custom resource will be automatically updated so that its scrape configuration includes the new custom metrics endpoint. Then Prometheus will begin scraping metrics from the endpoint.
|
||||
|
||||
You can also create a PodMonitor to expose the custom metrics endpoint, but ServiceMonitors are more appropriate for the majority of use cases.
|
||||
+23
@@ -0,0 +1,23 @@
|
||||
---
|
||||
title: Uninstall Monitoring
|
||||
---
|
||||
|
||||
<head>
|
||||
<link rel="canonical" href="https://ranchermanager.docs.rancher.com/how-to-guides/advanced-user-guides/monitoring-alerting-guides/uninstall-monitoring"/>
|
||||
</head>
|
||||
|
||||
1. Click **☰ > Cluster Management**.
|
||||
1. Go to the cluster that you created and click **Explore**.
|
||||
1. In the left navigation bar, click **Apps**.
|
||||
1. Click **Installed Apps**.
|
||||
1. Go to the `cattle-monitoring-system` namespace and check the boxes for `rancher-monitoring-crd` and `rancher-monitoring`.
|
||||
1. Click **Delete**.
|
||||
1. Confirm **Delete**.
|
||||
|
||||
**Result:** `rancher-monitoring` is uninstalled.
|
||||
|
||||
:::note Persistent Grafana Dashboards:
|
||||
|
||||
For users who are using Monitoring V2 v9.4.203 or below, uninstalling the Monitoring chart will delete the cattle-dashboards namespace, which will delete all persisted dashboards, unless the namespace is marked with the annotation `helm.sh/resource-policy: "keep"`. This annotation is added by default in Monitoring V2 v14.5.100+ but can be manually applied on the cattle-dashboards namespace before an uninstall if an older version of the Monitoring chart is currently installed onto your cluster.
|
||||
|
||||
:::
|
||||
Reference in New Issue
Block a user