From 509532c9bb55c55d977be83e50a5accdc1c6b454 Mon Sep 17 00:00:00 2001 From: Billy Tat Date: Fri, 10 May 2024 17:06:56 -0700 Subject: [PATCH 1/4] Document performance dashboard --- .../enable-monitoring.md | 78 +++++++++++++++++- .../enable-monitoring.md | 82 +++++++++++++++++++ .../enable-monitoring.md | 76 +++++++++++++++++ 3 files changed, 235 insertions(+), 1 deletion(-) diff --git a/docs/how-to-guides/advanced-user-guides/monitoring-alerting-guides/enable-monitoring.md b/docs/how-to-guides/advanced-user-guides/monitoring-alerting-guides/enable-monitoring.md index 2e4ab117ca0..bccbac40330 100644 --- a/docs/how-to-guides/advanced-user-guides/monitoring-alerting-guides/enable-monitoring.md +++ b/docs/how-to-guides/advanced-user-guides/monitoring-alerting-guides/enable-monitoring.md @@ -42,7 +42,7 @@ For more information about the default limits, see [this page.](../../../referen ### Enable Monitoring for use without SSL -1. Click **☰ > Cluster Management**. +1. Click **☰ > Cluster Management**. 1. Go to the cluster that you created and click **Explore**. 1. Click **Cluster Tools** (bottom left corner). 1. Click **Install** by Monitoring. @@ -77,3 +77,79 @@ key.pfx=`base64-content` ``` Then **Cert File Path** would be set to `/etc/alertmanager/secrets/cert.pem`. + +## Rancher Performance Dashboard + +When monitoring is installed, you are given basic health metrics about the local Rancher server, such as CPU and memory data. To get advanced metrics for your local Rancher server, you must enable the Rancher Performance Dashboard for Grafana. + +This dashboard provides access to the following advanced metrics: + +- Handler Average Execution Times Over Last 5 Minutes +- Rancher API Average Request Times Over Last 5 Minutes +- Subscribe Average Request Times Over Last 5 Minutes +- Lasso Controller Work Queue Depth +- Number of Rancher Requests +- Number of Failed Rancher API Requests +- K8s Proxy Store Average Request Times Over Last 5 Minutes +- K8s Proxy Client Average Request Times Over Last 5 Minutes +- Cached Objects by GroupVersionKind +- Lasso Handler Executions +- Handler Executions over last 2 minutes +- Total Handler Executions with Error +- Data Transmitted by Remote Dialer Sessions +- Errors for Remote Dialer Sessions +- Remote Dialer Connections Removed +- Remote Dialer Connections Added by Client + +:::note + +Advanced heap analysis is not present as it is a very context-dependent technique that's meant for debugging and not intended for normal observation. + +::: + +### Enabling the Rancher Performance Dashboard + +To enable the Rancher Performance Dashboard: + + + + +Use the following options with the Helm CLI: + +```bash +--set extraEnv\[0\].name="CATTLE_PROMETHEUS_METRICS" --set-string extraEnv\[0\].value=true +``` + +You can also include the following snippet in your Rancher Helm chart's values.yaml file: + +```yaml +extraEnv: + - name: "CATTLE_PROMETHEUS_METRICS" + value: "true" +``` + + + + +1. Click **☰ > Cluster Management**. +1. Go to the row of the `local` cluster and click **Explore**. +1. Click **Workloads > Deployments**. +1. Use the dropdown menu at the top to filter for **All Namespaces**. +1. Under the `cattle-system` namespace, go to the `rancher` row and click **⋮ > Edit Config** +1. Under **Environment Variables**, click **Add Variable**. +1. For **Type**, select `Key/Value Pair`. +1. For **Variable Name**, enter `CATTLE_PROMETHEUS_METRICS`. +1. For **Value**, enter `true`. +1. Click **Save** to apply the change. + + + + +### Accessing the Rancher Performance Dashboard + +1. Click **☰ > Cluster Management**. +1. Go to the row of the `local` cluster and click **Explore**. +1. Click **Monitoring** +1. Select the **Grafana** dashboard. +1. From the sidebar, click **Search dashboards**. +1. Enter `Rancher Performance Dashboard` and select it. diff --git a/versioned_docs/version-2.7/how-to-guides/advanced-user-guides/monitoring-alerting-guides/enable-monitoring.md b/versioned_docs/version-2.7/how-to-guides/advanced-user-guides/monitoring-alerting-guides/enable-monitoring.md index 2e4ab117ca0..d35c23d57a9 100644 --- a/versioned_docs/version-2.7/how-to-guides/advanced-user-guides/monitoring-alerting-guides/enable-monitoring.md +++ b/versioned_docs/version-2.7/how-to-guides/advanced-user-guides/monitoring-alerting-guides/enable-monitoring.md @@ -77,3 +77,85 @@ key.pfx=`base64-content` ``` Then **Cert File Path** would be set to `/etc/alertmanager/secrets/cert.pem`. + +## Rancher Performance Dashboard + +When monitoring is installed, you are given basic health metrics about the local Rancher server, such as CPU and memory data. To get advanced metrics for your local Rancher server, you must enable the Rancher Performance Dashboard for Grafana. + +This dashboard provides access to the following advanced metrics: + +- Handler Average Execution Times Over Last 5 Minutes +- Rancher API Average Request Times Over Last 5 Minutes +- Subscribe Average Request Times Over Last 5 Minutes +- Lasso Controller Work Queue Depth +- Number of Rancher Requests +- Number of Failed Rancher API Requests +- K8s Proxy Store Average Request Times Over Last 5 Minutes +- K8s Proxy Client Average Request Times Over Last 5 Minutes +- Cached Objects by GroupVersionKind +- Lasso Handler Executions +- Handler Executions over last 2 minutes +- Total Handler Executions with Error +- Data Transmitted by Remote Dialer Sessions +- Errors for Remote Dialer Sessions +- Remote Dialer Connections Removed +- Remote Dialer Connections Added by Client + +:::note + +Advanced heap analysis is not present as it is a very context-dependent technique that's meant for debugging and not intended for normal observation. + +::: + +:::note + +Advanced heap analysis is not present as it is a very context-dependent technique that's meant for debugging and not intended for normal observation. + +::: + +### Enabling the Rancher Performance Dashboard + +To enable the Rancher Performance Dashboard: + + + + +Use the following options with the Helm CLI: + +```bash +--set extraEnv\[0\].name="CATTLE_PROMETHEUS_METRICS" --set-string extraEnv\[0\].value=true +``` + +You can also include the following snippet in your Rancher Helm chart's values.yaml file: + +```yaml +extraEnv: + - name: "CATTLE_PROMETHEUS_METRICS" + value: "true" +``` + + + + +1. Click **☰ > Cluster Management**. +1. Go to the row of the `local` cluster and click **Explore**. +1. Click **Workloads > Deployments**. +1. Use the dropdown menu at the top to filter for **All Namespaces**. +1. Under the `cattle-system` namespace, go to the `rancher` row and click **⋮ > Edit Config** +1. Under **Environment Variables**, click **Add Variable**. +1. For **Type**, select `Key/Value Pair`. +1. For **Variable Name**, enter `CATTLE_PROMETHEUS_METRICS`. +1. For **Value**, enter `true`. +1. Click **Save** to apply the change. + + + + +### Accessing the Rancher Performance Dashboard + +1. Click **☰ > Cluster Management**. +1. Go to the row of the `local` cluster and click **Explore**. +1. Click **Monitoring** +1. Select the **Grafana** dashboard. +1. From the sidebar, click **Search dashboards**. +1. Enter `Rancher Performance Dashboard` and select it. diff --git a/versioned_docs/version-2.8/how-to-guides/advanced-user-guides/monitoring-alerting-guides/enable-monitoring.md b/versioned_docs/version-2.8/how-to-guides/advanced-user-guides/monitoring-alerting-guides/enable-monitoring.md index 2e4ab117ca0..0d21d70721a 100644 --- a/versioned_docs/version-2.8/how-to-guides/advanced-user-guides/monitoring-alerting-guides/enable-monitoring.md +++ b/versioned_docs/version-2.8/how-to-guides/advanced-user-guides/monitoring-alerting-guides/enable-monitoring.md @@ -77,3 +77,79 @@ key.pfx=`base64-content` ``` Then **Cert File Path** would be set to `/etc/alertmanager/secrets/cert.pem`. + +## Rancher Performance Dashboard + +When monitoring is installed, you are given basic health metrics about the local Rancher server, such as CPU and memory data. To get advanced metrics for your local Rancher server, you must enable the Rancher Performance Dashboard for Grafana. + +This dashboard provides access to the following advanced metrics: + +- Handler Average Execution Times Over Last 5 Minutes +- Rancher API Average Request Times Over Last 5 Minutes +- Subscribe Average Request Times Over Last 5 Minutes +- Lasso Controller Work Queue Depth +- Number of Rancher Requests +- Number of Failed Rancher API Requests +- K8s Proxy Store Average Request Times Over Last 5 Minutes +- K8s Proxy Client Average Request Times Over Last 5 Minutes +- Cached Objects by GroupVersionKind +- Lasso Handler Executions +- Handler Executions over last 2 minutes +- Total Handler Executions with Error +- Data Transmitted by Remote Dialer Sessions +- Errors for Remote Dialer Sessions +- Remote Dialer Connections Removed +- Remote Dialer Connections Added by Client + +:::note + +Advanced heap analysis is not present as it is a very context-dependent technique that's meant for debugging and not intended for normal observation. + +::: + +### Enabling the Rancher Performance Dashboard + +To enable the Rancher Performance Dashboard: + + + + +Use the following options with the Helm CLI: + +```bash +--set extraEnv\[0\].name="CATTLE_PROMETHEUS_METRICS" --set-string extraEnv\[0\].value=true +``` + +You can also include the following snippet in your Rancher Helm chart's values.yaml file: + +```yaml +extraEnv: + - name: "CATTLE_PROMETHEUS_METRICS" + value: "true" +``` + + + + +1. Click **☰ > Cluster Management**. +1. Go to the row of the `local` cluster and click **Explore**. +1. Click **Workloads > Deployments**. +1. Use the dropdown menu at the top to filter for **All Namespaces**. +1. Under the `cattle-system` namespace, go to the `rancher` row and click **⋮ > Edit Config** +1. Under **Environment Variables**, click **Add Variable**. +1. For **Type**, select `Key/Value Pair`. +1. For **Variable Name**, enter `CATTLE_PROMETHEUS_METRICS`. +1. For **Value**, enter `true`. +1. Click **Save** to apply the change. + + + + +### Accessing the Rancher Performance Dashboard + +1. Click **☰ > Cluster Management**. +1. Go to the row of the `local` cluster and click **Explore**. +1. Click **Monitoring** +1. Select the **Grafana** dashboard. +1. From the sidebar, click **Search dashboards**. +1. Enter `Rancher Performance Dashboard` and select it. From dbd40f23243d6b6bd0fe88b850e58a0c75829736 Mon Sep 17 00:00:00 2001 From: Billy Tat Date: Wed, 22 May 2024 09:20:03 -0700 Subject: [PATCH 2/4] Fix dashboard labels --- .../enable-monitoring.md | 28 +++++++++---------- .../enable-monitoring.md | 28 +++++++++---------- .../enable-monitoring.md | 28 +++++++++---------- 3 files changed, 42 insertions(+), 42 deletions(-) diff --git a/docs/how-to-guides/advanced-user-guides/monitoring-alerting-guides/enable-monitoring.md b/docs/how-to-guides/advanced-user-guides/monitoring-alerting-guides/enable-monitoring.md index bccbac40330..d52669e15e4 100644 --- a/docs/how-to-guides/advanced-user-guides/monitoring-alerting-guides/enable-monitoring.md +++ b/docs/how-to-guides/advanced-user-guides/monitoring-alerting-guides/enable-monitoring.md @@ -87,19 +87,19 @@ This dashboard provides access to the following advanced metrics: - Handler Average Execution Times Over Last 5 Minutes - Rancher API Average Request Times Over Last 5 Minutes - Subscribe Average Request Times Over Last 5 Minutes -- Lasso Controller Work Queue Depth -- Number of Rancher Requests -- Number of Failed Rancher API Requests -- K8s Proxy Store Average Request Times Over Last 5 Minutes -- K8s Proxy Client Average Request Times Over Last 5 Minutes -- Cached Objects by GroupVersionKind -- Lasso Handler Executions -- Handler Executions over last 2 minutes -- Total Handler Executions with Error -- Data Transmitted by Remote Dialer Sessions -- Errors for Remote Dialer Sessions -- Remote Dialer Connections Removed -- Remote Dialer Connections Added by Client +- Lasso Controller Work Queue Depth (Top 20) +- Number of Rancher Requests (Top 20) +- Number of Failed Rancher API Requests (Top 20) +- K8s Proxy Store Average Request Times Over Last 5 Minutes (Top 20) +- K8s Proxy Client Average Request Times Over Last 5 Minutes (Top 20) +- Cached Objects by GroupVersionKind (Top 20) +- Lasso Handler Executions (Top 20) +- Handler Executions Over Last 2 Minutes (Top 20) +- Total Handler Executions with Error (Top 20) +- Data Transmitted by Remote Dialer Sessions (Top 20) +- Errors for Remote Dialer Sessions (Top 20) +- Remote Dialer Connections Removed (Top 20) +- Remote Dialer Connections Added by Client (Top 20) :::note @@ -152,4 +152,4 @@ extraEnv: 1. Click **Monitoring** 1. Select the **Grafana** dashboard. 1. From the sidebar, click **Search dashboards**. -1. Enter `Rancher Performance Dashboard` and select it. +1. Enter `Rancher Performance Debugging` and select it. diff --git a/versioned_docs/version-2.7/how-to-guides/advanced-user-guides/monitoring-alerting-guides/enable-monitoring.md b/versioned_docs/version-2.7/how-to-guides/advanced-user-guides/monitoring-alerting-guides/enable-monitoring.md index d35c23d57a9..02ca13f1fb3 100644 --- a/versioned_docs/version-2.7/how-to-guides/advanced-user-guides/monitoring-alerting-guides/enable-monitoring.md +++ b/versioned_docs/version-2.7/how-to-guides/advanced-user-guides/monitoring-alerting-guides/enable-monitoring.md @@ -87,19 +87,19 @@ This dashboard provides access to the following advanced metrics: - Handler Average Execution Times Over Last 5 Minutes - Rancher API Average Request Times Over Last 5 Minutes - Subscribe Average Request Times Over Last 5 Minutes -- Lasso Controller Work Queue Depth -- Number of Rancher Requests -- Number of Failed Rancher API Requests -- K8s Proxy Store Average Request Times Over Last 5 Minutes -- K8s Proxy Client Average Request Times Over Last 5 Minutes -- Cached Objects by GroupVersionKind -- Lasso Handler Executions -- Handler Executions over last 2 minutes -- Total Handler Executions with Error -- Data Transmitted by Remote Dialer Sessions -- Errors for Remote Dialer Sessions -- Remote Dialer Connections Removed -- Remote Dialer Connections Added by Client +- Lasso Controller Work Queue Depth (Top 20) +- Number of Rancher Requests (Top 20) +- Number of Failed Rancher API Requests (Top 20) +- K8s Proxy Store Average Request Times Over Last 5 Minutes (Top 20) +- K8s Proxy Client Average Request Times Over Last 5 Minutes (Top 20) +- Cached Objects by GroupVersionKind (Top 20) +- Lasso Handler Executions (Top 20) +- Handler Executions Over Last 2 Minutes (Top 20) +- Total Handler Executions with Error (Top 20) +- Data Transmitted by Remote Dialer Sessions (Top 20) +- Errors for Remote Dialer Sessions (Top 20) +- Remote Dialer Connections Removed (Top 20) +- Remote Dialer Connections Added by Client (Top 20) :::note @@ -158,4 +158,4 @@ extraEnv: 1. Click **Monitoring** 1. Select the **Grafana** dashboard. 1. From the sidebar, click **Search dashboards**. -1. Enter `Rancher Performance Dashboard` and select it. +1. Enter `Rancher Performance Debugging` and select it. diff --git a/versioned_docs/version-2.8/how-to-guides/advanced-user-guides/monitoring-alerting-guides/enable-monitoring.md b/versioned_docs/version-2.8/how-to-guides/advanced-user-guides/monitoring-alerting-guides/enable-monitoring.md index 0d21d70721a..0a2fa2da193 100644 --- a/versioned_docs/version-2.8/how-to-guides/advanced-user-guides/monitoring-alerting-guides/enable-monitoring.md +++ b/versioned_docs/version-2.8/how-to-guides/advanced-user-guides/monitoring-alerting-guides/enable-monitoring.md @@ -87,19 +87,19 @@ This dashboard provides access to the following advanced metrics: - Handler Average Execution Times Over Last 5 Minutes - Rancher API Average Request Times Over Last 5 Minutes - Subscribe Average Request Times Over Last 5 Minutes -- Lasso Controller Work Queue Depth -- Number of Rancher Requests -- Number of Failed Rancher API Requests -- K8s Proxy Store Average Request Times Over Last 5 Minutes -- K8s Proxy Client Average Request Times Over Last 5 Minutes -- Cached Objects by GroupVersionKind -- Lasso Handler Executions -- Handler Executions over last 2 minutes -- Total Handler Executions with Error -- Data Transmitted by Remote Dialer Sessions -- Errors for Remote Dialer Sessions -- Remote Dialer Connections Removed -- Remote Dialer Connections Added by Client +- Lasso Controller Work Queue Depth (Top 20) +- Number of Rancher Requests (Top 20) +- Number of Failed Rancher API Requests (Top 20) +- K8s Proxy Store Average Request Times Over Last 5 Minutes (Top 20) +- K8s Proxy Client Average Request Times Over Last 5 Minutes (Top 20) +- Cached Objects by GroupVersionKind (Top 20) +- Lasso Handler Executions (Top 20) +- Handler Executions Over Last 2 Minutes (Top 20) +- Total Handler Executions with Error (Top 20) +- Data Transmitted by Remote Dialer Sessions (Top 20) +- Errors for Remote Dialer Sessions (Top 20) +- Remote Dialer Connections Removed (Top 20) +- Remote Dialer Connections Added by Client (Top 20) :::note @@ -152,4 +152,4 @@ extraEnv: 1. Click **Monitoring** 1. Select the **Grafana** dashboard. 1. From the sidebar, click **Search dashboards**. -1. Enter `Rancher Performance Dashboard` and select it. +1. Enter `Rancher Performance Debugging` and select it. From cfd8e386d0eb5a2e7fd4f6b4629fc257e94c7f0b Mon Sep 17 00:00:00 2001 From: Billy Tat Date: Thu, 23 May 2024 09:47:28 -0700 Subject: [PATCH 3/4] Apply suggestions from code review Co-authored-by: Silvio Moioli --- .../monitoring-alerting-guides/enable-monitoring.md | 4 ++-- .../monitoring-alerting-guides/enable-monitoring.md | 7 +------ .../monitoring-alerting-guides/enable-monitoring.md | 4 ++-- 3 files changed, 5 insertions(+), 10 deletions(-) diff --git a/docs/how-to-guides/advanced-user-guides/monitoring-alerting-guides/enable-monitoring.md b/docs/how-to-guides/advanced-user-guides/monitoring-alerting-guides/enable-monitoring.md index d52669e15e4..55d2220f747 100644 --- a/docs/how-to-guides/advanced-user-guides/monitoring-alerting-guides/enable-monitoring.md +++ b/docs/how-to-guides/advanced-user-guides/monitoring-alerting-guides/enable-monitoring.md @@ -80,7 +80,7 @@ Then **Cert File Path** would be set to `/etc/alertmanager/secrets/cert.pem`. ## Rancher Performance Dashboard -When monitoring is installed, you are given basic health metrics about the local Rancher server, such as CPU and memory data. To get advanced metrics for your local Rancher server, you must enable the Rancher Performance Dashboard for Grafana. +When monitoring is installed on the upstream (local) cluster, you are given basic health metrics about the Rancher pods, such as CPU and memory data. To get advanced metrics for your local Rancher server, you must additionally enable the Rancher Performance Dashboard for Grafana. This dashboard provides access to the following advanced metrics: @@ -103,7 +103,7 @@ This dashboard provides access to the following advanced metrics: :::note -Advanced heap analysis is not present as it is a very context-dependent technique that's meant for debugging and not intended for normal observation. +Profiling data (such as advanced memory or CPU analysis) is not present as it is a very context-dependent technique that's meant for debugging and not intended for normal observation. ::: diff --git a/versioned_docs/version-2.7/how-to-guides/advanced-user-guides/monitoring-alerting-guides/enable-monitoring.md b/versioned_docs/version-2.7/how-to-guides/advanced-user-guides/monitoring-alerting-guides/enable-monitoring.md index 02ca13f1fb3..900a814570e 100644 --- a/versioned_docs/version-2.7/how-to-guides/advanced-user-guides/monitoring-alerting-guides/enable-monitoring.md +++ b/versioned_docs/version-2.7/how-to-guides/advanced-user-guides/monitoring-alerting-guides/enable-monitoring.md @@ -80,7 +80,7 @@ Then **Cert File Path** would be set to `/etc/alertmanager/secrets/cert.pem`. ## Rancher Performance Dashboard -When monitoring is installed, you are given basic health metrics about the local Rancher server, such as CPU and memory data. To get advanced metrics for your local Rancher server, you must enable the Rancher Performance Dashboard for Grafana. +When monitoring is installed on the upstream (local) cluster, you are given basic health metrics about the Rancher pods, such as CPU and memory data. To get advanced metrics for your local Rancher server, you must additionally enable the Rancher Performance Dashboard for Grafana. This dashboard provides access to the following advanced metrics: @@ -107,11 +107,6 @@ Advanced heap analysis is not present as it is a very context-dependent techniqu ::: -:::note - -Advanced heap analysis is not present as it is a very context-dependent technique that's meant for debugging and not intended for normal observation. - -::: ### Enabling the Rancher Performance Dashboard diff --git a/versioned_docs/version-2.8/how-to-guides/advanced-user-guides/monitoring-alerting-guides/enable-monitoring.md b/versioned_docs/version-2.8/how-to-guides/advanced-user-guides/monitoring-alerting-guides/enable-monitoring.md index 0a2fa2da193..161d1495d5e 100644 --- a/versioned_docs/version-2.8/how-to-guides/advanced-user-guides/monitoring-alerting-guides/enable-monitoring.md +++ b/versioned_docs/version-2.8/how-to-guides/advanced-user-guides/monitoring-alerting-guides/enable-monitoring.md @@ -80,7 +80,7 @@ Then **Cert File Path** would be set to `/etc/alertmanager/secrets/cert.pem`. ## Rancher Performance Dashboard -When monitoring is installed, you are given basic health metrics about the local Rancher server, such as CPU and memory data. To get advanced metrics for your local Rancher server, you must enable the Rancher Performance Dashboard for Grafana. +When monitoring is installed on the upstream (local) cluster, you are given basic health metrics about the Rancher pods, such as CPU and memory data. To get advanced metrics for your local Rancher server, you must additionally enable the Rancher Performance Dashboard for Grafana. This dashboard provides access to the following advanced metrics: @@ -103,7 +103,7 @@ This dashboard provides access to the following advanced metrics: :::note -Advanced heap analysis is not present as it is a very context-dependent technique that's meant for debugging and not intended for normal observation. +Profiling data (such as advanced memory or CPU analysis) is not present as it is a very context-dependent technique that's meant for debugging and not intended for normal observation. ::: From a23e5823b5625ef28b8d6379ce2693cd0c9becff Mon Sep 17 00:00:00 2001 From: Billy Tat Date: Thu, 23 May 2024 10:14:32 -0700 Subject: [PATCH 4/4] Update versioned_docs/version-2.7/how-to-guides/advanced-user-guides/monitoring-alerting-guides/enable-monitoring.md Co-authored-by: Silvio Moioli --- .../monitoring-alerting-guides/enable-monitoring.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/versioned_docs/version-2.7/how-to-guides/advanced-user-guides/monitoring-alerting-guides/enable-monitoring.md b/versioned_docs/version-2.7/how-to-guides/advanced-user-guides/monitoring-alerting-guides/enable-monitoring.md index 900a814570e..a6524cb79ec 100644 --- a/versioned_docs/version-2.7/how-to-guides/advanced-user-guides/monitoring-alerting-guides/enable-monitoring.md +++ b/versioned_docs/version-2.7/how-to-guides/advanced-user-guides/monitoring-alerting-guides/enable-monitoring.md @@ -103,7 +103,7 @@ This dashboard provides access to the following advanced metrics: :::note -Advanced heap analysis is not present as it is a very context-dependent technique that's meant for debugging and not intended for normal observation. +Profiling data (such as advanced memory or CPU analysis) is not present as it is a very context-dependent technique that's meant for debugging and not intended for normal observation. :::