mirror of
https://github.com/rancher/rancher-docs.git
synced 2026-09-25 04:28:15 +00:00
Apply Divio and update links
This commit is contained in:
Billy Tat
+110
@@ -0,0 +1,110 @@
|
||||
---
|
||||
title: Skipped and Not Applicable Tests
|
||||
weight: 1
|
||||
aliases:
|
||||
- /rancher/v2.0-v2.4/en/cis-scans/legacy/skipped-tests
|
||||
- /rancher/v2.0-v2.4/en/cis-scans/v2.4/skipped-tests
|
||||
- /rancher/v2.0-v2.4/en/cis-scans/skipped-tests
|
||||
- /rancher/v2.x/en/cis-scans/v2.4/skipped-tests/
|
||||
---
|
||||
|
||||
This section lists the tests that are skipped in the permissive test profile for RKE.
|
||||
|
||||
All the tests that are skipped and not applicable on this page will be counted as Not Applicable in the generated report. The skipped test count will only mention the user-defined skipped tests. This allows user-skipped tests to be distinguished from the tests that are skipped by default in the RKE permissive test profile.
|
||||
|
||||
- [CIS Benchmark v1.5](#cis-benchmark-v1-5)
|
||||
- [CIS Benchmark v1.4](#cis-benchmark-v1-4)
|
||||
|
||||
# CIS Benchmark v1.5
|
||||
|
||||
### CIS Benchmark v1.5 Skipped Tests
|
||||
|
||||
| Number | Description | Reason for Skipping |
|
||||
| ---------- | ------------- | --------- |
|
||||
| 1.1.12 | Ensure that the etcd data directory ownership is set to etcd:etcd (Scored) | A system service account is required for etcd data directory ownership. Refer to Rancher's hardening guide for more details on how to configure this ownership. |
|
||||
| 1.2.6 | Ensure that the --kubelet-certificate-authority argument is set as appropriate (Scored) | When generating serving certificates, functionality could break in conjunction with hostname overrides which are required for certain cloud providers. |
|
||||
| 1.2.16 | Ensure that the admission control plugin PodSecurityPolicy is set (Scored) | Enabling Pod Security Policy can cause applications to unexpectedly fail. |
|
||||
| 1.2.33 | Ensure that the --encryption-provider-config argument is set as appropriate (Not Scored) | Enabling encryption changes how data can be recovered as data is encrypted. |
|
||||
| 1.2.34 | Ensure that encryption providers are appropriately configured (Not Scored) | Enabling encryption changes how data can be recovered as data is encrypted. |
|
||||
| 4.2.6 | Ensure that the --protect-kernel-defaults argument is set to true (Scored) | System level configurations are required before provisioning the cluster in order for this argument to be set to true. |
|
||||
| 4.2.10 | Ensure that the--tls-cert-file and --tls-private-key-file arguments are set as appropriate (Scored) | When generating serving certificates, functionality could break in conjunction with hostname overrides which are required for certain cloud providers. |
|
||||
| 5.1.5 | Ensure that default service accounts are not actively used. (Scored) | Kubernetes provides default service accounts to be used. |
|
||||
| 5.2.2 | Minimize the admission of containers wishing to share the host process ID namespace (Scored) | Enabling Pod Security Policy can cause applications to unexpectedly fail. |
|
||||
| 5.2.3 | Minimize the admission of containers wishing to share the host IPC namespace (Scored) | Enabling Pod Security Policy can cause applications to unexpectedly fail. |
|
||||
| 5.2.4 | Minimize the admission of containers wishing to share the host network namespace (Scored) | Enabling Pod Security Policy can cause applications to unexpectedly fail. |
|
||||
| 5.2.5 | Minimize the admission of containers with allowPrivilegeEscalation (Scored) | Enabling Pod Security Policy can cause applications to unexpectedly fail. |
|
||||
| 5.3.2 | Ensure that all Namespaces have Network Policies defined (Scored) | Enabling Network Policies can prevent certain applications from communicating with each other. |
|
||||
| 5.6.4 | The default namespace should not be used (Scored) | Kubernetes provides a default namespace. |
|
||||
|
||||
### CIS Benchmark v1.5 Not Applicable Tests
|
||||
|
||||
| Number | Description | Reason for being not applicable |
|
||||
| ---------- | ------------- | --------- |
|
||||
| 1.1.1 | Ensure that the API server pod specification file permissions are set to 644 or more restrictive (Scored) | Clusters provisioned by RKE doesn't require or maintain a configuration file for kube-apiserver. All configuration is passed in as arguments at container run time. |
|
||||
| 1.1.2 | Ensure that the API server pod specification file ownership is set to root:root (Scored) | Clusters provisioned by RKE doesn't require or maintain a configuration file for kube-apiserver. All configuration is passed in as arguments at container run time. |
|
||||
| 1.1.3 | Ensure that the controller manager pod specification file permissions are set to 644 or more restrictive (Scored) | Clusters provisioned by RKE doesn't require or maintain a configuration file for controller-manager. All configuration is passed in as arguments at container run time. |
|
||||
| 1.1.4 | Ensure that the controller manager pod specification file ownership is set to root:root (Scored) | Clusters provisioned by RKE doesn't require or maintain a configuration file for controller-manager. All configuration is passed in as arguments at container run time. |
|
||||
| 1.1.5 | Ensure that the scheduler pod specification file permissions are set to 644 or more restrictive (Scored) | Clusters provisioned by RKE doesn't require or maintain a configuration file for scheduler. All configuration is passed in as arguments at container run time. |
|
||||
| 1.1.6 | Ensure that the scheduler pod specification file ownership is set to root:root (Scored) | Clusters provisioned by RKE doesn't require or maintain a configuration file for scheduler. All configuration is passed in as arguments at container run time. |
|
||||
| 1.1.7 | Ensure that the etcd pod specification file permissions are set to 644 or more restrictive (Scored) | Clusters provisioned by RKE doesn't require or maintain a configuration file for etcd. All configuration is passed in as arguments at container run time. |
|
||||
| 1.1.8 | Ensure that the etcd pod specification file ownership is set to root:root (Scored) | Clusters provisioned by RKE doesn't require or maintain a configuration file for etcd. All configuration is passed in as arguments at container run time. |
|
||||
| 1.1.13 | Ensure that the admin.conf file permissions are set to 644 or more restrictive (Scored) | Clusters provisioned by RKE does not store the kubernetes default kubeconfig credentials file on the nodes. |
|
||||
| 1.1.14 | Ensure that the admin.conf file ownership is set to root:root (Scored) | Clusters provisioned by RKE does not store the kubernetes default kubeconfig credentials file on the nodes. |
|
||||
| 1.1.15 | Ensure that the scheduler.conf file permissions are set to 644 or more restrictive (Scored) | Clusters provisioned by RKE doesn't require or maintain a configuration file for scheduler. All configuration is passed in as arguments at container run time. |
|
||||
| 1.1.16 | Ensure that the scheduler.conf file ownership is set to root:root (Scored) | Clusters provisioned by RKE doesn't require or maintain a configuration file for scheduler. All configuration is passed in as arguments at container run time. |
|
||||
| 1.1.17 | Ensure that the controller-manager.conf file permissions are set to 644 or more restrictive (Scored) | Clusters provisioned by RKE doesn't require or maintain a configuration file for controller-manager. All configuration is passed in as arguments at container run time. |
|
||||
| 1.1.18 | Ensure that the controller-manager.conf file ownership is set to root:root (Scored) | Clusters provisioned by RKE doesn't require or maintain a configuration file for controller-manager. All configuration is passed in as arguments at container run time. |
|
||||
| 1.3.6 | Ensure that the RotateKubeletServerCertificate argument is set to true (Scored) | Clusters provisioned by RKE handles certificate rotation directly through RKE. |
|
||||
| 4.1.1 | Ensure that the kubelet service file permissions are set to 644 or more restrictive (Scored) | Clusters provisioned by RKE doesn’t require or maintain a configuration file for the kubelet service. All configuration is passed in as arguments at container run time. |
|
||||
| 4.1.2 | Ensure that the kubelet service file ownership is set to root:root (Scored) | Clusters provisioned by RKE doesn’t require or maintain a configuration file for the kubelet service. All configuration is passed in as arguments at container run time. |
|
||||
| 4.1.9 | Ensure that the kubelet configuration file has permissions set to 644 or more restrictive (Scored) | Clusters provisioned by RKE doesn’t require or maintain a configuration file for the kubelet. All configuration is passed in as arguments at container run time. |
|
||||
| 4.1.10 | Ensure that the kubelet configuration file ownership is set to root:root (Scored) | Clusters provisioned by RKE doesn’t require or maintain a configuration file for the kubelet. All configuration is passed in as arguments at container run time. |
|
||||
| 4.2.12 | Ensure that the RotateKubeletServerCertificate argument is set to true (Scored) | Clusters provisioned by RKE handles certificate rotation directly through RKE. |
|
||||
|
||||
# CIS Benchmark v1.4
|
||||
|
||||
The skipped and not applicable tests for CIS Benchmark v1.4 are as follows:
|
||||
|
||||
### CIS Benchmark v1.4 Skipped Tests
|
||||
|
||||
Number | Description | Reason for Skipping
|
||||
---|---|---
|
||||
1.1.11 | "Ensure that the admission control plugin AlwaysPullImages is set (Scored)" | Enabling AlwaysPullImages can use significant bandwidth.
|
||||
1.1.21 | "Ensure that the --kubelet-certificate-authority argument is set as appropriate (Scored)" | When generating serving certificates, functionality could break in conjunction with hostname overrides which are required for certain cloud providers.
|
||||
1.1.24 | "Ensure that the admission control plugin PodSecurityPolicy is set (Scored)" | Enabling Pod Security Policy can cause applications to unexpectedly fail.
|
||||
1.1.34 | "Ensure that the --encryption-provider-config argument is set as appropriate (Scored)" | Enabling encryption changes how data can be recovered as data is encrypted.
|
||||
1.1.35 | "Ensure that the encryption provider is set to aescbc (Scored)" | Enabling encryption changes how data can be recovered as data is encrypted.
|
||||
1.1.36 | "Ensure that the admission control plugin EventRateLimit is set (Scored)" | EventRateLimit needs to be tuned depending on the cluster.
|
||||
1.2.2 | "Ensure that the --address argument is set to 127.0.0.1 (Scored)" | Adding this argument prevents Rancher's monitoring tool to collect metrics on the scheduler.
|
||||
1.3.7 | "Ensure that the --address argument is set to 127.0.0.1 (Scored)" | Adding this argument prevents Rancher's monitoring tool to collect metrics on the controller manager.
|
||||
1.4.12 | "Ensure that the etcd data directory ownership is set to etcd:etcd (Scored)" | A system service account is required for etcd data directory ownership. Refer to Rancher's hardening guide for more details on how to configure this ownership.
|
||||
1.7.2 | "Do not admit containers wishing to share the host process ID namespace (Scored)" | Enabling Pod Security Policy can cause applications to unexpectedly fail.
|
||||
1.7.3 | "Do not admit containers wishing to share the host IPC namespace (Scored)" | Enabling Pod Security Policy can cause applications to unexpectedly fail.
|
||||
1.7.4 | "Do not admit containers wishing to share the host network namespace (Scored)" | Enabling Pod Security Policy can cause applications to unexpectedly fail.
|
||||
1.7.5 | " Do not admit containers with allowPrivilegeEscalation (Scored)" | Enabling Pod Security Policy can cause applications to unexpectedly fail.
|
||||
2.1.6 | "Ensure that the --protect-kernel-defaults argument is set to true (Scored)" | System level configurations are required before provisioning the cluster in order for this argument to be set to true.
|
||||
2.1.10 | "Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate (Scored)" | When generating serving certificates, functionality could break in conjunction with hostname overrides which are required for certain cloud providers.
|
||||
|
||||
### CIS Benchmark v1.4 Not Applicable Tests
|
||||
|
||||
Number | Description | Reason for being not applicable
|
||||
---|---|---
|
||||
1.1.9 | "Ensure that the --repair-malformed-updates argument is set to false (Scored)" | The argument --repair-malformed-updates has been removed as of Kubernetes version 1.14
|
||||
1.3.6 | "Ensure that the RotateKubeletServerCertificate argument is set to true" | Cluster provisioned by RKE handles certificate rotation directly through RKE.
|
||||
1.4.1 | "Ensure that the API server pod specification file permissions are set to 644 or more restrictive (Scored)" | Cluster provisioned by RKE doesn't require or maintain a configuration file for kube-apiserver.
|
||||
1.4.2 | "Ensure that the API server pod specification file ownership is set to root:root (Scored)" | Cluster provisioned by RKE doesn't require or maintain a configuration file for kube-apiserver.
|
||||
1.4.3 | "Ensure that the controller manager pod specification file permissions are set to 644 or more restrictive (Scored)" | Cluster provisioned by RKE doesn't require or maintain a configuration file for controller-manager.
|
||||
1.4.4 | "Ensure that the controller manager pod specification file ownership is set to root:root (Scored)" | Cluster provisioned by RKE doesn't require or maintain a configuration file for controller-manager.
|
||||
1.4.5 | "Ensure that the scheduler pod specification file permissions are set to 644 or more restrictive (Scored)" | Cluster provisioned by RKE doesn't require or maintain a configuration file for scheduler.
|
||||
1.4.6 | "Ensure that the scheduler pod specification file ownership is set to root:root (Scored)" | Cluster provisioned by RKE doesn't require or maintain a configuration file for scheduler.
|
||||
1.4.7 | "Ensure that the etcd pod specification file permissions are set to 644 or more restrictive (Scored)" | Cluster provisioned by RKE doesn't require or maintain a configuration file for etcd.
|
||||
1.4.8 | "Ensure that the etcd pod specification file ownership is set to root:root (Scored)" | Cluster provisioned by RKE doesn't require or maintain a configuration file for etcd.
|
||||
1.4.13 | "Ensure that the admin.conf file permissions are set to 644 or more restrictive (Scored)" | Cluster provisioned by RKE does not store the kubernetes default kubeconfig credentials file on the nodes.
|
||||
1.4.14 | "Ensure that the admin.conf file ownership is set to root:root (Scored)" | Cluster provisioned by RKE does not store the kubernetes default kubeconfig credentials file on the nodes.
|
||||
2.1.8 | "Ensure that the --hostname-override argument is not set (Scored)" | Clusters provisioned by RKE clusters and most cloud providers require hostnames.
|
||||
2.1.12 | "Ensure that the --rotate-certificates argument is not set to false (Scored)" | Cluster provisioned by RKE handles certificate rotation directly through RKE.
|
||||
2.1.13 | "Ensure that the RotateKubeletServerCertificate argument is set to true (Scored)" | Cluster provisioned by RKE handles certificate rotation directly through RKE.
|
||||
2.2.3 | "Ensure that the kubelet service file permissions are set to 644 or more restrictive (Scored)" | Cluster provisioned by RKE doesn’t require or maintain a configuration file for the kubelet service.
|
||||
2.2.4 | "Ensure that the kubelet service file ownership is set to root:root (Scored)" | Cluster provisioned by RKE doesn’t require or maintain a configuration file for the kubelet service.
|
||||
2.2.9 | "Ensure that the kubelet configuration file ownership is set to root:root (Scored)" | RKE doesn’t require or maintain a configuration file for the kubelet.
|
||||
2.2.10 | "Ensure that the kubelet configuration file has permissions set to 644 or more restrictive (Scored)" | RKE doesn’t require or maintain a configuration file for the kubelet.
|
||||
+60
@@ -0,0 +1,60 @@
|
||||
---
|
||||
title: Default Alerts for Cluster Monitoring
|
||||
weight: 1
|
||||
aliases:
|
||||
- /rancher/v2.0-v2.4/en/cluster-admin/tools/alerts/default-alerts
|
||||
- /rancher/v2.0-v2.4/en/monitoring-alerting/legacy/alerts/cluster-alerts/default-alerts
|
||||
- /rancher/v2.0-v2.4/en/monitoring-alerting/v2.0.x-v2.4.x/cluster-alerts/default-alerts
|
||||
---
|
||||
|
||||
When you create a cluster, some alert rules are predefined. These alerts notify you about signs that the cluster could be unhealthy. You can receive these alerts if you configure a [notifier](../notifiers.md) for them.
|
||||
|
||||
Several of the alerts use Prometheus expressions as the metric that triggers the alert. For more information on how expressions work, you can refer to the Rancher [documentation about Prometheus expressions](monitoring-alerting/legacy/monitoring/cluster-monitoring/expression/) or the Prometheus [documentation about querying metrics](https://prometheus.io/docs/prometheus/latest/querying/basics/).
|
||||
|
||||
# Alerts for etcd
|
||||
Etcd is the key-value store that contains the state of the Kubernetes cluster. Rancher provides default alerts if the built-in monitoring detects a potential problem with etcd. You don't have to enable monitoring to receive these alerts.
|
||||
|
||||
A leader is the node that handles all client requests that need cluster consensus. For more information, you can refer to this [explanation of how etcd works.](https://rancher.com/blog/2019/2019-01-29-what-is-etcd/#how-does-etcd-work)
|
||||
|
||||
The leader of the cluster can change in response to certain events. It is normal for the leader to change, but too many changes can indicate a problem with the network or a high CPU load. With longer latencies, the default etcd configuration may cause frequent heartbeat timeouts, which trigger a new leader election.
|
||||
|
||||
| Alert | Explanation |
|
||||
|-------|-------------|
|
||||
| A high number of leader changes within the etcd cluster are happening | A warning alert is triggered when the leader changes more than three times in one hour. |
|
||||
| Database usage close to the quota 500M | A warning alert is triggered when the size of etcd exceeds 500M.|
|
||||
| Etcd is unavailable | A critical alert is triggered when etcd becomes unavailable. |
|
||||
| Etcd member has no leader | A critical alert is triggered when the etcd cluster does not have a leader for at least three minutes. |
|
||||
|
||||
|
||||
# Alerts for Kubernetes Components
|
||||
Rancher provides alerts when core Kubernetes system components become unhealthy.
|
||||
|
||||
Controllers update Kubernetes resources based on changes in etcd. The [controller manager](https://kubernetes.io/docs/reference/command-line-tools-reference/kube-controller-manager/) monitors the cluster desired state through the Kubernetes API server and makes the necessary changes to the current state to reach the desired state.
|
||||
|
||||
The [scheduler](https://kubernetes.io/docs/reference/command-line-tools-reference/kube-scheduler/) service is a core component of Kubernetes. It is responsible for scheduling cluster workloads to nodes, based on various configurations, metrics, resource requirements and workload-specific requirements.
|
||||
|
||||
| Alert | Explanation |
|
||||
|-------|-------------|
|
||||
| Controller Manager is unavailable | A critical warning is triggered when the cluster’s controller-manager becomes unavailable. |
|
||||
| Scheduler is unavailable | A critical warning is triggered when the cluster’s scheduler becomes unavailable. |
|
||||
|
||||
|
||||
# Alerts for Events
|
||||
Kubernetes events are objects that provide insight into what is happening inside a cluster, such as what decisions were made by the scheduler or why some pods were evicted from the node. In the Rancher UI, from the project view, you can see events for each workload.
|
||||
|
||||
| Alert | Explanation |
|
||||
|-------|-------------|
|
||||
| Get warning deployment event | A warning alert is triggered when a warning event happens on a deployment. |
|
||||
|
||||
|
||||
# Alerts for Nodes
|
||||
Alerts can be triggered based on node metrics. Each computing resource in a Kubernetes cluster is called a node. Nodes can be either bare-metal servers or virtual machines.
|
||||
|
||||
| Alert | Explanation |
|
||||
|-------|-------------|
|
||||
| High CPU load | A warning alert is triggered if the node uses more than 100 percent of the node’s available CPU seconds for at least three minutes. |
|
||||
| High node memory utilization | A warning alert is triggered if the node uses more than 80 percent of its available memory for at least three minutes. |
|
||||
| Node disk is running full within 24 hours | A critical alert is triggered if the disk space on the node is expected to run out in the next 24 hours based on the disk growth over the last 6 hours. |
|
||||
|
||||
# Project-level Alerts
|
||||
When you enable monitoring for the project, some project-level alerts are provided. For details, refer to the [section on project-level alerts.](project-admin/tools/alerts/)
|
||||
+47
@@ -0,0 +1,47 @@
|
||||
---
|
||||
title: Elasticsearch
|
||||
weight: 200
|
||||
aliases:
|
||||
- /rancher/v2.0-v2.4/en/tools/logging/elasticsearch/
|
||||
- /rancher/v2.0-v2.4/en/cluster-admin/tools/logging/elasticsearch
|
||||
- /rancher/v2.0-v2.4/en/logging/legacy/cluster-logging/elasticsearch
|
||||
- /rancher/v2.0-v2.4/en/logging/v2.0.x-v2.4.x/cluster-logging/elasticsearch
|
||||
- /rancher/v2.x/en/logging/v2.0.x-v2.4.x/cluster-logging/elasticsearch/
|
||||
---
|
||||
|
||||
If your organization uses [Elasticsearch](https://www.elastic.co/), either on premise or in the cloud, you can configure Rancher to send it Kubernetes logs. Afterwards, you can log into your Elasticsearch deployment to view logs.
|
||||
|
||||
>**Prerequisites:** Configure an [Elasticsearch deployment](https://www.elastic.co/guide/en/cloud/saas-release/ec-create-deployment.html).
|
||||
|
||||
## Elasticsearch Deployment Configuration
|
||||
|
||||
1. In the **Endpoint** field, enter the IP address and port of your Elasticsearch instance. You can find this information from the dashboard of your Elasticsearch deployment.
|
||||
|
||||
* Elasticsearch usually uses port `9200` for HTTP and `9243` for HTTPS.
|
||||
|
||||
1. If you are using [X-Pack Security](https://www.elastic.co/guide/en/x-pack/current/xpack-introduction.html), enter your Elasticsearch **Username** and **Password** for authentication.
|
||||
|
||||
1. Enter an [Index Pattern](https://www.elastic.co/guide/en/kibana/current/index-patterns.html).
|
||||
|
||||
## SSL Configuration
|
||||
|
||||
If your instance of Elasticsearch uses SSL, your **Endpoint** will need to begin with `https://`. With the correct endpoint, the **SSL Configuration** form is enabled and ready to be completed.
|
||||
|
||||
1. Provide the **Client Private Key** and **Client Certificate**. You can either copy and paste them or upload them by using the **Read from a file** button.
|
||||
|
||||
- You can use either a self-signed certificate or one provided by a certificate authority.
|
||||
|
||||
- You can generate a self-signed certificate using an openssl command. For example:
|
||||
|
||||
```
|
||||
openssl req -x509 -newkey rsa:2048 -keyout myservice.key -out myservice.cert -days 365 -nodes -subj "/CN=myservice.example.com"
|
||||
```
|
||||
|
||||
1. Enter your **Client Key Password**.
|
||||
|
||||
1. Enter your **SSL Version**. The default version is `TLSv1_2`.
|
||||
|
||||
1. Select whether or not you want to verify your SSL.
|
||||
|
||||
* If you are using a self-signed certificate, select **Enabled - Input trusted server certificate**, provide the **CA Certificate PEM**. You can copy and paste the certificate or upload it using the **Read from a file** button.
|
||||
* If you are using a certificate from a certificate authority, select **Enabled - Input trusted server certificate**. You do not need to provide a **CA Certificate PEM**.
|
||||
+39
@@ -0,0 +1,39 @@
|
||||
---
|
||||
title: Fluentd
|
||||
weight: 600
|
||||
aliases:
|
||||
- /rancher/v2.0-v2.4/en/cluster-admin/tools/logging/fluentd
|
||||
- /rancher/v2.0-v2.4/en/logging/legacy/cluster-logging/fluentd
|
||||
- /rancher/v2.0-v2.4/en/logging/v2.0.x-v2.4.x/cluster-logging/fluentd
|
||||
- /rancher/v2.x/en/logging/v2.0.x-v2.4.x/cluster-logging/fluentd/
|
||||
---
|
||||
|
||||
If your organization uses [Fluentd](https://www.fluentd.org/), you can configure Rancher to send it Kubernetes logs. Afterwards, you can log into your Fluentd server to view logs.
|
||||
|
||||
>**Prerequisites:** Configure Fluentd input forward to receive the event stream.
|
||||
>
|
||||
>See [Fluentd Documentation](https://docs.fluentd.org/v1.0/articles/in_forward) for details.
|
||||
|
||||
## Fluentd Configuration
|
||||
|
||||
You can add multiple Fluentd Servers. If you want to add additional Fluentd servers, click **Add Fluentd Server**. For each Fluentd server, complete the configuration information:
|
||||
|
||||
1. In the **Endpoint** field, enter the address and port of your Fluentd instance, e.g. `http://Fluentd-server:24224`.
|
||||
|
||||
1. Enter the **Shared Key** if your Fluentd Server is using a shared key for authentication.
|
||||
|
||||
1. Enter the **Username** and **Password** if your Fluentd Server is using username and password for authentication.
|
||||
|
||||
1. **Optional:** Enter the **Hostname** of the Fluentd server.
|
||||
|
||||
1. Enter the load balancing **Weight** of the Fluentd server. If the weight of one server is 20 and the other server is 30, events will be sent in a 2:3 ratio. If you do not enter a weight, the default weight is 60.
|
||||
|
||||
1. If this server is a standby server, check **Use as Standby Only**. Standby servers are used when all other servers are not available.
|
||||
|
||||
After adding all the Fluentd servers, you have the option to select **Enable Gzip Compression**. By default, this is enabled because the transferred payload size will be reduced.
|
||||
|
||||
## SSL Configuration
|
||||
|
||||
If your Fluentd servers are using TLS, you need to select **Use TLS**. If you are using a self-signed certificate, provide the **CA Certificate PEM**. You can copy and paste the certificate or upload it using the **Read from a file** button.
|
||||
|
||||
>**Note:** Fluentd does not support self-signed certificates when client authentication is enabled.
|
||||
+47
@@ -0,0 +1,47 @@
|
||||
---
|
||||
title: Kafka
|
||||
weight: 400
|
||||
aliases:
|
||||
- /rancher/v2.0-v2.4/en/tools/logging/kafka/
|
||||
- /rancher/v2.0-v2.4/en/cluster-admin/tools/logging/kafka
|
||||
- /rancher/v2.0-v2.4/en/logging/legacy/cluster-logging/kafka
|
||||
- /rancher/v2.0-v2.4/en/logging/v2.0.x-v2.4.x/cluster-logging/kafka
|
||||
- /rancher/v2.x/en/logging/v2.0.x-v2.4.x/cluster-logging/kafka/
|
||||
---
|
||||
|
||||
If your organization uses [Kafka](https://kafka.apache.org/), you can configure Rancher to send it Kubernetes logs. Afterwards, you can log into your Kafka server to view logs.
|
||||
|
||||
>**Prerequisite:** You must have a Kafka server configured.
|
||||
|
||||
## Kafka Server Configuration
|
||||
|
||||
1. Select the type of **Endpoint** your Kafka server is using:
|
||||
|
||||
* **Zookeeper**: Enter the IP address and port. By default, Zookeeper uses port `2181`. Please note that a Zookeeper endpoint cannot enable TLS.
|
||||
* **Broker**: Click on **Add Endpoint**. For each Kafka broker, enter the IP address and port. By default, Kafka brokers use port `9092`.
|
||||
|
||||
1. In the **Topic** field, enter the name of a Kafka [topic](https://kafka.apache.org/documentation/#basic_ops_add_topic) that your Kubernetes cluster submits logs to.
|
||||
|
||||
## **Broker** Endpoint Type
|
||||
|
||||
### SSL Configuration
|
||||
|
||||
If your Kafka cluster is using SSL for the **Broker**, you need to complete the **SSL Configuration** form.
|
||||
|
||||
1. Provide the **Client Private Key** and **Client Certificate**. You can either copy and paste them or upload them by using the **Read from a file** button.
|
||||
|
||||
1. Provide the **CA Certificate PEM**. You can either copy and paste the certificate or upload it using the **Read from a file** button.
|
||||
|
||||
>**Note:** Kafka does not support self-signed certificates when client authentication is enabled.
|
||||
|
||||
### SASL configuration
|
||||
|
||||
If your Kafka cluster is using [SASL authentication](https://kafka.apache.org/documentation/#security_sasl) for the Broker, you need to complete the **SASL Configuration** form.
|
||||
|
||||
1. Enter the SASL **Username** and **Password**.
|
||||
|
||||
1. Select the **SASL Type** that your Kafka cluster is using.
|
||||
|
||||
* If your Kafka is using **Plain**, please ensure your Kafka cluster is using SSL.
|
||||
|
||||
* If your Kafka is using **Scram**, you need to select which **Scram Mechanism** Kafka is using.
|
||||
+81
@@ -0,0 +1,81 @@
|
||||
---
|
||||
title: Splunk
|
||||
weight: 300
|
||||
aliases:
|
||||
- /rancher/v2.0-v2.4/en/tasks/logging/splunk/
|
||||
- /rancher/v2.0-v2.4/en/tools/logging/splunk/
|
||||
- /rancher/v2.0-v2.4/en/cluster-admin/tools/logging/splunk
|
||||
- /rancher/v2.0-v2.4/en/logging/legacy/cluster-logging/splunk
|
||||
- /rancher/v2.0-v2.4/en/logging/v2.0.x-v2.4.x/cluster-logging/splunk
|
||||
- /rancher/v2.x/en/logging/v2.0.x-v2.4.x/cluster-logging/splunk/
|
||||
- /rancher/v2.x/en/cluster-admin/tools/logging/splunk
|
||||
---
|
||||
|
||||
If your organization uses [Splunk](https://www.splunk.com/), you can configure Rancher to send it Kubernetes logs. Afterwards, you can log into your Splunk server to view logs.
|
||||
|
||||
>**Prerequisites:**
|
||||
>
|
||||
>- Configure HTTP event collection for your Splunk Server (Splunk Enterprise or Splunk Cloud).
|
||||
>- Either create a new token or copy an existing token.
|
||||
>
|
||||
>For more information, see [Splunk Documentation](http://docs.splunk.com/Documentation/Splunk/7.1.2/Data/UsetheHTTPEventCollector#About_Event_Collector_tokens).
|
||||
|
||||
## Splunk Configuration
|
||||
|
||||
1. In the **Endpoint** field, enter the IP address and port for you Splunk instance (i.e. `http://splunk-server:8088`)
|
||||
|
||||
* Splunk usually uses port `8088`. If you're using Splunk Cloud, you'll need to work with [Splunk support](https://www.splunk.com/en_us/support-and-services.html) to get an endpoint URL.
|
||||
|
||||
1. Enter the **Token** you obtained while completing the prerequisites (i.e., when you created a token in Splunk).
|
||||
|
||||
1. In the **Source** field, enter the name of the token as entered in Splunk.
|
||||
|
||||
1. **Optional:** Provide one or more [index](http://docs.splunk.com/Documentation/Splunk/7.1.2/Indexer/Aboutindexesandindexers) that's allowed for your token.
|
||||
|
||||
## SSL Configuration
|
||||
|
||||
If your instance of Splunk uses SSL, your **Endpoint** will need to begin with `https://`. With the correct endpoint, the **SSL Configuration** form is enabled and ready to be completed.
|
||||
|
||||
1. Provide the **Client Private Key** and **Client Certificate**. You can either copy and paste them or upload them by using the **Read from a file** button.
|
||||
|
||||
- You can use either a self-signed certificate or one provided by a certificate authority.
|
||||
|
||||
- You can generate a self-signed certificate using an openssl command. For example:
|
||||
|
||||
```
|
||||
openssl req -x509 -newkey rsa:2048 -keyout myservice.key -out myservice.cert -days 365 -nodes -subj "/CN=myservice.example.com"
|
||||
```
|
||||
|
||||
1. Enter your **Client Key Password**.
|
||||
|
||||
1. Select whether or not you want to verify your SSL.
|
||||
|
||||
* If you are using a self-signed certificate, select **Enabled - Input trusted server certificate**, provide the **CA Certificate PEM**. You can copy and paste the certificate or upload it using the **Read from a file** button.
|
||||
* If you are using a certificate from a certificate authority, select **Enabled - Input trusted server certificate**. You do not need to provide a **CA Certificate PEM**.
|
||||
|
||||
## Viewing Logs
|
||||
|
||||
1. Log into your Splunk server.
|
||||
|
||||
1. Click on **Search & Reporting**. The number of **Indexed Events** listed should be increasing.
|
||||
|
||||
1. Click on Data Summary and select the Sources tab.
|
||||
|
||||
|
||||
1. To view the actual logs, click on the source that you declared earlier.
|
||||
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
You can use curl to see if **HEC** is listening for HTTP event data.
|
||||
|
||||
```
|
||||
$ curl http://splunk-server:8088/services/collector/event \
|
||||
-H 'Authorization: Splunk 8da70994-b1b0-4a79-b154-bfaae8f93432' \
|
||||
-d '{"event": "hello world"}'
|
||||
```
|
||||
|
||||
If Splunk is configured correctly, you should receive **json** data returning `success code 0`. You should be able
|
||||
to send logging data to HEC.
|
||||
|
||||
If you received an error, check your configuration in Splunk and Rancher.
|
||||
+47
@@ -0,0 +1,47 @@
|
||||
---
|
||||
title: Syslog
|
||||
weight: 500
|
||||
aliases:
|
||||
- /rancher/v2.0-v2.4/en/tools/logging/syslog/
|
||||
- /rancher/v2.0-v2.4/en/cluster-admin/tools/logging/syslog
|
||||
- /rancher/v2.0-v2.4/en/logging/legacy/cluster-logging/syslog
|
||||
- /rancher/v2.0-v2.4/en/logging/v2.0.x-v2.4.x/cluster-logging/syslog
|
||||
- /rancher/v2.x/en/logging/v2.0.x-v2.4.x/cluster-logging/syslog/
|
||||
---
|
||||
|
||||
If your organization uses [Syslog](https://tools.ietf.org/html/rfc5424), you can configure Rancher to send it Kubernetes logs. Afterwards, you can log into your Syslog server to view logs.
|
||||
|
||||
>**Prerequisite:** You must have a Syslog server configured.
|
||||
|
||||
If you are using rsyslog, please make sure your rsyslog authentication mode is `x509/name`.
|
||||
|
||||
## Syslog Server Configuration
|
||||
|
||||
1. In the **Endpoint** field, enter the IP address and port for your Syslog server. Additionally, in the dropdown, select the protocol that your Syslog server uses.
|
||||
|
||||
1. In the **Program** field, enter the name of the application sending logs to your Syslog server, e.g. `Rancher`.
|
||||
|
||||
1. If you are using a cloud logging service, e.g. [Sumologic](https://www.sumologic.com/), enter a **Token** that authenticates with your Syslog server. You will need to create this token in the cloud logging service.
|
||||
|
||||
1. Select a **Log Severity** for events that are logged to the Syslog server. For more information on each severity level, see the [Syslog protocol documentation](https://tools.ietf.org/html/rfc5424#page-11).
|
||||
|
||||
- By specifying a **Log Severity** does not mean that will act as a filtering mechanism for logs. To do that you should use a parser on the Syslog server.
|
||||
|
||||
## Encryption Configuration
|
||||
|
||||
If your Syslog server is using **TCP** protocol and uses TLS, you need to select **Use TLS** and complete the **Encryption Configuration** form.
|
||||
|
||||
1. Provide the **Client Private Key** and **Client Certificate**. You can either copy and paste them or upload them by using the **Read from a file** button.
|
||||
|
||||
- You can use either a self-signed certificate or one provided by a certificate authority.
|
||||
|
||||
- You can generate a self-signed certificate using an openssl command. For example:
|
||||
|
||||
```
|
||||
openssl req -x509 -newkey rsa:2048 -keyout myservice.key -out myservice.cert -days 365 -nodes -subj "/CN=myservice.example.com"
|
||||
```
|
||||
|
||||
1. Select whether or not you want to verify your SSL.
|
||||
|
||||
* If you are using a self-signed certificate, select **Enabled - Input trusted server certificate**, provide the **CA Certificate PEM**. You can copy and paste the certificate or upload it using the **Read from a file** button.
|
||||
* If you are using a certificate from a certificate authority, select **Enabled - Input trusted server certificate**. You do not need to provide a **CA Certificate PEM**.
|
||||
+119
@@ -0,0 +1,119 @@
|
||||
---
|
||||
title: Cluster Metrics
|
||||
weight: 3
|
||||
aliases:
|
||||
- /rancher/v2.0-v2.4/en/project-admin/tools/monitoring/cluster-metrics
|
||||
- /rancher/v2.0-v2.4/en/cluster-admin/tools/monitoring/cluster-metrics
|
||||
- /rancher/v2.0-v2.4/en/monitoring-alerting/legacy/monitoring/cluster-monitoring/cluster-metrics
|
||||
- /rancher/v2.0-v2.4/en/monitoring-alerting/v2.0.x-v2.4.x/cluster-monitoring/cluster-metrics
|
||||
- /rancher/v2.x/en/monitoring-alerting/v2.0.x-v2.4.x/cluster-monitoring/cluster-metrics/
|
||||
---
|
||||
|
||||
_Available as of v2.2.0_
|
||||
|
||||
Cluster metrics display the hardware utilization for all nodes in your cluster, regardless of its role. They give you a global monitoring insight into the cluster.
|
||||
|
||||
Some of the biggest metrics to look out for:
|
||||
|
||||
- **CPU Utilization**
|
||||
|
||||
High load either indicates that your cluster is running efficiently or that you're running out of CPU resources.
|
||||
|
||||
- **Disk Utilization**
|
||||
|
||||
Be on the lookout for increased read and write rates on nodes nearing their disk capacity. This advice is especially true for etcd nodes, as running out of storage on an etcd node leads to cluster failure.
|
||||
|
||||
- **Memory Utilization**
|
||||
|
||||
Deltas in memory utilization usually indicate a memory leak.
|
||||
|
||||
- **Load Average**
|
||||
|
||||
Generally, you want your load average to match your number of logical CPUs for the cluster. For example, if your cluster has 8 logical CPUs, the ideal load average would be 8 as well. If you load average is well under the number of logical CPUs for the cluster, you may want to reduce cluster resources. On the other hand, if your average is over 8, your cluster may need more resources.
|
||||
|
||||
## Finding Node Metrics
|
||||
|
||||
1. From the **Global** view, navigate to the cluster that you want to view metrics.
|
||||
|
||||
1. Select **Nodes** in the navigation bar.
|
||||
|
||||
1. Select a specific node and click on its name.
|
||||
|
||||
1. Click on **Node Metrics**.
|
||||
|
||||
[_Get expressions for Cluster Metrics_](monitoring-alerting/v2.0.x-v2.4.x/cluster-monitoring/expression/#cluster-metrics)
|
||||
|
||||
### Etcd Metrics
|
||||
|
||||
>**Note:** Only supported for [Rancher launched Kubernetes clusters](../../../pages-for-subheaders/launch-kubernetes-with-rancher.md).
|
||||
|
||||
Etcd metrics display the operations of the etcd database on each of your cluster nodes. After establishing a baseline of normal etcd operational metrics, observe them for abnormal deltas between metric refreshes, which indicate potential issues with etcd. Always address etcd issues immediately!
|
||||
|
||||
You should also pay attention to the text at the top of the etcd metrics, which displays leader election statistics. This text indicates if etcd currently has a leader, which is the etcd instance that coordinates the other etcd instances in your cluster. A large increase in leader changes implies etcd is unstable. If you notice a change in leader election statistics, you should investigate them for issues.
|
||||
|
||||
Some of the biggest metrics to look out for:
|
||||
|
||||
- **Etcd has a leader**
|
||||
|
||||
etcd is usually deployed on multiple nodes and elects a leader to coordinate its operations. If etcd does not have a leader, its operations are not being coordinated.
|
||||
|
||||
- **Number of leader changes**
|
||||
|
||||
If this statistic suddenly grows, it usually indicates network communication issues that constantly force the cluster to elect a new leader.
|
||||
|
||||
[_Get expressions for Etcd Metrics_](monitoring-alerting/v2.0.x-v2.4.x/cluster-monitoring/expression/#etcd-metrics)
|
||||
|
||||
### Kubernetes Components Metrics
|
||||
|
||||
Kubernetes components metrics display data about the cluster's individual Kubernetes components. Primarily, it displays information about connections and latency for each component: the API server, controller manager, scheduler, and ingress controller.
|
||||
|
||||
>**Note:** The metrics for the controller manager, scheduler and ingress controller are only supported for [Rancher launched Kubernetes clusters](../../../pages-for-subheaders/launch-kubernetes-with-rancher.md).
|
||||
|
||||
When analyzing Kubernetes component metrics, don't be concerned about any single standalone metric in the charts and graphs that display. Rather, you should establish a baseline for metrics considered normal following a period of observation, e.g. the range of values that your components usually operate within and are considered normal. After you establish this baseline, be on the lookout for large deltas in the charts and graphs, as these big changes usually indicate a problem that you need to investigate.
|
||||
|
||||
Some of the more important component metrics to monitor are:
|
||||
|
||||
- **API Server Request Latency**
|
||||
|
||||
Increasing API response times indicate there's a generalized problem that requires investigation.
|
||||
|
||||
- **API Server Request Rate**
|
||||
|
||||
Rising API request rates usually coincide with increased API response times. Increased request rates also indicate a generalized problem requiring investigation.
|
||||
|
||||
- **Scheduler Preemption Attempts**
|
||||
|
||||
If you see a spike in scheduler preemptions, it's an indication that you're running out of hardware resources, as Kubernetes is recognizing it doesn't have enough resources to run all your pods and is prioritizing the more important ones.
|
||||
|
||||
- **Scheduling Failed Pods**
|
||||
|
||||
Failed pods can have a variety of causes, such as unbound persistent volume claims, exhausted hardware resources, non-responsive nodes, etc.
|
||||
|
||||
- **Ingress Controller Request Process Time**
|
||||
|
||||
How fast ingress is routing connections to your cluster services.
|
||||
|
||||
[_Get expressions for Kubernetes Component Metrics_](monitoring-alerting/legacy/monitoring/cluster-monitoring/expression/#kubernetes-components-metrics)
|
||||
|
||||
## Rancher Logging Metrics
|
||||
|
||||
Although the Dashboard for a cluster primarily displays data sourced from Prometheus, it also displays information for cluster logging, provided that you have [configured Rancher to use a logging service](cluster-admin/tools/logging/).
|
||||
|
||||
[_Get expressions for Rancher Logging Metrics_](monitoring-alerting/v2.0.x-v2.4.x/cluster-monitoring/expression/#rancher-logging-metrics)
|
||||
|
||||
## Finding Workload Metrics
|
||||
|
||||
Workload metrics display the hardware utilization for a Kubernetes workload. You can also view metrics for [deployments](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/), [stateful sets](https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/) and so on.
|
||||
|
||||
1. From the **Global** view, navigate to the project that you want to view workload metrics.
|
||||
|
||||
1. From the main navigation bar, choose **Resources > Workloads.** In versions before v2.3.0, choose **Workloads** on the main navigation bar.
|
||||
|
||||
1. Select a specific workload and click on its name.
|
||||
|
||||
1. In the **Pods** section, select a specific pod and click on its name.
|
||||
|
||||
- **View the Pod Metrics:** Click on **Pod Metrics**.
|
||||
- **View the Container Metrics:** In the **Containers** section, select a specific container and click on its name. Click on **Container Metrics**.
|
||||
|
||||
[_Get expressions for Workload Metrics_](monitoring-alerting/v2.0.x-v2.4.x/cluster-monitoring/expression/#workload-metrics)
|
||||
+493
@@ -0,0 +1,493 @@
|
||||
---
|
||||
title: Prometheus Custom Metrics Adapter
|
||||
weight: 5
|
||||
aliases:
|
||||
- /rancher/v2.0-v2.4/en/project-admin/tools/monitoring/custom-metrics
|
||||
- /rancher/v2.0-v2.4/en/monitoring-alerting/legacy/monitoring/cluster-monitoring/custom-metrics
|
||||
- /rancher/v2.0-v2.4/en/cluster-admin/tools/monitoring/custom-metrics/
|
||||
- /rancher/v2.0-v2.4/en/monitoring-alerting/v2.0.x-v2.4.x/cluster-monitoring/custom-metrics
|
||||
- /rancher/v2.x/en/monitoring-alerting/v2.0.x-v2.4.x/cluster-monitoring/custom-metrics/
|
||||
---
|
||||
|
||||
After you've enabled [cluster level monitoring](monitoring-alerting/legacy/monitoring/cluster-monitoring/), You can view the metrics data from Rancher. You can also deploy the Prometheus custom metrics adapter then you can use the HPA with metrics stored in cluster monitoring.
|
||||
|
||||
## Deploy Prometheus Custom Metrics Adapter
|
||||
|
||||
We are going to use the [Prometheus custom metrics adapter](https://github.com/DirectXMan12/k8s-prometheus-adapter/releases/tag/v0.5.0), version v0.5.0. This is a great example for the [custom metrics server](https://github.com/kubernetes-incubator/custom-metrics-apiserver). And you must be the *cluster owner* to execute following steps.
|
||||
|
||||
- Get the service account of the cluster monitoring is using. It should be configured in the workload ID: `statefulset:cattle-prometheus:prometheus-cluster-monitoring`. And if you didn't customize anything, the service account name should be `cluster-monitoring`.
|
||||
|
||||
- Grant permission to that service account. You will need two kinds of permission.
|
||||
One role is `extension-apiserver-authentication-reader` in `kube-system`, so you will need to create a `Rolebinding` to in `kube-system`. This permission is to get api aggregation configuration from config map in `kube-system`.
|
||||
|
||||
```yaml
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: RoleBinding
|
||||
metadata:
|
||||
name: custom-metrics-auth-reader
|
||||
namespace: kube-system
|
||||
roleRef:
|
||||
apiGroup: rbac.authorization.k8s.io
|
||||
kind: Role
|
||||
name: extension-apiserver-authentication-reader
|
||||
subjects:
|
||||
- kind: ServiceAccount
|
||||
name: cluster-monitoring
|
||||
namespace: cattle-prometheus
|
||||
```
|
||||
|
||||
The other one is cluster role `system:auth-delegator`, so you will need to create a `ClusterRoleBinding`. This permission is to have subject access review permission.
|
||||
|
||||
```yaml
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: ClusterRoleBinding
|
||||
metadata:
|
||||
name: custom-metrics:system:auth-delegator
|
||||
roleRef:
|
||||
apiGroup: rbac.authorization.k8s.io
|
||||
kind: ClusterRole
|
||||
name: system:auth-delegator
|
||||
subjects:
|
||||
- kind: ServiceAccount
|
||||
name: cluster-monitoring
|
||||
namespace: cattle-prometheus
|
||||
```
|
||||
|
||||
- Create configuration for custom metrics adapter. Following is an example configuration. There will be a configuration details in next session.
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: adapter-config
|
||||
namespace: cattle-prometheus
|
||||
data:
|
||||
config.yaml: |
|
||||
rules:
|
||||
- seriesQuery: '{__name__=~"^container_.*",container_name!="POD",namespace!="",pod_name!=""}'
|
||||
seriesFilters: []
|
||||
resources:
|
||||
overrides:
|
||||
namespace:
|
||||
resource: namespace
|
||||
pod_name:
|
||||
resource: pod
|
||||
name:
|
||||
matches: ^container_(.*)_seconds_total$
|
||||
as: ""
|
||||
metricsQuery: sum(rate(<<.Series>>{<<.LabelMatchers>>,container_name!="POD"}[1m])) by (<<.GroupBy>>)
|
||||
- seriesQuery: '{__name__=~"^container_.*",container_name!="POD",namespace!="",pod_name!=""}'
|
||||
seriesFilters:
|
||||
- isNot: ^container_.*_seconds_total$
|
||||
resources:
|
||||
overrides:
|
||||
namespace:
|
||||
resource: namespace
|
||||
pod_name:
|
||||
resource: pod
|
||||
name:
|
||||
matches: ^container_(.*)_total$
|
||||
as: ""
|
||||
metricsQuery: sum(rate(<<.Series>>{<<.LabelMatchers>>,container_name!="POD"}[1m])) by (<<.GroupBy>>)
|
||||
- seriesQuery: '{__name__=~"^container_.*",container_name!="POD",namespace!="",pod_name!=""}'
|
||||
seriesFilters:
|
||||
- isNot: ^container_.*_total$
|
||||
resources:
|
||||
overrides:
|
||||
namespace:
|
||||
resource: namespace
|
||||
pod_name:
|
||||
resource: pod
|
||||
name:
|
||||
matches: ^container_(.*)$
|
||||
as: ""
|
||||
metricsQuery: sum(<<.Series>>{<<.LabelMatchers>>,container_name!="POD"}) by (<<.GroupBy>>)
|
||||
- seriesQuery: '{namespace!="",__name__!~"^container_.*"}'
|
||||
seriesFilters:
|
||||
- isNot: .*_total$
|
||||
resources:
|
||||
template: <<.Resource>>
|
||||
name:
|
||||
matches: ""
|
||||
as: ""
|
||||
metricsQuery: sum(<<.Series>>{<<.LabelMatchers>>}) by (<<.GroupBy>>)
|
||||
- seriesQuery: '{namespace!="",__name__!~"^container_.*"}'
|
||||
seriesFilters:
|
||||
- isNot: .*_seconds_total
|
||||
resources:
|
||||
template: <<.Resource>>
|
||||
name:
|
||||
matches: ^(.*)_total$
|
||||
as: ""
|
||||
metricsQuery: sum(rate(<<.Series>>{<<.LabelMatchers>>}[1m])) by (<<.GroupBy>>)
|
||||
- seriesQuery: '{namespace!="",__name__!~"^container_.*"}'
|
||||
seriesFilters: []
|
||||
resources:
|
||||
template: <<.Resource>>
|
||||
name:
|
||||
matches: ^(.*)_seconds_total$
|
||||
as: ""
|
||||
metricsQuery: sum(rate(<<.Series>>{<<.LabelMatchers>>}[1m])) by (<<.GroupBy>>)
|
||||
resourceRules:
|
||||
cpu:
|
||||
containerQuery: sum(rate(container_cpu_usage_seconds_total{<<.LabelMatchers>>}[1m])) by (<<.GroupBy>>)
|
||||
nodeQuery: sum(rate(container_cpu_usage_seconds_total{<<.LabelMatchers>>, id='/'}[1m])) by (<<.GroupBy>>)
|
||||
resources:
|
||||
overrides:
|
||||
instance:
|
||||
resource: node
|
||||
namespace:
|
||||
resource: namespace
|
||||
pod_name:
|
||||
resource: pod
|
||||
containerLabel: container_name
|
||||
memory:
|
||||
containerQuery: sum(container_memory_working_set_bytes{<<.LabelMatchers>>}) by (<<.GroupBy>>)
|
||||
nodeQuery: sum(container_memory_working_set_bytes{<<.LabelMatchers>>,id='/'}) by (<<.GroupBy>>)
|
||||
resources:
|
||||
overrides:
|
||||
instance:
|
||||
resource: node
|
||||
namespace:
|
||||
resource: namespace
|
||||
pod_name:
|
||||
resource: pod
|
||||
containerLabel: container_name
|
||||
window: 1m
|
||||
```
|
||||
|
||||
- Create HTTPS TLS certs for your api server. You can use following command to create a self-signed cert.
|
||||
|
||||
```bash
|
||||
openssl req -new -newkey rsa:4096 -x509 -sha256 -days 365 -nodes -out serving.crt -keyout serving.key -subj "/C=CN/CN=custom-metrics-apiserver.cattle-prometheus.svc.cluster.local"
|
||||
# And you will find serving.crt and serving.key in your path. And then you are going to create a secret in cattle-prometheus namespace.
|
||||
kubectl create secret generic -n cattle-prometheus cm-adapter-serving-certs --from-file=serving.key=./serving.key --from-file=serving.crt=./serving.crt
|
||||
```
|
||||
|
||||
- Then you can create the prometheus custom metrics adapter. And you will need a service for this deployment too. Creating it via Import YAML or Rancher would do. Please create those resources in `cattle-prometheus` namespaces.
|
||||
|
||||
Here is the prometheus custom metrics adapter deployment.
|
||||
```yaml
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
labels:
|
||||
app: custom-metrics-apiserver
|
||||
name: custom-metrics-apiserver
|
||||
namespace: cattle-prometheus
|
||||
spec:
|
||||
replicas: 1
|
||||
selector:
|
||||
matchLabels:
|
||||
app: custom-metrics-apiserver
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: custom-metrics-apiserver
|
||||
name: custom-metrics-apiserver
|
||||
spec:
|
||||
serviceAccountName: cluster-monitoring
|
||||
containers:
|
||||
- name: custom-metrics-apiserver
|
||||
image: directxman12/k8s-prometheus-adapter-amd64:v0.5.0
|
||||
args:
|
||||
- --secure-port=6443
|
||||
- --tls-cert-file=/var/run/serving-cert/serving.crt
|
||||
- --tls-private-key-file=/var/run/serving-cert/serving.key
|
||||
- --logtostderr=true
|
||||
- --prometheus-url=http://prometheus-operated/
|
||||
- --metrics-relist-interval=1m
|
||||
- --v=10
|
||||
- --config=/etc/adapter/config.yaml
|
||||
ports:
|
||||
- containerPort: 6443
|
||||
volumeMounts:
|
||||
- mountPath: /var/run/serving-cert
|
||||
name: volume-serving-cert
|
||||
readOnly: true
|
||||
- mountPath: /etc/adapter/
|
||||
name: config
|
||||
readOnly: true
|
||||
- mountPath: /tmp
|
||||
name: tmp-vol
|
||||
volumes:
|
||||
- name: volume-serving-cert
|
||||
secret:
|
||||
secretName: cm-adapter-serving-certs
|
||||
- name: config
|
||||
configMap:
|
||||
name: adapter-config
|
||||
- name: tmp-vol
|
||||
emptyDir: {}
|
||||
|
||||
```
|
||||
|
||||
Here is the service of the deployment.
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: custom-metrics-apiserver
|
||||
namespace: cattle-prometheus
|
||||
spec:
|
||||
ports:
|
||||
- port: 443
|
||||
targetPort: 6443
|
||||
selector:
|
||||
app: custom-metrics-apiserver
|
||||
```
|
||||
|
||||
- Create API service for your custom metric server.
|
||||
|
||||
```yaml
|
||||
apiVersion: apiregistration.k8s.io/v1beta1
|
||||
kind: APIService
|
||||
metadata:
|
||||
name: v1beta1.custom.metrics.k8s.io
|
||||
spec:
|
||||
service:
|
||||
name: custom-metrics-apiserver
|
||||
namespace: cattle-prometheus
|
||||
group: custom.metrics.k8s.io
|
||||
version: v1beta1
|
||||
insecureSkipTLSVerify: true
|
||||
groupPriorityMinimum: 100
|
||||
versionPriority: 100
|
||||
|
||||
```
|
||||
|
||||
- Then you can verify your custom metrics server by `kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1`. If you see the return datas from the api, it means that the metrics server has been successfully set up.
|
||||
|
||||
- You create HPA with custom metrics now. Here is an example of HPA. You will need to create a nginx deployment in your namespace first.
|
||||
|
||||
```yaml
|
||||
kind: HorizontalPodAutoscaler
|
||||
apiVersion: autoscaling/v2beta1
|
||||
metadata:
|
||||
name: nginx
|
||||
spec:
|
||||
scaleTargetRef:
|
||||
# point the HPA at the nginx deployment you just created
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
name: nginx
|
||||
# autoscale between 1 and 10 replicas
|
||||
minReplicas: 1
|
||||
maxReplicas: 10
|
||||
metrics:
|
||||
# use a "Pods" metric, which takes the average of the
|
||||
# given metric across all pods controlled by the autoscaling target
|
||||
- type: Pods
|
||||
pods:
|
||||
metricName: memory_usage_bytes
|
||||
targetAverageValue: 5000000
|
||||
```
|
||||
|
||||
And then, you should see your nginx is scaling up. HPA with custom metrics works.
|
||||
|
||||
## Configuration of prometheus custom metrics adapter
|
||||
|
||||
> Refer to https://github.com/DirectXMan12/k8s-prometheus-adapter/blob/master/docs/config.md
|
||||
|
||||
The adapter determines which metrics to expose, and how to expose them,
|
||||
through a set of "discovery" rules. Each rule is executed independently
|
||||
(so make sure that your rules are mutually exclusive), and specifies each
|
||||
of the steps the adapter needs to take to expose a metric in the API.
|
||||
|
||||
Each rule can be broken down into roughly four parts:
|
||||
|
||||
- *Discovery*, which specifies how the adapter should find all Prometheus
|
||||
metrics for this rule.
|
||||
|
||||
- *Association*, which specifies how the adapter should determine which
|
||||
Kubernetes resources a particular metric is associated with.
|
||||
|
||||
- *Naming*, which specifies how the adapter should expose the metric in
|
||||
the custom metrics API.
|
||||
|
||||
- *Querying*, which specifies how a request for a particular metric on one
|
||||
or more Kubernetes objects should be turned into a query to Prometheus.
|
||||
|
||||
A basic config with one rule might look like:
|
||||
|
||||
```yaml
|
||||
rules:
|
||||
# this rule matches cumulative cAdvisor metrics measured in seconds
|
||||
- seriesQuery: '{__name__=~"^container_.*",container_name!="POD",namespace!="",pod_name!=""}'
|
||||
resources:
|
||||
# skip specifying generic resource<->label mappings, and just
|
||||
# attach only pod and namespace resources by mapping label names to group-resources
|
||||
overrides:
|
||||
namespace: {resource: "namespace"},
|
||||
pod_name: {resource: "pod"},
|
||||
# specify that the `container_` and `_seconds_total` suffixes should be removed.
|
||||
# this also introduces an implicit filter on metric family names
|
||||
name:
|
||||
# we use the value of the capture group implicitly as the API name
|
||||
# we could also explicitly write `as: "$1"`
|
||||
matches: "^container_(.*)_seconds_total$"
|
||||
# specify how to construct a query to fetch samples for a given series
|
||||
# This is a Go template where the `.Series` and `.LabelMatchers` string values
|
||||
# are available, and the delimiters are `<<` and `>>` to avoid conflicts with
|
||||
# the prometheus query language
|
||||
metricsQuery: "sum(rate(<<.Series>>{<<.LabelMatchers>>,container_name!="POD"}[2m])) by (<<.GroupBy>>)"
|
||||
```
|
||||
|
||||
### Discovery
|
||||
|
||||
Discovery governs the process of finding the metrics that you want to
|
||||
expose in the custom metrics API. There are two fields that factor into
|
||||
discovery: `seriesQuery` and `seriesFilters`.
|
||||
|
||||
`seriesQuery` specifies Prometheus series query (as passed to the
|
||||
`/api/v1/series` endpoint in Prometheus) to use to find some set of
|
||||
Prometheus series. The adapter will strip the label values from this
|
||||
series, and then use the resulting metric-name-label-names combinations
|
||||
later on.
|
||||
|
||||
In many cases, `seriesQuery` will be sufficient to narrow down the list of
|
||||
Prometheus series. However, sometimes (especially if two rules might
|
||||
otherwise overlap), it's useful to do additional filtering on metric
|
||||
names. In this case, `seriesFilters` can be used. After the list of
|
||||
series is returned from `seriesQuery`, each series has its metric name
|
||||
filtered through any specified filters.
|
||||
|
||||
Filters may be either:
|
||||
|
||||
- `is: <regex>`, which matches any series whose name matches the specified
|
||||
regex.
|
||||
|
||||
- `isNot: <regex>`, which matches any series whose name does not match the
|
||||
specified regex.
|
||||
|
||||
For example:
|
||||
|
||||
```yaml
|
||||
# match all cAdvisor metrics that aren't measured in seconds
|
||||
seriesQuery: '{__name__=~"^container_.*_total",container_name!="POD",namespace!="",pod_name!=""}'
|
||||
seriesFilters:
|
||||
isNot: "^container_.*_seconds_total"
|
||||
```
|
||||
|
||||
### Association
|
||||
|
||||
Association governs the process of figuring out which Kubernetes resources
|
||||
a particular metric could be attached to. The `resources` field controls
|
||||
this process.
|
||||
|
||||
There are two ways to associate resources with a particular metric. In
|
||||
both cases, the value of the label becomes the name of the particular
|
||||
object.
|
||||
|
||||
One way is to specify that any label name that matches some particular
|
||||
pattern refers to some group-resource based on the label name. This can
|
||||
be done using the `template` field. The pattern is specified as a Go
|
||||
template, with the `Group` and `Resource` fields representing group and
|
||||
resource. You don't necessarily have to use the `Group` field (in which
|
||||
case the group is guessed by the system). For instance:
|
||||
|
||||
```yaml
|
||||
# any label `kube_<group>_<resource>` becomes <group>.<resource> in Kubernetes
|
||||
resources:
|
||||
template: "kube_<<.Group>>_<<.Resource>>"
|
||||
```
|
||||
|
||||
The other way is to specify that some particular label represents some
|
||||
particular Kubernetes resource. This can be done using the `overrides`
|
||||
field. Each override maps a Prometheus label to a Kubernetes
|
||||
group-resource. For instance:
|
||||
|
||||
```yaml
|
||||
# the microservice label corresponds to the apps.deployment resource
|
||||
resource:
|
||||
overrides:
|
||||
microservice: {group: "apps", resource: "deployment"}
|
||||
```
|
||||
|
||||
These two can be combined, so you can specify both a template and some
|
||||
individual overrides.
|
||||
|
||||
The resources mentioned can be any resource available in your kubernetes
|
||||
cluster, as long as you've got a corresponding label.
|
||||
|
||||
### Naming
|
||||
|
||||
Naming governs the process of converting a Prometheus metric name into
|
||||
a metric in the custom metrics API, and vice versa. It's controlled by
|
||||
the `name` field.
|
||||
|
||||
Naming is controlled by specifying a pattern to extract an API name from
|
||||
a Prometheus name, and potentially a transformation on that extracted
|
||||
value.
|
||||
|
||||
The pattern is specified in the `matches` field, and is just a regular
|
||||
expression. If not specified, it defaults to `.*`.
|
||||
|
||||
The transformation is specified by the `as` field. You can use any
|
||||
capture groups defined in the `matches` field. If the `matches` field
|
||||
doesn't contain capture groups, the `as` field defaults to `$0`. If it
|
||||
contains a single capture group, the `as` field defautls to `$1`.
|
||||
Otherwise, it's an error not to specify the as field.
|
||||
|
||||
For example:
|
||||
|
||||
```yaml
|
||||
# match turn any name <name>_total to <name>_per_second
|
||||
# e.g. http_requests_total becomes http_requests_per_second
|
||||
name:
|
||||
matches: "^(.*)_total$"
|
||||
as: "${1}_per_second"
|
||||
```
|
||||
|
||||
### Querying
|
||||
|
||||
Querying governs the process of actually fetching values for a particular
|
||||
metric. It's controlled by the `metricsQuery` field.
|
||||
|
||||
The `metricsQuery` field is a Go template that gets turned into
|
||||
a Prometheus query, using input from a particular call to the custom
|
||||
metrics API. A given call to the custom metrics API is distilled down to
|
||||
a metric name, a group-resource, and one or more objects of that
|
||||
group-resource. These get turned into the following fields in the
|
||||
template:
|
||||
|
||||
- `Series`: the metric name
|
||||
- `LabelMatchers`: a comma-separated list of label matchers matching the
|
||||
given objects. Currently, this is the label for the particular
|
||||
group-resource, plus the label for namespace, if the group-resource is
|
||||
namespaced.
|
||||
- `GroupBy`: a comma-separated list of labels to group by. Currently,
|
||||
this contains the group-resource label used in `LabelMatchers`.
|
||||
|
||||
For instance, suppose we had a series `http_requests_total` (exposed as
|
||||
`http_requests_per_second` in the API) with labels `service`, `pod`,
|
||||
`ingress`, `namespace`, and `verb`. The first four correspond to
|
||||
Kubernetes resources. Then, if someone requested the metric
|
||||
`pods/http_request_per_second` for the pods `pod1` and `pod2` in the
|
||||
`somens` namespace, we'd have:
|
||||
|
||||
- `Series: "http_requests_total"`
|
||||
- `LabelMatchers: "pod=~\"pod1|pod2",namespace="somens"`
|
||||
- `GroupBy`: `pod`
|
||||
|
||||
Additionally, there are two advanced fields that are "raw" forms of other
|
||||
fields:
|
||||
|
||||
- `LabelValuesByName`: a map mapping the labels and values from the
|
||||
`LabelMatchers` field. The values are pre-joined by `|`
|
||||
(for used with the `=~` matcher in Prometheus).
|
||||
- `GroupBySlice`: the slice form of `GroupBy`.
|
||||
|
||||
In general, you'll probably want to use the `Series`, `LabelMatchers`, and
|
||||
`GroupBy` fields. The other two are for advanced usage.
|
||||
|
||||
The query is expected to return one value for each object requested. The
|
||||
adapter will use the labels on the returned series to associate a given
|
||||
series back to its corresponding object.
|
||||
|
||||
For example:
|
||||
|
||||
```yaml
|
||||
# convert cumulative cAdvisor metrics into rates calculated over 2 minutes
|
||||
metricsQuery: "sum(rate(<<.Series>>{<<.LabelMatchers>>,container_name!="POD"}[2m])) by (<<.GroupBy>>)"
|
||||
```
|
||||
+436
@@ -0,0 +1,436 @@
|
||||
---
|
||||
title: Prometheus Expressions
|
||||
weight: 4
|
||||
aliases:
|
||||
- /rancher/v2.0-v2.4/en/project-admin/tools/monitoring/expression
|
||||
- /rancher/v2.0-v2.4/en/cluster-admin/tools/monitoring/expression
|
||||
- /rancher/v2.0-v2.4/en/monitoring-alerting/legacy/monitoring/cluster-monitoring/expression
|
||||
- /rancher/v2.0-v2.4/en/monitoring-alerting/v2.0.x-v2.4.x/cluster-monitoring/expression
|
||||
- /rancher/v2.x/en/monitoring-alerting/v2.0.x-v2.4.x/cluster-monitoring/expression/
|
||||
---
|
||||
|
||||
The PromQL expressions in this doc can be used to configure [alerts.](cluster-admin/tools/alerts/)
|
||||
|
||||
> Before expressions can be used in alerts, monitoring must be enabled. For more information, refer to the documentation on enabling monitoring [at the cluster level](monitoring-alerting/legacy/monitoring/cluster-monitoring/) or [at the project level.](project-admin/tools/monitoring/)
|
||||
|
||||
For more information about querying Prometheus, refer to the official [Prometheus documentation.](https://prometheus.io/docs/prometheus/latest/querying/basics/)
|
||||
|
||||
<!-- TOC -->
|
||||
|
||||
- [Cluster Metrics](#cluster-metrics)
|
||||
- [Cluster CPU Utilization](#cluster-cpu-utilization)
|
||||
- [Cluster Load Average](#cluster-load-average)
|
||||
- [Cluster Memory Utilization](#cluster-memory-utilization)
|
||||
- [Cluster Disk Utilization](#cluster-disk-utilization)
|
||||
- [Cluster Disk I/O](#cluster-disk-i-o)
|
||||
- [Cluster Network Packets](#cluster-network-packets)
|
||||
- [Cluster Network I/O](#cluster-network-i-o)
|
||||
- [Node Metrics](#node-metrics)
|
||||
- [Node CPU Utilization](#node-cpu-utilization)
|
||||
- [Node Load Average](#node-load-average)
|
||||
- [Node Memory Utilization](#node-memory-utilization)
|
||||
- [Node Disk Utilization](#node-disk-utilization)
|
||||
- [Node Disk I/O](#node-disk-i-o)
|
||||
- [Node Network Packets](#node-network-packets)
|
||||
- [Node Network I/O](#node-network-i-o)
|
||||
- [Etcd Metrics](#etcd-metrics)
|
||||
- [Etcd Has a Leader](#etcd-has-a-leader)
|
||||
- [Number of Times the Leader Changes](#number-of-times-the-leader-changes)
|
||||
- [Number of Failed Proposals](#number-of-failed-proposals)
|
||||
- [GRPC Client Traffic](#grpc-client-traffic)
|
||||
- [Peer Traffic](#peer-traffic)
|
||||
- [DB Size](#db-size)
|
||||
- [Active Streams](#active-streams)
|
||||
- [Raft Proposals](#raft-proposals)
|
||||
- [RPC Rate](#rpc-rate)
|
||||
- [Disk Operations](#disk-operations)
|
||||
- [Disk Sync Duration](#disk-sync-duration)
|
||||
- [Kubernetes Components Metrics](#kubernetes-components-metrics)
|
||||
- [API Server Request Latency](#api-server-request-latency)
|
||||
- [API Server Request Rate](#api-server-request-rate)
|
||||
- [Scheduling Failed Pods](#scheduling-failed-pods)
|
||||
- [Controller Manager Queue Depth](#controller-manager-queue-depth)
|
||||
- [Scheduler E2E Scheduling Latency](#scheduler-e2e-scheduling-latency)
|
||||
- [Scheduler Preemption Attempts](#scheduler-preemption-attempts)
|
||||
- [Ingress Controller Connections](#ingress-controller-connections)
|
||||
- [Ingress Controller Request Process Time](#ingress-controller-request-process-time)
|
||||
- [Rancher Logging Metrics](#rancher-logging-metrics)
|
||||
- [Fluentd Buffer Queue Rate](#fluentd-buffer-queue-rate)
|
||||
- [Fluentd Input Rate](#fluentd-input-rate)
|
||||
- [Fluentd Output Errors Rate](#fluentd-output-errors-rate)
|
||||
- [Fluentd Output Rate](#fluentd-output-rate)
|
||||
- [Workload Metrics](#workload-metrics)
|
||||
- [Workload CPU Utilization](#workload-cpu-utilization)
|
||||
- [Workload Memory Utilization](#workload-memory-utilization)
|
||||
- [Workload Network Packets](#workload-network-packets)
|
||||
- [Workload Network I/O](#workload-network-i-o)
|
||||
- [Workload Disk I/O](#workload-disk-i-o)
|
||||
- [Pod Metrics](#pod-metrics)
|
||||
- [Pod CPU Utilization](#pod-cpu-utilization)
|
||||
- [Pod Memory Utilization](#pod-memory-utilization)
|
||||
- [Pod Network Packets](#pod-network-packets)
|
||||
- [Pod Network I/O](#pod-network-i-o)
|
||||
- [Pod Disk I/O](#pod-disk-i-o)
|
||||
- [Container Metrics](#container-metrics)
|
||||
- [Container CPU Utilization](#container-cpu-utilization)
|
||||
- [Container Memory Utilization](#container-memory-utilization)
|
||||
- [Container Disk I/O](#container-disk-i-o)
|
||||
|
||||
<!-- /TOC -->
|
||||
|
||||
# Cluster Metrics
|
||||
|
||||
### Cluster CPU Utilization
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | `1 - (avg(irate(node_cpu_seconds_total{mode="idle"}[5m])) by (instance))` |
|
||||
| Summary | `1 - (avg(irate(node_cpu_seconds_total{mode="idle"}[5m])))` |
|
||||
|
||||
### Cluster Load Average
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | <table><tr><td>load1</td><td>`sum(node_load1) by (instance) / count(node_cpu_seconds_total{mode="system"}) by (instance)`</td></tr><tr><td>load5</td><td>`sum(node_load5) by (instance) / count(node_cpu_seconds_total{mode="system"}) by (instance)`</td></tr><tr><td>load15</td><td>`sum(node_load15) by (instance) / count(node_cpu_seconds_total{mode="system"}) by (instance)`</td></tr></table> |
|
||||
| Summary | <table><tr><td>load1</td><td>`sum(node_load1) by (instance) / count(node_cpu_seconds_total{mode="system"})`</td></tr><tr><td>load5</td><td>`sum(node_load5) by (instance) / count(node_cpu_seconds_total{mode="system"})`</td></tr><tr><td>load15</td><td>`sum(node_load15) by (instance) / count(node_cpu_seconds_total{mode="system"})`</td></tr></table> |
|
||||
|
||||
### Cluster Memory Utilization
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | `1 - sum(node_memory_MemAvailable_bytes) by (instance) / sum(node_memory_MemTotal_bytes) by (instance)` |
|
||||
| Summary | `1 - sum(node_memory_MemAvailable_bytes) / sum(node_memory_MemTotal_bytes)` |
|
||||
|
||||
### Cluster Disk Utilization
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | `(sum(node_filesystem_size_bytes{device!="rootfs"}) by (instance) - sum(node_filesystem_free_bytes{device!="rootfs"}) by (instance)) / sum(node_filesystem_size_bytes{device!="rootfs"}) by (instance)` |
|
||||
| Summary | `(sum(node_filesystem_size_bytes{device!="rootfs"}) - sum(node_filesystem_free_bytes{device!="rootfs"})) / sum(node_filesystem_size_bytes{device!="rootfs"})` |
|
||||
|
||||
### Cluster Disk I/O
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | <table><tr><td>read</td><td>`sum(rate(node_disk_read_bytes_total[5m])) by (instance)`</td></tr><tr><td>written</td><td>`sum(rate(node_disk_written_bytes_total[5m])) by (instance)`</td></tr></table> |
|
||||
| Summary | <table><tr><td>read</td><td>`sum(rate(node_disk_read_bytes_total[5m]))`</td></tr><tr><td>written</td><td>`sum(rate(node_disk_written_bytes_total[5m]))`</td></tr></table> |
|
||||
|
||||
### Cluster Network Packets
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | <table><tr><td>receive-dropped</td><td><code>sum(rate(node_network_receive_drop_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*"}[5m])) by (instance)</code></td></tr><tr><td>receive-errs</td><td><code>sum(rate(node_network_receive_errs_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*"}[5m])) by (instance)</code></td></tr><tr><td>receive-packets</td><td><code>sum(rate(node_network_receive_packets_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*"}[5m])) by (instance)</code></td></tr><tr><td>transmit-dropped</td><td><code>sum(rate(node_network_transmit_drop_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*"}[5m])) by (instance)</code></td></tr><tr><td>transmit-errs</td><td><code>sum(rate(node_network_transmit_errs_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*"}[5m])) by (instance)</code></td></tr><tr><td>transmit-packets</td><td><code>sum(rate(node_network_transmit_packets_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*"}[5m])) by (instance)</code></td></tr></table> |
|
||||
| Summary | <table><tr><td>receive-dropped</td><td><code>sum(rate(node_network_receive_drop_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*"}[5m]))</code></td></tr><tr><td>receive-errs</td><td><code>sum(rate(node_network_receive_errs_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*"}[5m]))</code></td></tr><tr><td>receive-packets</td><td><code>sum(rate(node_network_receive_packets_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*"}[5m]))</code></td></tr><tr><td>transmit-dropped</td><td><code>sum(rate(node_network_transmit_drop_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*"}[5m]))</code></td></tr><tr><td>transmit-errs</td><td><code>sum(rate(node_network_transmit_errs_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*"}[5m]))</code></td></tr><tr><td>transmit-packets</td><td><code>sum(rate(node_network_transmit_packets_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*"}[5m]))</code></td></tr></table> |
|
||||
|
||||
### Cluster Network I/O
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | <table><tr><td>receive</td><td><code>sum(rate(node_network_receive_bytes_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*"}[5m])) by (instance)</code></td></tr><tr><td>transmit</td><td><code>sum(rate(node_network_transmit_bytes_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*"}[5m])) by (instance)</code></td></tr></table> |
|
||||
| Summary | <table><tr><td>receive</td><td><code>sum(rate(node_network_receive_bytes_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*"}[5m]))</code></td></tr><tr><td>transmit</td><td><code>sum(rate(node_network_transmit_bytes_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*"}[5m]))</code></td></tr></table> |
|
||||
|
||||
# Node Metrics
|
||||
|
||||
### Node CPU Utilization
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | `avg(irate(node_cpu_seconds_total{mode!="idle", instance=~"$instance"}[5m])) by (mode)` |
|
||||
| Summary | `1 - (avg(irate(node_cpu_seconds_total{mode="idle", instance=~"$instance"}[5m])))` |
|
||||
|
||||
### Node Load Average
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | <table><tr><td>load1</td><td>`sum(node_load1{instance=~"$instance"}) / count(node_cpu_seconds_total{mode="system",instance=~"$instance"})`</td></tr><tr><td>load5</td><td>`sum(node_load5{instance=~"$instance"}) / count(node_cpu_seconds_total{mode="system",instance=~"$instance"})`</td></tr><tr><td>load15</td><td>`sum(node_load15{instance=~"$instance"}) / count(node_cpu_seconds_total{mode="system",instance=~"$instance"})`</td></tr></table> |
|
||||
| Summary | <table><tr><td>load1</td><td>`sum(node_load1{instance=~"$instance"}) / count(node_cpu_seconds_total{mode="system",instance=~"$instance"})`</td></tr><tr><td>load5</td><td>`sum(node_load5{instance=~"$instance"}) / count(node_cpu_seconds_total{mode="system",instance=~"$instance"})`</td></tr><tr><td>load15</td><td>`sum(node_load15{instance=~"$instance"}) / count(node_cpu_seconds_total{mode="system",instance=~"$instance"})`</td></tr></table> |
|
||||
|
||||
### Node Memory Utilization
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | `1 - sum(node_memory_MemAvailable_bytes{instance=~"$instance"}) / sum(node_memory_MemTotal_bytes{instance=~"$instance"})` |
|
||||
| Summary | `1 - sum(node_memory_MemAvailable_bytes{instance=~"$instance"}) / sum(node_memory_MemTotal_bytes{instance=~"$instance"}) ` |
|
||||
|
||||
### Node Disk Utilization
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | `(sum(node_filesystem_size_bytes{device!="rootfs",instance=~"$instance"}) by (device) - sum(node_filesystem_free_bytes{device!="rootfs",instance=~"$instance"}) by (device)) / sum(node_filesystem_size_bytes{device!="rootfs",instance=~"$instance"}) by (device)` |
|
||||
| Summary | `(sum(node_filesystem_size_bytes{device!="rootfs",instance=~"$instance"}) - sum(node_filesystem_free_bytes{device!="rootfs",instance=~"$instance"})) / sum(node_filesystem_size_bytes{device!="rootfs",instance=~"$instance"})` |
|
||||
|
||||
### Node Disk I/O
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | <table><tr><td>read</td><td>`sum(rate(node_disk_read_bytes_total{instance=~"$instance"}[5m]))`</td></tr><tr><td>written</td><td>`sum(rate(node_disk_written_bytes_total{instance=~"$instance"}[5m]))`</td></tr></table> |
|
||||
| Summary | <table><tr><td>read</td><td>`sum(rate(node_disk_read_bytes_total{instance=~"$instance"}[5m]))`</td></tr><tr><td>written</td><td>`sum(rate(node_disk_written_bytes_total{instance=~"$instance"}[5m]))`</td></tr></table> |
|
||||
|
||||
### Node Network Packets
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | <table><tr><td>receive-dropped</td><td><code>sum(rate(node_network_receive_drop_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*",instance=~"$instance"}[5m])) by (device)</code></td></tr><tr><td>receive-errs</td><td><code>sum(rate(node_network_receive_errs_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*",instance=~"$instance"}[5m])) by (device)</code></td></tr><tr><td>receive-packets</td><td><code>sum(rate(node_network_receive_packets_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*",instance=~"$instance"}[5m])) by (device)</code></td></tr><tr><td>transmit-dropped</td><td><code>sum(rate(node_network_transmit_drop_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*",instance=~"$instance"}[5m])) by (device)</code></td></tr><tr><td>transmit-errs</td><td><code>sum(rate(node_network_transmit_errs_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*",instance=~"$instance"}[5m])) by (device)</code></td></tr><tr><td>transmit-packets</td><td><code>sum(rate(node_network_transmit_packets_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*",instance=~"$instance"}[5m])) by (device)</code></td></tr></table> |
|
||||
| Summary | <table><tr><td>receive-dropped</td><td><code>sum(rate(node_network_receive_drop_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*",instance=~"$instance"}[5m]))</code></td></tr><tr><td>receive-errs</td><td><code>sum(rate(node_network_receive_errs_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*",instance=~"$instance"}[5m]))</code></td></tr><tr><td>receive-packets</td><td><code>sum(rate(node_network_receive_packets_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*",instance=~"$instance"}[5m]))</code></td></tr><tr><td>transmit-dropped</td><td><code>sum(rate(node_network_transmit_drop_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*",instance=~"$instance"}[5m]))</code></td></tr><tr><td>transmit-errs</td><td><code>sum(rate(node_network_transmit_errs_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*",instance=~"$instance"}[5m]))</code></td></tr><tr><td>transmit-packets</td><td><code>sum(rate(node_network_transmit_packets_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*",instance=~"$instance"}[5m]))</code></td></tr></table> |
|
||||
|
||||
### Node Network I/O
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | <table><tr><td>receive</td><td><code>sum(rate(node_network_receive_bytes_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*",instance=~"$instance"}[5m])) by (device)</code></td></tr><tr><td>transmit</td><td><code>sum(rate(node_network_transmit_bytes_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*",instance=~"$instance"}[5m])) by (device)</code></td></tr></table> |
|
||||
| Summary | <table><tr><td>receive</td><td><code>sum(rate(node_network_receive_bytes_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*",instance=~"$instance"}[5m]))</code></td></tr><tr><td>transmit</td><td><code>sum(rate(node_network_transmit_bytes_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*",instance=~"$instance"}[5m]))</code></td></tr></table> |
|
||||
|
||||
# Etcd Metrics
|
||||
|
||||
### Etcd Has a Leader
|
||||
|
||||
`max(etcd_server_has_leader)`
|
||||
|
||||
### Number of Times the Leader Changes
|
||||
|
||||
`max(etcd_server_leader_changes_seen_total)`
|
||||
|
||||
### Number of Failed Proposals
|
||||
|
||||
`sum(etcd_server_proposals_failed_total)`
|
||||
|
||||
### GRPC Client Traffic
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | <table><tr><td>in</td><td>`sum(rate(etcd_network_client_grpc_received_bytes_total[5m])) by (instance)`</td></tr><tr><td>out</td><td>`sum(rate(etcd_network_client_grpc_sent_bytes_total[5m])) by (instance)`</td></tr></table> |
|
||||
| Summary | <table><tr><td>in</td><td>`sum(rate(etcd_network_client_grpc_received_bytes_total[5m]))`</td></tr><tr><td>out</td><td>`sum(rate(etcd_network_client_grpc_sent_bytes_total[5m]))`</td></tr></table> |
|
||||
|
||||
### Peer Traffic
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | <table><tr><td>in</td><td>`sum(rate(etcd_network_peer_received_bytes_total[5m])) by (instance)`</td></tr><tr><td>out</td><td>`sum(rate(etcd_network_peer_sent_bytes_total[5m])) by (instance)`</td></tr></table> |
|
||||
| Summary | <table><tr><td>in</td><td>`sum(rate(etcd_network_peer_received_bytes_total[5m]))`</td></tr><tr><td>out</td><td>`sum(rate(etcd_network_peer_sent_bytes_total[5m]))`</td></tr></table> |
|
||||
|
||||
### DB Size
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | `sum(etcd_debugging_mvcc_db_total_size_in_bytes) by (instance)` |
|
||||
| Summary | `sum(etcd_debugging_mvcc_db_total_size_in_bytes)` |
|
||||
|
||||
### Active Streams
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | <table><tr><td>lease-watch</td><td>`sum(grpc_server_started_total{grpc_service="etcdserverpb.Lease",grpc_type="bidi_stream"}) by (instance) - sum(grpc_server_handled_total{grpc_service="etcdserverpb.Lease",grpc_type="bidi_stream"}) by (instance)`</td></tr><tr><td>watch</td><td>`sum(grpc_server_started_total{grpc_service="etcdserverpb.Watch",grpc_type="bidi_stream"}) by (instance) - sum(grpc_server_handled_total{grpc_service="etcdserverpb.Watch",grpc_type="bidi_stream"}) by (instance)`</td></tr></table> |
|
||||
| Summary | <table><tr><td>lease-watch</td><td>`sum(grpc_server_started_total{grpc_service="etcdserverpb.Lease",grpc_type="bidi_stream"}) - sum(grpc_server_handled_total{grpc_service="etcdserverpb.Lease",grpc_type="bidi_stream"})`</td></tr><tr><td>watch</td><td>`sum(grpc_server_started_total{grpc_service="etcdserverpb.Watch",grpc_type="bidi_stream"}) - sum(grpc_server_handled_total{grpc_service="etcdserverpb.Watch",grpc_type="bidi_stream"})`</td></tr></table> |
|
||||
|
||||
### Raft Proposals
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | <table><tr><td>applied</td><td>`sum(increase(etcd_server_proposals_applied_total[5m])) by (instance)`</td></tr><tr><td>committed</td><td>`sum(increase(etcd_server_proposals_committed_total[5m])) by (instance)`</td></tr><tr><td>pending</td><td>`sum(increase(etcd_server_proposals_pending[5m])) by (instance)`</td></tr><tr><td>failed</td><td>`sum(increase(etcd_server_proposals_failed_total[5m])) by (instance)`</td></tr></table> |
|
||||
| Summary | <table><tr><td>applied</td><td>`sum(increase(etcd_server_proposals_applied_total[5m]))`</td></tr><tr><td>committed</td><td>`sum(increase(etcd_server_proposals_committed_total[5m]))`</td></tr><tr><td>pending</td><td>`sum(increase(etcd_server_proposals_pending[5m]))`</td></tr><tr><td>failed</td><td>`sum(increase(etcd_server_proposals_failed_total[5m]))`</td></tr></table> |
|
||||
|
||||
### RPC Rate
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | <table><tr><td>total</td><td>`sum(rate(grpc_server_started_total{grpc_type="unary"}[5m])) by (instance)`</td></tr><tr><td>fail</td><td>`sum(rate(grpc_server_handled_total{grpc_type="unary",grpc_code!="OK"}[5m])) by (instance)`</td></tr></table> |
|
||||
| Summary | <table><tr><td>total</td><td>`sum(rate(grpc_server_started_total{grpc_type="unary"}[5m]))`</td></tr><tr><td>fail</td><td>`sum(rate(grpc_server_handled_total{grpc_type="unary",grpc_code!="OK"}[5m]))`</td></tr></table> |
|
||||
|
||||
### Disk Operations
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | <table><tr><td>commit-called-by-backend</td><td>`sum(rate(etcd_disk_backend_commit_duration_seconds_sum[1m])) by (instance)`</td></tr><tr><td>fsync-called-by-wal</td><td>`sum(rate(etcd_disk_wal_fsync_duration_seconds_sum[1m])) by (instance)`</td></tr></table> |
|
||||
| Summary | <table><tr><td>commit-called-by-backend</td><td>`sum(rate(etcd_disk_backend_commit_duration_seconds_sum[1m]))`</td></tr><tr><td>fsync-called-by-wal</td><td>`sum(rate(etcd_disk_wal_fsync_duration_seconds_sum[1m]))`</td></tr></table> |
|
||||
|
||||
### Disk Sync Duration
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | <table><tr><td>wal</td><td>`histogram_quantile(0.99, sum(rate(etcd_disk_wal_fsync_duration_seconds_bucket[5m])) by (instance, le))`</td></tr><tr><td>db</td><td>`histogram_quantile(0.99, sum(rate(etcd_disk_backend_commit_duration_seconds_bucket[5m])) by (instance, le))`</td></tr></table> |
|
||||
| Summary | <table><tr><td>wal</td><td>`sum(histogram_quantile(0.99, sum(rate(etcd_disk_wal_fsync_duration_seconds_bucket[5m])) by (instance, le)))`</td></tr><tr><td>db</td><td>`sum(histogram_quantile(0.99, sum(rate(etcd_disk_backend_commit_duration_seconds_bucket[5m])) by (instance, le)))`</td></tr></table> |
|
||||
|
||||
# Kubernetes Components Metrics
|
||||
|
||||
### API Server Request Latency
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | `avg(apiserver_request_latencies_sum / apiserver_request_latencies_count) by (instance, verb) /1e+06` |
|
||||
| Summary | `avg(apiserver_request_latencies_sum / apiserver_request_latencies_count) by (instance) /1e+06` |
|
||||
|
||||
### API Server Request Rate
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | `sum(rate(apiserver_request_count[5m])) by (instance, code)` |
|
||||
| Summary | `sum(rate(apiserver_request_count[5m])) by (instance)` |
|
||||
|
||||
### Scheduling Failed Pods
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | `sum(kube_pod_status_scheduled{condition="false"})` |
|
||||
| Summary | `sum(kube_pod_status_scheduled{condition="false"})` |
|
||||
|
||||
### Controller Manager Queue Depth
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | <table><tr><td>volumes</td><td>`sum(volumes_depth) by instance`</td></tr><tr><td>deployment</td><td>`sum(deployment_depth) by instance`</td></tr><tr><td>replicaset</td><td>`sum(replicaset_depth) by instance`</td></tr><tr><td>service</td><td>`sum(service_depth) by instance`</td></tr><tr><td>serviceaccount</td><td>`sum(serviceaccount_depth) by instance`</td></tr><tr><td>endpoint</td><td>`sum(endpoint_depth) by instance`</td></tr><tr><td>daemonset</td><td>`sum(daemonset_depth) by instance`</td></tr><tr><td>statefulset</td><td>`sum(statefulset_depth) by instance`</td></tr><tr><td>replicationmanager</td><td>`sum(replicationmanager_depth) by instance`</td></tr></table> |
|
||||
| Summary | <table><tr><td>volumes</td><td>`sum(volumes_depth)`</td></tr><tr><td>deployment</td><td>`sum(deployment_depth)`</td></tr><tr><td>replicaset</td><td>`sum(replicaset_depth)`</td></tr><tr><td>service</td><td>`sum(service_depth)`</td></tr><tr><td>serviceaccount</td><td>`sum(serviceaccount_depth)`</td></tr><tr><td>endpoint</td><td>`sum(endpoint_depth)`</td></tr><tr><td>daemonset</td><td>`sum(daemonset_depth)`</td></tr><tr><td>statefulset</td><td>`sum(statefulset_depth)`</td></tr><tr><td>replicationmanager</td><td>`sum(replicationmanager_depth)`</td></tr></table> |
|
||||
|
||||
### Scheduler E2E Scheduling Latency
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | `histogram_quantile(0.99, sum(scheduler_e2e_scheduling_latency_microseconds_bucket) by (le, instance)) / 1e+06` |
|
||||
| Summary | `sum(histogram_quantile(0.99, sum(scheduler_e2e_scheduling_latency_microseconds_bucket) by (le, instance)) / 1e+06)` |
|
||||
|
||||
### Scheduler Preemption Attempts
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | `sum(rate(scheduler_total_preemption_attempts[5m])) by (instance)` |
|
||||
| Summary | `sum(rate(scheduler_total_preemption_attempts[5m]))` |
|
||||
|
||||
### Ingress Controller Connections
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | <table><tr><td>reading</td><td>`sum(nginx_ingress_controller_nginx_process_connections{state="reading"}) by (instance)`</td></tr><tr><td>waiting</td><td>`sum(nginx_ingress_controller_nginx_process_connections{state="waiting"}) by (instance)`</td></tr><tr><td>writing</td><td>`sum(nginx_ingress_controller_nginx_process_connections{state="writing"}) by (instance)`</td></tr><tr><td>accepted</td><td>`sum(ceil(increase(nginx_ingress_controller_nginx_process_connections_total{state="accepted"}[5m]))) by (instance)`</td></tr><tr><td>active</td><td>`sum(ceil(increase(nginx_ingress_controller_nginx_process_connections_total{state="active"}[5m]))) by (instance)`</td></tr><tr><td>handled</td><td>`sum(ceil(increase(nginx_ingress_controller_nginx_process_connections_total{state="handled"}[5m]))) by (instance)`</td></tr></table> |
|
||||
| Summary | <table><tr><td>reading</td><td>`sum(nginx_ingress_controller_nginx_process_connections{state="reading"})`</td></tr><tr><td>waiting</td><td>`sum(nginx_ingress_controller_nginx_process_connections{state="waiting"})`</td></tr><tr><td>writing</td><td>`sum(nginx_ingress_controller_nginx_process_connections{state="writing"})`</td></tr><tr><td>accepted</td><td>`sum(ceil(increase(nginx_ingress_controller_nginx_process_connections_total{state="accepted"}[5m])))`</td></tr><tr><td>active</td><td>`sum(ceil(increase(nginx_ingress_controller_nginx_process_connections_total{state="active"}[5m])))`</td></tr><tr><td>handled</td><td>`sum(ceil(increase(nginx_ingress_controller_nginx_process_connections_total{state="handled"}[5m])))`</td></tr></table> |
|
||||
|
||||
### Ingress Controller Request Process Time
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | `topk(10, histogram_quantile(0.95,sum by (le, host, path)(rate(nginx_ingress_controller_request_duration_seconds_bucket{host!="_"}[5m]))))` |
|
||||
| Summary | `topk(10, histogram_quantile(0.95,sum by (le, host)(rate(nginx_ingress_controller_request_duration_seconds_bucket{host!="_"}[5m]))))` |
|
||||
|
||||
# Rancher Logging Metrics
|
||||
|
||||
|
||||
### Fluentd Buffer Queue Rate
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | `sum(rate(fluentd_output_status_buffer_queue_length[5m])) by (instance)` |
|
||||
| Summary | `sum(rate(fluentd_output_status_buffer_queue_length[5m]))` |
|
||||
|
||||
### Fluentd Input Rate
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | `sum(rate(fluentd_input_status_num_records_total[5m])) by (instance)` |
|
||||
| Summary | `sum(rate(fluentd_input_status_num_records_total[5m]))` |
|
||||
|
||||
### Fluentd Output Errors Rate
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | `sum(rate(fluentd_output_status_num_errors[5m])) by (type)` |
|
||||
| Summary | `sum(rate(fluentd_output_status_num_errors[5m]))` |
|
||||
|
||||
### Fluentd Output Rate
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | `sum(rate(fluentd_output_status_num_records_total[5m])) by (instance)` |
|
||||
| Summary | `sum(rate(fluentd_output_status_num_records_total[5m]))` |
|
||||
|
||||
# Workload Metrics
|
||||
|
||||
### Workload CPU Utilization
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | <table><tr><td>cfs throttled seconds</td><td>`sum(rate(container_cpu_cfs_throttled_seconds_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m])) by (pod_name)`</td></tr><tr><td>user seconds</td><td>`sum(rate(container_cpu_user_seconds_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m])) by (pod_name)`</td></tr><tr><td>system seconds</td><td>`sum(rate(container_cpu_system_seconds_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m])) by (pod_name)`</td></tr><tr><td>usage seconds</td><td>`sum(rate(container_cpu_usage_seconds_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m])) by (pod_name)`</td></tr></table> |
|
||||
| Summary | <table><tr><td>cfs throttled seconds</td><td>`sum(rate(container_cpu_cfs_throttled_seconds_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m]))`</td></tr><tr><td>user seconds</td><td>`sum(rate(container_cpu_user_seconds_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m]))`</td></tr><tr><td>system seconds</td><td>`sum(rate(container_cpu_system_seconds_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m]))`</td></tr><tr><td>usage seconds</td><td>`sum(rate(container_cpu_usage_seconds_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m]))`</td></tr></table> |
|
||||
|
||||
### Workload Memory Utilization
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | `sum(container_memory_working_set_bytes{namespace="$namespace",pod_name=~"$podName", container_name!=""}) by (pod_name)` |
|
||||
| Summary | `sum(container_memory_working_set_bytes{namespace="$namespace",pod_name=~"$podName", container_name!=""})` |
|
||||
|
||||
### Workload Network Packets
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | <table><tr><td>receive-packets</td><td>`sum(rate(container_network_receive_packets_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m])) by (pod_name)`</td></tr><tr><td>receive-dropped</td><td>`sum(rate(container_network_receive_packets_dropped_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m])) by (pod_name)`</td></tr><tr><td>receive-errors</td><td>`sum(rate(container_network_receive_errors_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m])) by (pod_name)`</td></tr><tr><td>transmit-packets</td><td>`sum(rate(container_network_transmit_packets_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m])) by (pod_name)`</td></tr><tr><td>transmit-dropped</td><td>`sum(rate(container_network_transmit_packets_dropped_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m])) by (pod_name)`</td></tr><tr><td>transmit-errors</td><td>`sum(rate(container_network_transmit_errors_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m])) by (pod_name)`</td></tr></table> |
|
||||
| Summary | <table><tr><td>receive-packets</td><td>`sum(rate(container_network_receive_packets_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m]))`</td></tr><tr><td>receive-dropped</td><td>`sum(rate(container_network_receive_packets_dropped_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m]))`</td></tr><tr><td>receive-errors</td><td>`sum(rate(container_network_receive_errors_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m]))`</td></tr><tr><td>transmit-packets</td><td>`sum(rate(container_network_transmit_packets_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m]))`</td></tr><tr><td>transmit-dropped</td><td>`sum(rate(container_network_transmit_packets_dropped_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m]))`</td></tr><tr><td>transmit-errors</td><td>`sum(rate(container_network_transmit_errors_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m]))`</td></tr></table> |
|
||||
|
||||
### Workload Network I/O
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | <table><tr><td>receive</td><td>`sum(rate(container_network_receive_bytes_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m])) by (pod_name)`</td></tr><tr><td>transmit</td><td>`sum(rate(container_network_transmit_bytes_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m])) by (pod_name)`</td></tr></table> |
|
||||
| Summary | <table><tr><td>receive</td><td>`sum(rate(container_network_receive_bytes_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m]))`</td></tr><tr><td>transmit</td><td>`sum(rate(container_network_transmit_bytes_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m]))`</td></tr></table> |
|
||||
|
||||
### Workload Disk I/O
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | <table><tr><td>read</td><td>`sum(rate(container_fs_reads_bytes_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m])) by (pod_name)`</td></tr><tr><td>write</td><td>`sum(rate(container_fs_writes_bytes_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m])) by (pod_name)`</td></tr></table> |
|
||||
| Summary | <table><tr><td>read</td><td>`sum(rate(container_fs_reads_bytes_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m]))`</td></tr><tr><td>write</td><td>`sum(rate(container_fs_writes_bytes_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m]))`</td></tr></table> |
|
||||
|
||||
# Pod Metrics
|
||||
|
||||
### Pod CPU Utilization
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | <table><tr><td>cfs throttled seconds</td><td>`sum(rate(container_cpu_cfs_throttled_seconds_total{container_name!="POD",namespace="$namespace",pod_name="$podName", container_name!=""}[5m])) by (container_name)`</td></tr><tr><td>usage seconds</td><td>`sum(rate(container_cpu_usage_seconds_total{container_name!="POD",namespace="$namespace",pod_name="$podName", container_name!=""}[5m])) by (container_name)`</td></tr><tr><td>system seconds</td><td>`sum(rate(container_cpu_system_seconds_total{container_name!="POD",namespace="$namespace",pod_name="$podName", container_name!=""}[5m])) by (container_name)`</td></tr><tr><td>user seconds</td><td>`sum(rate(container_cpu_user_seconds_total{container_name!="POD",namespace="$namespace",pod_name="$podName", container_name!=""}[5m])) by (container_name)`</td></tr></table> |
|
||||
| Summary | <table><tr><td>cfs throttled seconds</td><td>`sum(rate(container_cpu_cfs_throttled_seconds_total{container_name!="POD",namespace="$namespace",pod_name="$podName", container_name!=""}[5m]))`</td></tr><tr><td>usage seconds</td><td>`sum(rate(container_cpu_usage_seconds_total{container_name!="POD",namespace="$namespace",pod_name="$podName", container_name!=""}[5m]))`</td></tr><tr><td>system seconds</td><td>`sum(rate(container_cpu_system_seconds_total{container_name!="POD",namespace="$namespace",pod_name="$podName", container_name!=""}[5m]))`</td></tr><tr><td>user seconds</td><td>`sum(rate(container_cpu_user_seconds_total{container_name!="POD",namespace="$namespace",pod_name="$podName", container_name!=""}[5m]))`</td></tr></table> |
|
||||
|
||||
### Pod Memory Utilization
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | `sum(container_memory_working_set_bytes{container_name!="POD",namespace="$namespace",pod_name="$podName",container_name!=""}) by (container_name)` |
|
||||
| Summary | `sum(container_memory_working_set_bytes{container_name!="POD",namespace="$namespace",pod_name="$podName",container_name!=""})` |
|
||||
|
||||
### Pod Network Packets
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | <table><tr><td>receive-packets</td><td>`sum(rate(container_network_receive_packets_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`</td></tr><tr><td>receive-dropped</td><td>`sum(rate(container_network_receive_packets_dropped_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`</td></tr><tr><td>receive-errors</td><td>`sum(rate(container_network_receive_errors_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`</td></tr><tr><td>transmit-packets</td><td>`sum(rate(container_network_transmit_packets_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`</td></tr><tr><td>transmit-dropped</td><td>`sum(rate(container_network_transmit_packets_dropped_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`</td></tr><tr><td>transmit-errors</td><td>`sum(rate(container_network_transmit_errors_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`</td></tr></table> |
|
||||
| Summary | <table><tr><td>receive-packets</td><td>`sum(rate(container_network_receive_packets_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`</td></tr><tr><td>receive-dropped</td><td>`sum(rate(container_network_receive_packets_dropped_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`</td></tr><tr><td>receive-errors</td><td>`sum(rate(container_network_receive_errors_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`</td></tr><tr><td>transmit-packets</td><td>`sum(rate(container_network_transmit_packets_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`</td></tr><tr><td>transmit-dropped</td><td>`sum(rate(container_network_transmit_packets_dropped_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`</td></tr><tr><td>transmit-errors</td><td>`sum(rate(container_network_transmit_errors_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`</td></tr></table> |
|
||||
|
||||
### Pod Network I/O
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | <table><tr><td>receive</td><td>`sum(rate(container_network_receive_bytes_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`</td></tr><tr><td>transmit</td><td>`sum(rate(container_network_transmit_bytes_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`</td></tr></table> |
|
||||
| Summary | <table><tr><td>receive</td><td>`sum(rate(container_network_receive_bytes_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`</td></tr><tr><td>transmit</td><td>`sum(rate(container_network_transmit_bytes_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`</td></tr></table> |
|
||||
|
||||
### Pod Disk I/O
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | <table><tr><td>read</td><td>`sum(rate(container_fs_reads_bytes_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m])) by (container_name)`</td></tr><tr><td>write</td><td>`sum(rate(container_fs_writes_bytes_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m])) by (container_name)`</td></tr></table> |
|
||||
| Summary | <table><tr><td>read</td><td>`sum(rate(container_fs_reads_bytes_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`</td></tr><tr><td>write</td><td>`sum(rate(container_fs_writes_bytes_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`</td></tr></table> |
|
||||
|
||||
# Container Metrics
|
||||
|
||||
### Container CPU Utilization
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| cfs throttled seconds | `sum(rate(container_cpu_cfs_throttled_seconds_total{namespace="$namespace",pod_name="$podName",container_name="$containerName"}[5m]))` |
|
||||
| usage seconds | `sum(rate(container_cpu_usage_seconds_total{namespace="$namespace",pod_name="$podName",container_name="$containerName"}[5m]))` |
|
||||
| system seconds | `sum(rate(container_cpu_system_seconds_total{namespace="$namespace",pod_name="$podName",container_name="$containerName"}[5m]))` |
|
||||
| user seconds | `sum(rate(container_cpu_user_seconds_total{namespace="$namespace",pod_name="$podName",container_name="$containerName"}[5m]))` |
|
||||
|
||||
### Container Memory Utilization
|
||||
|
||||
`sum(container_memory_working_set_bytes{namespace="$namespace",pod_name="$podName",container_name="$containerName"})`
|
||||
|
||||
### Container Disk I/O
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| read | `sum(rate(container_fs_reads_bytes_total{namespace="$namespace",pod_name="$podName",container_name="$containerName"}[5m]))` |
|
||||
| write | `sum(rate(container_fs_writes_bytes_total{namespace="$namespace",pod_name="$podName",container_name="$containerName"}[5m]))` |
|
||||
+84
@@ -0,0 +1,84 @@
|
||||
---
|
||||
title: Project Monitoring
|
||||
weight: 2
|
||||
aliases:
|
||||
- /rancher/v2.0-v2.4/en/project-admin/tools/monitoring
|
||||
- /rancher/v2.0-v2.4/en/monitoring-alerting/v2.0.x-v2.4.x/monitoring/project-monitoring
|
||||
- /rancher/v2.0-v2.4/en/monitoring-alerting/v2.0.x-v2.4.x/cluster-monitoring/project-monitoring
|
||||
---
|
||||
|
||||
_Available as of v2.2.4_
|
||||
|
||||
Using Rancher, you can monitor the state and processes of your cluster nodes, Kubernetes components, and software deployments through integration with [Prometheus](https://prometheus.io/), a leading open-source monitoring solution.
|
||||
|
||||
This section covers the following topics:
|
||||
|
||||
- [Monitoring scope](#monitoring-scope)
|
||||
- [Permissions to configure project monitoring](#permissions-to-configure-project-monitoring)
|
||||
- [Enabling project monitoring](#enabling-project-monitoring)
|
||||
- [Project-level monitoring resource requirements](#project-level-monitoring-resource-requirements)
|
||||
- [Project metrics](#project-metrics)
|
||||
|
||||
### Monitoring Scope
|
||||
|
||||
Using Prometheus, you can monitor Rancher at both the [cluster level](monitoring-alerting/legacy/monitoring/cluster-monitoring/) and project level. For each cluster and project that is enabled for monitoring, Rancher deploys a Prometheus server.
|
||||
|
||||
- [Cluster monitoring](monitoring-alerting/legacy/monitoring/cluster-monitoring/) allows you to view the health of your Kubernetes cluster. Prometheus collects metrics from the cluster components below, which you can view in graphs and charts.
|
||||
|
||||
- Kubernetes control plane
|
||||
- etcd database
|
||||
- All nodes (including workers)
|
||||
|
||||
- Project monitoring allows you to view the state of pods running in a given project. Prometheus collects metrics from the project's deployed HTTP and TCP/UDP workloads.
|
||||
|
||||
### Permissions to Configure Project Monitoring
|
||||
|
||||
Only [administrators](../../../how-to-guides/advanced-user-guides/authentication-permissions-and-global-configuration/manage-role-based-access-control-rbac/global-permissions.md), [cluster owners or members](../../../how-to-guides/advanced-user-guides/authentication-permissions-and-global-configuration/manage-role-based-access-control-rbac/cluster-and-project-roles.md#cluster-roles), or [project owners](../../../how-to-guides/advanced-user-guides/authentication-permissions-and-global-configuration/manage-role-based-access-control-rbac/cluster-and-project-roles.md#project-roles) can configure project level monitoring. Project members can only view monitoring metrics.
|
||||
|
||||
### Enabling Project Monitoring
|
||||
|
||||
> **Prerequisite:** Cluster monitoring must be [enabled.](monitoring-alerting/legacy/monitoring/cluster-monitoring/)
|
||||
|
||||
1. Go to the project where monitoring should be enabled. Note: When cluster monitoring is enabled, monitoring is also enabled by default in the **System** project.
|
||||
|
||||
1. Select **Tools > Monitoring** in the navigation bar.
|
||||
|
||||
1. Select **Enable** to show the [Prometheus configuration options](monitoring-alerting/legacy/monitoring/cluster-monitoring/prometheus/). Enter in your desired configuration options.
|
||||
|
||||
1. Click **Save**.
|
||||
|
||||
### Project-Level Monitoring Resource Requirements
|
||||
|
||||
Container| CPU - Request | Mem - Request | CPU - Limit | Mem - Limit | Configurable
|
||||
---------|---------------|---------------|-------------|-------------|-------------
|
||||
Prometheus|750m| 750Mi | 1000m | 1000Mi | Yes
|
||||
Grafana | 100m | 100Mi | 200m | 200Mi | No
|
||||
|
||||
|
||||
**Result:** A single application,`project-monitoring`, is added as an [application](catalog/apps/) to the project. After the application is `active`, you can start viewing project metrics through the [Rancher dashboard](monitoring-alerting/legacy/monitoring/cluster-monitoring/) or directly from Grafana.
|
||||
|
||||
> The default username and password for the Grafana instance will be `admin/admin`. However, Grafana dashboards are served via the Rancher authentication proxy, so only users who are currently authenticated into the Rancher server have access to the Grafana dashboard.
|
||||
|
||||
### Project Metrics
|
||||
[Workload metrics](monitoring-alerting/v2.0.x-v2.4.x/cluster-monitoring/expression/#workload-metrics) are available for the project if monitoring is enabled at the [cluster level](monitoring-alerting/legacy/monitoring/cluster-monitoring/) and at the [project level.](#enabling-project-monitoring)
|
||||
|
||||
You can monitor custom metrics from any [exporters.](https://prometheus.io/docs/instrumenting/exporters/) You can also expose some custom endpoints on deployments without needing to configure Prometheus for your project.
|
||||
|
||||
> **Example:**
|
||||
> A [Redis](https://redis.io/) application is deployed in the namespace `redis-app` in the project `Datacenter`. It is monitored via [Redis exporter](https://github.com/oliver006/redis_exporter). After enabling project monitoring, you can edit the application to configure the <b>Advanced Options -> Custom Metrics</b> section. Enter the `Container Port` and `Path` and select the `Protocol`.
|
||||
|
||||
To access a project-level Grafana instance,
|
||||
|
||||
1. From the **Global** view, navigate to a cluster that has monitoring enabled.
|
||||
|
||||
1. Go to a project that has monitoring enabled.
|
||||
|
||||
1. From the project view, click **Apps.** In versions before v2.2.0, choose **Catalog Apps** on the main navigation bar.
|
||||
|
||||
1. Go to the `project-monitoring` application.
|
||||
|
||||
1. In the `project-monitoring` application, there are two `/index.html` links: one that leads to a Grafana instance and one that leads to a Prometheus instance. When you click the Grafana link, it will redirect you to a new webpage for Grafana, which shows metrics for the cluster.
|
||||
|
||||
1. You will be signed in to the Grafana instance automatically. The default username is `admin` and the default password is `admin`. For security, we recommend that you log out of Grafana, log back in with the `admin` password, and change your password.
|
||||
|
||||
**Results:** You will be logged into Grafana from the Grafana instance. After logging in, you can view the preset Grafana dashboards, which are imported via the [Grafana provisioning mechanism](http://docs.grafana.org/administration/provisioning/#dashboards), so you cannot modify them directly. For now, if you want to configure your own dashboards, clone the original and modify the new copy.
|
||||
+112
@@ -0,0 +1,112 @@
|
||||
---
|
||||
title: Prometheus Configuration
|
||||
weight: 1
|
||||
aliases:
|
||||
- /rancher/v2.0-v2.4/en/project-admin/tools/monitoring/prometheus
|
||||
- /rancher/v2.0-v2.4/en/cluster-admin/tools/monitoring/prometheus/
|
||||
- /rancher/v2.0-v2.4/en/monitoring-alerting/legacy/monitoring/cluster-monitoring/prometheus
|
||||
- /rancher/v2.0-v2.4/en/monitoring-alerting/v2.0.x-v2.4.x/cluster-monitoring/prometheus
|
||||
- /rancher/v2.x/en/monitoring-alerting/v2.0.x-v2.4.x/cluster-monitoring/prometheus/
|
||||
---
|
||||
|
||||
_Available as of v2.2.0_
|
||||
|
||||
While configuring monitoring at either the [cluster level](monitoring-alerting/legacy/monitoring/cluster-monitoring/) or [project level](project-admin/tools/monitoring/), there are multiple options that can be configured.
|
||||
|
||||
- [Basic Configuration](#basic-configuration)
|
||||
- [Advanced Options](#advanced-options)
|
||||
- [Node Exporter](#node-exporter)
|
||||
- [Persistent Storage](#persistent-storage)
|
||||
- [Remote Storage](#remote-storage)
|
||||
|
||||
# Basic Configuration
|
||||
|
||||
Option | Description
|
||||
-------|-------------
|
||||
Data Retention | How long your Prometheus instance retains monitoring data scraped from Rancher objects before it's purged.
|
||||
[Enable Node Exporter](#node-exporter) | Whether or not to deploy the node exporter.
|
||||
Node Exporter Host Port | The host port on which data is exposed, i.e. data that Prometheus collects from your node hardware. Required if you have enabled the node exporter.
|
||||
[Enable Persistent Storage](#persistent-storage) for Prometheus | Whether or not to configure storage for Prometheus so that metrics can be retained even if the Prometheus pod fails.
|
||||
[Enable Persistent Storage](#persistent-storage) for Grafana | Whether or not to configure storage for Grafana so that the Grafana dashboards and configuration can be retained even if the Grafana pod fails.
|
||||
Prometheus [CPU Limit](https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/#meaning-of-cpu) | CPU resource limit for the Prometheus pod.
|
||||
Prometheus [CPU Reservation](https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/#meaning-of-cpu) | CPU reservation for the Prometheus pod.
|
||||
Prometheus [Memory Limit](https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/#meaning-of-memory) | Memory resource limit for the Prometheus pod.
|
||||
Prometheus [Memory Reservation](https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/#meaning-of-memory) | Memory resource requests for the Prometheus pod.
|
||||
Selector | Ability to select the nodes in which Prometheus and Grafana pods are deployed to. To use this option, the nodes must have labels.
|
||||
|
||||
# Advanced Options
|
||||
|
||||
Since monitoring is an [application](https://github.com/rancher/system-charts/tree/dev/charts/rancher-monitoring) from the [Rancher catalog](catalog/), it can be configured like any other catalog application, by passing in values to Helm.
|
||||
|
||||
> **Warning:** Any modification to the application without understanding the entire application can lead to catastrophic errors.
|
||||
|
||||
### Prometheus RemoteRead and RemoteWrite
|
||||
|
||||
_Available as of v2.4.0_
|
||||
|
||||
Prometheus RemoteRead and RemoteWrite can be configured as custom answers in the **Advanced Options** section.
|
||||
|
||||
For more information on remote endpoints and storage, refer to the [Prometheus documentation.](https://prometheus.io/docs/operating/integrations/#remote-endpoints-and-storage)
|
||||
|
||||
The Prometheus operator documentation contains the full [RemoteReadSpec](https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/api.md#remotereadspec) and [RemoteWriteSpec.](https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/api.md#remotewritespec)
|
||||
|
||||
An example configuration would be:
|
||||
|
||||
| Variable | Value |
|
||||
|--------------|------------|
|
||||
| `prometheus.remoteWrite[0].url` | `http://mytarget.com` |
|
||||
|
||||
### LivenessProbe and ReadinessProbe
|
||||
|
||||
_Available as of v2.4.0_
|
||||
|
||||
Prometheus LivenessProbe and ReadinessProbe can be configured as custom answers in the **Advanced Options** section.
|
||||
|
||||
The Kubernetes probe spec is [here.](https://v1-17.docs.kubernetes.io/docs/reference/generated/kubernetes-api/v1.17/#probe-v1-core)
|
||||
|
||||
Some example key-value pairs are:
|
||||
|
||||
| Variable | Value |
|
||||
|--------------|------------|
|
||||
| `prometheus.livenessProbe.timeoutSeconds` | 60 |
|
||||
| `prometheus.readinessProbe.timeoutSeconds` | 60 |
|
||||
|
||||
# Node Exporter
|
||||
|
||||
The [node exporter](https://github.com/prometheus/node_exporter/blob/master/README.md) is a popular open source exporter, which exposes the metrics for hardware and \*NIX kernels OS. It is designed to monitor the host system. However, there are still issues with namespaces when running it in a container, mostly around filesystem mount spaces. In order to monitor actual network metrics for the container network, the node exporter must be deployed with the `hostNetwork` mode.
|
||||
|
||||
When configuring Prometheus and enabling the node exporter, enter a host port in the **Node Exporter Host Port** that will not produce port conflicts with existing applications. The host port chosen must be open to allow internal traffic between Prometheus and the Node Exporter.
|
||||
|
||||
>**Warning:** In order for Prometheus to collect the metrics of the node exporter, after enabling cluster monitoring, you must open the <b>Node Exporter Host Port</b> in the host firewall rules to allow intranet access. By default, `9796` is used as that host port.
|
||||
|
||||
# Persistent Storage
|
||||
|
||||
>**Prerequisite:** Configure one or more StorageClasses to use as [persistent storage](../../../pages-for-subheaders/create-kubernetes-persistent-storage.md) for your Prometheus or Grafana pod.
|
||||
|
||||
By default, when you enable Prometheus for either a cluster or project, all monitoring data that Prometheus collects is stored on its own pod. With local storage, if the Prometheus or Grafana pods fail, all the data is lost. Rancher recommends configuring an external persistent storage to the cluster. With the external persistent storage, if the Prometheus or Grafana pods fail, the new pods can recover using data from the persistent storage.
|
||||
|
||||
When enabling persistent storage for Prometheus or Grafana, specify the size of the persistent volume and select the StorageClass.
|
||||
|
||||
# Remote Storage
|
||||
|
||||
>**Prerequisite:** Need a remote storage endpoint to be available. The possible list of integrations is available [here](https://prometheus.io/docs/operating/integrations/)
|
||||
|
||||
Using advanced options, remote storage integration for the Prometheus installation can be configured as follows:
|
||||
|
||||
```
|
||||
prometheus.remoteWrite[0].url = http://remote1/push
|
||||
prometheus.remoteWrite[0].remoteTimeout = 33s
|
||||
|
||||
prometheus.remoteWrite[1].url = http://remote2/push
|
||||
|
||||
|
||||
prometheus.remoteRead[0].url = http://remote1/read
|
||||
prometheus.remoteRead[0].proxyUrl = http://proxy.url
|
||||
prometheus.remoteRead[0].bearerToken = token-value
|
||||
|
||||
prometheus.remoteRead[1].url = http://remote2/read
|
||||
prometheus.remoteRead[1].remoteTimeout = 33s
|
||||
prometheus.remoteRead[1].readRecent = true
|
||||
```
|
||||
|
||||
Additional fields can be set up based on the [ReadSpec](https://github.com/coreos/prometheus-operator/blob/master/Documentation/api.md#remotereadspec) and [RemoteWriteSpec](https://github.com/coreos/prometheus-operator/blob/master/Documentation/api.md#remotewritespec)
|
||||
+66
@@ -0,0 +1,66 @@
|
||||
---
|
||||
title: Viewing Metrics
|
||||
weight: 2
|
||||
aliases:
|
||||
- /rancher/v2.0-v2.4/en/project-admin/tools/monitoring/viewing-metrics
|
||||
- /rancher/v2.0-v2.4/en/cluster-admin/tools/monitoring/viewing-metrics
|
||||
- /rancher/v2.0-v2.4/en/monitoring-alerting/legacy/monitoring/cluster-monitoring/viewing-metrics
|
||||
- /rancher/v2.0-v2.4/en/monitoring-alerting/v2.0.x-v2.4.x/cluster-monitoring/viewing-metrics
|
||||
- /rancher/v2.x/en/monitoring-alerting/v2.0.x-v2.4.x/cluster-monitoring/viewing-metrics/
|
||||
---
|
||||
|
||||
_Available as of v2.2.0_
|
||||
|
||||
After you've enabled monitoring at either the [cluster level](monitoring-alerting/legacy/monitoring/cluster-monitoring/) or [project level](project-admin/tools/monitoring/), you will want to be start viewing the data being collected. There are multiple ways to view this data.
|
||||
|
||||
## Rancher Dashboard
|
||||
|
||||
>**Note:** This is only available if you've enabled monitoring at the [cluster level](monitoring-alerting/legacy/monitoring/cluster-monitoring/). Project specific analytics must be viewed using the project's Grafana instance.
|
||||
|
||||
Rancher's dashboards are available at multiple locations:
|
||||
|
||||
- **Cluster Dashboard**: From the **Global** view, navigate to the cluster.
|
||||
- **Node Metrics**: From the **Global** view, navigate to the cluster. Select **Nodes**. Find the individual node and click on its name. Click **Node Metrics.**
|
||||
- **Workload Metrics**: From the **Global** view, navigate to the project. From the main navigation bar, choose **Resources > Workloads.** (In versions before v2.3.0, choose **Workloads** on the main navigation bar.) Find the individual workload and click on its name. Click **Workload Metrics.**
|
||||
- **Pod Metrics**: From the **Global** view, navigate to the project. Select **Workloads > Workloads**. Find the individual workload and click on its name. Find the individual pod and click on its name. Click **Pod Metrics.**
|
||||
- **Container Metrics**: From the **Global** view, navigate to the project. From the main navigation bar, choose **Resources > Workloads.** (In versions before v2.3.0, choose **Workloads** on the main navigation bar.) Find the individual workload and click on its name. Find the individual pod and click on its name. Find the individual container and click on its name. Click **Container Metrics.**
|
||||
|
||||
Prometheus metrics are displayed and are denoted with the Grafana icon. If you click on the icon, the metrics will open a new tab in Grafana.
|
||||
|
||||
Within each Prometheus metrics widget, there are several ways to customize your view.
|
||||
|
||||
- Toggle between two views:
|
||||
- **Detail**: Displays graphs and charts that let you view each event in a Prometheus time series
|
||||
- **Summary** Displays events in a Prometheus time series that are outside the norm.
|
||||
- Change the range of the time series that you're viewing to see a more refined or expansive data sample.
|
||||
- Customize the data sample to display data between specific dates and times.
|
||||
|
||||
When analyzing these metrics, don't be concerned about any single standalone metric in the charts and graphs. Rather, you should establish a baseline for your metrics over the course of time, e.g. the range of values that your components usually operate within and are considered normal. After you establish the baseline, be on the lookout for any large deltas in the charts and graphs, as these big changes usually indicate a problem that you need to investigate.
|
||||
|
||||
## Grafana
|
||||
|
||||
If you've enabled monitoring at either the [cluster level](monitoring-alerting/legacy/monitoring/cluster-monitoring/) or [project level](project-admin/tools/monitoring/), Rancher automatically creates a link to Grafana instance. Use this link to view monitoring data.
|
||||
|
||||
Grafana allows you to query, visualize, alert, and ultimately, understand your cluster and workload data. For more information on Grafana and its capabilities, visit the [Grafana website](https://grafana.com/grafana).
|
||||
|
||||
### Authentication
|
||||
|
||||
Rancher determines which users can access the new Grafana instance, as well as the objects they can view within it, by validating them against the user's [cluster or project roles](../../../how-to-guides/advanced-user-guides/authentication-permissions-and-global-configuration/manage-role-based-access-control-rbac/cluster-and-project-roles.md). In other words, a user's access in Grafana mirrors their access in Rancher.
|
||||
|
||||
When you go to the Grafana instance, you will be logged in with the username `admin` and the password `admin`. If you log out and log in again, you will be prompted to change your password. You will only have access to the URL of the Grafana instance if you have access to view the corresponding metrics in Rancher. So for example, if your Rancher permissions are scoped to the project level, you won't be able to see the Grafana instance for cluster-level metrics.
|
||||
|
||||
### Accessing the Cluster-level Grafana Instance
|
||||
|
||||
1. From the **Global** view, navigate to a cluster that has monitoring enabled.
|
||||
|
||||
1. Go to the **System** project view. This project is where the cluster-level Grafana instance runs.
|
||||
|
||||
1. Click **Apps.** In versions before v2.2.0, choose **Catalog Apps** on the main navigation bar.
|
||||
|
||||
1. Go to the `cluster-monitoring` application.
|
||||
|
||||
1. In the `cluster-monitoring` application, there are two `/index.html` links: one that leads to a Grafana instance and one that leads to a Prometheus instance. When you click the Grafana link, it will redirect you to a new webpage for Grafana, which shows metrics for the cluster.
|
||||
|
||||
1. You will be signed in to the Grafana instance automatically. The default username is `admin` and the default password is `admin`. For security, we recommend that you log out of Grafana, log back in with the `admin` password, and change your password.
|
||||
|
||||
**Results:** You are logged into Grafana from the Grafana instance. After logging in, you can view the preset Grafana dashboards, which are imported via the [Grafana provisioning mechanism](http://docs.grafana.org/administration/provisioning/#dashboards), so you cannot modify them directly. For now, if you want to configure your own dashboards, clone the original and modify the new copy.
|
||||
+156
@@ -0,0 +1,156 @@
|
||||
---
|
||||
title: CPU and Memory Allocations
|
||||
weight: 1
|
||||
aliases:
|
||||
- /rancher/v2.0-v2.4/en/project-admin/istio/configuring-resource-allocations/
|
||||
- /rancher/v2.0-v2.4/en/project-admin/istio/config/
|
||||
- /rancher/v2.0-v2.4/en/cluster-admin/tools/istio/resources
|
||||
- /rancher/v2.0-v2.4/en/istio/legacy/resources
|
||||
- /rancher/v2.0-v2.4/en/istio/v2.3.x-v2.4.x/resources
|
||||
- /rancher/v2.x/en/istio/v2.3.x-v2.4.x/resources/
|
||||
---
|
||||
_Available as of v2.3.0_
|
||||
|
||||
This section describes the minimum recommended computing resources for the Istio components in a cluster.
|
||||
|
||||
The CPU and memory allocations for each component are [configurable.](#configuring-resource-allocations)
|
||||
|
||||
Before enabling Istio, we recommend that you confirm that your Rancher worker nodes have enough CPU and memory to run all of the components of Istio.
|
||||
|
||||
> **Tip:** In larger deployments, it is strongly advised that the infrastructure be placed on dedicated nodes in the cluster by adding a node selector for each Istio component.
|
||||
|
||||
The table below shows a summary of the minimum recommended resource requests and limits for the CPU and memory of each central Istio component.
|
||||
|
||||
In Kubernetes, the resource request indicates that the workload will not deployed on a node unless the node has at least the specified amount of memory and CPU available. If the workload surpasses the limit for CPU or memory, it can be terminated or evicted from the node. For more information on managing resource limits for containers, refer to the [Kubernetes documentation.](https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/)
|
||||
|
||||
Workload | Container | CPU - Request | Mem - Request | CPU - Limit | Mem - Limit | Configurable
|
||||
---------|-----------|---------------|---------------|-------------|-------------|-------------
|
||||
istio-pilot |discovery| 500m | 2048Mi | 1000m | 4096Mi | Y
|
||||
istio-telemetry |mixer| 1000m | 1024Mi | 4800m | 4096Mi | Y
|
||||
istio-policy | mixer | 1000m | 1024Mi | 4800m | 4096Mi | Y
|
||||
istio-tracing | jaeger | 100m | 100Mi | 500m | 1024Mi | Y
|
||||
prometheus | prometheus | 750m | 750Mi | 1000m | 1024Mi | Y
|
||||
grafana | grafana | 100m | 100Mi | 200m | 512Mi | Y
|
||||
Others | - | 500m | 500Mi | - | - | N
|
||||
**Total** | **-** | **3950m** | **5546Mi** | **>12300m** | **>14848Mi** | **-**
|
||||
|
||||
|
||||
# Configuring Resource Allocations
|
||||
|
||||
You can individually configure the resource allocation for each type of Istio component. This section includes the default resource allocations for each component.
|
||||
|
||||
To make it easier to schedule the workloads to a node, a cluster administrator can reduce the CPU and memory resource requests for the component. However, the default CPU and memory allocations are the minimum that we recommend.
|
||||
|
||||
You can find more information about Istio configuration in the [official Istio documentation](https://istio.io/docs/concepts/what-is-istio).
|
||||
|
||||
To configure the resources allocated to an Istio component,
|
||||
|
||||
1. In Rancher, go to the cluster where you have Istio installed.
|
||||
1. Click **Tools > Istio.** This opens the Istio configuration page.
|
||||
1. Change the CPU or memory allocations, the nodes where each component will be scheduled to, or the node tolerations.
|
||||
1. Click **Save.**
|
||||
|
||||
**Result:** The resource allocations for the Istio components are updated.
|
||||
|
||||
## Pilot
|
||||
|
||||
[Pilot](https://istio.io/docs/ops/deployment/architecture/#pilot) provides the following:
|
||||
|
||||
- Authentication configuration
|
||||
- Service discovery for the Envoy sidecars
|
||||
- Traffic management capabilities for intelligent routing (A/B tests and canary rollouts)
|
||||
- Configuration for resiliency (timeouts, retries, circuit breakers, etc)
|
||||
|
||||
For more information on Pilot, refer to the [documentation](https://istio.io/docs/concepts/traffic-management/#pilot-and-envoy).
|
||||
|
||||
Option | Description| Required | Default
|
||||
-------|------------|-------|-------
|
||||
Pilot CPU Limit | CPU resource limit for the istio-pilot pod.| Yes | 1000
|
||||
Pilot CPU Reservation | CPU reservation for the istio-pilot pod. | Yes | 500
|
||||
Pilot Memory Limit | Memory resource limit for the istio-pilot pod. | Yes | 4096
|
||||
Pilot Memory Reservation | Memory resource requests for the istio-pilot pod. | Yes | 2048
|
||||
Trace sampling Percentage | [Trace sampling percentage](https://istio.io/docs/tasks/telemetry/distributed-tracing/overview/#trace-sampling) | Yes | 1
|
||||
Pilot Selector | Ability to select the nodes in which istio-pilot pod is deployed to. To use this option, the nodes must have labels. | No | n/a
|
||||
|
||||
## Mixer
|
||||
|
||||
[Mixer](https://istio.io/docs/ops/deployment/architecture/#mixer) enforces access control and usage policies across the service mesh. It also integrates with plugins for monitoring tools such as Prometheus. The Envoy sidecar proxy passes telemetry data and monitoring data to Mixer, and Mixer passes the monitoring data to Prometheus.
|
||||
|
||||
For more information on Mixer, policies and telemetry, refer to the [documentation](https://istio.io/docs/concepts/policies-and-telemetry/).
|
||||
|
||||
Option | Description| Required | Default
|
||||
-------|------------|-------|-------
|
||||
Mixer Telemetry CPU Limit | CPU resource limit for the istio-telemetry pod.| Yes | 4800
|
||||
Mixer Telemetry CPU Reservation | CPU reservation for the istio-telemetry pod.| Yes | 1000
|
||||
Mixer Telemetry Memory Limit | Memory resource limit for the istio-telemetry pod.| Yes | 4096
|
||||
Mixer Telemetry Memory Reservation | Memory resource requests for the istio-telemetry pod.| Yes | 1024
|
||||
Enable Mixer Policy | Whether or not to deploy the istio-policy. | Yes | False
|
||||
Mixer Policy CPU Limit | CPU resource limit for the istio-policy pod. | Yes, when policy enabled | 4800
|
||||
Mixer Policy CPU Reservation | CPU reservation for the istio-policy pod. | Yes, when policy enabled | 1000
|
||||
Mixer Policy Memory Limit | Memory resource limit for the istio-policy pod. | Yes, when policy enabled | 4096
|
||||
Mixer Policy Memory Reservation | Memory resource requests for the istio-policy pod. | Yes, when policy enabled | 1024
|
||||
Mixer Selector | Ability to select the nodes in which istio-policy and istio-telemetry pods are deployed to. To use this option, the nodes must have labels. | No | n/a
|
||||
|
||||
## Tracing
|
||||
|
||||
[Distributed tracing](https://istio.io/docs/tasks/telemetry/distributed-tracing/overview/) enables users to track a request through a service mesh. This makes it easier to troubleshoot problems with latency, parallelism and serialization.
|
||||
|
||||
Option | Description| Required | Default
|
||||
-------|------------|-------|-------
|
||||
Enable Tracing | Whether or not to deploy the istio-tracing. | Yes | True
|
||||
Tracing CPU Limit | CPU resource limit for the istio-tracing pod. | Yes | 500
|
||||
Tracing CPU Reservation | CPU reservation for the istio-tracing pod. | Yes | 100
|
||||
Tracing Memory Limit | Memory resource limit for the istio-tracing pod. | Yes | 1024
|
||||
Tracing Memory Reservation | Memory resource requests for the istio-tracing pod. | Yes | 100
|
||||
Tracing Selector | Ability to select the nodes in which tracing pod is deployed to. To use this option, the nodes must have labels. | No | n/a
|
||||
|
||||
## Ingress Gateway
|
||||
|
||||
The Istio gateway allows Istio features such as monitoring and route rules to be applied to traffic entering the cluster. This gateway is a prerequisite for outside traffic to make requests to Istio.
|
||||
|
||||
For more information, refer to the [documentation](https://istio.io/docs/tasks/traffic-management/ingress/).
|
||||
|
||||
Option | Description| Required | Default
|
||||
-------|------------|-------|-------
|
||||
Enable Ingress Gateway | Whether or not to deploy the istio-ingressgateway. | Yes | False
|
||||
Service Type of Istio Ingress Gateway | How to expose the gateway. You can choose NodePort or Loadbalancer | Yes | NodePort
|
||||
Http2 Port | The NodePort for http2 requests | Yes | 31380
|
||||
Https Port | The NodePort for https requests | Yes | 31390
|
||||
Load Balancer IP | Ingress Gateway Load Balancer IP | No | n/a
|
||||
Load Balancer Source Ranges | Ingress Gateway Load Balancer Source Ranges | No | n/a
|
||||
Ingress Gateway CPU Limit | CPU resource limit for the istio-ingressgateway pod. | Yes | 2000
|
||||
Ingress Gateway CPU Reservation | CPU reservation for the istio-ingressgateway pod. | Yes | 100
|
||||
Ingress Gateway Memory Limit | Memory resource limit for the istio-ingressgateway pod. | Yes | 1024
|
||||
Ingress Gateway Memory Reservation | Memory resource requests for the istio-ingressgateway pod. | Yes | 128
|
||||
Ingress Gateway Selector | Ability to select the nodes in which istio-ingressgateway pod is deployed to. To use this option, the nodes must have labels. | No | n/a
|
||||
|
||||
## Prometheus
|
||||
|
||||
You can query for Istio metrics using Prometheus. Prometheus is an open-source systems monitoring and alerting toolkit.
|
||||
|
||||
Option | Description| Required | Default
|
||||
-------|------------|-------|-------
|
||||
Prometheus CPU Limit | CPU resource limit for the Prometheus pod.| Yes | 1000
|
||||
Prometheus CPU Reservation | CPU reservation for the Prometheus pod.| Yes | 750
|
||||
Prometheus Memory Limit | Memory resource limit for the Prometheus pod.| Yes | 1024
|
||||
Prometheus Memory Reservation | Memory resource requests for the Prometheus pod.| Yes | 750
|
||||
Retention for Prometheus | How long your Prometheus instance retains data | Yes | 6
|
||||
Prometheus Selector | Ability to select the nodes in which Prometheus pod is deployed to. To use this option, the nodes must have labels.| No | n/a
|
||||
|
||||
## Grafana
|
||||
|
||||
You can visualize metrics with Grafana. Grafana lets you visualize Istio traffic data scraped by Prometheus.
|
||||
|
||||
Option | Description| Required | Default
|
||||
-------|------------|-------|-------
|
||||
Enable Grafana | Whether or not to deploy the Grafana.| Yes | True
|
||||
Grafana CPU Limit | CPU resource limit for the Grafana pod.| Yes, when Grafana enabled | 200
|
||||
Grafana CPU Reservation | CPU reservation for the Grafana pod.| Yes, when Grafana enabled | 100
|
||||
Grafana Memory Limit | Memory resource limit for the Grafana pod.| Yes, when Grafana enabled | 512
|
||||
Grafana Memory Reservation | Memory resource requests for the Grafana pod.| Yes, when Grafana enabled | 100
|
||||
Grafana Selector | Ability to select the nodes in which Grafana pod is deployed to. To use this option, the nodes must have labels. | No | n/a
|
||||
Enable Persistent Storage for Grafana | Enable Persistent Storage for Grafana | Yes, when Grafana enabled | False
|
||||
Source | Use a Storage Class to provision a new persistent volume or Use an existing persistent volume claim | Yes, when Grafana enabled and enabled PV | Use SC
|
||||
Storage Class | Storage Class for provisioning PV for Grafana | Yes, when Grafana enabled, enabled PV and use storage class | Use the default class
|
||||
Persistent Volume Size | The size for the PV you would like to provision for Grafana | Yes, when Grafana enabled, enabled PV and use storage class | 5Gi
|
||||
Existing Claim | Use existing PVC for Grafana | Yes, when Grafana enabled, enabled PV and use existing PVC | n/a
|
||||
+32
@@ -0,0 +1,32 @@
|
||||
---
|
||||
title: Disabling Istio
|
||||
weight: 4
|
||||
aliases:
|
||||
- /rancher/v2.0-v2.4/en/cluster-admin/tools/istio/disabling-istio
|
||||
- /rancher/v2.0-v2.4/en/istio/legacy/disabling-istio
|
||||
- /rancher/v2.0-v2.4/en/istio/v2.3.x-v2.4.x/disabling-istio
|
||||
- /rancher/v2.x/en/istio/v2.3.x-v2.4.x/disabling-istio/
|
||||
---
|
||||
|
||||
This section describes how to disable Istio in a cluster, namespace, or workload.
|
||||
|
||||
# Disable Istio in a Cluster
|
||||
|
||||
To disable Istio,
|
||||
|
||||
1. From the **Global** view, navigate to the cluster that you want to disable Istio for.
|
||||
1. Click **Tools > Istio.**
|
||||
1. Click **Disable,** then click the red button again to confirm the disable action.
|
||||
|
||||
**Result:** The `cluster-istio` application in the cluster's `system` project gets removed. The Istio sidecar cannot be deployed on any workloads in the cluster.
|
||||
|
||||
# Disable Istio in a Namespace
|
||||
|
||||
1. In the Rancher UI, go to the project that has the namespace where you want to disable Istio.
|
||||
1. On the **Workloads** tab, you will see a list of namespaces and the workloads deployed in them. Go to the namespace where you want to disable and click the **⋮ > Disable Istio Auto Injection.**
|
||||
|
||||
**Result:** When workloads are deployed in this namespace, they will not have the Istio sidecar.
|
||||
|
||||
# Remove the Istio Sidecar from a Workload
|
||||
|
||||
Disable Istio in the namespace, then redeploy the workloads with in it. They will be deployed without the Istio sidecar.
|
||||
+63
@@ -0,0 +1,63 @@
|
||||
---
|
||||
title: Role-based Access Control
|
||||
weight: 3
|
||||
aliases:
|
||||
- /rancher/v2.0-v2.4/en/cluster-admin/tools/istio/rbac
|
||||
- /rancher/v2.0-v2.4/en/istio/legacy/rbac
|
||||
- /rancher/v2.0-v2.4/en/istio/v2.3.x-v2.4.x/rbac
|
||||
- /rancher/v2.x/en/istio/v2.3.x-v2.4.x/rbac/
|
||||
---
|
||||
|
||||
This section describes the permissions required to access Istio features and how to configure access to the Kiali and Jaeger visualizations.
|
||||
|
||||
# Cluster-level Access
|
||||
|
||||
By default, only cluster administrators can:
|
||||
|
||||
- Enable Istio for the cluster
|
||||
- Configure resource allocations for Istio
|
||||
- View each UI for Prometheus, Grafana, Kiali, and Jaeger
|
||||
|
||||
# Project-level Access
|
||||
|
||||
After Istio is enabled in a cluster, project owners and members have permission to:
|
||||
|
||||
- Enable and disable Istio sidecar auto-injection for namespaces
|
||||
- Add the Istio sidecar to workloads
|
||||
- View the traffic metrics and traffic graph for the cluster
|
||||
- View the Kiali and Jaeger visualizations if cluster administrators give access to project members
|
||||
- Configure Istio's resources (such as the gateway, destination rules, or virtual services) with `kubectl` (This does not apply to read-only project members)
|
||||
|
||||
# Access to Visualizations
|
||||
|
||||
By default, the Kiali and Jaeger visualizations are restricted to the cluster owner because the information in them could be sensitive.
|
||||
|
||||
**Jaeger** provides a UI for a distributed tracing system, which is useful for root cause analysis and for determining what causes poor performance.
|
||||
|
||||
**Kiali** provides a diagram that shows the services within a service mesh and how they are connected.
|
||||
|
||||
Rancher supports giving groups permission to access Kiali and Jaeger, but not individuals.
|
||||
|
||||
To configure who has permission to access the Kiali and Jaeger UI,
|
||||
|
||||
1. Go to the cluster view and click **Tools > Istio.**
|
||||
1. Then go to the **Member Access** section. If you want to restrict access to certain groups, choose **Allow cluster owner and specified members to access Kiali and Jaeger UI.** Search for the groups that you want to have access to Kiali and Jaeger. If you want all members to have access to the tools, click **Allow all members to access Kiali and Jaeger UI.**
|
||||
1. Click **Save.**
|
||||
|
||||
**Result:** The access levels for Kiali and Jaeger have been updated.
|
||||
|
||||
# Summary of Default Permissions for Istio Users
|
||||
|
||||
| Permission | Cluster Administrators | Project Owners | Project Members | Read-only Project Members |
|
||||
|------------------------------------------|----------------|----------------|-----------------|---------------------------|
|
||||
| Enable and disable Istio for the cluster | ✓ | | | |
|
||||
| Configure Istio resource limits | ✓ | | | |
|
||||
| Control who has access to Kiali and the Jaeger UI | ✓ | | | |
|
||||
| Enable and disable Istio for a namespace | ✓ | ✓ | ✓ | |
|
||||
| Enable and disable Istio on workloads | ✓ | ✓ | ✓ | |
|
||||
| Configure Istio with `kubectl` | ✓ | ✓ | ✓ | |
|
||||
| View Prometheus UI and Grafana UI | ✓ | | | |
|
||||
| View Kiali UI and Jaeger UI ([Configurable](#access-to-visualizations)) | ✓ | | | |
|
||||
| View Istio project dashboard, including traffic metrics* | ✓ | ✓ | ✓ | ✓ |
|
||||
|
||||
* By default, only the cluster owner will see the traffic graph. Project members will see only a subset of traffic metrics. Project members cannot see the traffic graph because it comes from Kiali, and access to Kiali is restricted to cluster owners by default.
|
||||
+23
@@ -0,0 +1,23 @@
|
||||
---
|
||||
title: Release Notes
|
||||
aliases:
|
||||
- /rancher/v2.0-v2.4/en/cluster-admin/tools/istio/release-notes
|
||||
- /rancher/v2.0-v2.4/en/istio/legacy/release-notes
|
||||
- /rancher/v2.0-v2.4/en/istio/v2.3.x-v2.4.x/release-notes
|
||||
- /rancher/v2.x/en/istio/v2.3.x-v2.4.x/release-notes/
|
||||
---
|
||||
|
||||
|
||||
# Istio 1.5.8
|
||||
|
||||
### Important note on 1.5.x versions
|
||||
|
||||
When upgrading from any 1.4 version of Istio to any 1.5 version, the Rancher installer will delete several resources in order to complete the upgrade, at which point they will be immediately re-installed. This includes the `istio-reader-service-account`. If your Istio installation is using this service account be aware that any secrets tied to the service account will be deleted. Most notably this will **break specific [multi-cluster deployments](https://archive.istio.io/v1.4/docs/setup/install/multicluster/)**. Downgrades back to 1.4 are not possible.
|
||||
|
||||
See the official upgrade notes for additional information on the 1.5 release and upgrading from 1.4: https://istio.io/latest/news/releases/1.5.x/announcing-1.5/upgrade-notes/
|
||||
|
||||
> **Note:** Rancher continues to use the Helm installation method, which produces a different architecture from an istioctl installation.
|
||||
|
||||
### Known Issues
|
||||
|
||||
* The Kiali traffic graph is currently not working [#24924](https://github.com/istio/istio/issues/24924)
|
||||
@@ -0,0 +1,207 @@
|
||||
---
|
||||
title: Notifiers
|
||||
weight: 4
|
||||
aliases:
|
||||
- /rancher/v2.0-v2.4/en/project-admin/tools/notifiers
|
||||
- /rancher/v2.0-v2.4/en/cluster-admin/tools/notifiers
|
||||
- /rancher/v2.0-v2.4/en/monitoring-alerting/legacy/notifiers
|
||||
- /rancher/v2.0-v2.4/en/monitoring-alerting/v2.0.x-v2.4.x/notifiers
|
||||
- /rancher/v2.x/en/monitoring-alerting/v2.0.x-v2.4.x/notifiers/
|
||||
- /rancher/v2.x/en/monitoring-alerting/v2.0.x-v2.4.x/cluster-alerts/default-alerts/
|
||||
---
|
||||
|
||||
Notifiers are services that inform you of alert events. You can configure notifiers to send alert notifications to staff best suited to take corrective action.
|
||||
|
||||
Rancher integrates with a variety of popular IT services, including:
|
||||
|
||||
- **Slack**: Send alert notifications to your Slack channels.
|
||||
- **Email**: Choose email recipients for alert notifications.
|
||||
- **PagerDuty**: Route notifications to staff by phone, SMS, or personal email.
|
||||
- **WebHooks**: Update a webpage with alert notifications.
|
||||
- **WeChat**: (Available as of v2.2.0) Send alert notifications to your Enterprise WeChat contacts.
|
||||
- **DingTalk**: (Available as of v2.4.6) Send alert notifications to DingTalk using a webhook.
|
||||
- **Microsoft Teams**: (Available as of v2.4.6) Send alert notifications to Teams using a webhook.
|
||||
|
||||
This section covers the following topics:
|
||||
|
||||
- [Roles-based access control for notifiers](#roles-based-access-control-for-notifiers)
|
||||
- [Adding notifiers](#adding-notifiers)
|
||||
- [Configuration](#configuration)
|
||||
- [Managing notifiers](#managing-notifiers)
|
||||
- [Example payload for a webhook alert notifier](#example-payload-for-a-webhook-alert-notifier)
|
||||
|
||||
# Roles-based Access Control for Notifiers
|
||||
|
||||
Notifiers are configured at the cluster level. This model ensures that only cluster owners need to configure notifiers, leaving project owners to simply configure alerts in the scope of their projects. You don't need to dispense privileges like SMTP server access or cloud account access.
|
||||
|
||||
# Adding Notifiers
|
||||
|
||||
Set up a notifier so that you can begin configuring and sending alerts.
|
||||
|
||||
1. From the **Global View**, open the cluster that you want to add a notifier.
|
||||
1. From the main menu, select **Tools > Notifiers**. Then click **Add Notifier**.
|
||||
1. Select the service you want to use as your notifier, and then fill out the form. For help filling out the form, refer to the configuration section below.
|
||||
1. Click **Test.** You should receive a notification confirming that the notifier is configured correctly.
|
||||
1. Click **Add** to complete adding the notifier.
|
||||
|
||||
**Result:** Your notifier is added to Rancher.
|
||||
|
||||
# Configuration
|
||||
|
||||
- [Slack](#slack)
|
||||
- [Email](#email)
|
||||
- [PagerDuty](#pagerduty)
|
||||
- [Webhook](#webhook)
|
||||
- [WeChat](#wechat)
|
||||
- [DingTalk](#dingtalk)
|
||||
- [Microsoft Teams](#microsoft-teams)
|
||||
|
||||
### Slack
|
||||
|
||||
| Field | Explanation |
|
||||
|----------|----------------------|
|
||||
| Name | Enter a **Name** for the notifier. |
|
||||
| URL | From Slack, create a webhook. For instructions, see the [Slack Documentation](https://get.slack.help/hc/en-us/articles/115005265063-Incoming-WebHooks-for-Slack). Then enter the Slack webhook URL. |
|
||||
| Default Channel | Enter the name of the channel that you want to send alert notifications in the following format: `#<channelname>`. Both public and private channels are supported. |
|
||||
| Proxy URL | Proxy for the Slack webhook. |
|
||||
| Send Resolved Alerts | _Available as of v2.3.0_ Whether to send a follow-up notification if an alert has been resolved (e.g. [Resolved] High CPU Usage) |
|
||||
|
||||
**Validation:** Click **Test**. If the test is successful, the Slack channel you're configuring for the notifier outputs **Slack setting validated.**
|
||||
|
||||
### Email
|
||||
|
||||
| Field | Explanation |
|
||||
|----------|----------------------|
|
||||
| Name | Enter a **Name** for the notifier. |
|
||||
| Default Recipient Address | Enter the email address that you want to receive the notification. |
|
||||
| Send Resolved Alerts | _Available as of v2.3.0_ Whether to send a follow-up notification if an alert has been resolved (e.g. [Resolved] High CPU Usage) |
|
||||
|
||||
SMTP Server Configuration:
|
||||
|
||||
| Field | Explanation |
|
||||
|----------|----------------------|
|
||||
| Sender | Enter an email address available on your mail server that you want to send the notification. |
|
||||
| Host | Enter the IP address or hostname for your SMTP server. Example: `smtp.email.com` |
|
||||
| Port | In the **Port** field, enter the port used for email. Typically, TLS uses `587` and SSL uses `465`. |
|
||||
| Use TLS | If you're using TLS, make sure **Use TLS** is selected. |
|
||||
| Username | Username to authenticate with the SMTP server. |
|
||||
| Password | Password to authenticate with the SMTP server. |
|
||||
|
||||
**Validation:** Click **Test**. If the test is successful, Rancher prints **settings validated** and you receive a test notification email.
|
||||
|
||||
### PagerDuty
|
||||
|
||||
| Field | Explanation |
|
||||
|----------|----------------------|
|
||||
| Name | Enter a **Name** for the notifier. |
|
||||
| Default Integration Key | From PagerDuty, create a Prometheus integration. For instructions, see the [PagerDuty Documentation](https://www.pagerduty.com/docs/guides/prometheus-integration-guide/). Then enter the integration key.
|
||||
| Service Key | The same as the integration key. For instructions on creating a Prometheus integration, see the [PagerDuty Documentation](https://www.pagerduty.com/docs/guides/prometheus-integration-guide/). Then enter the integration key. |
|
||||
| Send Resolved Alerts | _Available as of v2.3.0_ Whether to send a follow-up notification if an alert has been resolved (e.g. [Resolved] High CPU Usage) |
|
||||
|
||||
**Validation:** Click **Test**. If the test is successful, your PagerDuty endpoint outputs **PagerDuty setting validated.**
|
||||
|
||||
### Webhook
|
||||
|
||||
| Field | Explanation |
|
||||
|----------|----------------------|
|
||||
| Name | Enter a **Name** for the notifier. |
|
||||
| URL | Using the app of your choice, create a webhook URL. |
|
||||
| Proxy URL | Proxy for the webhook. |
|
||||
| Send Resolved Alerts | _Available as of v2.3.0_ Whether to send a follow-up notification if an alert has been resolved (e.g. [Resolved] High CPU Usage) |
|
||||
|
||||
**Validation:** Click **Test**. If the test is successful, the URL you're configuring as a notifier outputs **Webhook setting validated.**
|
||||
|
||||
### WeChat
|
||||
|
||||
_Available as of v2.2.0_
|
||||
|
||||
| Field | Explanation |
|
||||
|----------|----------------------|
|
||||
| Name | Enter a **Name** for the notifier. |
|
||||
| Corporation ID | Enter the "EnterpriseID" of your corporation. You can get it fro the [Profile page](https://work.weixin.qq.com/wework_admin/frame#profile). |
|
||||
| Application Agent ID | From Enterprise WeChat, create an application in the [Application page](https://work.weixin.qq.com/wework_admin/frame#apps), and then enter the "AgentId" of this application. You will also need to enter the application secret. |
|
||||
| Application Secret | The secret that corresponds to the Application Agent ID. |
|
||||
| Recipient Type | Party, tag, or user. |
|
||||
| Default Recipient | The default recipient ID should correspond to the recipient type. It should be the party ID, tag ID or user account that you want to receive the notification. You could get contact information from [Contacts page](https://work.weixin.qq.com/wework_admin/frame#contacts). |
|
||||
| Proxy URL | If you are using a proxy, enter the proxy URL. |
|
||||
| Send Resolved Alerts | _Available as of v2.3.0_ Whether to send a follow-up notification if an alert has been resolved (e.g. [Resolved] High CPU Usage) |
|
||||
|
||||
**Validation:** Click **Test.** If the test is successful, you should receive an alert message.
|
||||
|
||||
### DingTalk
|
||||
|
||||
_Available as of v2.4.6_
|
||||
|
||||
| Field | Explanation |
|
||||
|----------|----------------------|
|
||||
| Name | Enter a **Name** for the notifier. |
|
||||
| Webhook URL | Enter the DingTalk webhook URL. For help setting up the webhook, refer to the [DingTalk documentation.](https://www.alibabacloud.com/help/doc-detail/52872.htm) |
|
||||
| Secret | Optional: Enter a secret for the DingTalk webhook. |
|
||||
| Proxy URL | Optional: Enter a proxy for the DingTalk webhook. |
|
||||
| Send Resolved Alerts | Whether to send a follow-up notification if an alert has been resolved (e.g. [Resolved] High CPU Usage) |
|
||||
|
||||
**Validation:** Click **Test.** If the test is successful, the DingTalk notifier output is **DingTalk setting validated.**
|
||||
|
||||
### Microsoft Teams
|
||||
|
||||
_Available as of v2.4.6_
|
||||
|
||||
| Field | Explanation |
|
||||
|----------|----------------------|
|
||||
| Name | Enter a **Name** for the notifier. |
|
||||
| Webhook URL | Enter the Microsoft Teams webhook URL. For help setting up the webhook, refer to the [Teams Documentation.](https://docs.microsoft.com/en-us/microsoftteams/platform/webhooks-and-connectors/how-to/add-incoming-webhook) |
|
||||
| Proxy URL | Optional: Enter a proxy for the Teams webhook. |
|
||||
| Send Resolved Alerts | Whether to send a follow-up notification if an alert has been resolved (e.g. [Resolved] High CPU Usage) |
|
||||
|
||||
**Validation:** Click **Test.** If the test is successful, the Teams notifier output is **MicrosoftTeams setting validated.**
|
||||
|
||||
# Managing Notifiers
|
||||
|
||||
After you set up notifiers, you can manage them. From the **Global** view, open the cluster that you want to manage your notifiers. Select **Tools > Notifiers**. You can:
|
||||
|
||||
- **Edit** their settings that you configured during their initial setup.
|
||||
- **Clone** them, to quickly setup slightly different notifiers.
|
||||
- **Delete** them when they're no longer necessary.
|
||||
|
||||
# Example Payload for a Webhook Alert Notifier
|
||||
|
||||
```json
|
||||
{
|
||||
"receiver": "c-2a3bc:kube-components-alert",
|
||||
"status": "firing",
|
||||
"alerts": [
|
||||
{
|
||||
"status": "firing",
|
||||
"labels": {
|
||||
"alert_name": "Scheduler is unavailable",
|
||||
"alert_type": "systemService",
|
||||
"cluster_name": "mycluster (ID: c-2a3bc)",
|
||||
"component_name": "scheduler",
|
||||
"group_id": "c-2a3bc:kube-components-alert",
|
||||
"logs": "Get http://127.0.0.1:10251/healthz: dial tcp 127.0.0.1:10251: connect: connection refused",
|
||||
"rule_id": "c-2a3bc:kube-components-alert_scheduler-system-service",
|
||||
"severity": "critical"
|
||||
},
|
||||
"annotations": {},
|
||||
"startsAt": "2020-01-30T19:18:13.321684733Z",
|
||||
"endsAt": "0001-01-01T00:00:00Z",
|
||||
"generatorURL": ""
|
||||
}
|
||||
],
|
||||
"groupLabels": {
|
||||
"component_name": "scheduler",
|
||||
"rule_id": "c-2a3bc:kube-components-alert_scheduler-system-service"
|
||||
},
|
||||
"commonLabels": {
|
||||
"alert_name": "Scheduler is unavailable",
|
||||
"alert_type": "systemService",
|
||||
"cluster_name": "mycluster (ID: c-2a3bc)"
|
||||
}
|
||||
}
|
||||
```
|
||||
# What's Next?
|
||||
|
||||
After creating a notifier, set up alerts to receive notifications of Rancher system events.
|
||||
|
||||
- [Cluster owners](../../how-to-guides/advanced-user-guides/authentication-permissions-and-global-configuration/manage-role-based-access-control-rbac/cluster-and-project-roles.md#cluster-roles) can set up alerts at the [cluster level](cluster-admin/tools/alerts/).
|
||||
- [Project owners](../../how-to-guides/advanced-user-guides/authentication-permissions-and-global-configuration/manage-role-based-access-control-rbac/cluster-and-project-roles.md#project-roles) can set up alerts at the [project level](project-admin/tools/alerts/).
|
||||
@@ -0,0 +1,99 @@
|
||||
---
|
||||
title: OPA Gatekeeper
|
||||
weight: 17
|
||||
aliases:
|
||||
- /rancher/v2.0-v2.4/en/cluster-admin/tools/opa-gatekeeper
|
||||
- /rancher/v2.0-v2.4/en/opa-gatekeper/Open%20Policy%20Agent
|
||||
- /rancher/v2.0-v2.4/en/opa-gatekeper
|
||||
---
|
||||
_Available as of v2.4.0_
|
||||
|
||||
To ensure consistency and compliance, every organization needs the ability to define and enforce policies in its environment in an automated way. [OPA (Open Policy Agent)](https://www.openpolicyagent.org/) is a policy engine that facilitates policy-based control for cloud native environments. Rancher provides the ability to enable OPA Gatekeeper in Kubernetes clusters, and also installs a couple of built-in policy definitions, which are also called constraint templates.
|
||||
|
||||
OPA provides a high-level declarative language that lets you specify policy as code and ability to extend simple APIs to offload policy decision-making.
|
||||
|
||||
[OPA Gatekeeper](https://github.com/open-policy-agent/gatekeeper) is a project that provides integration between OPA and Kubernetes. OPA Gatekeeper provides:
|
||||
|
||||
- An extensible, parameterized policy library.
|
||||
- Native Kubernetes CRDs for instantiating the policy library, also called “constraints."
|
||||
- Native Kubernetes CRDs for extending the policy library, also called "constraint templates."
|
||||
- Audit functionality.
|
||||
|
||||
To read more about OPA, please refer to the [official documentation.](https://www.openpolicyagent.org/docs/latest/)
|
||||
|
||||
# How the OPA Gatekeeper Integration Works
|
||||
|
||||
Kubernetes provides the ability to extend API server functionality via admission controller webhooks, which are invoked whenever a resource is created, updated or deleted. Gatekeeper is installed as a validating webhook and enforces policies defined by Kubernetes custom resource definitions. In addition to the admission control usage, Gatekeeper provides the capability to audit existing resources in Kubernetes clusters and mark current violations of enabled policies.
|
||||
|
||||
OPA Gatekeeper is made available via Rancher's Helm system chart, and it is installed in a namespace named `gatekeeper-system.`
|
||||
|
||||
# Enabling OPA Gatekeeper in a Cluster
|
||||
|
||||
> **Prerequisites:**
|
||||
>
|
||||
> - Only administrators and cluster owners can enable OPA Gatekeeper.
|
||||
> - The dashboard needs to be enabled using the `dashboard` feature flag. For more information, refer to the [section on enabling experimental features.](installation/options/feature-flags/)
|
||||
|
||||
1. Navigate to the cluster's **Dashboard** view.
|
||||
1. On the left side menu, expand the cluster menu and click on **OPA Gatekeeper.**
|
||||
1. To install Gatekeeper with the default configuration, click on **Enable Gatekeeper (v0.1.0) with defaults.**
|
||||
1. To change any default configuration, click on **Customize Gatekeeper yaml configuration.**
|
||||
|
||||
# Constraint Templates
|
||||
|
||||
[Constraint templates](https://github.com/open-policy-agent/gatekeeper#constraint-templates) are Kubernetes custom resources that define the schema and Rego logic of the OPA policy to be applied by Gatekeeper. For more information on the Rego policy language, refer to the [official documentation.](https://www.openpolicyagent.org/docs/latest/policy-language/)
|
||||
|
||||
When OPA Gatekeeper is enabled, Rancher installs some templates by default.
|
||||
|
||||
To list the constraint templates installed in the cluster, go to the left side menu under OPA Gatekeeper and click on **Templates.**
|
||||
|
||||
Rancher also provides the ability to create your own constraint templates by importing YAML definitions.
|
||||
|
||||
# Creating and Configuring Constraints
|
||||
|
||||
[Constraints](https://github.com/open-policy-agent/gatekeeper#constraints) are Kubernetes custom resources that define the scope of objects to which a specific constraint template applies to. The complete policy is defined by constraint templates and constraints together.
|
||||
|
||||
> **Prerequisites:** OPA Gatekeeper must be enabled in the cluster.
|
||||
|
||||
To list the constraints installed, go to the left side menu under OPA Gatekeeper, and click on **Constraints.**
|
||||
|
||||
New constraints can be created from a constraint template.
|
||||
|
||||
Rancher provides the ability to create a constraint by using a convenient form that lets you input the various constraint fields.
|
||||
|
||||
The **Edit as yaml** option is also available to configure the the constraint's yaml definition.
|
||||
|
||||
### Exempting Rancher's System Namespaces from Constraints
|
||||
|
||||
When a constraint is created, ensure that it does not apply to any Rancher or Kubernetes system namespaces. If the system namespaces are not excluded, then it is possible to see many resources under them marked as violations of the constraint.
|
||||
|
||||
To limit the scope of the constraint only to user namespaces, always specify these namespaces under the **Match** field of the constraint.
|
||||
|
||||
Also, the constraint may interfere with other Rancher functionality and deny system workloads from being deployed. To avoid this, exclude all Rancher-specific namespaces from your constraints.
|
||||
|
||||
# Enforcing Constraints in your Cluster
|
||||
|
||||
When the **Enforcement Action** is **Deny,** the constraint is immediately enabled and will deny any requests that violate the policy defined. By default, the enforcement value is **Deny.**
|
||||
|
||||
When the **Enforcement Action** is **Dryrun,** then any resources that violate the policy are only recorded under the constraint's status field.
|
||||
|
||||
To enforce constraints, create a constraint using the form. In the **Enforcement Action** field, choose **Deny.**
|
||||
|
||||
# Audit and Violations in your Cluster
|
||||
|
||||
OPA Gatekeeper runs a periodic audit to check if any existing resource violates any enforced constraint. The audit-interval (default 300s) can be configured while installing Gatekeeper.
|
||||
|
||||
On the Gatekeeper page, any violations of the defined constraints are listed.
|
||||
|
||||
Also under **Constraints,** the number of violations of the constraint can be found.
|
||||
|
||||
The detail view of each constraint lists information about the resource that violated the constraint.
|
||||
|
||||
# Disabling Gatekeeper
|
||||
|
||||
1. Navigate to the cluster's Dashboard view
|
||||
1. On the left side menu, expand the cluster menu and click on **OPA Gatekeeper.**
|
||||
1. Click the **⋮ > Disable**.
|
||||
|
||||
**Result:** Upon disabling OPA Gatekeeper, all constraint templates and constraints will also be deleted.
|
||||
|
||||
Reference in New Issue
Block a user