public/rancher-docs
1
0
Fork 0
mirror of https://github.com/rancher/rancher-docs.git synced 2026-09-26 04:58:07 +00:00
Code Issues Packages Projects Releases Wiki Activity

Remove unneeded intermediate folders

Browse Source
This commit is contained in:
Billy Tat
2022-08-17 10:23:03 -07:00
parent 506e174643
commit 07355d1446
1146 changed files with 0 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
versioned_docs/version-2.0-2.4/cluster-admin/tools/cis-scans/cis-scans.md
+156
View File
@@ -0,0 +1,156 @@
---
title: CIS Scans
weight: 18
aliases:
- /rancher/v2.0-v2.4/en/cis-scans/legacy
- /rancher/v2.0-v2.4/en/cis-scans
- /rancher/v2.x/en/cis-scans/v2.4/
---
_Available as of v2.4.0_
- [Prerequisites](#prerequisites)
- [Running a scan](#running-a-scan)
- [Scheduling recurring scans](#scheduling-recurring-scans)
- [Skipping tests](#skipping-tests)
- [Setting alerts](#setting-alerts)
- [Deleting a report](#deleting-a-report)
- [Downloading a report](#downloading-a-report)
- [List of skipped and not applicable tests](#list-of-skipped-and-not-applicable-tests)
# Prerequisites
To run security scans on a cluster and access the generated reports, you must be an [Administrator]({{<baseurl>}}/rancher/v2.0-v2.4/en/admin-settings/rbac/global-permissions/) or [Cluster Owner.]({{<baseurl>}}/rancher/v2.0-v2.4/en/admin-settings/rbac/cluster-project-roles/)
Rancher can only run security scans on clusters that were created with RKE, which includes custom clusters and clusters that Rancher created in an infrastructure provider such as Amazon EC2 or GCE. Imported clusters and clusters in hosted Kubernetes providers can't be scanned by Rancher.
The security scan cannot run in a cluster that has Windows nodes.
You will only be able to see the CIS scan reports for clusters that you have access to.
# Running a Scan
1. From the cluster view in Rancher, click **Tools > CIS Scans.**
1. Click **Run Scan.**
1. Choose a CIS scan profile.
**Result:** A report is generated and displayed in the **CIS Scans** page. To see details of the report, click the report's name.
# Scheduling Recurring Scans
Recurring scans can be scheduled to run on any RKE Kubernetes cluster.
To enable recurring scans, edit the advanced options in the cluster configuration during cluster creation or after the cluster has been created.
To schedule scans for an existing cluster:
1. Go to the cluster view in Rancher.
1. Click **Tools > CIS Scans.**
1. Click **Add Schedule.** This takes you to the section of the cluster editing page that is applicable to configuring a schedule for CIS scans. (This section can also be reached by going to the cluster view, clicking **&#8942; > Edit,** and going to the **Advanced Options.**)
1. In the **CIS Scan Enabled** field, click **Yes.**
1. In the **CIS Scan Profile** field, choose a **Permissive** or **Hardened** profile. The corresponding CIS Benchmark version is included in the profile name. Note: Any skipped tests [defined in a separate ConfigMap](#skipping-tests) will be skipped regardless of whether a **Permissive** or **Hardened** profile is selected. When selecting the the permissive profile, you should see which tests were skipped by Rancher (tests that are skipped by default for RKE clusters) and which tests were skipped by a Rancher user. In the hardened test profile, the only skipped tests will be skipped by users.
1. In the **CIS Scan Interval (cron)** job, enter a [cron expression](https://en.wikipedia.org/wiki/Cron#CRON_expression) to define how often the cluster will be scanned.
1. In the **CIS Scan Report Retention** field, enter the number of past reports that should be kept.
**Result:** The security scan will run and generate reports at the scheduled intervals.
The test schedule can be configured in the `cluster.yml`:
```yaml
scheduled_cluster_scan:
    enabled: true
    scan_config:
        cis_scan_config:
            override_benchmark_version: rke-cis-1.4
            profile: permissive
    schedule_config:
        cron_schedule: 0 0 * * *
        retention: 24
```
# Skipping Tests
You can define a set of tests that will be skipped by the CIS scan when the next report is generated.
These tests will be skipped for subsequent CIS scans, including both manually triggered and scheduled scans, and the tests will be skipped with any profile.
The skipped tests will be listed alongside the test profile name in the cluster configuration options when a test profile is selected for a recurring cluster scan. The skipped tests will also be shown every time a scan is triggered manually from the Rancher UI by clicking **Run Scan.** The display of skipped tests allows you to know ahead of time which tests will be run in each scan.
To skip tests, you will need to define them in a Kubernetes ConfigMap resource. Each skipped CIS scan test is listed in the ConfigMap alongside the version of the CIS benchmark that the test belongs to.
To skip tests by editing a ConfigMap resource,
1. Create a `security-scan` namespace.
1. Create a ConfigMap named `security-scan-cfg`.
1. Enter the skip information under the key `config.json` in the following format:
```json
{
"skip": {
"rke-cis-1.4": [
"1.1.1",
"1.2.2"
]
}
}
```
In the example above, the CIS benchmark version is specified alongside the tests to be skipped for that version.
**Result:** These tests will be skipped on subsequent scans that use the defined CIS Benchmark version.
# Setting Alerts
Rancher provides a set of alerts for cluster scans. which are not configured to have notifiers by default:
- A manual cluster scan was completed
- A manual cluster scan has failures
- A scheduled cluster scan was completed
- A scheduled cluster scan has failures
> **Prerequisite:** You need to configure a [notifier]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/tools/notifiers/) before configuring, sending, or receiving alerts.
To activate an existing alert for a CIS scan result,
1. From the cluster view in Rancher, click **Tools > Alerts.**
1. Go to the section called **A set of alerts for cluster scans.**
1. Go to the alert you want to activate and click **&#8942; > Activate.**
1. Go to the alert rule group **A set of alerts for cluster scans** and click **&#8942; > Edit.**
1. Scroll down to the **Alert** section. In the **To** field, select the notifier that you would like to use for sending alert notifications.
1. Optional: To limit the frequency of the notifications, click on **Show advanced options** and configure the time interval of the alerts.
1. Click **Save.**
**Result:** The notifications will be triggered when the a scan is run on a cluster and the active alerts have satisfied conditions.
To create a new alert,
1. Go to the cluster view and click **Tools > CIS Scans.**
1. Click **Add Alert.**
1. Fill out the form.
1. Enter a name for the alert.
1. In the **Is** field, set the alert to be triggered when a scan is completed or when a scan has a failure.
1. In the **Send a** field, set the alert as a **Critical,** **Warning,** or **Info** alert level.
1. Choose a [notifier]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/tools/notifiers/) for the alert.
**Result:** The alert is created and activated. The notifications will be triggered when the a scan is run on a cluster and the active alerts have satisfied conditions.
For more information about alerts, refer to [this page.]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/tools/alerts/)
# Deleting a Report
1. From the cluster view in Rancher, click **Tools > CIS Scans.**
1. Go to the report that should be deleted.
1. Click the **&#8942; > Delete.**
1. Click **Delete.**
# Downloading a Report
1. From the cluster view in Rancher, click **Tools > CIS Scans.**
1. Go to the report that you want to download. Click **&#8942; > Download.**
**Result:** The report is downloaded in CSV format.
# List of Skipped and Not Applicable Tests
For a list of skipped and not applicable tests, refer to <a href="{{<baseurl>}}/rancher/v2.0-v2.4/en/cis-scans/legacy/skipped-tests" target="_blank">this page.</a>
versioned_docs/version-2.0-2.4/cluster-admin/tools/cis-scans/skipped-tests/skipped-tests.md
+110
View File
@@ -0,0 +1,110 @@
---
title: Skipped and Not Applicable Tests
weight: 1
aliases:
- /rancher/v2.0-v2.4/en/cis-scans/legacy/skipped-tests
- /rancher/v2.0-v2.4/en/cis-scans/v2.4/skipped-tests
- /rancher/v2.0-v2.4/en/cis-scans/skipped-tests
- /rancher/v2.x/en/cis-scans/v2.4/skipped-tests/
---
This section lists the tests that are skipped in the permissive test profile for RKE.
All the tests that are skipped and not applicable on this page will be counted as Not Applicable in the generated report. The skipped test count will only mention the user-defined skipped tests. This allows user-skipped tests to be distinguished from the tests that are skipped by default in the RKE permissive test profile.
- [CIS Benchmark v1.5](#cis-benchmark-v1-5)
- [CIS Benchmark v1.4](#cis-benchmark-v1-4)
# CIS Benchmark v1.5
### CIS Benchmark v1.5 Skipped Tests
| Number | Description | Reason for Skipping |
| ---------- | ------------- | --------- |
| 1.1.12 | Ensure that the etcd data directory ownership is set to etcd:etcd (Scored) | A system service account is required for etcd data directory ownership. Refer to Rancher's hardening guide for more details on how to configure this ownership. |
| 1.2.6 | Ensure that the --kubelet-certificate-authority argument is set as appropriate (Scored) | When generating serving certificates, functionality could break in conjunction with hostname overrides which are required for certain cloud providers. |
| 1.2.16 | Ensure that the admission control plugin PodSecurityPolicy is set (Scored) | Enabling Pod Security Policy can cause applications to unexpectedly fail. |
| 1.2.33 | Ensure that the --encryption-provider-config argument is set as appropriate (Not Scored) | Enabling encryption changes how data can be recovered as data is encrypted. |
| 1.2.34 | Ensure that encryption providers are appropriately configured (Not Scored) | Enabling encryption changes how data can be recovered as data is encrypted. |
| 4.2.6 | Ensure that the --protect-kernel-defaults argument is set to true (Scored) | System level configurations are required before provisioning the cluster in order for this argument to be set to true. |
| 4.2.10 | Ensure that the--tls-cert-file and --tls-private-key-file arguments are set as appropriate (Scored) | When generating serving certificates, functionality could break in conjunction with hostname overrides which are required for certain cloud providers. |
| 5.1.5 | Ensure that default service accounts are not actively used. (Scored) | Kubernetes provides default service accounts to be used. |
| 5.2.2 | Minimize the admission of containers wishing to share the host process ID namespace (Scored) | Enabling Pod Security Policy can cause applications to unexpectedly fail. |
| 5.2.3 | Minimize the admission of containers wishing to share the host IPC namespace (Scored) | Enabling Pod Security Policy can cause applications to unexpectedly fail. |
| 5.2.4 | Minimize the admission of containers wishing to share the host network namespace (Scored) | Enabling Pod Security Policy can cause applications to unexpectedly fail. |
| 5.2.5 | Minimize the admission of containers with allowPrivilegeEscalation (Scored) | Enabling Pod Security Policy can cause applications to unexpectedly fail. |
| 5.3.2 | Ensure that all Namespaces have Network Policies defined (Scored) | Enabling Network Policies can prevent certain applications from communicating with each other. |
| 5.6.4 | The default namespace should not be used (Scored) | Kubernetes provides a default namespace. |
### CIS Benchmark v1.5 Not Applicable Tests
| Number | Description | Reason for being not applicable |
| ---------- | ------------- | --------- |
| 1.1.1 | Ensure that the API server pod specification file permissions are set to 644 or more restrictive (Scored) | Clusters provisioned by RKE doesn't require or maintain a configuration file for kube-apiserver. All configuration is passed in as arguments at container run time. |
| 1.1.2 | Ensure that the API server pod specification file ownership is set to root:root (Scored) | Clusters provisioned by RKE doesn't require or maintain a configuration file for kube-apiserver. All configuration is passed in as arguments at container run time. |
| 1.1.3 | Ensure that the controller manager pod specification file permissions are set to 644 or more restrictive (Scored) | Clusters provisioned by RKE doesn't require or maintain a configuration file for controller-manager. All configuration is passed in as arguments at container run time. |
| 1.1.4 | Ensure that the controller manager pod specification file ownership is set to root:root (Scored) | Clusters provisioned by RKE doesn't require or maintain a configuration file for controller-manager. All configuration is passed in as arguments at container run time. |
| 1.1.5 | Ensure that the scheduler pod specification file permissions are set to 644 or more restrictive (Scored) | Clusters provisioned by RKE doesn't require or maintain a configuration file for scheduler. All configuration is passed in as arguments at container run time. |
| 1.1.6 | Ensure that the scheduler pod specification file ownership is set to root:root (Scored) | Clusters provisioned by RKE doesn't require or maintain a configuration file for scheduler. All configuration is passed in as arguments at container run time. |
| 1.1.7 | Ensure that the etcd pod specification file permissions are set to 644 or more restrictive (Scored) | Clusters provisioned by RKE doesn't require or maintain a configuration file for etcd. All configuration is passed in as arguments at container run time. |
| 1.1.8 | Ensure that the etcd pod specification file ownership is set to root:root (Scored) | Clusters provisioned by RKE doesn't require or maintain a configuration file for etcd. All configuration is passed in as arguments at container run time. |
| 1.1.13 | Ensure that the admin.conf file permissions are set to 644 or more restrictive (Scored) | Clusters provisioned by RKE does not store the kubernetes default kubeconfig credentials file on the nodes. |
| 1.1.14 | Ensure that the admin.conf file ownership is set to root:root (Scored) | Clusters provisioned by RKE does not store the kubernetes default kubeconfig credentials file on the nodes. |
| 1.1.15 | Ensure that the scheduler.conf file permissions are set to 644 or more restrictive (Scored) | Clusters provisioned by RKE doesn't require or maintain a configuration file for scheduler. All configuration is passed in as arguments at container run time. |
| 1.1.16 | Ensure that the scheduler.conf file ownership is set to root:root (Scored) | Clusters provisioned by RKE doesn't require or maintain a configuration file for scheduler. All configuration is passed in as arguments at container run time. |
| 1.1.17 | Ensure that the controller-manager.conf file permissions are set to 644 or more restrictive (Scored) | Clusters provisioned by RKE doesn't require or maintain a configuration file for controller-manager. All configuration is passed in as arguments at container run time. |
| 1.1.18 | Ensure that the controller-manager.conf file ownership is set to root:root (Scored) | Clusters provisioned by RKE doesn't require or maintain a configuration file for controller-manager. All configuration is passed in as arguments at container run time. |
| 1.3.6 | Ensure that the RotateKubeletServerCertificate argument is set to true (Scored) | Clusters provisioned by RKE handles certificate rotation directly through RKE. |
| 4.1.1 | Ensure that the kubelet service file permissions are set to 644 or more restrictive (Scored) | Clusters provisioned by RKE doesn’t require or maintain a configuration file for the kubelet service. All configuration is passed in as arguments at container run time. |
| 4.1.2 | Ensure that the kubelet service file ownership is set to root:root (Scored) | Clusters provisioned by RKE doesn’t require or maintain a configuration file for the kubelet service. All configuration is passed in as arguments at container run time. |
| 4.1.9 | Ensure that the kubelet configuration file has permissions set to 644 or more restrictive (Scored) | Clusters provisioned by RKE doesn’t require or maintain a configuration file for the kubelet. All configuration is passed in as arguments at container run time. |
| 4.1.10 | Ensure that the kubelet configuration file ownership is set to root:root (Scored) | Clusters provisioned by RKE doesn’t require or maintain a configuration file for the kubelet. All configuration is passed in as arguments at container run time. |
| 4.2.12 | Ensure that the RotateKubeletServerCertificate argument is set to true (Scored) | Clusters provisioned by RKE handles certificate rotation directly through RKE. |
# CIS Benchmark v1.4
The skipped and not applicable tests for CIS Benchmark v1.4 are as follows:
### CIS Benchmark v1.4 Skipped Tests
Number | Description | Reason for Skipping
---|---|---
1.1.11 | "Ensure that the admission control plugin AlwaysPullImages is set (Scored)" | Enabling AlwaysPullImages can use significant bandwidth.
1.1.21 | "Ensure that the --kubelet-certificate-authority argument is set as appropriate (Scored)" | When generating serving certificates, functionality could break in conjunction with hostname overrides which are required for certain cloud providers.
1.1.24 | "Ensure that the admission control plugin PodSecurityPolicy is set (Scored)" | Enabling Pod Security Policy can cause applications to unexpectedly fail.
1.1.34 | "Ensure that the --encryption-provider-config argument is set as appropriate (Scored)" | Enabling encryption changes how data can be recovered as data is encrypted.
1.1.35 | "Ensure that the encryption provider is set to aescbc (Scored)" | Enabling encryption changes how data can be recovered as data is encrypted.
1.1.36 | "Ensure that the admission control plugin EventRateLimit is set (Scored)" | EventRateLimit needs to be tuned depending on the cluster.
1.2.2 | "Ensure that the --address argument is set to 127.0.0.1 (Scored)" | Adding this argument prevents Rancher's monitoring tool to collect metrics on the scheduler.
1.3.7 | "Ensure that the --address argument is set to 127.0.0.1 (Scored)" | Adding this argument prevents Rancher's monitoring tool to collect metrics on the controller manager.
1.4.12 | "Ensure that the etcd data directory ownership is set to etcd:etcd (Scored)" | A system service account is required for etcd data directory ownership. Refer to Rancher's hardening guide for more details on how to configure this ownership.
1.7.2 | "Do not admit containers wishing to share the host process ID namespace (Scored)" | Enabling Pod Security Policy can cause applications to unexpectedly fail.
1.7.3 | "Do not admit containers wishing to share the host IPC namespace (Scored)" | Enabling Pod Security Policy can cause applications to unexpectedly fail.
1.7.4 | "Do not admit containers wishing to share the host network namespace (Scored)" | Enabling Pod Security Policy can cause applications to unexpectedly fail.
1.7.5 | " Do not admit containers with allowPrivilegeEscalation (Scored)" | Enabling Pod Security Policy can cause applications to unexpectedly fail.
2.1.6 | "Ensure that the --protect-kernel-defaults argument is set to true (Scored)" | System level configurations are required before provisioning the cluster in order for this argument to be set to true.
2.1.10 | "Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate (Scored)" | When generating serving certificates, functionality could break in conjunction with hostname overrides which are required for certain cloud providers.
### CIS Benchmark v1.4 Not Applicable Tests
Number | Description | Reason for being not applicable
---|---|---
1.1.9 | "Ensure that the --repair-malformed-updates argument is set to false (Scored)" | The argument --repair-malformed-updates has been removed as of Kubernetes version 1.14
1.3.6 | "Ensure that the RotateKubeletServerCertificate argument is set to true" | Cluster provisioned by RKE handles certificate rotation directly through RKE.
1.4.1 | "Ensure that the API server pod specification file permissions are set to 644 or more restrictive (Scored)" | Cluster provisioned by RKE doesn't require or maintain a configuration file for kube-apiserver.
1.4.2 | "Ensure that the API server pod specification file ownership is set to root:root (Scored)" | Cluster provisioned by RKE doesn't require or maintain a configuration file for kube-apiserver.
1.4.3 | "Ensure that the controller manager pod specification file permissions are set to 644 or more restrictive (Scored)" | Cluster provisioned by RKE doesn't require or maintain a configuration file for controller-manager.
1.4.4 | "Ensure that the controller manager pod specification file ownership is set to root:root (Scored)" | Cluster provisioned by RKE doesn't require or maintain a configuration file for controller-manager.
1.4.5 | "Ensure that the scheduler pod specification file permissions are set to 644 or more restrictive (Scored)" | Cluster provisioned by RKE doesn't require or maintain a configuration file for scheduler.
1.4.6 | "Ensure that the scheduler pod specification file ownership is set to root:root (Scored)" | Cluster provisioned by RKE doesn't require or maintain a configuration file for scheduler.
1.4.7 | "Ensure that the etcd pod specification file permissions are set to 644 or more restrictive (Scored)" | Cluster provisioned by RKE doesn't require or maintain a configuration file for etcd.
1.4.8 | "Ensure that the etcd pod specification file ownership is set to root:root (Scored)" | Cluster provisioned by RKE doesn't require or maintain a configuration file for etcd.
1.4.13 | "Ensure that the admin.conf file permissions are set to 644 or more restrictive (Scored)" | Cluster provisioned by RKE does not store the kubernetes default kubeconfig credentials file on the nodes.
1.4.14 | "Ensure that the admin.conf file ownership is set to root:root (Scored)" | Cluster provisioned by RKE does not store the kubernetes default kubeconfig credentials file on the nodes.
2.1.8 | "Ensure that the --hostname-override argument is not set (Scored)" | Clusters provisioned by RKE clusters and most cloud providers require hostnames.
2.1.12 | "Ensure that the --rotate-certificates argument is not set to false (Scored)" | Cluster provisioned by RKE handles certificate rotation directly through RKE.
2.1.13 | "Ensure that the RotateKubeletServerCertificate argument is set to true (Scored)" | Cluster provisioned by RKE handles certificate rotation directly through RKE.
2.2.3 | "Ensure that the kubelet service file permissions are set to 644 or more restrictive (Scored)" | Cluster provisioned by RKE doesn’t require or maintain a configuration file for the kubelet service.
2.2.4 | "Ensure that the kubelet service file ownership is set to root:root (Scored)" | Cluster provisioned by RKE doesn’t require or maintain a configuration file for the kubelet service.
2.2.9 | "Ensure that the kubelet configuration file ownership is set to root:root (Scored)" | RKE doesn’t require or maintain a configuration file for the kubelet.
2.2.10 | "Ensure that the kubelet configuration file has permissions set to 644 or more restrictive (Scored)" | RKE doesn’t require or maintain a configuration file for the kubelet.
versioned_docs/version-2.0-2.4/cluster-admin/tools/cluster-alerts/cluster-alerts.md
+347
View File
@@ -0,0 +1,347 @@
---
title: Cluster Alerts
shortTitle: Alerts
weight: 2
aliases:
- /rancher/v2.0-v2.4/en/cluster-admin/tools/alerts
- /rancher/v2.0-v2.4/en/monitoring-alerting/legacy/alerts/cluster-alerts
- /rancher/v2.0-v2.4/en/monitoring-alerting/v2.0.x-v2.4.x/cluster-alerts
- /rancher/v2.x/en/monitoring-alerting/v2.0.x-v2.4.x/cluster-alerts/
---
To keep your clusters and applications healthy and driving your organizational productivity forward, you need to stay informed of events occurring in your clusters and projects, both planned and unplanned. When an event occurs, your alert is triggered, and you are sent a notification. You can then, if necessary, follow up with corrective actions.
This section covers the following topics:
- [About Alerts](#about-alerts)
- [Alert Event Examples](#alert-event-examples)
- [Alerts Triggered by Prometheus Queries](#alerts-triggered-by-prometheus-queries)
- [Urgency Levels](#urgency-levels)
- [Scope of Alerts](#scope-of-alerts)
- [Managing Cluster Alerts](#managing-cluster-alerts)
- [Adding Cluster Alerts](#adding-cluster-alerts)
- [Cluster Alert Configuration](#cluster-alert-configuration)
- [System Service Alerts](#system-service-alerts)
- [Resource Event Alerts](#resource-event-alerts)
- [Node Alerts](#node-alerts)
- [Node Selector Alerts](#node-selector-alerts)
- [CIS Scan Alerts](#cis-scan-alerts)
- [Metric Expression Alerts](#metric-expression-alerts)
# About Alerts
Notifiers and alerts are built on top of the [Prometheus Alertmanager](https://prometheus.io/docs/alerting/alertmanager/). Leveraging these tools, Rancher can notify [cluster owners]({{<baseurl>}}/rancher/v2.0-v2.4/en/admin-settings/rbac/cluster-project-roles/#cluster-roles) and [project owners]({{<baseurl>}}/rancher/v2.0-v2.4/en/admin-settings/rbac/cluster-project-roles/#project-roles) of events they need to address.
Before you can receive alerts, you must configure one or more notifier in Rancher.
When you create a cluster, some alert rules are predefined. You can receive these alerts if you configure a [notifier]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/tools/notifiers) for them.
For details about what triggers the predefined alerts, refer to the [documentation on default alerts.]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/tools/alerts/default-alerts)
### Alert Event Examples
Some examples of alert events are:
- A Kubernetes master component entering an unhealthy state.
- A node or workload error occurring.
- A scheduled deployment taking place as planned.
- A node's hardware resources becoming overstressed.
### Alerts Triggered by Prometheus Queries
When you edit an alert rule, you will have the opportunity to configure the alert to be triggered based on a Prometheus expression. For examples of expressions, refer to [this page.]({{<baseurl>}}/rancher/v2.0-v2.4/en/monitoring-alerting/v2.0.x-v2.4.x/cluster-monitoring/expression/)
Monitoring must be [enabled]({{<baseurl>}}/rancher/v2.0-v2.4/en/monitoring-alerting/v2.0.x-v2.4.x/cluster-monitoring/) before you can trigger alerts with custom Prometheus queries or expressions.
### Urgency Levels
You can set an urgency level for each alert. This urgency appears in the notification you receive, helping you to prioritize your response actions. For example, if you have an alert configured to inform you of a routine deployment, no action is required. These alerts can be assigned a low priority level. However, if a deployment fails, it can critically impact your organization, and you need to react quickly. Assign these alerts a high priority level.
### Scope of Alerts
The scope for alerts can be set at either the cluster level or [project level]({{<baseurl>}}/rancher/v2.0-v2.4/en/project-admin/tools/alerts/).
At the cluster level, Rancher monitors components in your Kubernetes cluster, and sends you alerts related to:
- The state of your nodes.
- The system services that manage your Kubernetes cluster.
- The resource events from specific system services.
- The Prometheus expression cross the thresholds
### Managing Cluster Alerts
After you set up cluster alerts, you can manage each alert object. To manage alerts, browse to the cluster containing the alerts, and then select **Tools > Alerts** that you want to manage. You can:
- Deactivate/Reactive alerts
- Edit alert settings
- Delete unnecessary alerts
- Mute firing alerts
- Unmute muted alerts
# Adding Cluster Alerts
As a [cluster owner]({{<baseurl>}}/rancher/v2.0-v2.4/en/admin-settings/rbac/cluster-project-roles/#cluster-roles), you can configure Rancher to send you alerts for cluster events.
>**Prerequisite:** Before you can receive cluster alerts, you must [add a notifier]({{<baseurl>}}/rancher/v2.0-v2.4/en/monitoring-alerting/legacy/notifiers/).
1. From the **Global** view, navigate to the cluster that you want to configure cluster alerts for. Select **Tools > Alerts**. Then click **Add Alert Group**.
1. Enter a **Name** for the alert that describes its purpose, you could group alert rules for the different purpose.
1. Based on the type of alert you want to create, refer to the [cluster alert configuration section.](#cluster-alert-configuration)
1. Continue adding more **Alert Rule** to the group.
1. Finally, choose the [notifiers]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/tools/notifiers/) to send the alerts to.
- You can set up multiple notifiers.
- You can change notifier recipients on the fly.
1. Click **Create.**
**Result:** Your alert is configured. A notification is sent when the alert is triggered.
# Cluster Alert Configuration
- [System Service Alerts](#system-service-alerts)
- [Resource Event Alerts](#resource-event-alerts)
- [Node Alerts](#node-alerts)
- [Node Selector Alerts](#node-selector-alerts)
- [CIS Scan Alerts](#cis-scan-alerts)
- [Metric Expression Alerts](#metric-expression-alerts)
# System Service Alerts
This alert type monitor for events that affect one of the Kubernetes master components, regardless of the node it occurs on.
Each of the below sections corresponds to a part of the alert rule configuration section in the Rancher UI.
### When a
Select the **System Services** option, and then select an option from the dropdown:
- [controller-manager](https://kubernetes.io/docs/concepts/overview/components/#kube-controller-manager)
- [etcd](https://kubernetes.io/docs/concepts/overview/components/#etcd)
- [scheduler](https://kubernetes.io/docs/concepts/overview/components/#kube-scheduler)
### Is
The alert will be triggered when the selected Kubernetes master component is unhealthy.
### Send a
Select the urgency level of the alert. The options are:
- **Critical**: Most urgent
- **Warning**: Normal urgency
- **Info**: Least urgent
Select the urgency level based on the importance of the service and how many nodes fill the role within your cluster. For example, if you're making an alert for the `etcd` service, select **Critical**. If you're making an alert for redundant schedulers, **Warning** is more appropriate.
### Advanced Options
By default, the below options will apply to all alert rules within the group. You can disable these advanced options when configuring a specific rule.
- **Group Wait Time**: How long to wait to buffer alerts of the same group before sending initially, default to 30 seconds.
- **Group Interval Time**: How long to wait before sending an alert that has been added to a group which contains already fired alerts, default to 30 seconds.
- **Repeat Wait Time**: How long to wait before re-sending a given alert that has already been sent, default to 1 hour.
# Resource Event Alerts
This alert type monitors for specific events that are thrown from a resource type.
Each of the below sections corresponds to a part of the alert rule configuration section in the Rancher UI.
### When a
Choose the type of resource event that triggers an alert. The options are:
- **Normal**: triggers an alert when any standard resource event occurs.
- **Warning**: triggers an alert when unexpected resource events occur.
Select a resource type from the **Choose a Resource** drop-down that you want to trigger an alert.
- [DaemonSet](https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/)
- [Deployment](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/)
- [Node](https://kubernetes.io/docs/concepts/architecture/nodes/)
- [Pod](https://kubernetes.io/docs/concepts/workloads/pods/pod/)
- [StatefulSet](https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/)
### Send a
Select the urgency level of the alert.
- **Critical**: Most urgent
- **Warning**: Normal urgency
- **Info**: Least urgent
Select the urgency level of the alert by considering factors such as how often the event occurs or its importance. For example:
- If you set a normal alert for pods, you're likely to receive alerts often, and individual pods usually self-heal, so select an urgency of **Info**.
- If you set a warning alert for StatefulSets, it's very likely to impact operations, so select an urgency of **Critical**.
### Advanced Options
By default, the below options will apply to all alert rules within the group. You can disable these advanced options when configuring a specific rule.
- **Group Wait Time**: How long to wait to buffer alerts of the same group before sending initially, default to 30 seconds.
- **Group Interval Time**: How long to wait before sending an alert that has been added to a group which contains already fired alerts, default to 30 seconds.
- **Repeat Wait Time**: How long to wait before re-sending a given alert that has already been sent, default to 1 hour.
# Node Alerts
This alert type monitors for events that occur on a specific node.
Each of the below sections corresponds to a part of the alert rule configuration section in the Rancher UI.
### When a
Select the **Node** option, and then make a selection from the **Choose a Node** drop-down.
### Is
Choose an event to trigger the alert.
- **Not Ready**: Sends you an alert when the node is unresponsive.
- **CPU usage over**: Sends you an alert when the node raises above an entered percentage of its processing allocation.
- **Mem usage over**: Sends you an alert when the node raises above an entered percentage of its memory allocation.
### Send a
Select the urgency level of the alert.
- **Critical**: Most urgent
- **Warning**: Normal urgency
- **Info**: Least urgent
Select the urgency level of the alert based on its impact on operations. For example, an alert triggered when a node's CPU raises above 60% deems an urgency of **Info**, but a node that is **Not Ready** deems an urgency of **Critical**.
### Advanced Options
By default, the below options will apply to all alert rules within the group. You can disable these advanced options when configuring a specific rule.
- **Group Wait Time**: How long to wait to buffer alerts of the same group before sending initially, default to 30 seconds.
- **Group Interval Time**: How long to wait before sending an alert that has been added to a group which contains already fired alerts, default to 30 seconds.
- **Repeat Wait Time**: How long to wait before re-sending a given alert that has already been sent, default to 1 hour.
# Node Selector Alerts
This alert type monitors for events that occur on any node on marked with a label. For more information, see the Kubernetes documentation for [Labels](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/).
Each of the below sections corresponds to a part of the alert rule configuration section in the Rancher UI.
### When a
Select the **Node Selector** option, and then click **Add Selector** to enter a key value pair for a label. This label should be applied to one or more of your nodes. Add as many selectors as you'd like.
### Is
Choose an event to trigger the alert.
- **Not Ready**: Sends you an alert when selected nodes are unresponsive.
- **CPU usage over**: Sends you an alert when selected nodes raise above an entered percentage of processing allocation.
- **Mem usage over**: Sends you an alert when selected nodes raise above an entered percentage of memory allocation.
### Send a
Select the urgency level of the alert.
- **Critical**: Most urgent
- **Warning**: Normal urgency
- **Info**: Least urgent
Select the urgency level of the alert based on its impact on operations. For example, an alert triggered when a node's CPU raises above 60% deems an urgency of **Info**, but a node that is **Not Ready** deems an urgency of **Critical**.
### Advanced Options
By default, the below options will apply to all alert rules within the group. You can disable these advanced options when configuring a specific rule.
- **Group Wait Time**: How long to wait to buffer alerts of the same group before sending initially, default to 30 seconds.
- **Group Interval Time**: How long to wait before sending an alert that has been added to a group which contains already fired alerts, default to 30 seconds.
- **Repeat Wait Time**: How long to wait before re-sending a given alert that has already been sent, default to 1 hour.
# CIS Scan Alerts
_Available as of v2.4.0_
This alert type is triggered based on the results of a CIS scan.
Each of the below sections corresponds to a part of the alert rule configuration section in the Rancher UI.
### When a
Select **CIS Scan.**
### Is
Choose an event to trigger the alert:
- Completed Scan
- Has Failure
### Send a
Select the urgency level of the alert.
- **Critical**: Most urgent
- **Warning**: Normal urgency
- **Info**: Least urgent
Select the urgency level of the alert based on its impact on operations. For example, an alert triggered when a node's CPU raises above 60% deems an urgency of **Info**, but a node that is **Not Ready** deems an urgency of **Critical**.
### Advanced Options
By default, the below options will apply to all alert rules within the group. You can disable these advanced options when configuring a specific rule.
- **Group Wait Time**: How long to wait to buffer alerts of the same group before sending initially, default to 30 seconds.
- **Group Interval Time**: How long to wait before sending an alert that has been added to a group which contains already fired alerts, default to 30 seconds.
- **Repeat Wait Time**: How long to wait before re-sending a given alert that has already been sent, default to 1 hour.
# Metric Expression Alerts
This alert type monitors for the overload from Prometheus expression querying, it would be available after you enable monitoring.
Each of the below sections corresponds to a part of the alert rule configuration section in the Rancher UI.
### When a
Input or select an **Expression**, the dropdown shows the original metrics from Prometheus, including:
- [**Node**](https://github.com/prometheus/node_exporter)
- [**Container**](https://github.com/google/cadvisor)
- [**ETCD**](https://etcd.io/docs/v3.4.0/op-guide/monitoring/)
- [**Kubernetes Components**](https://github.com/kubernetes/metrics)
- [**Kubernetes Resources**](https://github.com/kubernetes/kube-state-metrics)
- [**Fluentd**](https://docs.fluentd.org/v1.0/articles/monitoring-prometheus) (supported by [Logging]({{<baseurl>}}/rancher/v2.0-v2.4//en/cluster-admin/tools/logging))
- [**Cluster Level Grafana**](http://docs.grafana.org/administration/metrics/)
- **Cluster Level Prometheus**
### Is
Choose a comparison:
- **Equal**: Trigger alert when expression value equal to the threshold.
- **Not Equal**: Trigger alert when expression value not equal to the threshold.
- **Greater Than**: Trigger alert when expression value greater than to threshold.
- **Less Than**: Trigger alert when expression value equal or less than the threshold.
- **Greater or Equal**: Trigger alert when expression value greater to equal to the threshold.
- **Less or Equal**: Trigger alert when expression value less or equal to the threshold.
If applicable, choose a comparison value or a threshold for the alert to be triggered.
### For
Select a duration for a trigger alert when the expression value crosses the threshold longer than the configured duration.
### Send a
Select the urgency level of the alert.
- **Critical**: Most urgent
- **Warning**: Normal urgency
- **Info**: Least urgent
Select the urgency level of the alert based on its impact on operations. For example, an alert triggered when a node's load expression ```sum(node_load5) / count(node_cpu_seconds_total{mode="system"})``` raises above 0.6 deems an urgency of **Info**, but 1 deems an urgency of **Critical**.
### Advanced Options
By default, the below options will apply to all alert rules within the group. You can disable these advanced options when configuring a specific rule.
- **Group Wait Time**: How long to wait to buffer alerts of the same group before sending initially, default to 30 seconds.
- **Group Interval Time**: How long to wait before sending an alert that has been added to a group which contains already fired alerts, default to 30 seconds.
- **Repeat Wait Time**: How long to wait before re-sending a given alert that has already been sent, default to 1 hour.
versioned_docs/version-2.0-2.4/cluster-admin/tools/cluster-alerts/default-alerts/default-alerts.md
+60
View File
@@ -0,0 +1,60 @@
---
title: Default Alerts for Cluster Monitoring
weight: 1
aliases:
- /rancher/v2.0-v2.4/en/cluster-admin/tools/alerts/default-alerts
- /rancher/v2.0-v2.4/en/monitoring-alerting/legacy/alerts/cluster-alerts/default-alerts
- /rancher/v2.0-v2.4/en/monitoring-alerting/v2.0.x-v2.4.x/cluster-alerts/default-alerts
---
When you create a cluster, some alert rules are predefined. These alerts notify you about signs that the cluster could be unhealthy. You can receive these alerts if you configure a [notifier]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/tools/notifiers) for them.
Several of the alerts use Prometheus expressions as the metric that triggers the alert. For more information on how expressions work, you can refer to the Rancher [documentation about Prometheus expressions]({{<baseurl>}}/rancher/v2.0-v2.4/en/monitoring-alerting/legacy/monitoring/cluster-monitoring/expression/) or the Prometheus [documentation about querying metrics](https://prometheus.io/docs/prometheus/latest/querying/basics/).
# Alerts for etcd
Etcd is the key-value store that contains the state of the Kubernetes cluster. Rancher provides default alerts if the built-in monitoring detects a potential problem with etcd. You don't have to enable monitoring to receive these alerts.
A leader is the node that handles all client requests that need cluster consensus. For more information, you can refer to this [explanation of how etcd works.](https://rancher.com/blog/2019/2019-01-29-what-is-etcd/#how-does-etcd-work)
The leader of the cluster can change in response to certain events. It is normal for the leader to change, but too many changes can indicate a problem with the network or a high CPU load. With longer latencies, the default etcd configuration may cause frequent heartbeat timeouts, which trigger a new leader election.
| Alert | Explanation |
|-------|-------------|
| A high number of leader changes within the etcd cluster are happening | A warning alert is triggered when the leader changes more than three times in one hour. |
| Database usage close to the quota 500M | A warning alert is triggered when the size of etcd exceeds 500M.|
| Etcd is unavailable | A critical alert is triggered when etcd becomes unavailable. |
| Etcd member has no leader | A critical alert is triggered when the etcd cluster does not have a leader for at least three minutes. |
# Alerts for Kubernetes Components
Rancher provides alerts when core Kubernetes system components become unhealthy.
Controllers update Kubernetes resources based on changes in etcd. The [controller manager](https://kubernetes.io/docs/reference/command-line-tools-reference/kube-controller-manager/) monitors the cluster desired state through the Kubernetes API server and makes the necessary changes to the current state to reach the desired state.
The [scheduler](https://kubernetes.io/docs/reference/command-line-tools-reference/kube-scheduler/) service is a core component of Kubernetes. It is responsible for scheduling cluster workloads to nodes, based on various configurations, metrics, resource requirements and workload-specific requirements.
| Alert | Explanation |
|-------|-------------|
| Controller Manager is unavailable | A critical warning is triggered when the cluster’s controller-manager becomes unavailable. |
| Scheduler is unavailable | A critical warning is triggered when the cluster’s scheduler becomes unavailable. |
# Alerts for Events
Kubernetes events are objects that provide insight into what is happening inside a cluster, such as what decisions were made by the scheduler or why some pods were evicted from the node. In the Rancher UI, from the project view, you can see events for each workload.
| Alert | Explanation |
|-------|-------------|
| Get warning deployment event | A warning alert is triggered when a warning event happens on a deployment. |
# Alerts for Nodes
Alerts can be triggered based on node metrics. Each computing resource in a Kubernetes cluster is called a node. Nodes can be either bare-metal servers or virtual machines.
| Alert | Explanation |
|-------|-------------|
| High CPU load | A warning alert is triggered if the node uses more than 100 percent of the node’s available CPU seconds for at least three minutes. |
| High node memory utilization | A warning alert is triggered if the node uses more than 80 percent of its available memory for at least three minutes. |
| Node disk is running full within 24 hours | A critical alert is triggered if the disk space on the node is expected to run out in the next 24 hours based on the disk growth over the last 6 hours. |
# Project-level Alerts
When you enable monitoring for the project, some project-level alerts are provided. For details, refer to the [section on project-level alerts.]({{<baseurl>}}/rancher/v2.0-v2.4/en/project-admin/tools/alerts/)
versioned_docs/version-2.0-2.4/cluster-admin/tools/cluster-logging/cluster-logging.md
+128
View File
@@ -0,0 +1,128 @@
---
title: Cluster Logging
shortTitle: Logging
description: Rancher integrates with popular logging services. Learn the requirements and benefits of integrating with logging services, and enable logging on your cluster.
metaDescription: "Rancher integrates with popular logging services. Learn the requirements and benefits of integrating with logging services, and enable logging on your cluster."
weight: 3
aliases:
- /rancher/v2.0-v2.4/en/tasks/logging/
- /rancher/v2.0-v2.4/en/cluster-admin/tools/logging
- /rancher/v2.0-v2.4/en/logging/legacy/cluster-logging
- /rancher/v2.0-v2.4/en/logging/v2.0.x-v2.4.x/cluster-logging/
- /rancher/v2.x/en/logging/v2.0.x-v2.4.x/
- /rancher/v2.x/en/logging/v2.0.x-v2.4.x/cluster-logging/
---
Logging is helpful because it allows you to:
- Capture and analyze the state of your cluster
- Look for trends in your environment
- Save your logs to a safe location outside of your cluster
- Stay informed of events like a container crashing, a pod eviction, or a node dying
- More easily debug and troubleshoot problems
Rancher supports integration with the following services:
- Elasticsearch
- Splunk
- Kafka
- Syslog
- Fluentd
This section covers the following topics:
- [How logging integrations work](#how-logging-integrations-work)
- [Requirements](#requirements)
- [Logging scope](#logging-scope)
- [Enabling cluster logging](#enabling-cluster-logging)
# How Logging Integrations Work
Rancher can integrate with popular external services used for event streams, telemetry, or search. These services can log errors and warnings in your Kubernetes infrastructure to a stream.
These services collect container log events, which are saved to the `/var/log/containers` directory on each of your nodes. The service collects both standard and error events. You can then log into your services to review the events collected, leveraging each service's unique features.
When configuring Rancher to integrate with these services, you'll have to point Rancher toward the service's endpoint and provide authentication information.
Additionally, you'll have the opportunity to enter key-value pairs to filter the log events collected. The service will only collect events for containers marked with your configured key-value pairs.
>**Note:** You can only configure one logging service per cluster or per project.
# Requirements
The Docker daemon on each node in the cluster should be [configured](https://docs.docker.com/config/containers/logging/configure/) with the (default) log-driver: `json-file`. You can check the log-driver by running the following command:
```
$ docker info | grep 'Logging Driver'
Logging Driver: json-file
```
# Logging Scope
You can configure logging at either cluster level or project level.
- Cluster logging writes logs for every pod in the cluster, i.e. in all the projects. For [RKE clusters]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-provisioning/rke-clusters), it also writes logs for all the Kubernetes system components.
- [Project logging]({{<baseurl>}}/rancher/v2.0-v2.4/en/project-admin/tools/logging/) writes logs for every pod in that particular project.
Logs that are sent to your logging service are from the following locations:
- Pod logs stored at `/var/log/containers`.
- Kubernetes system components logs stored at `/var/lib/rancher/rke/log/`.
# Enabling Cluster Logging
As an [administrator]({{<baseurl>}}/rancher/v2.0-v2.4/en/admin-settings/rbac/global-permissions/) or [cluster owner]({{<baseurl>}}/rancher/v2.0-v2.4/en/admin-settings/rbac/cluster-project-roles/#cluster-roles), you can configure Rancher to send Kubernetes logs to a logging service.
1. From the **Global** view, navigate to the cluster that you want to configure cluster logging.
1. Select **Tools > Logging** in the navigation bar.
1. Select a logging service and enter the configuration. Refer to the specific service for detailed configuration. Rancher supports integration with the following services:
- [Elasticsearch]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/tools/logging/elasticsearch/)
- [Splunk]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/tools/logging/splunk/)
- [Kafka]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/tools/logging/kafka/)
- [Syslog]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/tools/logging/syslog/)
- [Fluentd]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/tools/logging/fluentd/)
1. (Optional) Instead of using the UI to configure the logging services, you can enter custom advanced configurations by clicking on **Edit as File**, which is located above the logging targets. This link is only visible after you select a logging service.
- With the file editor, enter raw fluentd configuration for any logging service. Refer to the documentation for each logging service on how to setup the output configuration.
- [Elasticsearch Documentation](https://github.com/uken/fluent-plugin-elasticsearch)
- [Splunk Documentation](https://github.com/fluent/fluent-plugin-splunk)
- [Kafka Documentation](https://github.com/fluent/fluent-plugin-kafka)
- [Syslog Documentation](https://github.com/dlackty/fluent-plugin-remote_syslog)
- [Fluentd Documentation](https://docs.fluentd.org/v1.0/articles/out_forward)
- If the logging service is using TLS, you also need to complete the **SSL Configuration** form.
1. Provide the **Client Private Key** and **Client Certificate**. You can either copy and paste them or upload them by using the **Read from a file** button.
- You can use either a self-signed certificate or one provided by a certificate authority.
- You can generate a self-signed certificate using an openssl command. For example:
```
openssl req -x509 -newkey rsa:2048 -keyout myservice.key -out myservice.cert -days 365 -nodes -subj "/CN=myservice.example.com"
```
2. If you are using a self-signed certificate, provide the **CA Certificate PEM**.
1. (Optional) Complete the **Additional Logging Configuration** form.
1. **Optional:** Use the **Add Field** button to add custom log fields to your logging configuration. These fields are key value pairs (such as `foo=bar`) that you can use to filter the logs from another system.
1. Enter a **Flush Interval**. This value determines how often [Fluentd](https://www.fluentd.org/) flushes data to the logging server. Intervals are measured in seconds.
1. **Include System Log**. The logs from pods in system project and RKE components will be sent to the target. Uncheck it to exclude the system logs.
1. Click **Test**. Rancher sends a test log to the service.
> **Note:** This button is replaced with _Dry Run_ if you are using the custom configuration editor. In this case, Rancher calls the fluentd dry run command to validate the configuration.
1. Click **Save**.
**Result:** Rancher is now configured to send logs to the selected service. Log into the logging service so that you can start viewing the logs.
## Related Links
[Logging Architecture](https://kubernetes.io/docs/concepts/cluster-administration/logging/)
versioned_docs/version-2.0-2.4/cluster-admin/tools/cluster-logging/elasticsearch/elasticsearch.md
+47
View File
@@ -0,0 +1,47 @@
---
title: Elasticsearch
weight: 200
aliases:
- /rancher/v2.0-v2.4/en/tools/logging/elasticsearch/
- /rancher/v2.0-v2.4/en/cluster-admin/tools/logging/elasticsearch
- /rancher/v2.0-v2.4/en/logging/legacy/cluster-logging/elasticsearch
- /rancher/v2.0-v2.4/en/logging/v2.0.x-v2.4.x/cluster-logging/elasticsearch
- /rancher/v2.x/en/logging/v2.0.x-v2.4.x/cluster-logging/elasticsearch/
---
If your organization uses [Elasticsearch](https://www.elastic.co/), either on premise or in the cloud, you can configure Rancher to send it Kubernetes logs. Afterwards, you can log into your Elasticsearch deployment to view logs.
>**Prerequisites:** Configure an [Elasticsearch deployment](https://www.elastic.co/guide/en/cloud/saas-release/ec-create-deployment.html).
## Elasticsearch Deployment Configuration
1. In the **Endpoint** field, enter the IP address and port of your Elasticsearch instance. You can find this information from the dashboard of your Elasticsearch deployment.
* Elasticsearch usually uses port `9200` for HTTP and `9243` for HTTPS.
1. If you are using [X-Pack Security](https://www.elastic.co/guide/en/x-pack/current/xpack-introduction.html), enter your Elasticsearch **Username** and **Password** for authentication.
1. Enter an [Index Pattern](https://www.elastic.co/guide/en/kibana/current/index-patterns.html).
## SSL Configuration
If your instance of Elasticsearch uses SSL, your **Endpoint** will need to begin with `https://`. With the correct endpoint, the **SSL Configuration** form is enabled and ready to be completed.
1. Provide the **Client Private Key** and **Client Certificate**. You can either copy and paste them or upload them by using the **Read from a file** button.
- You can use either a self-signed certificate or one provided by a certificate authority.
- You can generate a self-signed certificate using an openssl command. For example:
```
openssl req -x509 -newkey rsa:2048 -keyout myservice.key -out myservice.cert -days 365 -nodes -subj "/CN=myservice.example.com"
```
1. Enter your **Client Key Password**.
1. Enter your **SSL Version**. The default version is `TLSv1_2`.
1. Select whether or not you want to verify your SSL.
* If you are using a self-signed certificate, select **Enabled - Input trusted server certificate**, provide the **CA Certificate PEM**. You can copy and paste the certificate or upload it using the **Read from a file** button.
* If you are using a certificate from a certificate authority, select **Enabled - Input trusted server certificate**. You do not need to provide a **CA Certificate PEM**.
versioned_docs/version-2.0-2.4/cluster-admin/tools/cluster-logging/fluentd/fluentd.md
+39
View File
@@ -0,0 +1,39 @@
---
title: Fluentd
weight: 600
aliases:
- /rancher/v2.0-v2.4/en/cluster-admin/tools/logging/fluentd
- /rancher/v2.0-v2.4/en/logging/legacy/cluster-logging/fluentd
- /rancher/v2.0-v2.4/en/logging/v2.0.x-v2.4.x/cluster-logging/fluentd
- /rancher/v2.x/en/logging/v2.0.x-v2.4.x/cluster-logging/fluentd/
---
If your organization uses [Fluentd](https://www.fluentd.org/), you can configure Rancher to send it Kubernetes logs. Afterwards, you can log into your Fluentd server to view logs.
>**Prerequisites:** Configure Fluentd input forward to receive the event stream.
>
>See [Fluentd Documentation](https://docs.fluentd.org/v1.0/articles/in_forward) for details.
## Fluentd Configuration
You can add multiple Fluentd Servers. If you want to add additional Fluentd servers, click **Add Fluentd Server**. For each Fluentd server, complete the configuration information:
1. In the **Endpoint** field, enter the address and port of your Fluentd instance, e.g. `http://Fluentd-server:24224`.
1. Enter the **Shared Key** if your Fluentd Server is using a shared key for authentication.
1. Enter the **Username** and **Password** if your Fluentd Server is using username and password for authentication.
1. **Optional:** Enter the **Hostname** of the Fluentd server.
1. Enter the load balancing **Weight** of the Fluentd server. If the weight of one server is 20 and the other server is 30, events will be sent in a 2:3 ratio. If you do not enter a weight, the default weight is 60.
1. If this server is a standby server, check **Use as Standby Only**. Standby servers are used when all other servers are not available.
After adding all the Fluentd servers, you have the option to select **Enable Gzip Compression**. By default, this is enabled because the transferred payload size will be reduced.
## SSL Configuration
If your Fluentd servers are using TLS, you need to select **Use TLS**. If you are using a self-signed certificate, provide the **CA Certificate PEM**. You can copy and paste the certificate or upload it using the **Read from a file** button.
>**Note:** Fluentd does not support self-signed certificates when client authentication is enabled.
versioned_docs/version-2.0-2.4/cluster-admin/tools/cluster-logging/kafka/kafka.md
+47
View File
@@ -0,0 +1,47 @@
---
title: Kafka
weight: 400
aliases:
- /rancher/v2.0-v2.4/en/tools/logging/kafka/
- /rancher/v2.0-v2.4/en/cluster-admin/tools/logging/kafka
- /rancher/v2.0-v2.4/en/logging/legacy/cluster-logging/kafka
- /rancher/v2.0-v2.4/en/logging/v2.0.x-v2.4.x/cluster-logging/kafka
- /rancher/v2.x/en/logging/v2.0.x-v2.4.x/cluster-logging/kafka/
---
If your organization uses [Kafka](https://kafka.apache.org/), you can configure Rancher to send it Kubernetes logs. Afterwards, you can log into your Kafka server to view logs.
>**Prerequisite:** You must have a Kafka server configured.
## Kafka Server Configuration
1. Select the type of **Endpoint** your Kafka server is using:
* **Zookeeper**: Enter the IP address and port. By default, Zookeeper uses port `2181`. Please note that a Zookeeper endpoint cannot enable TLS.
* **Broker**: Click on **Add Endpoint**. For each Kafka broker, enter the IP address and port. By default, Kafka brokers use port `9092`.
1. In the **Topic** field, enter the name of a Kafka [topic](https://kafka.apache.org/documentation/#basic_ops_add_topic) that your Kubernetes cluster submits logs to.
## **Broker** Endpoint Type
### SSL Configuration
If your Kafka cluster is using SSL for the **Broker**, you need to complete the **SSL Configuration** form.
1. Provide the **Client Private Key** and **Client Certificate**. You can either copy and paste them or upload them by using the **Read from a file** button.
1. Provide the **CA Certificate PEM**. You can either copy and paste the certificate or upload it using the **Read from a file** button.
>**Note:** Kafka does not support self-signed certificates when client authentication is enabled.
### SASL configuration
If your Kafka cluster is using [SASL authentication](https://kafka.apache.org/documentation/#security_sasl) for the Broker, you need to complete the **SASL Configuration** form.
1. Enter the SASL **Username** and **Password**.
1. Select the **SASL Type** that your Kafka cluster is using.
* If your Kafka is using **Plain**, please ensure your Kafka cluster is using SSL.
* If your Kafka is using **Scram**, you need to select which **Scram Mechanism** Kafka is using.
versioned_docs/version-2.0-2.4/cluster-admin/tools/cluster-logging/splunk/splunk.md
+81
View File
@@ -0,0 +1,81 @@
---
title: Splunk
weight: 300
aliases:
- /rancher/v2.0-v2.4/en/tasks/logging/splunk/
- /rancher/v2.0-v2.4/en/tools/logging/splunk/
- /rancher/v2.0-v2.4/en/cluster-admin/tools/logging/splunk
- /rancher/v2.0-v2.4/en/logging/legacy/cluster-logging/splunk
- /rancher/v2.0-v2.4/en/logging/v2.0.x-v2.4.x/cluster-logging/splunk
- /rancher/v2.x/en/logging/v2.0.x-v2.4.x/cluster-logging/splunk/
- /rancher/v2.x/en/cluster-admin/tools/logging/splunk
---
If your organization uses [Splunk](https://www.splunk.com/), you can configure Rancher to send it Kubernetes logs. Afterwards, you can log into your Splunk server to view logs.
>**Prerequisites:**
>
>- Configure HTTP event collection for your Splunk Server (Splunk Enterprise or Splunk Cloud).
>- Either create a new token or copy an existing token.
>
>For more information, see [Splunk Documentation](http://docs.splunk.com/Documentation/Splunk/7.1.2/Data/UsetheHTTPEventCollector#About_Event_Collector_tokens).
## Splunk Configuration
1. In the **Endpoint** field, enter the IP address and port for you Splunk instance (i.e. `http://splunk-server:8088`)
* Splunk usually uses port `8088`. If you're using Splunk Cloud, you'll need to work with [Splunk support](https://www.splunk.com/en_us/support-and-services.html) to get an endpoint URL.
1. Enter the **Token** you obtained while completing the prerequisites (i.e., when you created a token in Splunk).
1. In the **Source** field, enter the name of the token as entered in Splunk.
1. **Optional:** Provide one or more [index](http://docs.splunk.com/Documentation/Splunk/7.1.2/Indexer/Aboutindexesandindexers) that's allowed for your token.
## SSL Configuration
If your instance of Splunk uses SSL, your **Endpoint** will need to begin with `https://`. With the correct endpoint, the **SSL Configuration** form is enabled and ready to be completed.
1. Provide the **Client Private Key** and **Client Certificate**. You can either copy and paste them or upload them by using the **Read from a file** button.
- You can use either a self-signed certificate or one provided by a certificate authority.
- You can generate a self-signed certificate using an openssl command. For example:
```
openssl req -x509 -newkey rsa:2048 -keyout myservice.key -out myservice.cert -days 365 -nodes -subj "/CN=myservice.example.com"
```
1. Enter your **Client Key Password**.
1. Select whether or not you want to verify your SSL.
* If you are using a self-signed certificate, select **Enabled - Input trusted server certificate**, provide the **CA Certificate PEM**. You can copy and paste the certificate or upload it using the **Read from a file** button.
* If you are using a certificate from a certificate authority, select **Enabled - Input trusted server certificate**. You do not need to provide a **CA Certificate PEM**.
## Viewing Logs
1. Log into your Splunk server.
1. Click on **Search & Reporting**. The number of **Indexed Events** listed should be increasing.
1. Click on Data Summary and select the Sources tab.
![View Logs]({{<baseurl>}}/img/rancher/splunk/splunk4.jpg)
1. To view the actual logs, click on the source that you declared earlier.
![View Logs]({{<baseurl>}}/img/rancher/splunk/splunk5.jpg)
## Troubleshooting
You can use curl to see if **HEC** is listening for HTTP event data.
```
$ curl http://splunk-server:8088/services/collector/event \
-H 'Authorization: Splunk 8da70994-b1b0-4a79-b154-bfaae8f93432' \
-d '{"event": "hello world"}'
```
If Splunk is configured correctly, you should receive **json** data returning `success code 0`. You should be able
to send logging data to HEC.
If you received an error, check your configuration in Splunk and Rancher.
versioned_docs/version-2.0-2.4/cluster-admin/tools/cluster-logging/syslog/syslog.md
+47
View File
@@ -0,0 +1,47 @@
---
title: Syslog
weight: 500
aliases:
- /rancher/v2.0-v2.4/en/tools/logging/syslog/
- /rancher/v2.0-v2.4/en/cluster-admin/tools/logging/syslog
- /rancher/v2.0-v2.4/en/logging/legacy/cluster-logging/syslog
- /rancher/v2.0-v2.4/en/logging/v2.0.x-v2.4.x/cluster-logging/syslog
- /rancher/v2.x/en/logging/v2.0.x-v2.4.x/cluster-logging/syslog/
---
If your organization uses [Syslog](https://tools.ietf.org/html/rfc5424), you can configure Rancher to send it Kubernetes logs. Afterwards, you can log into your Syslog server to view logs.
>**Prerequisite:** You must have a Syslog server configured.
If you are using rsyslog, please make sure your rsyslog authentication mode is `x509/name`.
## Syslog Server Configuration
1. In the **Endpoint** field, enter the IP address and port for your Syslog server. Additionally, in the dropdown, select the protocol that your Syslog server uses.
1. In the **Program** field, enter the name of the application sending logs to your Syslog server, e.g. `Rancher`.
1. If you are using a cloud logging service, e.g. [Sumologic](https://www.sumologic.com/), enter a **Token** that authenticates with your Syslog server. You will need to create this token in the cloud logging service.
1. Select a **Log Severity** for events that are logged to the Syslog server. For more information on each severity level, see the [Syslog protocol documentation](https://tools.ietf.org/html/rfc5424#page-11).
- By specifying a **Log Severity** does not mean that will act as a filtering mechanism for logs. To do that you should use a parser on the Syslog server.
## Encryption Configuration
If your Syslog server is using **TCP** protocol and uses TLS, you need to select **Use TLS** and complete the **Encryption Configuration** form.
1. Provide the **Client Private Key** and **Client Certificate**. You can either copy and paste them or upload them by using the **Read from a file** button.
- You can use either a self-signed certificate or one provided by a certificate authority.
- You can generate a self-signed certificate using an openssl command. For example:
```
openssl req -x509 -newkey rsa:2048 -keyout myservice.key -out myservice.cert -days 365 -nodes -subj "/CN=myservice.example.com"
```
1. Select whether or not you want to verify your SSL.
* If you are using a self-signed certificate, select **Enabled - Input trusted server certificate**, provide the **CA Certificate PEM**. You can copy and paste the certificate or upload it using the **Read from a file** button.
* If you are using a certificate from a certificate authority, select **Enabled - Input trusted server certificate**. You do not need to provide a **CA Certificate PEM**.
versioned_docs/version-2.0-2.4/cluster-admin/tools/cluster-monitoring/cluster-metrics/cluster-metrics.md
+119
View File
@@ -0,0 +1,119 @@
---
title: Cluster Metrics
weight: 3
aliases:
- /rancher/v2.0-v2.4/en/project-admin/tools/monitoring/cluster-metrics
- /rancher/v2.0-v2.4/en/cluster-admin/tools/monitoring/cluster-metrics
- /rancher/v2.0-v2.4/en/monitoring-alerting/legacy/monitoring/cluster-monitoring/cluster-metrics
- /rancher/v2.0-v2.4/en/monitoring-alerting/v2.0.x-v2.4.x/cluster-monitoring/cluster-metrics
- /rancher/v2.x/en/monitoring-alerting/v2.0.x-v2.4.x/cluster-monitoring/cluster-metrics/
---
_Available as of v2.2.0_
Cluster metrics display the hardware utilization for all nodes in your cluster, regardless of its role. They give you a global monitoring insight into the cluster.
Some of the biggest metrics to look out for:
- **CPU Utilization**
High load either indicates that your cluster is running efficiently or that you're running out of CPU resources.
- **Disk Utilization**
Be on the lookout for increased read and write rates on nodes nearing their disk capacity. This advice is especially true for etcd nodes, as running out of storage on an etcd node leads to cluster failure.
- **Memory Utilization**
Deltas in memory utilization usually indicate a memory leak.
- **Load Average**
Generally, you want your load average to match your number of logical CPUs for the cluster. For example, if your cluster has 8 logical CPUs, the ideal load average would be 8 as well. If you load average is well under the number of logical CPUs for the cluster, you may want to reduce cluster resources. On the other hand, if your average is over 8, your cluster may need more resources.
## Finding Node Metrics
1. From the **Global** view, navigate to the cluster that you want to view metrics.
1. Select **Nodes** in the navigation bar.
1. Select a specific node and click on its name.
1. Click on **Node Metrics**.
[_Get expressions for Cluster Metrics_]({{<baseurl>}}/rancher/v2.0-v2.4/en/monitoring-alerting/v2.0.x-v2.4.x/cluster-monitoring/expression/#cluster-metrics)
### Etcd Metrics
>**Note:** Only supported for [Rancher launched Kubernetes clusters]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-provisioning/rke-clusters/).
Etcd metrics display the operations of the etcd database on each of your cluster nodes. After establishing a baseline of normal etcd operational metrics, observe them for abnormal deltas between metric refreshes, which indicate potential issues with etcd. Always address etcd issues immediately!
You should also pay attention to the text at the top of the etcd metrics, which displays leader election statistics. This text indicates if etcd currently has a leader, which is the etcd instance that coordinates the other etcd instances in your cluster. A large increase in leader changes implies etcd is unstable. If you notice a change in leader election statistics, you should investigate them for issues.
Some of the biggest metrics to look out for:
- **Etcd has a leader**
etcd is usually deployed on multiple nodes and elects a leader to coordinate its operations. If etcd does not have a leader, its operations are not being coordinated.
- **Number of leader changes**
If this statistic suddenly grows, it usually indicates network communication issues that constantly force the cluster to elect a new leader.
[_Get expressions for Etcd Metrics_]({{<baseurl>}}/rancher/v2.0-v2.4/en/monitoring-alerting/v2.0.x-v2.4.x/cluster-monitoring/expression/#etcd-metrics)
### Kubernetes Components Metrics
Kubernetes components metrics display data about the cluster's individual Kubernetes components. Primarily, it displays information about connections and latency for each component: the API server, controller manager, scheduler, and ingress controller.
>**Note:** The metrics for the controller manager, scheduler and ingress controller are only supported for [Rancher launched Kubernetes clusters]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-provisioning/rke-clusters/).
When analyzing Kubernetes component metrics, don't be concerned about any single standalone metric in the charts and graphs that display. Rather, you should establish a baseline for metrics considered normal following a period of observation, e.g. the range of values that your components usually operate within and are considered normal. After you establish this baseline, be on the lookout for large deltas in the charts and graphs, as these big changes usually indicate a problem that you need to investigate.
Some of the more important component metrics to monitor are:
- **API Server Request Latency**
Increasing API response times indicate there's a generalized problem that requires investigation.
- **API Server Request Rate**
Rising API request rates usually coincide with increased API response times. Increased request rates also indicate a generalized problem requiring investigation.
- **Scheduler Preemption Attempts**
If you see a spike in scheduler preemptions, it's an indication that you're running out of hardware resources, as Kubernetes is recognizing it doesn't have enough resources to run all your pods and is prioritizing the more important ones.
- **Scheduling Failed Pods**
Failed pods can have a variety of causes, such as unbound persistent volume claims, exhausted hardware resources, non-responsive nodes, etc.
- **Ingress Controller Request Process Time**
How fast ingress is routing connections to your cluster services.
[_Get expressions for Kubernetes Component Metrics_]({{<baseurl>}}/rancher/v2.0-v2.4/en/monitoring-alerting/legacy/monitoring/cluster-monitoring/expression/#kubernetes-components-metrics)
## Rancher Logging Metrics
Although the Dashboard for a cluster primarily displays data sourced from Prometheus, it also displays information for cluster logging, provided that you have [configured Rancher to use a logging service]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/tools/logging/).
[_Get expressions for Rancher Logging Metrics_]({{<baseurl>}}/rancher/v2.0-v2.4/en/monitoring-alerting/v2.0.x-v2.4.x/cluster-monitoring/expression/#rancher-logging-metrics)
## Finding Workload Metrics
Workload metrics display the hardware utilization for a Kubernetes workload. You can also view metrics for [deployments](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/), [stateful sets](https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/) and so on.
1. From the **Global** view, navigate to the project that you want to view workload metrics.
1. From the main navigation bar, choose **Resources > Workloads.** In versions before v2.3.0, choose **Workloads** on the main navigation bar.
1. Select a specific workload and click on its name.
1. In the **Pods** section, select a specific pod and click on its name.
- **View the Pod Metrics:** Click on **Pod Metrics**.
- **View the Container Metrics:** In the **Containers** section, select a specific container and click on its name. Click on **Container Metrics**.
[_Get expressions for Workload Metrics_]({{<baseurl>}}/rancher/v2.0-v2.4/en/monitoring-alerting/v2.0.x-v2.4.x/cluster-monitoring/expression/#workload-metrics)
versioned_docs/version-2.0-2.4/cluster-admin/tools/cluster-monitoring/cluster-monitoring.md
+124
View File
@@ -0,0 +1,124 @@
---
title: Integrating Rancher and Prometheus for Cluster Monitoring
shortTitle: Monitoring
description: Prometheus lets you view metrics from your different Rancher and Kubernetes objects. Learn about the scope of monitoring and how to enable cluster monitoring
weight: 1
aliases:
- /rancher/v2.0-v2.4/en/project-admin/tools/monitoring
- /rancher/v2.0-v2.4/en/monitoring-alerting/legacy/monitoring/cluster-monitoring
- /rancher/v2.0-v2.4/en/monitoring-alerting/v2.0.x-v2.4.x/monitoring/cluster-monitoring
- /rancher/v2.0-v2.4/en/monitoring-alerting/v2.0.x-v2.4.x/cluster-monitoring
- /rancher/v2.x/en/monitoring-alerting/v2.0.x-v2.4.x/
- /rancher/v2.x/en/monitoring-alerting/v2.0.x-v2.4.x/cluster-monitoring/
---
_Available as of v2.2.0_
Using Rancher, you can monitor the state and processes of your cluster nodes, Kubernetes components, and software deployments through integration with [Prometheus](https://prometheus.io/), a leading open-source monitoring solution.
This section covers the following topics:
- [About Prometheus](#about-prometheus)
- [Monitoring scope](#monitoring-scope)
- [Enabling cluster monitoring](#enabling-cluster-monitoring)
- [Resource consumption](#resource-consumption)
- [Resource consumption of Prometheus pods](#resource-consumption-of-prometheus-pods)
- [Resource consumption of other pods](#resource-consumption-of-other-pods)
# About Prometheus
Prometheus provides a _time series_ of your data, which is, according to [Prometheus documentation](https://prometheus.io/docs/concepts/data_model/):
You can configure these services to collect logs at either the cluster level or the project level. This page describes how to enable monitoring for a cluster. For details on enabling monitoring for a project, refer to the [project administration section]({{<baseurl>}}/rancher/v2.0-v2.4/en/project-admin/tools/monitoring/).
>A stream of timestamped values belonging to the same metric and the same set of labeled dimensions, along with comprehensive statistics and metrics of the monitored cluster.
In other words, Prometheus lets you view metrics from your different Rancher and Kubernetes objects. Using timestamps, Prometheus lets you query and view these metrics in easy-to-read graphs and visuals, either through the Rancher UI or [Grafana](https://grafana.com/), which is an analytics viewing platform deployed along with Prometheus.
By viewing data that Prometheus scrapes from your cluster control plane, nodes, and deployments, you can stay on top of everything happening in your cluster. You can then use these analytics to better run your organization: stop system emergencies before they start, develop maintenance strategies, restore crashed servers, etc.
Multi-tenancy support in terms of cluster-only and project-only Prometheus instances are also supported.
# Monitoring Scope
Using Prometheus, you can monitor Rancher at both the cluster level and [project level]({{<baseurl>}}/rancher/v2.0-v2.4/en/project-admin/tools/monitoring/). For each cluster and project that is enabled for monitoring, Rancher deploys a Prometheus server.
- Cluster monitoring allows you to view the health of your Kubernetes cluster. Prometheus collects metrics from the cluster components below, which you can view in graphs and charts.
- Kubernetes control plane
- etcd database
- All nodes (including workers)
- [Project monitoring]({{<baseurl>}}/rancher/v2.0-v2.4/en/project-admin/tools/monitoring/) allows you to view the state of pods running in a given project. Prometheus collects metrics from the project's deployed HTTP and TCP/UDP workloads.
# Enabling Cluster Monitoring
As an [administrator]({{<baseurl>}}/rancher/v2.0-v2.4/en/admin-settings/rbac/global-permissions/) or [cluster owner]({{<baseurl>}}/rancher/v2.0-v2.4/en/admin-settings/rbac/cluster-project-roles/#cluster-roles), you can configure Rancher to deploy Prometheus to monitor your Kubernetes cluster.
> **Prerequisites:** The following TCP ports need to be opened for metrics scraping:
>
> | Port | Node type | Component |
> | --- | --- | --- |
> | 9796 | Worker | Node exporter |
> | 10254 | Worker | Nginx Ingress Controller |
> | 10250 | Worker/Controlplane | Kubelet |
> | 10251 | Controlplane | Kube scheduler |
> | 10252 | Controlplane | Kube controller manager |
> | 2379 | Etcd | Etcd server |
> Monitoring V1 requires a Kubernetes verison less than or equal to v1.20.x. To install monitoring on Kubernetes v1.21+, you will need to [migrate to Monitoring V2.]({{<baseurl>}}/rancher/v2.5/en/monitoring-alerting/migrating/)
1. From the **Global** view, navigate to the cluster that you want to configure cluster monitoring.
1. Select **Tools > Monitoring** in the navigation bar.
1. Select **Enable** to show the [Prometheus configuration options]({{<baseurl>}}/rancher/v2.0-v2.4/en/monitoring-alerting/legacy/monitoring/cluster-monitoring/prometheus/). Review the [resource consumption recommendations](#resource-consumption) to ensure you have enough resources for Prometheus and on your worker nodes to enable monitoring. Enter in your desired configuration options.
1. Click **Save**.
**Result:** The Prometheus server will be deployed as well as two monitoring applications. The two monitoring applications, `cluster-monitoring` and `monitoring-operator`, are added as an [application]({{<baseurl>}}/rancher/v2.0-v2.4/en/catalog/apps/) to the cluster's `system` project. After the applications are `active`, you can start viewing [cluster metrics]({{<baseurl>}}/rancher/v2.0-v2.4/en/monitoring-alerting/legacy/monitoring/cluster-monitoring/cluster-metrics/) through the Rancher dashboard or directly from Grafana.
> The default username and password for the Grafana instance will be `admin/admin`. However, Grafana dashboards are served via the Rancher authentication proxy, so only users who are currently authenticated into the Rancher server have access to the Grafana dashboard.
# Resource Consumption
When enabling cluster monitoring, you need to ensure your worker nodes and Prometheus pod have enough resources. The tables below provides a guide of how much resource consumption will be used. In larger deployments, it is strongly advised that the monitoring infrastructure be placed on dedicated nodes in the cluster.
### Resource Consumption of Prometheus Pods
This table is the resource consumption of the Prometheus pod, which is based on the number of all the nodes in the cluster. The count of nodes includes the worker, control plane and etcd nodes. Total disk space allocation should be approximated by the `rate * retention` period set at the cluster level. When enabling cluster level monitoring, you should adjust the CPU and Memory limits and reservation.
Number of Cluster Nodes | CPU (milli CPU) | Memory | Disk
------------------------|-----|--------|------
5 | 500 | 650 MB | ~1 GB/Day
50| 2000 | 2 GB | ~5 GB/Day
256| 4000 | 6 GB | ~18 GB/Day
Additional pod resource requirements for cluster level monitoring.
| Workload | Container | CPU - Request | Mem - Request | CPU - Limit | Mem - Limit | Configurable |
|---------------------|---------------------------------|---------------|---------------|-------------|-------------|--------------|
| Prometheus | prometheus | 750m | 750Mi | 1000m | 1000Mi | Y |
| | prometheus-proxy | 50m | 50Mi | 100m | 100Mi | Y |
| | prometheus-auth | 100m | 100Mi | 500m | 200Mi | Y |
| | prometheus-config-reloader | - | - | 50m | 50Mi | N |
| | rules-configmap-reloader | - | - | 100m | 25Mi | N |
| Grafana | grafana-init-plugin-json-copy | 50m | 50Mi | 50m | 50Mi | Y |
| | grafana-init-plugin-json-modify | 50m | 50Mi | 50m | 50Mi | Y |
| | grafana | 100m | 100Mi | 200m | 200Mi | Y |
| | grafana-proxy | 50m | 50Mi | 100m | 100Mi | Y |
| Kube-State Exporter | kube-state | 100m | 130Mi | 100m | 200Mi | Y |
| Node Exporter | exporter-node | 200m | 200Mi | 200m | 200Mi | Y |
| Operator | prometheus-operator | 100m | 50Mi | 200m | 100Mi | Y |
### Resource Consumption of Other Pods
Besides the Prometheus pod, there are components that are deployed that require additional resources on the worker nodes.
Pod | CPU (milli CPU) | Memory (MB)
----|-----------------|------------
Node Exporter (Per Node) | 100 | 30
Kube State Cluster Monitor | 100 | 130
Grafana | 100 | 150
Prometheus Cluster Monitoring Nginx | 50 | 50
versioned_docs/version-2.0-2.4/cluster-admin/tools/cluster-monitoring/custom-metrics/custom-metrics.md
+493
View File
@@ -0,0 +1,493 @@
---
title: Prometheus Custom Metrics Adapter
weight: 5
aliases:
- /rancher/v2.0-v2.4/en/project-admin/tools/monitoring/custom-metrics
- /rancher/v2.0-v2.4/en/monitoring-alerting/legacy/monitoring/cluster-monitoring/custom-metrics
- /rancher/v2.0-v2.4/en/cluster-admin/tools/monitoring/custom-metrics/
- /rancher/v2.0-v2.4/en/monitoring-alerting/v2.0.x-v2.4.x/cluster-monitoring/custom-metrics
- /rancher/v2.x/en/monitoring-alerting/v2.0.x-v2.4.x/cluster-monitoring/custom-metrics/
---
After you've enabled [cluster level monitoring]({{< baseurl >}}/rancher/v2.0-v2.4/en/monitoring-alerting/legacy/monitoring/cluster-monitoring/), You can view the metrics data from Rancher. You can also deploy the Prometheus custom metrics adapter then you can use the HPA with metrics stored in cluster monitoring.
## Deploy Prometheus Custom Metrics Adapter
We are going to use the [Prometheus custom metrics adapter](https://github.com/DirectXMan12/k8s-prometheus-adapter/releases/tag/v0.5.0), version v0.5.0. This is a great example for the [custom metrics server](https://github.com/kubernetes-incubator/custom-metrics-apiserver). And you must be the *cluster owner* to execute following steps.
- Get the service account of the cluster monitoring is using. It should be configured in the workload ID: `statefulset:cattle-prometheus:prometheus-cluster-monitoring`. And if you didn't customize anything, the service account name should be `cluster-monitoring`.
- Grant permission to that service account. You will need two kinds of permission.
One role is `extension-apiserver-authentication-reader` in `kube-system`, so you will need to create a `Rolebinding` to in `kube-system`. This permission is to get api aggregation configuration from config map in `kube-system`.
```yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: custom-metrics-auth-reader
namespace: kube-system
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: extension-apiserver-authentication-reader
subjects:
- kind: ServiceAccount
name: cluster-monitoring
namespace: cattle-prometheus
```
The other one is cluster role `system:auth-delegator`, so you will need to create a `ClusterRoleBinding`. This permission is to have subject access review permission.
```yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: custom-metrics:system:auth-delegator
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:auth-delegator
subjects:
- kind: ServiceAccount
name: cluster-monitoring
namespace: cattle-prometheus
```
- Create configuration for custom metrics adapter. Following is an example configuration. There will be a configuration details in next session.
```yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: adapter-config
namespace: cattle-prometheus
data:
config.yaml: |
rules:
- seriesQuery: '{__name__=~"^container_.*",container_name!="POD",namespace!="",pod_name!=""}'
seriesFilters: []
resources:
overrides:
namespace:
resource: namespace
pod_name:
resource: pod
name:
matches: ^container_(.*)_seconds_total$
as: ""
metricsQuery: sum(rate(<<.Series>>{<<.LabelMatchers>>,container_name!="POD"}[1m])) by (<<.GroupBy>>)
- seriesQuery: '{__name__=~"^container_.*",container_name!="POD",namespace!="",pod_name!=""}'
seriesFilters:
- isNot: ^container_.*_seconds_total$
resources:
overrides:
namespace:
resource: namespace
pod_name:
resource: pod
name:
matches: ^container_(.*)_total$
as: ""
metricsQuery: sum(rate(<<.Series>>{<<.LabelMatchers>>,container_name!="POD"}[1m])) by (<<.GroupBy>>)
- seriesQuery: '{__name__=~"^container_.*",container_name!="POD",namespace!="",pod_name!=""}'
seriesFilters:
- isNot: ^container_.*_total$
resources:
overrides:
namespace:
resource: namespace
pod_name:
resource: pod
name:
matches: ^container_(.*)$
as: ""
metricsQuery: sum(<<.Series>>{<<.LabelMatchers>>,container_name!="POD"}) by (<<.GroupBy>>)
- seriesQuery: '{namespace!="",__name__!~"^container_.*"}'
seriesFilters:
- isNot: .*_total$
resources:
template: <<.Resource>>
name:
matches: ""
as: ""
metricsQuery: sum(<<.Series>>{<<.LabelMatchers>>}) by (<<.GroupBy>>)
- seriesQuery: '{namespace!="",__name__!~"^container_.*"}'
seriesFilters:
- isNot: .*_seconds_total
resources:
template: <<.Resource>>
name:
matches: ^(.*)_total$
as: ""
metricsQuery: sum(rate(<<.Series>>{<<.LabelMatchers>>}[1m])) by (<<.GroupBy>>)
- seriesQuery: '{namespace!="",__name__!~"^container_.*"}'
seriesFilters: []
resources:
template: <<.Resource>>
name:
matches: ^(.*)_seconds_total$
as: ""
metricsQuery: sum(rate(<<.Series>>{<<.LabelMatchers>>}[1m])) by (<<.GroupBy>>)
resourceRules:
cpu:
containerQuery: sum(rate(container_cpu_usage_seconds_total{<<.LabelMatchers>>}[1m])) by (<<.GroupBy>>)
nodeQuery: sum(rate(container_cpu_usage_seconds_total{<<.LabelMatchers>>, id='/'}[1m])) by (<<.GroupBy>>)
resources:
overrides:
instance:
resource: node
namespace:
resource: namespace
pod_name:
resource: pod
containerLabel: container_name
memory:
containerQuery: sum(container_memory_working_set_bytes{<<.LabelMatchers>>}) by (<<.GroupBy>>)
nodeQuery: sum(container_memory_working_set_bytes{<<.LabelMatchers>>,id='/'}) by (<<.GroupBy>>)
resources:
overrides:
instance:
resource: node
namespace:
resource: namespace
pod_name:
resource: pod
containerLabel: container_name
window: 1m
```
- Create HTTPS TLS certs for your api server. You can use following command to create a self-signed cert.
```bash
openssl req -new -newkey rsa:4096 -x509 -sha256 -days 365 -nodes -out serving.crt -keyout serving.key -subj "/C=CN/CN=custom-metrics-apiserver.cattle-prometheus.svc.cluster.local"
# And you will find serving.crt and serving.key in your path. And then you are going to create a secret in cattle-prometheus namespace.
kubectl create secret generic -n cattle-prometheus cm-adapter-serving-certs --from-file=serving.key=./serving.key --from-file=serving.crt=./serving.crt
```
- Then you can create the prometheus custom metrics adapter. And you will need a service for this deployment too. Creating it via Import YAML or Rancher would do. Please create those resources in `cattle-prometheus` namespaces.
Here is the prometheus custom metrics adapter deployment.
```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: custom-metrics-apiserver
name: custom-metrics-apiserver
namespace: cattle-prometheus
spec:
replicas: 1
selector:
matchLabels:
app: custom-metrics-apiserver
template:
metadata:
labels:
app: custom-metrics-apiserver
name: custom-metrics-apiserver
spec:
serviceAccountName: cluster-monitoring
containers:
- name: custom-metrics-apiserver
image: directxman12/k8s-prometheus-adapter-amd64:v0.5.0
args:
- --secure-port=6443
- --tls-cert-file=/var/run/serving-cert/serving.crt
- --tls-private-key-file=/var/run/serving-cert/serving.key
- --logtostderr=true
- --prometheus-url=http://prometheus-operated/
- --metrics-relist-interval=1m
- --v=10
- --config=/etc/adapter/config.yaml
ports:
- containerPort: 6443
volumeMounts:
- mountPath: /var/run/serving-cert
name: volume-serving-cert
readOnly: true
- mountPath: /etc/adapter/
name: config
readOnly: true
- mountPath: /tmp
name: tmp-vol
volumes:
- name: volume-serving-cert
secret:
secretName: cm-adapter-serving-certs
- name: config
configMap:
name: adapter-config
- name: tmp-vol
emptyDir: {}
```
Here is the service of the deployment.
```yaml
apiVersion: v1
kind: Service
metadata:
name: custom-metrics-apiserver
namespace: cattle-prometheus
spec:
ports:
- port: 443
targetPort: 6443
selector:
app: custom-metrics-apiserver
```
- Create API service for your custom metric server.
```yaml
apiVersion: apiregistration.k8s.io/v1beta1
kind: APIService
metadata:
name: v1beta1.custom.metrics.k8s.io
spec:
service:
name: custom-metrics-apiserver
namespace: cattle-prometheus
group: custom.metrics.k8s.io
version: v1beta1
insecureSkipTLSVerify: true
groupPriorityMinimum: 100
versionPriority: 100
```
- Then you can verify your custom metrics server by `kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1`. If you see the return datas from the api, it means that the metrics server has been successfully set up.
- You create HPA with custom metrics now. Here is an example of HPA. You will need to create a nginx deployment in your namespace first.
```yaml
kind: HorizontalPodAutoscaler
apiVersion: autoscaling/v2beta1
metadata:
name: nginx
spec:
scaleTargetRef:
# point the HPA at the nginx deployment you just created
apiVersion: apps/v1
kind: Deployment
name: nginx
# autoscale between 1 and 10 replicas
minReplicas: 1
maxReplicas: 10
metrics:
# use a "Pods" metric, which takes the average of the
# given metric across all pods controlled by the autoscaling target
- type: Pods
pods:
metricName: memory_usage_bytes
targetAverageValue: 5000000
```
And then, you should see your nginx is scaling up. HPA with custom metrics works.
## Configuration of prometheus custom metrics adapter
> Refer to https://github.com/DirectXMan12/k8s-prometheus-adapter/blob/master/docs/config.md
The adapter determines which metrics to expose, and how to expose them,
through a set of "discovery" rules. Each rule is executed independently
(so make sure that your rules are mutually exclusive), and specifies each
of the steps the adapter needs to take to expose a metric in the API.
Each rule can be broken down into roughly four parts:
- *Discovery*, which specifies how the adapter should find all Prometheus
metrics for this rule.
- *Association*, which specifies how the adapter should determine which
Kubernetes resources a particular metric is associated with.
- *Naming*, which specifies how the adapter should expose the metric in
the custom metrics API.
- *Querying*, which specifies how a request for a particular metric on one
or more Kubernetes objects should be turned into a query to Prometheus.
A basic config with one rule might look like:
```yaml
rules:
# this rule matches cumulative cAdvisor metrics measured in seconds
- seriesQuery: '{__name__=~"^container_.*",container_name!="POD",namespace!="",pod_name!=""}'
resources:
# skip specifying generic resource<->label mappings, and just
# attach only pod and namespace resources by mapping label names to group-resources
overrides:
namespace: {resource: "namespace"},
pod_name: {resource: "pod"},
# specify that the `container_` and `_seconds_total` suffixes should be removed.
# this also introduces an implicit filter on metric family names
name:
# we use the value of the capture group implicitly as the API name
# we could also explicitly write `as: "$1"`
matches: "^container_(.*)_seconds_total$"
# specify how to construct a query to fetch samples for a given series
# This is a Go template where the `.Series` and `.LabelMatchers` string values
# are available, and the delimiters are `<<` and `>>` to avoid conflicts with
# the prometheus query language
metricsQuery: "sum(rate(<<.Series>>{<<.LabelMatchers>>,container_name!="POD"}[2m])) by (<<.GroupBy>>)"
```
### Discovery
Discovery governs the process of finding the metrics that you want to
expose in the custom metrics API. There are two fields that factor into
discovery: `seriesQuery` and `seriesFilters`.
`seriesQuery` specifies Prometheus series query (as passed to the
`/api/v1/series` endpoint in Prometheus) to use to find some set of
Prometheus series. The adapter will strip the label values from this
series, and then use the resulting metric-name-label-names combinations
later on.
In many cases, `seriesQuery` will be sufficient to narrow down the list of
Prometheus series. However, sometimes (especially if two rules might
otherwise overlap), it's useful to do additional filtering on metric
names. In this case, `seriesFilters` can be used. After the list of
series is returned from `seriesQuery`, each series has its metric name
filtered through any specified filters.
Filters may be either:
- `is: <regex>`, which matches any series whose name matches the specified
regex.
- `isNot: <regex>`, which matches any series whose name does not match the
specified regex.
For example:
```yaml
# match all cAdvisor metrics that aren't measured in seconds
seriesQuery: '{__name__=~"^container_.*_total",container_name!="POD",namespace!="",pod_name!=""}'
seriesFilters:
isNot: "^container_.*_seconds_total"
```
### Association
Association governs the process of figuring out which Kubernetes resources
a particular metric could be attached to. The `resources` field controls
this process.
There are two ways to associate resources with a particular metric. In
both cases, the value of the label becomes the name of the particular
object.
One way is to specify that any label name that matches some particular
pattern refers to some group-resource based on the label name. This can
be done using the `template` field. The pattern is specified as a Go
template, with the `Group` and `Resource` fields representing group and
resource. You don't necessarily have to use the `Group` field (in which
case the group is guessed by the system). For instance:
```yaml
# any label `kube_<group>_<resource>` becomes <group>.<resource> in Kubernetes
resources:
template: "kube_<<.Group>>_<<.Resource>>"
```
The other way is to specify that some particular label represents some
particular Kubernetes resource. This can be done using the `overrides`
field. Each override maps a Prometheus label to a Kubernetes
group-resource. For instance:
```yaml
# the microservice label corresponds to the apps.deployment resource
resource:
overrides:
microservice: {group: "apps", resource: "deployment"}
```
These two can be combined, so you can specify both a template and some
individual overrides.
The resources mentioned can be any resource available in your kubernetes
cluster, as long as you've got a corresponding label.
### Naming
Naming governs the process of converting a Prometheus metric name into
a metric in the custom metrics API, and vice versa. It's controlled by
the `name` field.
Naming is controlled by specifying a pattern to extract an API name from
a Prometheus name, and potentially a transformation on that extracted
value.
The pattern is specified in the `matches` field, and is just a regular
expression. If not specified, it defaults to `.*`.
The transformation is specified by the `as` field. You can use any
capture groups defined in the `matches` field. If the `matches` field
doesn't contain capture groups, the `as` field defaults to `$0`. If it
contains a single capture group, the `as` field defautls to `$1`.
Otherwise, it's an error not to specify the as field.
For example:
```yaml
# match turn any name <name>_total to <name>_per_second
# e.g. http_requests_total becomes http_requests_per_second
name:
matches: "^(.*)_total$"
as: "${1}_per_second"
```
### Querying
Querying governs the process of actually fetching values for a particular
metric. It's controlled by the `metricsQuery` field.
The `metricsQuery` field is a Go template that gets turned into
a Prometheus query, using input from a particular call to the custom
metrics API. A given call to the custom metrics API is distilled down to
a metric name, a group-resource, and one or more objects of that
group-resource. These get turned into the following fields in the
template:
- `Series`: the metric name
- `LabelMatchers`: a comma-separated list of label matchers matching the
given objects. Currently, this is the label for the particular
group-resource, plus the label for namespace, if the group-resource is
namespaced.
- `GroupBy`: a comma-separated list of labels to group by. Currently,
this contains the group-resource label used in `LabelMatchers`.
For instance, suppose we had a series `http_requests_total` (exposed as
`http_requests_per_second` in the API) with labels `service`, `pod`,
`ingress`, `namespace`, and `verb`. The first four correspond to
Kubernetes resources. Then, if someone requested the metric
`pods/http_request_per_second` for the pods `pod1` and `pod2` in the
`somens` namespace, we'd have:
- `Series: "http_requests_total"`
- `LabelMatchers: "pod=~\"pod1|pod2",namespace="somens"`
- `GroupBy`: `pod`
Additionally, there are two advanced fields that are "raw" forms of other
fields:
- `LabelValuesByName`: a map mapping the labels and values from the
`LabelMatchers` field. The values are pre-joined by `|`
(for used with the `=~` matcher in Prometheus).
- `GroupBySlice`: the slice form of `GroupBy`.
In general, you'll probably want to use the `Series`, `LabelMatchers`, and
`GroupBy` fields. The other two are for advanced usage.
The query is expected to return one value for each object requested. The
adapter will use the labels on the returned series to associate a given
series back to its corresponding object.
For example:
```yaml
# convert cumulative cAdvisor metrics into rates calculated over 2 minutes
metricsQuery: "sum(rate(<<.Series>>{<<.LabelMatchers>>,container_name!="POD"}[2m])) by (<<.GroupBy>>)"
```
versioned_docs/version-2.0-2.4/cluster-admin/tools/cluster-monitoring/expression/expression.md
+436
View File
@@ -0,0 +1,436 @@
---
title: Prometheus Expressions
weight: 4
aliases:
- /rancher/v2.0-v2.4/en/project-admin/tools/monitoring/expression
- /rancher/v2.0-v2.4/en/cluster-admin/tools/monitoring/expression
- /rancher/v2.0-v2.4/en/monitoring-alerting/legacy/monitoring/cluster-monitoring/expression
- /rancher/v2.0-v2.4/en/monitoring-alerting/v2.0.x-v2.4.x/cluster-monitoring/expression
- /rancher/v2.x/en/monitoring-alerting/v2.0.x-v2.4.x/cluster-monitoring/expression/
---
The PromQL expressions in this doc can be used to configure [alerts.]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/tools/alerts/)
> Before expressions can be used in alerts, monitoring must be enabled. For more information, refer to the documentation on enabling monitoring [at the cluster level]({{<baseurl>}}/rancher/v2.0-v2.4/en/monitoring-alerting/legacy/monitoring/cluster-monitoring/) or [at the project level.]({{<baseurl>}}/rancher/v2.0-v2.4/en/project-admin/tools/monitoring/)
For more information about querying Prometheus, refer to the official [Prometheus documentation.](https://prometheus.io/docs/prometheus/latest/querying/basics/)
<!-- TOC -->
- [Cluster Metrics](#cluster-metrics)
- [Cluster CPU Utilization](#cluster-cpu-utilization)
- [Cluster Load Average](#cluster-load-average)
- [Cluster Memory Utilization](#cluster-memory-utilization)
- [Cluster Disk Utilization](#cluster-disk-utilization)
- [Cluster Disk I/O](#cluster-disk-i-o)
- [Cluster Network Packets](#cluster-network-packets)
- [Cluster Network I/O](#cluster-network-i-o)
- [Node Metrics](#node-metrics)
- [Node CPU Utilization](#node-cpu-utilization)
- [Node Load Average](#node-load-average)
- [Node Memory Utilization](#node-memory-utilization)
- [Node Disk Utilization](#node-disk-utilization)
- [Node Disk I/O](#node-disk-i-o)
- [Node Network Packets](#node-network-packets)
- [Node Network I/O](#node-network-i-o)
- [Etcd Metrics](#etcd-metrics)
- [Etcd Has a Leader](#etcd-has-a-leader)
- [Number of Times the Leader Changes](#number-of-times-the-leader-changes)
- [Number of Failed Proposals](#number-of-failed-proposals)
- [GRPC Client Traffic](#grpc-client-traffic)
- [Peer Traffic](#peer-traffic)
- [DB Size](#db-size)
- [Active Streams](#active-streams)
- [Raft Proposals](#raft-proposals)
- [RPC Rate](#rpc-rate)
- [Disk Operations](#disk-operations)
- [Disk Sync Duration](#disk-sync-duration)
- [Kubernetes Components Metrics](#kubernetes-components-metrics)
- [API Server Request Latency](#api-server-request-latency)
- [API Server Request Rate](#api-server-request-rate)
- [Scheduling Failed Pods](#scheduling-failed-pods)
- [Controller Manager Queue Depth](#controller-manager-queue-depth)
- [Scheduler E2E Scheduling Latency](#scheduler-e2e-scheduling-latency)
- [Scheduler Preemption Attempts](#scheduler-preemption-attempts)
- [Ingress Controller Connections](#ingress-controller-connections)
- [Ingress Controller Request Process Time](#ingress-controller-request-process-time)
- [Rancher Logging Metrics](#rancher-logging-metrics)
- [Fluentd Buffer Queue Rate](#fluentd-buffer-queue-rate)
- [Fluentd Input Rate](#fluentd-input-rate)
- [Fluentd Output Errors Rate](#fluentd-output-errors-rate)
- [Fluentd Output Rate](#fluentd-output-rate)
- [Workload Metrics](#workload-metrics)
- [Workload CPU Utilization](#workload-cpu-utilization)
- [Workload Memory Utilization](#workload-memory-utilization)
- [Workload Network Packets](#workload-network-packets)
- [Workload Network I/O](#workload-network-i-o)
- [Workload Disk I/O](#workload-disk-i-o)
- [Pod Metrics](#pod-metrics)
- [Pod CPU Utilization](#pod-cpu-utilization)
- [Pod Memory Utilization](#pod-memory-utilization)
- [Pod Network Packets](#pod-network-packets)
- [Pod Network I/O](#pod-network-i-o)
- [Pod Disk I/O](#pod-disk-i-o)
- [Container Metrics](#container-metrics)
- [Container CPU Utilization](#container-cpu-utilization)
- [Container Memory Utilization](#container-memory-utilization)
- [Container Disk I/O](#container-disk-i-o)
<!-- /TOC -->
# Cluster Metrics
### Cluster CPU Utilization
| Catalog | Expression |
| --- | --- |
| Detail | `1 - (avg(irate(node_cpu_seconds_total{mode="idle"}[5m])) by (instance))` |
| Summary | `1 - (avg(irate(node_cpu_seconds_total{mode="idle"}[5m])))` |
### Cluster Load Average
| Catalog | Expression |
| --- | --- |
| Detail | <table><tr><td>load1</td><td>`sum(node_load1) by (instance) / count(node_cpu_seconds_total{mode="system"}) by (instance)`</td></tr><tr><td>load5</td><td>`sum(node_load5) by (instance) / count(node_cpu_seconds_total{mode="system"}) by (instance)`</td></tr><tr><td>load15</td><td>`sum(node_load15) by (instance) / count(node_cpu_seconds_total{mode="system"}) by (instance)`</td></tr></table> |
| Summary | <table><tr><td>load1</td><td>`sum(node_load1) by (instance) / count(node_cpu_seconds_total{mode="system"})`</td></tr><tr><td>load5</td><td>`sum(node_load5) by (instance) / count(node_cpu_seconds_total{mode="system"})`</td></tr><tr><td>load15</td><td>`sum(node_load15) by (instance) / count(node_cpu_seconds_total{mode="system"})`</td></tr></table> |
### Cluster Memory Utilization
| Catalog | Expression |
| --- | --- |
| Detail | `1 - sum(node_memory_MemAvailable_bytes) by (instance) / sum(node_memory_MemTotal_bytes) by (instance)` |
| Summary | `1 - sum(node_memory_MemAvailable_bytes) / sum(node_memory_MemTotal_bytes)` |
### Cluster Disk Utilization
| Catalog | Expression |
| --- | --- |
| Detail | `(sum(node_filesystem_size_bytes{device!="rootfs"}) by (instance) - sum(node_filesystem_free_bytes{device!="rootfs"}) by (instance)) / sum(node_filesystem_size_bytes{device!="rootfs"}) by (instance)` |
| Summary | `(sum(node_filesystem_size_bytes{device!="rootfs"}) - sum(node_filesystem_free_bytes{device!="rootfs"})) / sum(node_filesystem_size_bytes{device!="rootfs"})` |
### Cluster Disk I/O
| Catalog | Expression |
| --- | --- |
| Detail | <table><tr><td>read</td><td>`sum(rate(node_disk_read_bytes_total[5m])) by (instance)`</td></tr><tr><td>written</td><td>`sum(rate(node_disk_written_bytes_total[5m])) by (instance)`</td></tr></table> |
| Summary | <table><tr><td>read</td><td>`sum(rate(node_disk_read_bytes_total[5m]))`</td></tr><tr><td>written</td><td>`sum(rate(node_disk_written_bytes_total[5m]))`</td></tr></table> |
### Cluster Network Packets
| Catalog | Expression |
| --- | --- |
| Detail | <table><tr><td>receive-dropped</td><td><code>sum(rate(node_network_receive_drop_total{device!~"lo &#124; veth.* &#124; docker.* &#124; flannel.* &#124; cali.* &#124; cbr.*"}[5m])) by (instance)</code></td></tr><tr><td>receive-errs</td><td><code>sum(rate(node_network_receive_errs_total{device!~"lo &#124; veth.* &#124; docker.* &#124; flannel.* &#124; cali.* &#124; cbr.*"}[5m])) by (instance)</code></td></tr><tr><td>receive-packets</td><td><code>sum(rate(node_network_receive_packets_total{device!~"lo &#124; veth.* &#124; docker.* &#124; flannel.* &#124; cali.* &#124; cbr.*"}[5m])) by (instance)</code></td></tr><tr><td>transmit-dropped</td><td><code>sum(rate(node_network_transmit_drop_total{device!~"lo &#124; veth.* &#124; docker.* &#124; flannel.* &#124; cali.* &#124; cbr.*"}[5m])) by (instance)</code></td></tr><tr><td>transmit-errs</td><td><code>sum(rate(node_network_transmit_errs_total{device!~"lo &#124; veth.* &#124; docker.* &#124; flannel.* &#124; cali.* &#124; cbr.*"}[5m])) by (instance)</code></td></tr><tr><td>transmit-packets</td><td><code>sum(rate(node_network_transmit_packets_total{device!~"lo &#124; veth.* &#124; docker.* &#124; flannel.* &#124; cali.* &#124; cbr.*"}[5m])) by (instance)</code></td></tr></table> |
| Summary | <table><tr><td>receive-dropped</td><td><code>sum(rate(node_network_receive_drop_total{device!~"lo &#124; veth.* &#124; docker.* &#124; flannel.* &#124; cali.* &#124; cbr.*"}[5m]))</code></td></tr><tr><td>receive-errs</td><td><code>sum(rate(node_network_receive_errs_total{device!~"lo &#124; veth.* &#124; docker.* &#124; flannel.* &#124; cali.* &#124; cbr.*"}[5m]))</code></td></tr><tr><td>receive-packets</td><td><code>sum(rate(node_network_receive_packets_total{device!~"lo &#124; veth.* &#124; docker.* &#124; flannel.* &#124; cali.* &#124; cbr.*"}[5m]))</code></td></tr><tr><td>transmit-dropped</td><td><code>sum(rate(node_network_transmit_drop_total{device!~"lo &#124; veth.* &#124; docker.* &#124; flannel.* &#124; cali.* &#124; cbr.*"}[5m]))</code></td></tr><tr><td>transmit-errs</td><td><code>sum(rate(node_network_transmit_errs_total{device!~"lo &#124; veth.* &#124; docker.* &#124; flannel.* &#124; cali.* &#124; cbr.*"}[5m]))</code></td></tr><tr><td>transmit-packets</td><td><code>sum(rate(node_network_transmit_packets_total{device!~"lo &#124; veth.* &#124; docker.* &#124; flannel.* &#124; cali.* &#124; cbr.*"}[5m]))</code></td></tr></table> |
### Cluster Network I/O
| Catalog | Expression |
| --- | --- |
| Detail | <table><tr><td>receive</td><td><code>sum(rate(node_network_receive_bytes_total{device!~"lo &#124; veth.* &#124; docker.* &#124; flannel.* &#124; cali.* &#124; cbr.*"}[5m])) by (instance)</code></td></tr><tr><td>transmit</td><td><code>sum(rate(node_network_transmit_bytes_total{device!~"lo &#124; veth.* &#124; docker.* &#124; flannel.* &#124; cali.* &#124; cbr.*"}[5m])) by (instance)</code></td></tr></table> |
| Summary | <table><tr><td>receive</td><td><code>sum(rate(node_network_receive_bytes_total{device!~"lo &#124; veth.* &#124; docker.* &#124; flannel.* &#124; cali.* &#124; cbr.*"}[5m]))</code></td></tr><tr><td>transmit</td><td><code>sum(rate(node_network_transmit_bytes_total{device!~"lo &#124; veth.* &#124; docker.* &#124; flannel.* &#124; cali.* &#124; cbr.*"}[5m]))</code></td></tr></table> |
# Node Metrics
### Node CPU Utilization
| Catalog | Expression |
| --- | --- |
| Detail | `avg(irate(node_cpu_seconds_total{mode!="idle", instance=~"$instance"}[5m])) by (mode)` |
| Summary | `1 - (avg(irate(node_cpu_seconds_total{mode="idle", instance=~"$instance"}[5m])))` |
### Node Load Average
| Catalog | Expression |
| --- | --- |
| Detail | <table><tr><td>load1</td><td>`sum(node_load1{instance=~"$instance"}) / count(node_cpu_seconds_total{mode="system",instance=~"$instance"})`</td></tr><tr><td>load5</td><td>`sum(node_load5{instance=~"$instance"}) / count(node_cpu_seconds_total{mode="system",instance=~"$instance"})`</td></tr><tr><td>load15</td><td>`sum(node_load15{instance=~"$instance"}) / count(node_cpu_seconds_total{mode="system",instance=~"$instance"})`</td></tr></table> |
| Summary | <table><tr><td>load1</td><td>`sum(node_load1{instance=~"$instance"}) / count(node_cpu_seconds_total{mode="system",instance=~"$instance"})`</td></tr><tr><td>load5</td><td>`sum(node_load5{instance=~"$instance"}) / count(node_cpu_seconds_total{mode="system",instance=~"$instance"})`</td></tr><tr><td>load15</td><td>`sum(node_load15{instance=~"$instance"}) / count(node_cpu_seconds_total{mode="system",instance=~"$instance"})`</td></tr></table> |
### Node Memory Utilization
| Catalog | Expression |
| --- | --- |
| Detail | `1 - sum(node_memory_MemAvailable_bytes{instance=~"$instance"}) / sum(node_memory_MemTotal_bytes{instance=~"$instance"})` |
| Summary | `1 - sum(node_memory_MemAvailable_bytes{instance=~"$instance"}) / sum(node_memory_MemTotal_bytes{instance=~"$instance"}) ` |
### Node Disk Utilization
| Catalog | Expression |
| --- | --- |
| Detail | `(sum(node_filesystem_size_bytes{device!="rootfs",instance=~"$instance"}) by (device) - sum(node_filesystem_free_bytes{device!="rootfs",instance=~"$instance"}) by (device)) / sum(node_filesystem_size_bytes{device!="rootfs",instance=~"$instance"}) by (device)` |
| Summary | `(sum(node_filesystem_size_bytes{device!="rootfs",instance=~"$instance"}) - sum(node_filesystem_free_bytes{device!="rootfs",instance=~"$instance"})) / sum(node_filesystem_size_bytes{device!="rootfs",instance=~"$instance"})` |
### Node Disk I/O
| Catalog | Expression |
| --- | --- |
| Detail | <table><tr><td>read</td><td>`sum(rate(node_disk_read_bytes_total{instance=~"$instance"}[5m]))`</td></tr><tr><td>written</td><td>`sum(rate(node_disk_written_bytes_total{instance=~"$instance"}[5m]))`</td></tr></table> |
| Summary | <table><tr><td>read</td><td>`sum(rate(node_disk_read_bytes_total{instance=~"$instance"}[5m]))`</td></tr><tr><td>written</td><td>`sum(rate(node_disk_written_bytes_total{instance=~"$instance"}[5m]))`</td></tr></table> |
### Node Network Packets
| Catalog | Expression |
| --- | --- |
| Detail | <table><tr><td>receive-dropped</td><td><code>sum(rate(node_network_receive_drop_total{device!~"lo &#124; veth.* &#124; docker.* &#124; flannel.* &#124; cali.* &#124; cbr.*",instance=~"$instance"}[5m])) by (device)</code></td></tr><tr><td>receive-errs</td><td><code>sum(rate(node_network_receive_errs_total{device!~"lo &#124; veth.* &#124; docker.* &#124; flannel.* &#124; cali.* &#124; cbr.*",instance=~"$instance"}[5m])) by (device)</code></td></tr><tr><td>receive-packets</td><td><code>sum(rate(node_network_receive_packets_total{device!~"lo &#124; veth.* &#124; docker.* &#124; flannel.* &#124; cali.* &#124; cbr.*",instance=~"$instance"}[5m])) by (device)</code></td></tr><tr><td>transmit-dropped</td><td><code>sum(rate(node_network_transmit_drop_total{device!~"lo &#124; veth.* &#124; docker.* &#124; flannel.* &#124; cali.* &#124; cbr.*",instance=~"$instance"}[5m])) by (device)</code></td></tr><tr><td>transmit-errs</td><td><code>sum(rate(node_network_transmit_errs_total{device!~"lo &#124; veth.* &#124; docker.* &#124; flannel.* &#124; cali.* &#124; cbr.*",instance=~"$instance"}[5m])) by (device)</code></td></tr><tr><td>transmit-packets</td><td><code>sum(rate(node_network_transmit_packets_total{device!~"lo &#124; veth.* &#124; docker.* &#124; flannel.* &#124; cali.* &#124; cbr.*",instance=~"$instance"}[5m])) by (device)</code></td></tr></table> |
| Summary | <table><tr><td>receive-dropped</td><td><code>sum(rate(node_network_receive_drop_total{device!~"lo &#124; veth.* &#124; docker.* &#124; flannel.* &#124; cali.* &#124; cbr.*",instance=~"$instance"}[5m]))</code></td></tr><tr><td>receive-errs</td><td><code>sum(rate(node_network_receive_errs_total{device!~"lo &#124; veth.* &#124; docker.* &#124; flannel.* &#124; cali.* &#124; cbr.*",instance=~"$instance"}[5m]))</code></td></tr><tr><td>receive-packets</td><td><code>sum(rate(node_network_receive_packets_total{device!~"lo &#124; veth.* &#124; docker.* &#124; flannel.* &#124; cali.* &#124; cbr.*",instance=~"$instance"}[5m]))</code></td></tr><tr><td>transmit-dropped</td><td><code>sum(rate(node_network_transmit_drop_total{device!~"lo &#124; veth.* &#124; docker.* &#124; flannel.* &#124; cali.* &#124; cbr.*",instance=~"$instance"}[5m]))</code></td></tr><tr><td>transmit-errs</td><td><code>sum(rate(node_network_transmit_errs_total{device!~"lo &#124; veth.* &#124; docker.* &#124; flannel.* &#124; cali.* &#124; cbr.*",instance=~"$instance"}[5m]))</code></td></tr><tr><td>transmit-packets</td><td><code>sum(rate(node_network_transmit_packets_total{device!~"lo &#124; veth.* &#124; docker.* &#124; flannel.* &#124; cali.* &#124; cbr.*",instance=~"$instance"}[5m]))</code></td></tr></table> |
### Node Network I/O
| Catalog | Expression |
| --- | --- |
| Detail | <table><tr><td>receive</td><td><code>sum(rate(node_network_receive_bytes_total{device!~"lo &#124; veth.* &#124; docker.* &#124; flannel.* &#124; cali.* &#124; cbr.*",instance=~"$instance"}[5m])) by (device)</code></td></tr><tr><td>transmit</td><td><code>sum(rate(node_network_transmit_bytes_total{device!~"lo &#124; veth.* &#124; docker.* &#124; flannel.* &#124; cali.* &#124; cbr.*",instance=~"$instance"}[5m])) by (device)</code></td></tr></table> |
| Summary | <table><tr><td>receive</td><td><code>sum(rate(node_network_receive_bytes_total{device!~"lo &#124; veth.* &#124; docker.* &#124; flannel.* &#124; cali.* &#124; cbr.*",instance=~"$instance"}[5m]))</code></td></tr><tr><td>transmit</td><td><code>sum(rate(node_network_transmit_bytes_total{device!~"lo &#124; veth.* &#124; docker.* &#124; flannel.* &#124; cali.* &#124; cbr.*",instance=~"$instance"}[5m]))</code></td></tr></table> |
# Etcd Metrics
### Etcd Has a Leader
`max(etcd_server_has_leader)`
### Number of Times the Leader Changes
`max(etcd_server_leader_changes_seen_total)`
### Number of Failed Proposals
`sum(etcd_server_proposals_failed_total)`
### GRPC Client Traffic
| Catalog | Expression |
| --- | --- |
| Detail | <table><tr><td>in</td><td>`sum(rate(etcd_network_client_grpc_received_bytes_total[5m])) by (instance)`</td></tr><tr><td>out</td><td>`sum(rate(etcd_network_client_grpc_sent_bytes_total[5m])) by (instance)`</td></tr></table> |
| Summary | <table><tr><td>in</td><td>`sum(rate(etcd_network_client_grpc_received_bytes_total[5m]))`</td></tr><tr><td>out</td><td>`sum(rate(etcd_network_client_grpc_sent_bytes_total[5m]))`</td></tr></table> |
### Peer Traffic
| Catalog | Expression |
| --- | --- |
| Detail | <table><tr><td>in</td><td>`sum(rate(etcd_network_peer_received_bytes_total[5m])) by (instance)`</td></tr><tr><td>out</td><td>`sum(rate(etcd_network_peer_sent_bytes_total[5m])) by (instance)`</td></tr></table> |
| Summary | <table><tr><td>in</td><td>`sum(rate(etcd_network_peer_received_bytes_total[5m]))`</td></tr><tr><td>out</td><td>`sum(rate(etcd_network_peer_sent_bytes_total[5m]))`</td></tr></table> |
### DB Size
| Catalog | Expression |
| --- | --- |
| Detail | `sum(etcd_debugging_mvcc_db_total_size_in_bytes) by (instance)` |
| Summary | `sum(etcd_debugging_mvcc_db_total_size_in_bytes)` |
### Active Streams
| Catalog | Expression |
| --- | --- |
| Detail | <table><tr><td>lease-watch</td><td>`sum(grpc_server_started_total{grpc_service="etcdserverpb.Lease",grpc_type="bidi_stream"}) by (instance) - sum(grpc_server_handled_total{grpc_service="etcdserverpb.Lease",grpc_type="bidi_stream"}) by (instance)`</td></tr><tr><td>watch</td><td>`sum(grpc_server_started_total{grpc_service="etcdserverpb.Watch",grpc_type="bidi_stream"}) by (instance) - sum(grpc_server_handled_total{grpc_service="etcdserverpb.Watch",grpc_type="bidi_stream"}) by (instance)`</td></tr></table> |
| Summary | <table><tr><td>lease-watch</td><td>`sum(grpc_server_started_total{grpc_service="etcdserverpb.Lease",grpc_type="bidi_stream"}) - sum(grpc_server_handled_total{grpc_service="etcdserverpb.Lease",grpc_type="bidi_stream"})`</td></tr><tr><td>watch</td><td>`sum(grpc_server_started_total{grpc_service="etcdserverpb.Watch",grpc_type="bidi_stream"}) - sum(grpc_server_handled_total{grpc_service="etcdserverpb.Watch",grpc_type="bidi_stream"})`</td></tr></table> |
### Raft Proposals
| Catalog | Expression |
| --- | --- |
| Detail | <table><tr><td>applied</td><td>`sum(increase(etcd_server_proposals_applied_total[5m])) by (instance)`</td></tr><tr><td>committed</td><td>`sum(increase(etcd_server_proposals_committed_total[5m])) by (instance)`</td></tr><tr><td>pending</td><td>`sum(increase(etcd_server_proposals_pending[5m])) by (instance)`</td></tr><tr><td>failed</td><td>`sum(increase(etcd_server_proposals_failed_total[5m])) by (instance)`</td></tr></table> |
| Summary | <table><tr><td>applied</td><td>`sum(increase(etcd_server_proposals_applied_total[5m]))`</td></tr><tr><td>committed</td><td>`sum(increase(etcd_server_proposals_committed_total[5m]))`</td></tr><tr><td>pending</td><td>`sum(increase(etcd_server_proposals_pending[5m]))`</td></tr><tr><td>failed</td><td>`sum(increase(etcd_server_proposals_failed_total[5m]))`</td></tr></table> |
### RPC Rate
| Catalog | Expression |
| --- | --- |
| Detail | <table><tr><td>total</td><td>`sum(rate(grpc_server_started_total{grpc_type="unary"}[5m])) by (instance)`</td></tr><tr><td>fail</td><td>`sum(rate(grpc_server_handled_total{grpc_type="unary",grpc_code!="OK"}[5m])) by (instance)`</td></tr></table> |
| Summary | <table><tr><td>total</td><td>`sum(rate(grpc_server_started_total{grpc_type="unary"}[5m]))`</td></tr><tr><td>fail</td><td>`sum(rate(grpc_server_handled_total{grpc_type="unary",grpc_code!="OK"}[5m]))`</td></tr></table> |
### Disk Operations
| Catalog | Expression |
| --- | --- |
| Detail | <table><tr><td>commit-called-by-backend</td><td>`sum(rate(etcd_disk_backend_commit_duration_seconds_sum[1m])) by (instance)`</td></tr><tr><td>fsync-called-by-wal</td><td>`sum(rate(etcd_disk_wal_fsync_duration_seconds_sum[1m])) by (instance)`</td></tr></table> |
| Summary | <table><tr><td>commit-called-by-backend</td><td>`sum(rate(etcd_disk_backend_commit_duration_seconds_sum[1m]))`</td></tr><tr><td>fsync-called-by-wal</td><td>`sum(rate(etcd_disk_wal_fsync_duration_seconds_sum[1m]))`</td></tr></table> |
### Disk Sync Duration
| Catalog | Expression |
| --- | --- |
| Detail | <table><tr><td>wal</td><td>`histogram_quantile(0.99, sum(rate(etcd_disk_wal_fsync_duration_seconds_bucket[5m])) by (instance, le))`</td></tr><tr><td>db</td><td>`histogram_quantile(0.99, sum(rate(etcd_disk_backend_commit_duration_seconds_bucket[5m])) by (instance, le))`</td></tr></table> |
| Summary | <table><tr><td>wal</td><td>`sum(histogram_quantile(0.99, sum(rate(etcd_disk_wal_fsync_duration_seconds_bucket[5m])) by (instance, le)))`</td></tr><tr><td>db</td><td>`sum(histogram_quantile(0.99, sum(rate(etcd_disk_backend_commit_duration_seconds_bucket[5m])) by (instance, le)))`</td></tr></table> |
# Kubernetes Components Metrics
### API Server Request Latency
| Catalog | Expression |
| --- | --- |
| Detail | `avg(apiserver_request_latencies_sum / apiserver_request_latencies_count) by (instance, verb) /1e+06` |
| Summary | `avg(apiserver_request_latencies_sum / apiserver_request_latencies_count) by (instance) /1e+06` |
### API Server Request Rate
| Catalog | Expression |
| --- | --- |
| Detail | `sum(rate(apiserver_request_count[5m])) by (instance, code)` |
| Summary | `sum(rate(apiserver_request_count[5m])) by (instance)` |
### Scheduling Failed Pods
| Catalog | Expression |
| --- | --- |
| Detail | `sum(kube_pod_status_scheduled{condition="false"})` |
| Summary | `sum(kube_pod_status_scheduled{condition="false"})` |
### Controller Manager Queue Depth
| Catalog | Expression |
| --- | --- |
| Detail | <table><tr><td>volumes</td><td>`sum(volumes_depth) by instance`</td></tr><tr><td>deployment</td><td>`sum(deployment_depth) by instance`</td></tr><tr><td>replicaset</td><td>`sum(replicaset_depth) by instance`</td></tr><tr><td>service</td><td>`sum(service_depth) by instance`</td></tr><tr><td>serviceaccount</td><td>`sum(serviceaccount_depth) by instance`</td></tr><tr><td>endpoint</td><td>`sum(endpoint_depth) by instance`</td></tr><tr><td>daemonset</td><td>`sum(daemonset_depth) by instance`</td></tr><tr><td>statefulset</td><td>`sum(statefulset_depth) by instance`</td></tr><tr><td>replicationmanager</td><td>`sum(replicationmanager_depth) by instance`</td></tr></table> |
| Summary | <table><tr><td>volumes</td><td>`sum(volumes_depth)`</td></tr><tr><td>deployment</td><td>`sum(deployment_depth)`</td></tr><tr><td>replicaset</td><td>`sum(replicaset_depth)`</td></tr><tr><td>service</td><td>`sum(service_depth)`</td></tr><tr><td>serviceaccount</td><td>`sum(serviceaccount_depth)`</td></tr><tr><td>endpoint</td><td>`sum(endpoint_depth)`</td></tr><tr><td>daemonset</td><td>`sum(daemonset_depth)`</td></tr><tr><td>statefulset</td><td>`sum(statefulset_depth)`</td></tr><tr><td>replicationmanager</td><td>`sum(replicationmanager_depth)`</td></tr></table> |
### Scheduler E2E Scheduling Latency
| Catalog | Expression |
| --- | --- |
| Detail | `histogram_quantile(0.99, sum(scheduler_e2e_scheduling_latency_microseconds_bucket) by (le, instance)) / 1e+06` |
| Summary | `sum(histogram_quantile(0.99, sum(scheduler_e2e_scheduling_latency_microseconds_bucket) by (le, instance)) / 1e+06)` |
### Scheduler Preemption Attempts
| Catalog | Expression |
| --- | --- |
| Detail | `sum(rate(scheduler_total_preemption_attempts[5m])) by (instance)` |
| Summary | `sum(rate(scheduler_total_preemption_attempts[5m]))` |
### Ingress Controller Connections
| Catalog | Expression |
| --- | --- |
| Detail | <table><tr><td>reading</td><td>`sum(nginx_ingress_controller_nginx_process_connections{state="reading"}) by (instance)`</td></tr><tr><td>waiting</td><td>`sum(nginx_ingress_controller_nginx_process_connections{state="waiting"}) by (instance)`</td></tr><tr><td>writing</td><td>`sum(nginx_ingress_controller_nginx_process_connections{state="writing"}) by (instance)`</td></tr><tr><td>accepted</td><td>`sum(ceil(increase(nginx_ingress_controller_nginx_process_connections_total{state="accepted"}[5m]))) by (instance)`</td></tr><tr><td>active</td><td>`sum(ceil(increase(nginx_ingress_controller_nginx_process_connections_total{state="active"}[5m]))) by (instance)`</td></tr><tr><td>handled</td><td>`sum(ceil(increase(nginx_ingress_controller_nginx_process_connections_total{state="handled"}[5m]))) by (instance)`</td></tr></table> |
| Summary | <table><tr><td>reading</td><td>`sum(nginx_ingress_controller_nginx_process_connections{state="reading"})`</td></tr><tr><td>waiting</td><td>`sum(nginx_ingress_controller_nginx_process_connections{state="waiting"})`</td></tr><tr><td>writing</td><td>`sum(nginx_ingress_controller_nginx_process_connections{state="writing"})`</td></tr><tr><td>accepted</td><td>`sum(ceil(increase(nginx_ingress_controller_nginx_process_connections_total{state="accepted"}[5m])))`</td></tr><tr><td>active</td><td>`sum(ceil(increase(nginx_ingress_controller_nginx_process_connections_total{state="active"}[5m])))`</td></tr><tr><td>handled</td><td>`sum(ceil(increase(nginx_ingress_controller_nginx_process_connections_total{state="handled"}[5m])))`</td></tr></table> |
### Ingress Controller Request Process Time
| Catalog | Expression |
| --- | --- |
| Detail | `topk(10, histogram_quantile(0.95,sum by (le, host, path)(rate(nginx_ingress_controller_request_duration_seconds_bucket{host!="_"}[5m]))))` |
| Summary | `topk(10, histogram_quantile(0.95,sum by (le, host)(rate(nginx_ingress_controller_request_duration_seconds_bucket{host!="_"}[5m]))))` |
# Rancher Logging Metrics
### Fluentd Buffer Queue Rate
| Catalog | Expression |
| --- | --- |
| Detail | `sum(rate(fluentd_output_status_buffer_queue_length[5m])) by (instance)` |
| Summary | `sum(rate(fluentd_output_status_buffer_queue_length[5m]))` |
### Fluentd Input Rate
| Catalog | Expression |
| --- | --- |
| Detail | `sum(rate(fluentd_input_status_num_records_total[5m])) by (instance)` |
| Summary | `sum(rate(fluentd_input_status_num_records_total[5m]))` |
### Fluentd Output Errors Rate
| Catalog | Expression |
| --- | --- |
| Detail | `sum(rate(fluentd_output_status_num_errors[5m])) by (type)` |
| Summary | `sum(rate(fluentd_output_status_num_errors[5m]))` |
### Fluentd Output Rate
| Catalog | Expression |
| --- | --- |
| Detail | `sum(rate(fluentd_output_status_num_records_total[5m])) by (instance)` |
| Summary | `sum(rate(fluentd_output_status_num_records_total[5m]))` |
# Workload Metrics
### Workload CPU Utilization
| Catalog | Expression |
| --- | --- |
| Detail | <table><tr><td>cfs throttled seconds</td><td>`sum(rate(container_cpu_cfs_throttled_seconds_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m])) by (pod_name)`</td></tr><tr><td>user seconds</td><td>`sum(rate(container_cpu_user_seconds_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m])) by (pod_name)`</td></tr><tr><td>system seconds</td><td>`sum(rate(container_cpu_system_seconds_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m])) by (pod_name)`</td></tr><tr><td>usage seconds</td><td>`sum(rate(container_cpu_usage_seconds_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m])) by (pod_name)`</td></tr></table> |
| Summary | <table><tr><td>cfs throttled seconds</td><td>`sum(rate(container_cpu_cfs_throttled_seconds_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m]))`</td></tr><tr><td>user seconds</td><td>`sum(rate(container_cpu_user_seconds_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m]))`</td></tr><tr><td>system seconds</td><td>`sum(rate(container_cpu_system_seconds_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m]))`</td></tr><tr><td>usage seconds</td><td>`sum(rate(container_cpu_usage_seconds_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m]))`</td></tr></table> |
### Workload Memory Utilization
| Catalog | Expression |
| --- | --- |
| Detail | `sum(container_memory_working_set_bytes{namespace="$namespace",pod_name=~"$podName", container_name!=""}) by (pod_name)` |
| Summary | `sum(container_memory_working_set_bytes{namespace="$namespace",pod_name=~"$podName", container_name!=""})` |
### Workload Network Packets
| Catalog | Expression |
| --- | --- |
| Detail | <table><tr><td>receive-packets</td><td>`sum(rate(container_network_receive_packets_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m])) by (pod_name)`</td></tr><tr><td>receive-dropped</td><td>`sum(rate(container_network_receive_packets_dropped_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m])) by (pod_name)`</td></tr><tr><td>receive-errors</td><td>`sum(rate(container_network_receive_errors_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m])) by (pod_name)`</td></tr><tr><td>transmit-packets</td><td>`sum(rate(container_network_transmit_packets_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m])) by (pod_name)`</td></tr><tr><td>transmit-dropped</td><td>`sum(rate(container_network_transmit_packets_dropped_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m])) by (pod_name)`</td></tr><tr><td>transmit-errors</td><td>`sum(rate(container_network_transmit_errors_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m])) by (pod_name)`</td></tr></table> |
| Summary | <table><tr><td>receive-packets</td><td>`sum(rate(container_network_receive_packets_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m]))`</td></tr><tr><td>receive-dropped</td><td>`sum(rate(container_network_receive_packets_dropped_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m]))`</td></tr><tr><td>receive-errors</td><td>`sum(rate(container_network_receive_errors_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m]))`</td></tr><tr><td>transmit-packets</td><td>`sum(rate(container_network_transmit_packets_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m]))`</td></tr><tr><td>transmit-dropped</td><td>`sum(rate(container_network_transmit_packets_dropped_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m]))`</td></tr><tr><td>transmit-errors</td><td>`sum(rate(container_network_transmit_errors_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m]))`</td></tr></table> |
### Workload Network I/O
| Catalog | Expression |
| --- | --- |
| Detail | <table><tr><td>receive</td><td>`sum(rate(container_network_receive_bytes_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m])) by (pod_name)`</td></tr><tr><td>transmit</td><td>`sum(rate(container_network_transmit_bytes_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m])) by (pod_name)`</td></tr></table> |
| Summary | <table><tr><td>receive</td><td>`sum(rate(container_network_receive_bytes_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m]))`</td></tr><tr><td>transmit</td><td>`sum(rate(container_network_transmit_bytes_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m]))`</td></tr></table> |
### Workload Disk I/O
| Catalog | Expression |
| --- | --- |
| Detail | <table><tr><td>read</td><td>`sum(rate(container_fs_reads_bytes_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m])) by (pod_name)`</td></tr><tr><td>write</td><td>`sum(rate(container_fs_writes_bytes_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m])) by (pod_name)`</td></tr></table> |
| Summary | <table><tr><td>read</td><td>`sum(rate(container_fs_reads_bytes_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m]))`</td></tr><tr><td>write</td><td>`sum(rate(container_fs_writes_bytes_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m]))`</td></tr></table> |
# Pod Metrics
### Pod CPU Utilization
| Catalog | Expression |
| --- | --- |
| Detail | <table><tr><td>cfs throttled seconds</td><td>`sum(rate(container_cpu_cfs_throttled_seconds_total{container_name!="POD",namespace="$namespace",pod_name="$podName", container_name!=""}[5m])) by (container_name)`</td></tr><tr><td>usage seconds</td><td>`sum(rate(container_cpu_usage_seconds_total{container_name!="POD",namespace="$namespace",pod_name="$podName", container_name!=""}[5m])) by (container_name)`</td></tr><tr><td>system seconds</td><td>`sum(rate(container_cpu_system_seconds_total{container_name!="POD",namespace="$namespace",pod_name="$podName", container_name!=""}[5m])) by (container_name)`</td></tr><tr><td>user seconds</td><td>`sum(rate(container_cpu_user_seconds_total{container_name!="POD",namespace="$namespace",pod_name="$podName", container_name!=""}[5m])) by (container_name)`</td></tr></table> |
| Summary | <table><tr><td>cfs throttled seconds</td><td>`sum(rate(container_cpu_cfs_throttled_seconds_total{container_name!="POD",namespace="$namespace",pod_name="$podName", container_name!=""}[5m]))`</td></tr><tr><td>usage seconds</td><td>`sum(rate(container_cpu_usage_seconds_total{container_name!="POD",namespace="$namespace",pod_name="$podName", container_name!=""}[5m]))`</td></tr><tr><td>system seconds</td><td>`sum(rate(container_cpu_system_seconds_total{container_name!="POD",namespace="$namespace",pod_name="$podName", container_name!=""}[5m]))`</td></tr><tr><td>user seconds</td><td>`sum(rate(container_cpu_user_seconds_total{container_name!="POD",namespace="$namespace",pod_name="$podName", container_name!=""}[5m]))`</td></tr></table> |
### Pod Memory Utilization
| Catalog | Expression |
| --- | --- |
| Detail | `sum(container_memory_working_set_bytes{container_name!="POD",namespace="$namespace",pod_name="$podName",container_name!=""}) by (container_name)` |
| Summary | `sum(container_memory_working_set_bytes{container_name!="POD",namespace="$namespace",pod_name="$podName",container_name!=""})` |
### Pod Network Packets
| Catalog | Expression |
| --- | --- |
| Detail | <table><tr><td>receive-packets</td><td>`sum(rate(container_network_receive_packets_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`</td></tr><tr><td>receive-dropped</td><td>`sum(rate(container_network_receive_packets_dropped_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`</td></tr><tr><td>receive-errors</td><td>`sum(rate(container_network_receive_errors_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`</td></tr><tr><td>transmit-packets</td><td>`sum(rate(container_network_transmit_packets_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`</td></tr><tr><td>transmit-dropped</td><td>`sum(rate(container_network_transmit_packets_dropped_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`</td></tr><tr><td>transmit-errors</td><td>`sum(rate(container_network_transmit_errors_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`</td></tr></table> |
| Summary | <table><tr><td>receive-packets</td><td>`sum(rate(container_network_receive_packets_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`</td></tr><tr><td>receive-dropped</td><td>`sum(rate(container_network_receive_packets_dropped_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`</td></tr><tr><td>receive-errors</td><td>`sum(rate(container_network_receive_errors_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`</td></tr><tr><td>transmit-packets</td><td>`sum(rate(container_network_transmit_packets_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`</td></tr><tr><td>transmit-dropped</td><td>`sum(rate(container_network_transmit_packets_dropped_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`</td></tr><tr><td>transmit-errors</td><td>`sum(rate(container_network_transmit_errors_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`</td></tr></table> |
### Pod Network I/O
| Catalog | Expression |
| --- | --- |
| Detail | <table><tr><td>receive</td><td>`sum(rate(container_network_receive_bytes_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`</td></tr><tr><td>transmit</td><td>`sum(rate(container_network_transmit_bytes_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`</td></tr></table> |
| Summary | <table><tr><td>receive</td><td>`sum(rate(container_network_receive_bytes_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`</td></tr><tr><td>transmit</td><td>`sum(rate(container_network_transmit_bytes_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`</td></tr></table> |
### Pod Disk I/O
| Catalog | Expression |
| --- | --- |
| Detail | <table><tr><td>read</td><td>`sum(rate(container_fs_reads_bytes_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m])) by (container_name)`</td></tr><tr><td>write</td><td>`sum(rate(container_fs_writes_bytes_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m])) by (container_name)`</td></tr></table> |
| Summary | <table><tr><td>read</td><td>`sum(rate(container_fs_reads_bytes_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`</td></tr><tr><td>write</td><td>`sum(rate(container_fs_writes_bytes_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`</td></tr></table> |
# Container Metrics
### Container CPU Utilization
| Catalog | Expression |
| --- | --- |
| cfs throttled seconds | `sum(rate(container_cpu_cfs_throttled_seconds_total{namespace="$namespace",pod_name="$podName",container_name="$containerName"}[5m]))` |
| usage seconds | `sum(rate(container_cpu_usage_seconds_total{namespace="$namespace",pod_name="$podName",container_name="$containerName"}[5m]))` |
| system seconds | `sum(rate(container_cpu_system_seconds_total{namespace="$namespace",pod_name="$podName",container_name="$containerName"}[5m]))` |
| user seconds | `sum(rate(container_cpu_user_seconds_total{namespace="$namespace",pod_name="$podName",container_name="$containerName"}[5m]))` |
### Container Memory Utilization
`sum(container_memory_working_set_bytes{namespace="$namespace",pod_name="$podName",container_name="$containerName"})`
### Container Disk I/O
| Catalog | Expression |
| --- | --- |
| read | `sum(rate(container_fs_reads_bytes_total{namespace="$namespace",pod_name="$podName",container_name="$containerName"}[5m]))` |
| write | `sum(rate(container_fs_writes_bytes_total{namespace="$namespace",pod_name="$podName",container_name="$containerName"}[5m]))` |
versioned_docs/version-2.0-2.4/cluster-admin/tools/cluster-monitoring/project-monitoring/project-monitoring.md
+84
View File
@@ -0,0 +1,84 @@
---
title: Project Monitoring
weight: 2
aliases:
- /rancher/v2.0-v2.4/en/project-admin/tools/monitoring
- /rancher/v2.0-v2.4/en/monitoring-alerting/v2.0.x-v2.4.x/monitoring/project-monitoring
- /rancher/v2.0-v2.4/en/monitoring-alerting/v2.0.x-v2.4.x/cluster-monitoring/project-monitoring
---
_Available as of v2.2.4_
Using Rancher, you can monitor the state and processes of your cluster nodes, Kubernetes components, and software deployments through integration with [Prometheus](https://prometheus.io/), a leading open-source monitoring solution.
This section covers the following topics:
- [Monitoring scope](#monitoring-scope)
- [Permissions to configure project monitoring](#permissions-to-configure-project-monitoring)
- [Enabling project monitoring](#enabling-project-monitoring)
- [Project-level monitoring resource requirements](#project-level-monitoring-resource-requirements)
- [Project metrics](#project-metrics)
### Monitoring Scope
Using Prometheus, you can monitor Rancher at both the [cluster level]({{<baseurl>}}/rancher/v2.0-v2.4/en/monitoring-alerting/legacy/monitoring/cluster-monitoring/) and project level. For each cluster and project that is enabled for monitoring, Rancher deploys a Prometheus server.
- [Cluster monitoring]({{<baseurl>}}/rancher/v2.0-v2.4/en/monitoring-alerting/legacy/monitoring/cluster-monitoring/) allows you to view the health of your Kubernetes cluster. Prometheus collects metrics from the cluster components below, which you can view in graphs and charts.
- Kubernetes control plane
- etcd database
- All nodes (including workers)
- Project monitoring allows you to view the state of pods running in a given project. Prometheus collects metrics from the project's deployed HTTP and TCP/UDP workloads.
### Permissions to Configure Project Monitoring
Only [administrators]({{<baseurl>}}/rancher/v2.0-v2.4/en/admin-settings/rbac/global-permissions/), [cluster owners or members]({{<baseurl>}}/rancher/v2.0-v2.4/en/admin-settings/rbac/cluster-project-roles/#cluster-roles), or [project owners]({{<baseurl>}}/rancher/v2.0-v2.4/en/admin-settings/rbac/cluster-project-roles/#project-roles) can configure project level monitoring. Project members can only view monitoring metrics.
### Enabling Project Monitoring
> **Prerequisite:** Cluster monitoring must be [enabled.]({{<baseurl>}}/rancher/v2.0-v2.4/en/monitoring-alerting/legacy/monitoring/cluster-monitoring/)
1. Go to the project where monitoring should be enabled. Note: When cluster monitoring is enabled, monitoring is also enabled by default in the **System** project.
1. Select **Tools > Monitoring** in the navigation bar.
1. Select **Enable** to show the [Prometheus configuration options]({{<baseurl>}}/rancher/v2.0-v2.4/en/monitoring-alerting/legacy/monitoring/cluster-monitoring/prometheus/). Enter in your desired configuration options.
1. Click **Save**.
### Project-Level Monitoring Resource Requirements
Container| CPU - Request | Mem - Request | CPU - Limit | Mem - Limit | Configurable
---------|---------------|---------------|-------------|-------------|-------------
Prometheus|750m| 750Mi | 1000m | 1000Mi | Yes
Grafana | 100m | 100Mi | 200m | 200Mi | No
**Result:** A single application,`project-monitoring`, is added as an [application]({{<baseurl>}}/rancher/v2.0-v2.4/en/catalog/apps/) to the project. After the application is `active`, you can start viewing project metrics through the [Rancher dashboard]({{<baseurl>}}/rancher/v2.0-v2.4/en/monitoring-alerting/legacy/monitoring/cluster-monitoring/) or directly from Grafana.
> The default username and password for the Grafana instance will be `admin/admin`. However, Grafana dashboards are served via the Rancher authentication proxy, so only users who are currently authenticated into the Rancher server have access to the Grafana dashboard.
### Project Metrics
[Workload metrics]({{<baseurl>}}/rancher/v2.0-v2.4/en/monitoring-alerting/v2.0.x-v2.4.x/cluster-monitoring/expression/#workload-metrics) are available for the project if monitoring is enabled at the [cluster level]({{<baseurl>}}/rancher/v2.0-v2.4/en/monitoring-alerting/legacy/monitoring/cluster-monitoring/) and at the [project level.](#enabling-project-monitoring)
You can monitor custom metrics from any [exporters.](https://prometheus.io/docs/instrumenting/exporters/) You can also expose some custom endpoints on deployments without needing to configure Prometheus for your project.
> **Example:**
> A [Redis](https://redis.io/) application is deployed in the namespace `redis-app` in the project `Datacenter`. It is monitored via [Redis exporter](https://github.com/oliver006/redis_exporter). After enabling project monitoring, you can edit the application to configure the <b>Advanced Options -> Custom Metrics</b> section. Enter the `Container Port` and `Path` and select the `Protocol`.
To access a project-level Grafana instance,
1. From the **Global** view, navigate to a cluster that has monitoring enabled.
1. Go to a project that has monitoring enabled.
1. From the project view, click **Apps.** In versions before v2.2.0, choose **Catalog Apps** on the main navigation bar.
1. Go to the `project-monitoring` application.
1. In the `project-monitoring` application, there are two `/index.html` links: one that leads to a Grafana instance and one that leads to a Prometheus instance. When you click the Grafana link, it will redirect you to a new webpage for Grafana, which shows metrics for the cluster.
1. You will be signed in to the Grafana instance automatically. The default username is `admin` and the default password is `admin`. For security, we recommend that you log out of Grafana, log back in with the `admin` password, and change your password.
**Results:** You will be logged into Grafana from the Grafana instance. After logging in, you can view the preset Grafana dashboards, which are imported via the [Grafana provisioning mechanism](http://docs.grafana.org/administration/provisioning/#dashboards), so you cannot modify them directly. For now, if you want to configure your own dashboards, clone the original and modify the new copy.
versioned_docs/version-2.0-2.4/cluster-admin/tools/cluster-monitoring/prometheus/prometheus.md
+112
View File
@@ -0,0 +1,112 @@
---
title: Prometheus Configuration
weight: 1
aliases:
- /rancher/v2.0-v2.4/en/project-admin/tools/monitoring/prometheus
- /rancher/v2.0-v2.4/en/cluster-admin/tools/monitoring/prometheus/
- /rancher/v2.0-v2.4/en/monitoring-alerting/legacy/monitoring/cluster-monitoring/prometheus
- /rancher/v2.0-v2.4/en/monitoring-alerting/v2.0.x-v2.4.x/cluster-monitoring/prometheus
- /rancher/v2.x/en/monitoring-alerting/v2.0.x-v2.4.x/cluster-monitoring/prometheus/
---
_Available as of v2.2.0_
While configuring monitoring at either the [cluster level]({{<baseurl>}}/rancher/v2.0-v2.4/en/monitoring-alerting/legacy/monitoring/cluster-monitoring/) or [project level]({{<baseurl>}}/rancher/v2.0-v2.4/en/project-admin/tools/monitoring/), there are multiple options that can be configured.
- [Basic Configuration](#basic-configuration)
- [Advanced Options](#advanced-options)
- [Node Exporter](#node-exporter)
- [Persistent Storage](#persistent-storage)
- [Remote Storage](#remote-storage)
# Basic Configuration
Option | Description
-------|-------------
Data Retention | How long your Prometheus instance retains monitoring data scraped from Rancher objects before it's purged.
[Enable Node Exporter](#node-exporter) | Whether or not to deploy the node exporter.
Node Exporter Host Port | The host port on which data is exposed, i.e. data that Prometheus collects from your node hardware. Required if you have enabled the node exporter.
[Enable Persistent Storage](#persistent-storage) for Prometheus | Whether or not to configure storage for Prometheus so that metrics can be retained even if the Prometheus pod fails.
[Enable Persistent Storage](#persistent-storage) for Grafana | Whether or not to configure storage for Grafana so that the Grafana dashboards and configuration can be retained even if the Grafana pod fails.
Prometheus [CPU Limit](https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/#meaning-of-cpu) | CPU resource limit for the Prometheus pod.
Prometheus [CPU Reservation](https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/#meaning-of-cpu) | CPU reservation for the Prometheus pod.
Prometheus [Memory Limit](https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/#meaning-of-memory) | Memory resource limit for the Prometheus pod.
Prometheus [Memory Reservation](https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/#meaning-of-memory) | Memory resource requests for the Prometheus pod.
Selector | Ability to select the nodes in which Prometheus and Grafana pods are deployed to. To use this option, the nodes must have labels.
# Advanced Options
Since monitoring is an [application](https://github.com/rancher/system-charts/tree/dev/charts/rancher-monitoring) from the [Rancher catalog]({{<baseurl>}}/rancher/v2.0-v2.4/en/catalog/), it can be configured like any other catalog application, by passing in values to Helm.
> **Warning:** Any modification to the application without understanding the entire application can lead to catastrophic errors.
### Prometheus RemoteRead and RemoteWrite
_Available as of v2.4.0_
Prometheus RemoteRead and RemoteWrite can be configured as custom answers in the **Advanced Options** section.
For more information on remote endpoints and storage, refer to the [Prometheus documentation.](https://prometheus.io/docs/operating/integrations/#remote-endpoints-and-storage)
The Prometheus operator documentation contains the full [RemoteReadSpec](https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/api.md#remotereadspec) and [RemoteWriteSpec.](https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/api.md#remotewritespec)
An example configuration would be:
| Variable | Value |
|--------------|------------|
| `prometheus.remoteWrite[0].url` | `http://mytarget.com` |
### LivenessProbe and ReadinessProbe
_Available as of v2.4.0_
Prometheus LivenessProbe and ReadinessProbe can be configured as custom answers in the **Advanced Options** section.
The Kubernetes probe spec is [here.](https://v1-17.docs.kubernetes.io/docs/reference/generated/kubernetes-api/v1.17/#probe-v1-core)
Some example key-value pairs are:
| Variable | Value |
|--------------|------------|
| `prometheus.livenessProbe.timeoutSeconds` | 60 |
| `prometheus.readinessProbe.timeoutSeconds` | 60 |
# Node Exporter
The [node exporter](https://github.com/prometheus/node_exporter/blob/master/README.md) is a popular open source exporter, which exposes the metrics for hardware and \*NIX kernels OS. It is designed to monitor the host system. However, there are still issues with namespaces when running it in a container, mostly around filesystem mount spaces. In order to monitor actual network metrics for the container network, the node exporter must be deployed with the `hostNetwork` mode.
When configuring Prometheus and enabling the node exporter, enter a host port in the **Node Exporter Host Port** that will not produce port conflicts with existing applications. The host port chosen must be open to allow internal traffic between Prometheus and the Node Exporter.
>**Warning:** In order for Prometheus to collect the metrics of the node exporter, after enabling cluster monitoring, you must open the <b>Node Exporter Host Port</b> in the host firewall rules to allow intranet access. By default, `9796` is used as that host port.
# Persistent Storage
>**Prerequisite:** Configure one or more StorageClasses to use as [persistent storage]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/volumes-and-storage/) for your Prometheus or Grafana pod.
By default, when you enable Prometheus for either a cluster or project, all monitoring data that Prometheus collects is stored on its own pod. With local storage, if the Prometheus or Grafana pods fail, all the data is lost. Rancher recommends configuring an external persistent storage to the cluster. With the external persistent storage, if the Prometheus or Grafana pods fail, the new pods can recover using data from the persistent storage.
When enabling persistent storage for Prometheus or Grafana, specify the size of the persistent volume and select the StorageClass.
# Remote Storage
>**Prerequisite:** Need a remote storage endpoint to be available. The possible list of integrations is available [here](https://prometheus.io/docs/operating/integrations/)
Using advanced options, remote storage integration for the Prometheus installation can be configured as follows:
```
prometheus.remoteWrite[0].url = http://remote1/push
prometheus.remoteWrite[0].remoteTimeout = 33s
prometheus.remoteWrite[1].url = http://remote2/push
prometheus.remoteRead[0].url = http://remote1/read
prometheus.remoteRead[0].proxyUrl = http://proxy.url
prometheus.remoteRead[0].bearerToken = token-value
prometheus.remoteRead[1].url = http://remote2/read
prometheus.remoteRead[1].remoteTimeout = 33s
prometheus.remoteRead[1].readRecent = true
```
Additional fields can be set up based on the [ReadSpec](https://github.com/coreos/prometheus-operator/blob/master/Documentation/api.md#remotereadspec) and [RemoteWriteSpec](https://github.com/coreos/prometheus-operator/blob/master/Documentation/api.md#remotewritespec)
versioned_docs/version-2.0-2.4/cluster-admin/tools/cluster-monitoring/viewing-metrics/viewing-metrics.md
+66
View File
@@ -0,0 +1,66 @@
---
title: Viewing Metrics
weight: 2
aliases:
- /rancher/v2.0-v2.4/en/project-admin/tools/monitoring/viewing-metrics
- /rancher/v2.0-v2.4/en/cluster-admin/tools/monitoring/viewing-metrics
- /rancher/v2.0-v2.4/en/monitoring-alerting/legacy/monitoring/cluster-monitoring/viewing-metrics
- /rancher/v2.0-v2.4/en/monitoring-alerting/v2.0.x-v2.4.x/cluster-monitoring/viewing-metrics
- /rancher/v2.x/en/monitoring-alerting/v2.0.x-v2.4.x/cluster-monitoring/viewing-metrics/
---
_Available as of v2.2.0_
After you've enabled monitoring at either the [cluster level]({{<baseurl>}}/rancher/v2.0-v2.4/en/monitoring-alerting/legacy/monitoring/cluster-monitoring/) or [project level]({{<baseurl>}}/rancher/v2.0-v2.4/en/project-admin/tools/monitoring/), you will want to be start viewing the data being collected. There are multiple ways to view this data.
## Rancher Dashboard
>**Note:** This is only available if you've enabled monitoring at the [cluster level]({{<baseurl>}}/rancher/v2.0-v2.4/en/monitoring-alerting/legacy/monitoring/cluster-monitoring/). Project specific analytics must be viewed using the project's Grafana instance.
Rancher's dashboards are available at multiple locations:
- **Cluster Dashboard**: From the **Global** view, navigate to the cluster.
- **Node Metrics**: From the **Global** view, navigate to the cluster. Select **Nodes**. Find the individual node and click on its name. Click **Node Metrics.**
- **Workload Metrics**: From the **Global** view, navigate to the project. From the main navigation bar, choose **Resources > Workloads.** (In versions before v2.3.0, choose **Workloads** on the main navigation bar.) Find the individual workload and click on its name. Click **Workload Metrics.**
- **Pod Metrics**: From the **Global** view, navigate to the project. Select **Workloads > Workloads**. Find the individual workload and click on its name. Find the individual pod and click on its name. Click **Pod Metrics.**
- **Container Metrics**: From the **Global** view, navigate to the project. From the main navigation bar, choose **Resources > Workloads.** (In versions before v2.3.0, choose **Workloads** on the main navigation bar.) Find the individual workload and click on its name. Find the individual pod and click on its name. Find the individual container and click on its name. Click **Container Metrics.**
Prometheus metrics are displayed and are denoted with the Grafana icon. If you click on the icon, the metrics will open a new tab in Grafana.
Within each Prometheus metrics widget, there are several ways to customize your view.
- Toggle between two views:
- **Detail**: Displays graphs and charts that let you view each event in a Prometheus time series
- **Summary** Displays events in a Prometheus time series that are outside the norm.
- Change the range of the time series that you're viewing to see a more refined or expansive data sample.
- Customize the data sample to display data between specific dates and times.
When analyzing these metrics, don't be concerned about any single standalone metric in the charts and graphs. Rather, you should establish a baseline for your metrics over the course of time, e.g. the range of values that your components usually operate within and are considered normal. After you establish the baseline, be on the lookout for any large deltas in the charts and graphs, as these big changes usually indicate a problem that you need to investigate.
## Grafana
If you've enabled monitoring at either the [cluster level]({{<baseurl>}}/rancher/v2.0-v2.4/en/monitoring-alerting/legacy/monitoring/cluster-monitoring/) or [project level]({{<baseurl>}}/rancher/v2.0-v2.4/en/project-admin/tools/monitoring/), Rancher automatically creates a link to Grafana instance. Use this link to view monitoring data.
Grafana allows you to query, visualize, alert, and ultimately, understand your cluster and workload data. For more information on Grafana and its capabilities, visit the [Grafana website](https://grafana.com/grafana).
### Authentication
Rancher determines which users can access the new Grafana instance, as well as the objects they can view within it, by validating them against the user's [cluster or project roles]({{<baseurl>}}/rancher/v2.0-v2.4/en/admin-settings/rbac/cluster-project-roles/). In other words, a user's access in Grafana mirrors their access in Rancher.
When you go to the Grafana instance, you will be logged in with the username `admin` and the password `admin`. If you log out and log in again, you will be prompted to change your password. You will only have access to the URL of the Grafana instance if you have access to view the corresponding metrics in Rancher. So for example, if your Rancher permissions are scoped to the project level, you won't be able to see the Grafana instance for cluster-level metrics.
### Accessing the Cluster-level Grafana Instance
1. From the **Global** view, navigate to a cluster that has monitoring enabled.
1. Go to the **System** project view. This project is where the cluster-level Grafana instance runs.
1. Click **Apps.** In versions before v2.2.0, choose **Catalog Apps** on the main navigation bar.
1. Go to the `cluster-monitoring` application.
1. In the `cluster-monitoring` application, there are two `/index.html` links: one that leads to a Grafana instance and one that leads to a Prometheus instance. When you click the Grafana link, it will redirect you to a new webpage for Grafana, which shows metrics for the cluster.
1. You will be signed in to the Grafana instance automatically. The default username is `admin` and the default password is `admin`. For security, we recommend that you log out of Grafana, log back in with the `admin` password, and change your password.
**Results:** You are logged into Grafana from the Grafana instance. After logging in, you can view the preset Grafana dashboards, which are imported via the [Grafana provisioning mechanism](http://docs.grafana.org/administration/provisioning/#dashboards), so you cannot modify them directly. For now, if you want to configure your own dashboards, clone the original and modify the new copy.
versioned_docs/version-2.0-2.4/cluster-admin/tools/istio/disabling-istio/disabling-istio.md
+32
View File
@@ -0,0 +1,32 @@
---
title: Disabling Istio
weight: 4
aliases:
- /rancher/v2.0-v2.4/en/cluster-admin/tools/istio/disabling-istio
- /rancher/v2.0-v2.4/en/istio/legacy/disabling-istio
- /rancher/v2.0-v2.4/en/istio/v2.3.x-v2.4.x/disabling-istio
- /rancher/v2.x/en/istio/v2.3.x-v2.4.x/disabling-istio/
---
This section describes how to disable Istio in a cluster, namespace, or workload.
# Disable Istio in a Cluster
To disable Istio,
1. From the **Global** view, navigate to the cluster that you want to disable Istio for.
1. Click **Tools > Istio.**
1. Click **Disable,** then click the red button again to confirm the disable action.
**Result:** The `cluster-istio` application in the cluster's `system` project gets removed. The Istio sidecar cannot be deployed on any workloads in the cluster.
# Disable Istio in a Namespace
1. In the Rancher UI, go to the project that has the namespace where you want to disable Istio.
1. On the **Workloads** tab, you will see a list of namespaces and the workloads deployed in them. Go to the namespace where you want to disable and click the **&#8942; > Disable Istio Auto Injection.**
**Result:** When workloads are deployed in this namespace, they will not have the Istio sidecar.
# Remove the Istio Sidecar from a Workload
Disable Istio in the namespace, then redeploy the workloads with in it. They will be deployed without the Istio sidecar.
versioned_docs/version-2.0-2.4/cluster-admin/tools/istio/istio.md
+93
View File
@@ -0,0 +1,93 @@
---
title: Istio
weight: 15
aliases:
- /rancher/v2.0-v2.4/en/dashboard/istio
- /rancher/v2.0-v2.4/en/project-admin/istio/configuring-resource-allocations/
- /rancher/v2.0-v2.4/en/cluster-admin/tools/istio/
- /rancher/v2.0-v2.4/en/project-admin/istio
- /rancher/v2.0-v2.4/en/istio/legacy/cluster-istio
- /rancher/v2.x/en/istio/v2.3.x-v2.4.x/
---
_Available as of v2.3.0_
[Istio](https://istio.io/) is an open-source tool that makes it easier for DevOps teams to observe, control, troubleshoot, and secure the traffic within a complex network of microservices.
As a network of microservices changes and grows, the interactions between them can become more difficult to manage and understand. In such a situation, it is useful to have a service mesh as a separate infrastructure layer. Istio's service mesh lets you manipulate traffic between microservices without changing the microservices directly.
Our integration of Istio is designed so that a Rancher operator, such as an administrator or cluster owner, can deliver Istio to developers. Then developers can use Istio to enforce security policies, troubleshoot problems, or manage traffic for green/blue deployments, canary deployments, or A/B testing.
This service mesh provides features that include but are not limited to the following:
- Traffic management features
- Enhanced monitoring and tracing
- Service discovery and routing
- Secure connections and service-to-service authentication with mutual TLS
- Load balancing
- Automatic retries, backoff, and circuit breaking
After Istio is enabled in a cluster, you can leverage Istio's control plane functionality with `kubectl`.
Rancher's Istio integration comes with comprehensive visualization aids:
- **Trace the root cause of errors with Jaeger.** [Jaeger](https://www.jaegertracing.io/) is an open-source tool that provides a UI for a distributed tracing system, which is useful for root cause analysis and for determining what causes poor performance. Distributed tracing allows you to view an entire chain of calls, which might originate with a user request and traverse dozens of microservices.
- **Get the full picture of your microservice architecture with Kiali.** [Kiali](https://www.kiali.io/) provides a diagram that shows the services within a service mesh and how they are connected, including the traffic rates and latencies between them. You can check the health of the service mesh, or drill down to see the incoming and outgoing requests to a single component.
- **Gain insights from time series analytics with Grafana dashboards.** [Grafana](https://grafana.com/) is an analytics platform that allows you to query, visualize, alert on and understand the data gathered by Prometheus.
- **Write custom queries for time series data with the Prometheus UI.** [Prometheus](https://prometheus.io/) is a systems monitoring and alerting toolkit. Prometheus scrapes data from your cluster, which is then used by Grafana. A Prometheus UI is also integrated into Rancher, and lets you write custom queries for time series data and see the results in the UI.
Istio needs to be set up by a Rancher administrator or cluster administrator before it can be used in a project.
# Prerequisites
Before enabling Istio, we recommend that you confirm that your Rancher worker nodes have enough [CPU and memory]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/tools/istio/resources) to run all of the components of Istio.
# Setup Guide
Refer to the [setup guide]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/tools/istio/setup) for instructions on how to set up Istio and use it in a project.
# Disabling Istio
To remove Istio components from a cluster, namespace, or workload, refer to the section on [disabling Istio.]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/tools/istio/disabling-istio)
# Accessing Visualizations
> By default, only cluster owners have access to Jaeger and Kiali. For instructions on how to allow project members to access them, see [this section.]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/tools/istio/rbac/)
After Istio is set up in a cluster, Grafana, Prometheus, Jaeger, and Kiali are available in the Rancher UI.
Your access to the visualizations depend on your role. Grafana and Prometheus are only available for cluster owners. The Kiali and Jaeger UIs are available only to cluster owners by default, but cluster owners can allow project members to access them by editing the Istio settings. When you go to your project and click **Resources > Istio,** you can go to each UI for Kiali, Jaeger, Grafana, and Prometheus by clicking their icons in the top right corner of the page.
To see the visualizations, go to the cluster where Istio is set up and click **Tools > Istio.** You should see links to each UI at the top of the page.
You can also get to the visualization tools from the project view.
# Viewing the Kiali Traffic Graph
1. From the project view in Rancher, click **Resources > Istio.**
1. If you are a cluster owner, you can go to the **Traffic Graph** tab. This tab has the Kiali network visualization integrated into the UI.
# Viewing Traffic Metrics
Istio’s monitoring features provide visibility into the performance of all your services.
1. From the project view in Rancher, click **Resources > Istio.**
1. Go to the **Traffic Metrics** tab. After traffic is generated in your cluster, you should be able to see metrics for **Success Rate, Request Volume, 4xx Response Count, Project 5xx Response Count** and **Request Duration.** Cluster owners can see all of the metrics, while project members can see a subset of the metrics.
# Architecture
Istio installs a service mesh that uses [Envoy](https://www.envoyproxy.io/learn/service-mesh) sidecar proxies to intercept traffic to each workload. These sidecars intercept and manage service-to-service communication, allowing fine-grained observation and control over traffic within the cluster.
Only workloads that have the Istio sidecar injected can be tracked and controlled by Istio.
Enabling Istio in Rancher enables monitoring in the cluster, and enables Istio in all new namespaces that are created in a cluster. You need to manually enable Istio in preexisting namespaces.
When a namespace has Istio enabled, new workloads deployed in the namespace will automatically have the Istio sidecar. You need to manually enable Istio in preexisting workloads.
For more information on the Istio sidecar, refer to the [Istio docs](https://istio.io/docs/setup/kubernetes/additional-setup/sidecar-injection/).
### Two Ingresses
By default, each Rancher-provisioned cluster has one NGINX ingress controller allowing traffic into the cluster. To allow Istio to receive external traffic, you need to enable the Istio ingress gateway for the cluster. The result is that your cluster will have two ingresses.
![In an Istio-enabled cluster, you can have two ingresses: the default Nginx ingress, and the default Istio controller.]({{<baseurl>}}/img/rancher/istio-ingress.svg)
versioned_docs/version-2.0-2.4/cluster-admin/tools/istio/rbac/rbac.md
+63
View File
@@ -0,0 +1,63 @@
---
title: Role-based Access Control
weight: 3
aliases:
- /rancher/v2.0-v2.4/en/cluster-admin/tools/istio/rbac
- /rancher/v2.0-v2.4/en/istio/legacy/rbac
- /rancher/v2.0-v2.4/en/istio/v2.3.x-v2.4.x/rbac
- /rancher/v2.x/en/istio/v2.3.x-v2.4.x/rbac/
---
This section describes the permissions required to access Istio features and how to configure access to the Kiali and Jaeger visualizations.
# Cluster-level Access
By default, only cluster administrators can:
- Enable Istio for the cluster
- Configure resource allocations for Istio
- View each UI for Prometheus, Grafana, Kiali, and Jaeger
# Project-level Access
After Istio is enabled in a cluster, project owners and members have permission to:
- Enable and disable Istio sidecar auto-injection for namespaces
- Add the Istio sidecar to workloads
- View the traffic metrics and traffic graph for the cluster
- View the Kiali and Jaeger visualizations if cluster administrators give access to project members
- Configure Istio's resources (such as the gateway, destination rules, or virtual services) with `kubectl` (This does not apply to read-only project members)
# Access to Visualizations
By default, the Kiali and Jaeger visualizations are restricted to the cluster owner because the information in them could be sensitive.
**Jaeger** provides a UI for a distributed tracing system, which is useful for root cause analysis and for determining what causes poor performance.
**Kiali** provides a diagram that shows the services within a service mesh and how they are connected.
Rancher supports giving groups permission to access Kiali and Jaeger, but not individuals.
To configure who has permission to access the Kiali and Jaeger UI,
1. Go to the cluster view and click **Tools > Istio.**
1. Then go to the **Member Access** section. If you want to restrict access to certain groups, choose **Allow cluster owner and specified members to access Kiali and Jaeger UI.** Search for the groups that you want to have access to Kiali and Jaeger. If you want all members to have access to the tools, click **Allow all members to access Kiali and Jaeger UI.**
1. Click **Save.**
**Result:** The access levels for Kiali and Jaeger have been updated.
# Summary of Default Permissions for Istio Users
| Permission | Cluster Administrators | Project Owners | Project Members | Read-only Project Members |
|------------------------------------------|----------------|----------------|-----------------|---------------------------|
| Enable and disable Istio for the cluster | ✓ | | | |
| Configure Istio resource limits | ✓ | | | |
| Control who has access to Kiali and the Jaeger UI | ✓ | | | |
| Enable and disable Istio for a namespace | ✓ | ✓ | ✓ | |
| Enable and disable Istio on workloads | ✓ | ✓ | ✓ | |
| Configure Istio with `kubectl` | ✓ | ✓ | ✓ | |
| View Prometheus UI and Grafana UI | ✓ | | | |
| View Kiali UI and Jaeger UI ([Configurable](#access-to-visualizations)) | ✓ | | | |
| View Istio project dashboard, including traffic metrics* | ✓ | ✓ | ✓ | ✓ |
* By default, only the cluster owner will see the traffic graph. Project members will see only a subset of traffic metrics. Project members cannot see the traffic graph because it comes from Kiali, and access to Kiali is restricted to cluster owners by default.
versioned_docs/version-2.0-2.4/cluster-admin/tools/istio/release-notes/release-notes.md
+23
View File
@@ -0,0 +1,23 @@
---
title: Release Notes
aliases:
- /rancher/v2.0-v2.4/en/cluster-admin/tools/istio/release-notes
- /rancher/v2.0-v2.4/en/istio/legacy/release-notes
- /rancher/v2.0-v2.4/en/istio/v2.3.x-v2.4.x/release-notes
- /rancher/v2.x/en/istio/v2.3.x-v2.4.x/release-notes/
---
# Istio 1.5.8
### Important note on 1.5.x versions
When upgrading from any 1.4 version of Istio to any 1.5 version, the Rancher installer will delete several resources in order to complete the upgrade, at which point they will be immediately re-installed. This includes the `istio-reader-service-account`. If your Istio installation is using this service account be aware that any secrets tied to the service account will be deleted. Most notably this will **break specific [multi-cluster deployments](https://archive.istio.io/v1.4/docs/setup/install/multicluster/)**. Downgrades back to 1.4 are not possible.
See the official upgrade notes for additional information on the 1.5 release and upgrading from 1.4: https://istio.io/latest/news/releases/1.5.x/announcing-1.5/upgrade-notes/
> **Note:** Rancher continues to use the Helm installation method, which produces a different architecture from an istioctl installation.
### Known Issues
* The Kiali traffic graph is currently not working [#24924](https://github.com/istio/istio/issues/24924)
versioned_docs/version-2.0-2.4/cluster-admin/tools/istio/resources/resources.md
+156
View File
@@ -0,0 +1,156 @@
---
title: CPU and Memory Allocations
weight: 1
aliases:
- /rancher/v2.0-v2.4/en/project-admin/istio/configuring-resource-allocations/
- /rancher/v2.0-v2.4/en/project-admin/istio/config/
- /rancher/v2.0-v2.4/en/cluster-admin/tools/istio/resources
- /rancher/v2.0-v2.4/en/istio/legacy/resources
- /rancher/v2.0-v2.4/en/istio/v2.3.x-v2.4.x/resources
- /rancher/v2.x/en/istio/v2.3.x-v2.4.x/resources/
---
_Available as of v2.3.0_
This section describes the minimum recommended computing resources for the Istio components in a cluster.
The CPU and memory allocations for each component are [configurable.](#configuring-resource-allocations)
Before enabling Istio, we recommend that you confirm that your Rancher worker nodes have enough CPU and memory to run all of the components of Istio.
> **Tip:** In larger deployments, it is strongly advised that the infrastructure be placed on dedicated nodes in the cluster by adding a node selector for each Istio component.
The table below shows a summary of the minimum recommended resource requests and limits for the CPU and memory of each central Istio component.
In Kubernetes, the resource request indicates that the workload will not deployed on a node unless the node has at least the specified amount of memory and CPU available. If the workload surpasses the limit for CPU or memory, it can be terminated or evicted from the node. For more information on managing resource limits for containers, refer to the [Kubernetes documentation.](https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/)
Workload | Container | CPU - Request | Mem - Request | CPU - Limit | Mem - Limit | Configurable
---------|-----------|---------------|---------------|-------------|-------------|-------------
istio-pilot |discovery| 500m | 2048Mi | 1000m | 4096Mi | Y
istio-telemetry |mixer| 1000m | 1024Mi | 4800m | 4096Mi | Y
istio-policy | mixer | 1000m | 1024Mi | 4800m | 4096Mi | Y
istio-tracing | jaeger | 100m | 100Mi | 500m | 1024Mi | Y
prometheus | prometheus | 750m | 750Mi | 1000m | 1024Mi | Y
grafana | grafana | 100m | 100Mi | 200m | 512Mi | Y
Others | - | 500m | 500Mi | - | - | N
**Total** | **-** | **3950m** | **5546Mi** | **>12300m** | **>14848Mi** | **-**
# Configuring Resource Allocations
You can individually configure the resource allocation for each type of Istio component. This section includes the default resource allocations for each component.
To make it easier to schedule the workloads to a node, a cluster administrator can reduce the CPU and memory resource requests for the component. However, the default CPU and memory allocations are the minimum that we recommend.
You can find more information about Istio configuration in the [official Istio documentation](https://istio.io/docs/concepts/what-is-istio).
To configure the resources allocated to an Istio component,
1. In Rancher, go to the cluster where you have Istio installed.
1. Click **Tools > Istio.** This opens the Istio configuration page.
1. Change the CPU or memory allocations, the nodes where each component will be scheduled to, or the node tolerations.
1. Click **Save.**
**Result:** The resource allocations for the Istio components are updated.
## Pilot
[Pilot](https://istio.io/docs/ops/deployment/architecture/#pilot) provides the following:
- Authentication configuration
- Service discovery for the Envoy sidecars
- Traffic management capabilities for intelligent routing (A/B tests and canary rollouts)
- Configuration for resiliency (timeouts, retries, circuit breakers, etc)
For more information on Pilot, refer to the [documentation](https://istio.io/docs/concepts/traffic-management/#pilot-and-envoy).
Option | Description| Required | Default
-------|------------|-------|-------
Pilot CPU Limit | CPU resource limit for the istio-pilot pod.| Yes | 1000
Pilot CPU Reservation | CPU reservation for the istio-pilot pod. | Yes | 500
Pilot Memory Limit | Memory resource limit for the istio-pilot pod. | Yes | 4096
Pilot Memory Reservation | Memory resource requests for the istio-pilot pod. | Yes | 2048
Trace sampling Percentage | [Trace sampling percentage](https://istio.io/docs/tasks/telemetry/distributed-tracing/overview/#trace-sampling) | Yes | 1
Pilot Selector | Ability to select the nodes in which istio-pilot pod is deployed to. To use this option, the nodes must have labels. | No | n/a
## Mixer
[Mixer](https://istio.io/docs/ops/deployment/architecture/#mixer) enforces access control and usage policies across the service mesh. It also integrates with plugins for monitoring tools such as Prometheus. The Envoy sidecar proxy passes telemetry data and monitoring data to Mixer, and Mixer passes the monitoring data to Prometheus.
For more information on Mixer, policies and telemetry, refer to the [documentation](https://istio.io/docs/concepts/policies-and-telemetry/).
Option | Description| Required | Default
-------|------------|-------|-------
Mixer Telemetry CPU Limit | CPU resource limit for the istio-telemetry pod.| Yes | 4800
Mixer Telemetry CPU Reservation | CPU reservation for the istio-telemetry pod.| Yes | 1000
Mixer Telemetry Memory Limit | Memory resource limit for the istio-telemetry pod.| Yes | 4096
Mixer Telemetry Memory Reservation | Memory resource requests for the istio-telemetry pod.| Yes | 1024
Enable Mixer Policy | Whether or not to deploy the istio-policy. | Yes | False
Mixer Policy CPU Limit | CPU resource limit for the istio-policy pod. | Yes, when policy enabled | 4800
Mixer Policy CPU Reservation | CPU reservation for the istio-policy pod. | Yes, when policy enabled | 1000
Mixer Policy Memory Limit | Memory resource limit for the istio-policy pod. | Yes, when policy enabled | 4096
Mixer Policy Memory Reservation | Memory resource requests for the istio-policy pod. | Yes, when policy enabled | 1024
Mixer Selector | Ability to select the nodes in which istio-policy and istio-telemetry pods are deployed to. To use this option, the nodes must have labels. | No | n/a
## Tracing
[Distributed tracing](https://istio.io/docs/tasks/telemetry/distributed-tracing/overview/) enables users to track a request through a service mesh. This makes it easier to troubleshoot problems with latency, parallelism and serialization.
Option | Description| Required | Default
-------|------------|-------|-------
Enable Tracing | Whether or not to deploy the istio-tracing. | Yes | True
Tracing CPU Limit | CPU resource limit for the istio-tracing pod. | Yes | 500
Tracing CPU Reservation | CPU reservation for the istio-tracing pod. | Yes | 100
Tracing Memory Limit | Memory resource limit for the istio-tracing pod. | Yes | 1024
Tracing Memory Reservation | Memory resource requests for the istio-tracing pod. | Yes | 100
Tracing Selector | Ability to select the nodes in which tracing pod is deployed to. To use this option, the nodes must have labels. | No | n/a
## Ingress Gateway
The Istio gateway allows Istio features such as monitoring and route rules to be applied to traffic entering the cluster. This gateway is a prerequisite for outside traffic to make requests to Istio.
For more information, refer to the [documentation](https://istio.io/docs/tasks/traffic-management/ingress/).
Option | Description| Required | Default
-------|------------|-------|-------
Enable Ingress Gateway | Whether or not to deploy the istio-ingressgateway. | Yes | False
Service Type of Istio Ingress Gateway | How to expose the gateway. You can choose NodePort or Loadbalancer | Yes | NodePort
Http2 Port | The NodePort for http2 requests | Yes | 31380
Https Port | The NodePort for https requests | Yes | 31390
Load Balancer IP | Ingress Gateway Load Balancer IP | No | n/a
Load Balancer Source Ranges | Ingress Gateway Load Balancer Source Ranges | No | n/a
Ingress Gateway CPU Limit | CPU resource limit for the istio-ingressgateway pod. | Yes | 2000
Ingress Gateway CPU Reservation | CPU reservation for the istio-ingressgateway pod. | Yes | 100
Ingress Gateway Memory Limit | Memory resource limit for the istio-ingressgateway pod. | Yes | 1024
Ingress Gateway Memory Reservation | Memory resource requests for the istio-ingressgateway pod. | Yes | 128
Ingress Gateway Selector | Ability to select the nodes in which istio-ingressgateway pod is deployed to. To use this option, the nodes must have labels. | No | n/a
## Prometheus
You can query for Istio metrics using Prometheus. Prometheus is an open-source systems monitoring and alerting toolkit.
Option | Description| Required | Default
-------|------------|-------|-------
Prometheus CPU Limit | CPU resource limit for the Prometheus pod.| Yes | 1000
Prometheus CPU Reservation | CPU reservation for the Prometheus pod.| Yes | 750
Prometheus Memory Limit | Memory resource limit for the Prometheus pod.| Yes | 1024
Prometheus Memory Reservation | Memory resource requests for the Prometheus pod.| Yes | 750
Retention for Prometheus | How long your Prometheus instance retains data | Yes | 6
Prometheus Selector | Ability to select the nodes in which Prometheus pod is deployed to. To use this option, the nodes must have labels.| No | n/a
## Grafana
You can visualize metrics with Grafana. Grafana lets you visualize Istio traffic data scraped by Prometheus.
Option | Description| Required | Default
-------|------------|-------|-------
Enable Grafana | Whether or not to deploy the Grafana.| Yes | True
Grafana CPU Limit | CPU resource limit for the Grafana pod.| Yes, when Grafana enabled | 200
Grafana CPU Reservation | CPU reservation for the Grafana pod.| Yes, when Grafana enabled | 100
Grafana Memory Limit | Memory resource limit for the Grafana pod.| Yes, when Grafana enabled | 512
Grafana Memory Reservation | Memory resource requests for the Grafana pod.| Yes, when Grafana enabled | 100
Grafana Selector | Ability to select the nodes in which Grafana pod is deployed to. To use this option, the nodes must have labels. | No | n/a
Enable Persistent Storage for Grafana | Enable Persistent Storage for Grafana | Yes, when Grafana enabled | False
Source | Use a Storage Class to provision a new persistent volume or Use an existing persistent volume claim | Yes, when Grafana enabled and enabled PV | Use SC
Storage Class | Storage Class for provisioning PV for Grafana | Yes, when Grafana enabled, enabled PV and use storage class | Use the default class
Persistent Volume Size | The size for the PV you would like to provision for Grafana | Yes, when Grafana enabled, enabled PV and use storage class | 5Gi
Existing Claim | Use existing PVC for Grafana | Yes, when Grafana enabled, enabled PV and use existing PVC | n/a
versioned_docs/version-2.0-2.4/cluster-admin/tools/istio/setup/deploy-workloads/deploy-workloads.md
+327
View File
@@ -0,0 +1,327 @@
---
title: 4. Add Deployments and Services with the Istio Sidecar
weight: 4
aliases:
- /rancher/v2.0-v2.4/en/cluster-admin/tools/istio/setup/deploy-workloads
- /rancher/v2.0-v2.4/en/istio/legacy/setup/deploy-workloads
- /rancher/v2.0-v2.4/en/istio/v2.3.x-v2.4.x/setup/deploy-workloads
- /rancher/v2.x/en/istio/v2.3.x-v2.4.x/setup/deploy-workloads/
---
> **Prerequisite:** To enable Istio for a workload, the cluster and namespace must have Istio enabled.
Enabling Istio in a namespace only enables automatic sidecar injection for new workloads. To enable the Envoy sidecar for existing workloads, you need to enable it manually for each workload.
To inject the Istio sidecar on an existing workload in the namespace, go to the workload, click the **&#8942;,** and click **Redeploy.** When the workload is redeployed, it will have the Envoy sidecar automatically injected.
Wait a few minutes for the workload to upgrade to have the istio sidecar. Click it and go to the Containers section. You should be able to see istio-init and istio-proxy alongside your original workload. This means the Istio sidecar is enabled for the workload. Istio is doing all the wiring for the sidecar envoy. Now Istio can do all the features automatically if you enable them in the yaml.
### 3. Add Deployments and Services
Next we add the Kubernetes resources for the sample deployments and services for the BookInfo app in Istio's documentation.
1. Go to the project inside the cluster you want to deploy the workload on.
1. In Workloads, click **Import YAML.**
1. Copy the below resources into the form.
1. Click **Import.**
This will set up the following sample resources from Istio's example BookInfo app:
Details service and deployment:
- A `details` Service
- A ServiceAccount for `bookinfo-details`
- A `details-v1` Deployment
Ratings service and deployment:
- A `ratings` Service
- A ServiceAccount for `bookinfo-ratings`
- A `ratings-v1` Deployment
Reviews service and deployments (three versions):
- A `reviews` Service
- A ServiceAccount for `bookinfo-reviews`
- A `reviews-v1` Deployment
- A `reviews-v2` Deployment
- A `reviews-v3` Deployment
Productpage service and deployment:
This is the main page of the app, which will be visible from a web browser. The other services will be called from this page.
- A `productpage` service
- A ServiceAccount for `bookinfo-productpage`
- A `productpage-v1` Deployment
### Resource YAML
```yaml
# Copyright 2017 Istio Authors
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
##################################################################################################
# Details service
##################################################################################################
apiVersion: v1
kind: Service
metadata:
name: details
labels:
app: details
service: details
spec:
ports:
- port: 9080
name: http
selector:
app: details
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: bookinfo-details
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: details-v1
labels:
app: details
version: v1
spec:
replicas: 1
selector:
matchLabels:
app: details
version: v1
template:
metadata:
labels:
app: details
version: v1
spec:
serviceAccountName: bookinfo-details
containers:
- name: details
image: docker.io/istio/examples-bookinfo-details-v1:1.15.0
imagePullPolicy: IfNotPresent
ports:
- containerPort: 9080
---
##################################################################################################
# Ratings service
##################################################################################################
apiVersion: v1
kind: Service
metadata:
name: ratings
labels:
app: ratings
service: ratings
spec:
ports:
- port: 9080
name: http
selector:
app: ratings
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: bookinfo-ratings
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: ratings-v1
labels:
app: ratings
version: v1
spec:
replicas: 1
selector:
matchLabels:
app: ratings
version: v1
template:
metadata:
labels:
app: ratings
version: v1
spec:
serviceAccountName: bookinfo-ratings
containers:
- name: ratings
image: docker.io/istio/examples-bookinfo-ratings-v1:1.15.0
imagePullPolicy: IfNotPresent
ports:
- containerPort: 9080
---
##################################################################################################
# Reviews service
##################################################################################################
apiVersion: v1
kind: Service
metadata:
name: reviews
labels:
app: reviews
service: reviews
spec:
ports:
- port: 9080
name: http
selector:
app: reviews
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: bookinfo-reviews
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: reviews-v1
labels:
app: reviews
version: v1
spec:
replicas: 1
selector:
matchLabels:
app: reviews
version: v1
template:
metadata:
labels:
app: reviews
version: v1
spec:
serviceAccountName: bookinfo-reviews
containers:
- name: reviews
image: docker.io/istio/examples-bookinfo-reviews-v1:1.15.0
imagePullPolicy: IfNotPresent
ports:
- containerPort: 9080
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: reviews-v2
labels:
app: reviews
version: v2
spec:
replicas: 1
selector:
matchLabels:
app: reviews
version: v2
template:
metadata:
labels:
app: reviews
version: v2
spec:
serviceAccountName: bookinfo-reviews
containers:
- name: reviews
image: docker.io/istio/examples-bookinfo-reviews-v2:1.15.0
imagePullPolicy: IfNotPresent
ports:
- containerPort: 9080
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: reviews-v3
labels:
app: reviews
version: v3
spec:
replicas: 1
selector:
matchLabels:
app: reviews
version: v3
template:
metadata:
labels:
app: reviews
version: v3
spec:
serviceAccountName: bookinfo-reviews
containers:
- name: reviews
image: docker.io/istio/examples-bookinfo-reviews-v3:1.15.0
imagePullPolicy: IfNotPresent
ports:
- containerPort: 9080
---
##################################################################################################
# Productpage services
##################################################################################################
apiVersion: v1
kind: Service
metadata:
name: productpage
labels:
app: productpage
service: productpage
spec:
ports:
- port: 9080
name: http
selector:
app: productpage
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: bookinfo-productpage
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: productpage-v1
labels:
app: productpage
version: v1
spec:
replicas: 1
selector:
matchLabels:
app: productpage
version: v1
template:
metadata:
labels:
app: productpage
version: v1
spec:
serviceAccountName: bookinfo-productpage
containers:
- name: productpage
image: docker.io/istio/examples-bookinfo-productpage-v1:1.15.0
imagePullPolicy: IfNotPresent
ports:
- containerPort: 9080
---
```
### [Next: Set up the Istio Gateway]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/tools/istio/setup/gateway)
versioned_docs/version-2.0-2.4/cluster-admin/tools/istio/setup/enable-istio-in-cluster/enable-istio-in-cluster.md
+39
View File
@@ -0,0 +1,39 @@
---
title: 1. Enable Istio in the Cluster
weight: 1
aliases:
- /rancher/v2.0-v2.4/en/cluster-admin/tools/istio/setup/enable-istio-in-cluster
- /rancher/v2.0-v2.4/en/istio/legacy/setup/enable-istio-in-cluster
- /rancher/v2.0-v2.4/en/istio/v2.3.x-v2.4.x/setup/enable-istio-in-cluster
- /rancher/v2.x/en/istio/v2.3.x-v2.4.x/setup/enable-istio-in-cluster/
---
This cluster uses the default Nginx controller to allow traffic into the cluster.
A Rancher [administrator]({{<baseurl>}}/rancher/v2.0-v2.4/en/admin-settings/rbac/global-permissions/) or [cluster owner]({{<baseurl>}}/rancher/v2.0-v2.4/en/admin-settings/rbac/cluster-project-roles/#cluster-roles) can configure Rancher to deploy Istio in a Kubernetes cluster.
# Prerequisites
This guide assumes you have already [installed Rancher,]({{<baseurl>}}/rancher/v2.0-v2.4/en/installation) and you have already [provisioned a separate Kubernetes cluster]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-provisioning) on which you will install Istio.
The nodes in your cluster must meet the [CPU and memory requirements.]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/tools/istio/resources/)
The workloads and services that you want to be controlled by Istio must meet [Istio's requirements.](https://istio.io/docs/setup/additional-setup/requirements/)
> If the cluster has a Pod Security Policy enabled there are [additional prerequisites steps]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/tools/istio/setup/enable-istio-in-cluster/enable-istio-with-psp/)
# Enable Istio in the Cluster
1. From the **Global** view, navigate to the **cluster** where you want to enable Istio.
1. Click **Tools > Istio.**
1. Optional: Configure member access and [resource limits]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/tools/istio/resources/) for the Istio components. Ensure you have enough resources on your worker nodes to enable Istio.
1. Click **Enable**.
1. Click **Save**.
**Result:** Istio is enabled at the cluster level.
The Istio application, `cluster-istio`, is added as an application to the cluster's `system` project.
When Istio is enabled in the cluster, the label for Istio sidecar auto injection,`istio-injection=enabled`, will be automatically added to each new namespace in this cluster. This automatically enables Istio sidecar injection in all new workloads that are deployed in those namespaces. You will need to manually enable Istio in preexisting namespaces and workloads.
### [Next: Enable Istio in a Namespace]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/tools/istio/setup/enable-istio-in-namespace)
versioned_docs/version-2.0-2.4/cluster-admin/tools/istio/setup/enable-istio-in-cluster/enable-istio-with-psp/enable-istio-with-psp.md
+53
View File
@@ -0,0 +1,53 @@
---
title: Enable Istio with Pod Security Policies
aliases:
- /rancher/v2.0-v2.4/en/cluster-admin/tools/istio/setup/enable-istio-in-cluster/enable-istio-with-psp
- /rancher/v2.0-v2.4/en/istio/legacy/setup/enable-istio-in-cluster/enable-istio-with-psp
- /rancher/v2.0-v2.4/en/istio/v2.3.x-v2.4.x/setup/enable-istio-in-cluster/enable-istio-with-psp
- /rancher/v2.x/en/istio/v2.3.x-v2.4.x/setup/enable-istio-in-cluster/enable-istio-with-psp/
---
>**Note:** The following guide is only for RKE provisioned clusters.
If you have restrictive Pod Security Policies enabled, then Istio may not be able to function correctly, because it needs certain permissions in order to install itself and manage pod infrastructure. In this section, we will configure a cluster with PSPs enabled for an Istio install, and also set up the Istio CNI plugin.
The Istio CNI plugin removes the need for each application pod to have a privileged `NET_ADMIN` container. For further information, see the [Istio CNI Plugin docs](https://istio.io/docs/setup/additional-setup/cni). Please note that the [Istio CNI Plugin is in alpha](https://istio.io/about/feature-stages/).
- 1. [Configure the System Project Policy to allow Istio install.](#1-configure-the-system-project-policy-to-allow-istio-install)
- 2. [Install the CNI plugin in the System project.](#2-install-the-cni-plugin-in-the-system-project)
- 3. [Install Istio.](#3-install-istio)
### 1. Configure the System Project Policy to allow Istio install
1. From the main menu of the **Dashboard**, select **Projects/Namespaces**.
1. Find the **Project: System** project and select the **&#8942; > Edit**.
1. Change the Pod Security Policy option to be unrestricted, then click Save.
### 2. Install the CNI Plugin in the System Project
1. From the main menu of the **Dashboard**, select **Projects/Namespaces**.
1. Select the **Project: System** project.
1. Choose **Tools > Catalogs** in the navigation bar.
1. Add a catalog with the following:
1. Name: istio-cni
1. Catalog URL: https://github.com/istio/cni
1. Branch: The branch that matches your current release, for example: `release-1.4`.
1. From the main menu select **Apps**
1. Click Launch and select istio-cni
1. Update the namespace to be "kube-system"
1. In the answers section, click "Edit as YAML" and paste in the following, then click launch:
```
---
logLevel: "info"
excludeNamespaces:
- "istio-system"
- "kube-system"
```
### 3. Install Istio
Follow the [primary instructions]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/tools/istio/setup/enable-istio-in-cluster/), adding a custom answer: `istio_cni.enabled: true`.
After Istio has finished installing, the Apps page in System Projects should show both istio and `istio-cni` applications deployed successfully. Sidecar injection will now be functional.
versioned_docs/version-2.0-2.4/cluster-admin/tools/istio/setup/enable-istio-in-namespace/enable-istio-in-namespace.md
+53
View File
@@ -0,0 +1,53 @@
---
title: 2. Enable Istio in a Namespace
weight: 2
aliases:
- /rancher/v2.0-v2.4/en/cluster-admin/tools/istio/setup/enable-istio-in-namespace
- /rancher/v2.0-v2.4/en/istio/legacy/setup/enable-istio-in-namespace
- /rancher/v2.0-v2.4/en/istio/v2.3.x-v2.4.x/setup/enable-istio-in-namespace
- /rancher/v2.x/en/istio/v2.3.x-v2.4.x/setup/enable-istio-in-namespace/
---
You will need to manually enable Istio in each namespace that you want to be tracked or controlled by Istio. When Istio is enabled in a namespace, the Envoy sidecar proxy will be automatically injected into all new workloads that are deployed in the namespace.
This namespace setting will only affect new workloads in the namespace. Any preexisting workloads will need to be re-deployed to leverage the sidecar auto injection.
> **Prerequisite:** To enable Istio in a namespace, the cluster must have Istio enabled.
1. In the Rancher UI, go to the cluster view. Click the **Projects/Namespaces** tab.
1. Go to the namespace where you want to enable the Istio sidecar auto injection and click the **&#8942;.**
1. Click **Edit.**
1. In the **Istio sidecar auto injection** section, click **Enable.**
1. Click **Save.**
**Result:** The namespace now has the label `istio-injection=enabled`. All new workloads deployed in this namespace will have the Istio sidecar injected by default.
### Verifying that Automatic Istio Sidecar Injection is Enabled
To verify that Istio is enabled, deploy a hello-world workload in the namespace. Go to the workload and click the pod name. In the **Containers** section, you should see the `istio-proxy` container.
### Excluding Workloads from Being Injected with the Istio Sidecar
If you need to exclude a workload from getting injected with the Istio sidecar, use the following annotation on the workload:
```
sidecar.istio.io/inject: “false”
```
To add the annotation to a workload,
1. From the **Global** view, open the project that has the workload that should not have the sidecar.
1. Click **Resources > Workloads.**
1. Go to the workload that should not have the sidecar and click **&#8942; > Edit.**
1. Click **Show Advanced Options.** Then expand the **Labels & Annotations** section.
1. Click **Add Annotation.**
1. In the **Key** field, enter `sidecar.istio.io/inject`.
1. In the **Value** field, enter `false`.
1. Click **Save.**
**Result:** The Istio sidecar will not be injected into the workload.
> **NOTE:** If you are having issues with a Job you deployed not completing, you will need to add this annotation to your pod using the provided steps. Since Istio Sidecars run indefinitely, a Job cannot be considered complete even after its task has completed.
### [Next: Select the Nodes ]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/tools/istio/setup/node-selectors)
versioned_docs/version-2.0-2.4/cluster-admin/tools/istio/setup/gateway/gateway.md
+135
View File
@@ -0,0 +1,135 @@
---
title: 5. Set up the Istio Gateway
weight: 5
aliases:
- /rancher/v2.0-v2.4/en/cluster-admin/tools/istio/setup/gateway
- /rancher/v2.0-v2.4/en/istio/legacy/setup/gateway
- /rancher/v2.0-v2.4/en/istio/v2.3.x-v2.4.x/setup/gateway
- /rancher/v2.x/en/istio/v2.3.x-v2.4.x/setup/gateway/
---
The gateway to each cluster can have its own port or load balancer, which is unrelated to a service mesh. By default, each Rancher-provisioned cluster has one NGINX ingress controller allowing traffic into the cluster.
You can use the NGINX ingress controller with or without Istio installed. If this is the only gateway to your cluster, Istio will be able to route traffic from service to service, but Istio will not be able to receive traffic from outside the cluster.
To allow Istio to receive external traffic, you need to enable Istio's gateway, which works as a north-south proxy for external traffic. When you enable the Istio gateway, the result is that your cluster will have two ingresses.
You will also need to set up a Kubernetes gateway for your services. This Kubernetes resource points to Istio's implementation of the ingress gateway to the cluster.
You can route traffic into the service mesh with a load balancer or just Istio's NodePort gateway. This section describes how to set up the NodePort gateway.
For more information on the Istio gateway, refer to the [Istio documentation.](https://istio.io/docs/reference/config/networking/v1alpha3/gateway/)
![In an Istio-enabled cluster, you can have two ingresses: the default Nginx ingress, and the default Istio controller.]({{<baseurl>}}/img/rancher/istio-ingress.svg)
# Enable the Istio Gateway
The ingress gateway is a Kubernetes service that will be deployed in your cluster. There is only one Istio gateway per cluster.
1. Go to the cluster where you want to allow outside traffic into Istio.
1. Click **Tools > Istio.**
1. Expand the **Ingress Gateway** section.
1. Under **Enable Ingress Gateway,** click **True.** The default type of service for the Istio gateway is NodePort. You can also configure it as a [load balancer.]({{<baseurl>}}/rancher/v2.0-v2.4/en/k8s-in-rancher/load-balancers-and-ingress/load-balancers/)
1. Optionally, configure the ports, service types, node selectors and tolerations, and resource requests and limits for this service. The default resource requests for CPU and memory are the minimum recommended resources.
1. Click **Save.**
**Result:** The gateway is deployed, which allows Istio to receive traffic from outside the cluster.
# Add a Kubernetes Gateway that Points to the Istio Gateway
To allow traffic to reach Ingress, you will also need to provide a Kubernetes gateway resource in your YAML that points to Istio's implementation of the ingress gateway to the cluster.
1. Go to the namespace where you want to deploy the Kubernetes gateway and click **Import YAML.**
1. Upload the gateway YAML as a file or paste it into the form. An example gateway YAML is provided below.
1. Click **Import.**
```yaml
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: bookinfo-gateway
spec:
selector:
istio: ingressgateway # use istio default controller
servers:
- port:
number: 80
name: http
protocol: HTTP
hosts:
- "*"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: bookinfo
spec:
hosts:
- "*"
gateways:
- bookinfo-gateway
http:
- match:
- uri:
exact: /productpage
- uri:
prefix: /static
- uri:
exact: /login
- uri:
exact: /logout
- uri:
prefix: /api/v1/products
route:
- destination:
host: productpage
port:
number: 9080
```
**Result:** You have configured your gateway resource so that Istio can receive traffic from outside the cluster.
Confirm that the resource exists by running:
```
kubectl get gateway -A
```
The result should be something like this:
```
NAME AGE
bookinfo-gateway 64m
```
### Access the ProductPage Service from a Web Browser
To test and see if the BookInfo app deployed correctly, the app can be viewed a web browser using the Istio controller IP and port, combined with the request name specified in your Kubernetes gateway resource:
`http://<IP of Istio controller>:<Port of istio controller>/productpage`
To get the ingress gateway URL and port,
1. Go to the `System` project in your cluster.
1. Within the `System` project, go to `Resources` > `Workloads` then scroll down to the `istio-system` namespace.
1. Within `istio-system`, there is a workload named `istio-ingressgateway`. Under the name of this workload, you should see links, such as `80/tcp`.
1. Click one of those links. This should show you the URL of the ingress gateway in your web browser. Append `/productpage` to the URL.
**Result:** You should see the BookInfo app in the web browser.
For help inspecting the Istio controller URL and ports, try the commands the [Istio documentation.](https://istio.io/docs/tasks/traffic-management/ingress/ingress-control/#determining-the-ingress-ip-and-ports)
# Troubleshooting
The [official Istio documentation](https://istio.io/docs/tasks/traffic-management/ingress/ingress-control/#troubleshooting) suggests `kubectl` commands to inspect the correct ingress host and ingress port for external requests.
### Confirming that the Kubernetes Gateway Matches Istio's Ingress Controller
You can try the steps in this section to make sure the Kubernetes gateway is configured properly.
In the gateway resource, the selector refers to Istio's default ingress controller by its label, in which the key of the label is `istio` and the value is `ingressgateway`. To make sure the label is appropriate for the gateway, do the following:
1. Go to the `System` project in your cluster.
1. Within the `System` project, go to the namespace `istio-system`.
1. Within `istio-system`, there is a workload named `istio-ingressgateway`.
1. Click the name of this workload and go to the **Labels and Annotations** section. You should see that it has the key `istio` and the value `ingressgateway`. This confirms that the selector in the Gateway resource matches Istio's default ingress controller.
### [Next: Set up Istio's Components for Traffic Management]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/tools/istio/setup/set-up-traffic-management)
versioned_docs/version-2.0-2.4/cluster-admin/tools/istio/setup/node-selectors/node-selectors.md
+43
View File
@@ -0,0 +1,43 @@
---
title: 3. Select the Nodes Where Istio Components Will be Deployed
weight: 3
aliases:
- /rancher/v2.0-v2.4/en/cluster-admin/tools/istio/setup/node-selectors
- /rancher/v2.0-v2.4/en/istio/legacy/setup/node-selectors
- /rancher/v2.0-v2.4/en/istio/v2.3.x-v2.4.x/setup/node-selectors
- /rancher/v2.x/en/istio/v2.3.x-v2.4.x/setup/node-selectors/
---
> **Prerequisite:** Your cluster needs a worker node that can designated for Istio. The worker node should meet the [resource requirements.]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/tools/istio/resources)
This section describes how use node selectors to configure Istio components to be deployed on a designated node.
In larger deployments, it is strongly advised that Istio's infrastructure be placed on dedicated nodes in the cluster by adding a node selector for each Istio component.
# Adding a Label to the Istio Node
First, add a label to the node where Istio components should be deployed. This label can have any key-value pair. For this example, we will use the key `istio` and the value `enabled`.
1. From the cluster view, go to the **Nodes** tab.
1. Go to a worker node that will host the Istio components and click **&#8942; > Edit.**
1. Expand the **Labels & Annotations** section.
1. Click **Add Label.**
1. In the fields that appear, enter `istio` for the key and `enabled` for the value.
1. Click **Save.**
**Result:** A worker node has the label that will allow you to designate it for Istio components.
# Configuring Istio Components to Use the Labeled Node
Configure each Istio component to be deployed to the node with the Istio label. Each Istio component can be configured individually, but in this tutorial, we will configure all of the components to be scheduled on the same node for the sake of simplicity.
For larger deployments, it is recommended to schedule each component of Istio onto separate nodes.
1. From the cluster view, click **Tools > Istio.**
1. Expand the **Pilot** section and click **Add Selector** in the form that appears. Enter the node selector label that you added to the Istio node. In our case, we are using the key `istio` and the value `enabled.`
1. Repeat the previous step for the **Mixer** and **Tracing** sections.
1. Click **Save.**
**Result:** The Istio components will be deployed on the Istio node.
### [Next: Add Deployments and Services]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/tools/istio/setup/deploy-workloads)
versioned_docs/version-2.0-2.4/cluster-admin/tools/istio/setup/set-up-traffic-management/set-up-traffic-management.md
+66
View File
@@ -0,0 +1,66 @@
---
title: 6. Set up Istio's Components for Traffic Management
weight: 6
aliases:
- /rancher/v2.0-v2.4/en/cluster-admin/tools/istio/setup/set-up-traffic-management
- /rancher/v2.0-v2.4/en/istio/legacy/setup/set-up-traffic-management
- /rancher/v2.0-v2.4/en/istio/v2.3.x-v2.4.x/setup/set-up-traffic-management
- /rancher/v2.x/en/istio/v2.3.x-v2.4.x/setup/set-up-traffic-management/
---
A central advantage of traffic management in Istio is that it allows dynamic request routing. Some common applications for dynamic request routing include canary deployments and blue/green deployments. The two key resources in Istio traffic management are *virtual services* and *destination rules*.
- [Virtual services](https://istio.io/docs/reference/config/networking/v1alpha3/virtual-service/) intercept and direct traffic to your Kubernetes services, allowing you to divide percentages of traffic from a request to different services. You can use them to define a set of routing rules to apply when a host is addressed.
- [Destination rules](https://istio.io/docs/reference/config/networking/v1alpha3/destination-rule/) serve as the single source of truth about which service versions are available to receive traffic from virtual services. You can use these resources to define policies that apply to traffic that is intended for a service after routing has occurred.
This section describes how to add an example virtual service that corresponds to the `reviews` microservice in the sample BookInfo app. The purpose of this service is to divide traffic between two versions of the `reviews` service.
In this example, we take the traffic to the `reviews` service and intercept it so that 50 percent of it goes to `v1` of the service and 50 percent goes to `v2`.
After this virtual service is deployed, we will generate traffic and see from the Kiali visualization that traffic is being routed evenly between the two versions of the service.
To deploy the virtual service and destination rules for the `reviews` service,
1. Go to the project view and click **Import YAML.**
1. Copy resources below into the form.
1. Click **Import.**
```
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: reviews
spec:
hosts:
- reviews
http:
- route:
- destination:
host: reviews
subset: v1
weight: 50
- destination:
host: reviews
subset: v3
weight: 50
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: reviews
spec:
host: reviews
subsets:
- name: v1
labels:
version: v1
- name: v2
labels:
version: v2
- name: v3
labels:
version: v3
```
**Result:** When you generate traffic to this service (for example, by refreshing the ingress gateway URL), the Kiali traffic graph will reflect that traffic to the `reviews` service is divided evenly between `v1` and `v3`.
### [Next: Generate and View Traffic]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/tools/istio/setup/view-traffic)
versioned_docs/version-2.0-2.4/cluster-admin/tools/istio/setup/setup.md
+26
View File
@@ -0,0 +1,26 @@
---
title: Setup Guide
weight: 2
aliases:
- /rancher/v2.0-v2.4/en/cluster-admin/tools/istio/setup
- /rancher/v2.0-v2.4/en/istio/legacy/setup
- /rancher/v2.0-v2.4/en/istio/v2.3.x-v2.4.x/setup
- /rancher/v2.x/en/istio/v2.3.x-v2.4.x/setup/
---
This section describes how to enable Istio and start using it in your projects.
This section assumes that you have Rancher installed, and you have a Rancher-provisioned Kubernetes cluster where you would like to set up Istio.
If you use Istio for traffic management, you will need to allow external traffic to the cluster. In that case, you will need to follow all of the steps below.
> **Quick Setup** If you don't need external traffic to reach Istio, and you just want to set up Istio for monitoring and tracing traffic within the cluster, skip the steps for [setting up the Istio gateway]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/tools/istio/setup/gateway) and [setting up Istio's components for traffic management.]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/tools/istio/setup/set-up-traffic-management)
1. [Enable Istio in the cluster.]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/tools/istio/setup/enable-istio-in-cluster)
1. [Enable Istio in all the namespaces where you want to use it.]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/tools/istio/setup/enable-istio-in-namespace)
1. [Select the nodes where the main Istio components will be deployed.]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/tools/istio/setup/node-selectors)
1. [Add deployments and services that have the Istio sidecar injected.]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/tools/istio/setup/deploy-workloads)
1. [Set up the Istio gateway. ]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/tools/istio/setup/gateway)
1. [Set up Istio's components for traffic management.]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/tools/istio/setup/set-up-traffic-management)
1. [Generate traffic and see Istio in action.]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/tools/istio/setup/#view-traffic)
versioned_docs/version-2.0-2.4/cluster-admin/tools/istio/setup/view-traffic/view-traffic.md
+31
View File
@@ -0,0 +1,31 @@
---
title: 7. Generate and View Traffic
weight: 7
aliases:
- /rancher/v2.0-v2.4/en/cluster-admin/tools/istio/setup/view-traffic
- /rancher/v2.0-v2.4/en/istio/legacy/setup/view-traffic
- /rancher/v2.0-v2.4/en/istio/v2.3.x-v2.4.x/setup/view-traffic
- /rancher/v2.x/en/istio/v2.3.x-v2.4.x/setup/view-traffic/
---
This section describes how to view the traffic that is being managed by Istio.
# The Kiali Traffic Graph
Rancher integrates a Kiali graph into the Rancher UI. The Kiali graph provides a powerful way to visualize the topology of your Istio service mesh. It shows you which services communicate with each other.
To see the traffic graph,
1. From the project view in Rancher, click **Resources > Istio.**
1. Go to the **Traffic Graph** tab. This tab has the Kiali network visualization integrated into the UI.
If you refresh the URL to the BookInfo app several times, you should be able to see green arrows on the Kiali graph showing traffic to `v1` and `v3` of the `reviews` service. The control panel on the right side of the graph lets you configure details including how many minutes of the most recent traffic should be shown on the graph.
For additional tools and visualizations, you can go to each UI for Kiali, Jaeger, Grafana, and Prometheus by clicking their icons in the top right corner of the page.
# Viewing Traffic Metrics
Istio’s monitoring features provide visibility into the performance of all your services.
1. From the project view in Rancher, click **Resources > Istio.**
1. Go to the **Traffic Metrics** tab. After traffic is generated in your cluster, you should be able to see metrics for **Success Rate, Request Volume, 4xx Response Count, Project 5xx Response Count** and **Request Duration.**
versioned_docs/version-2.0-2.4/cluster-admin/tools/notifiers/notifiers.md
+207
View File
@@ -0,0 +1,207 @@
---
title: Notifiers
weight: 4
aliases:
- /rancher/v2.0-v2.4/en/project-admin/tools/notifiers
- /rancher/v2.0-v2.4/en/cluster-admin/tools/notifiers
- /rancher/v2.0-v2.4/en/monitoring-alerting/legacy/notifiers
- /rancher/v2.0-v2.4/en/monitoring-alerting/v2.0.x-v2.4.x/notifiers
- /rancher/v2.x/en/monitoring-alerting/v2.0.x-v2.4.x/notifiers/
- /rancher/v2.x/en/monitoring-alerting/v2.0.x-v2.4.x/cluster-alerts/default-alerts/
---
Notifiers are services that inform you of alert events. You can configure notifiers to send alert notifications to staff best suited to take corrective action.
Rancher integrates with a variety of popular IT services, including:
- **Slack**: Send alert notifications to your Slack channels.
- **Email**: Choose email recipients for alert notifications.
- **PagerDuty**: Route notifications to staff by phone, SMS, or personal email.
- **WebHooks**: Update a webpage with alert notifications.
- **WeChat**: (Available as of v2.2.0) Send alert notifications to your Enterprise WeChat contacts.
- **DingTalk**: (Available as of v2.4.6) Send alert notifications to DingTalk using a webhook.
- **Microsoft Teams**: (Available as of v2.4.6) Send alert notifications to Teams using a webhook.
This section covers the following topics:
- [Roles-based access control for notifiers](#roles-based-access-control-for-notifiers)
- [Adding notifiers](#adding-notifiers)
- [Configuration](#configuration)
- [Managing notifiers](#managing-notifiers)
- [Example payload for a webhook alert notifier](#example-payload-for-a-webhook-alert-notifier)
# Roles-based Access Control for Notifiers
Notifiers are configured at the cluster level. This model ensures that only cluster owners need to configure notifiers, leaving project owners to simply configure alerts in the scope of their projects. You don't need to dispense privileges like SMTP server access or cloud account access.
# Adding Notifiers
Set up a notifier so that you can begin configuring and sending alerts.
1. From the **Global View**, open the cluster that you want to add a notifier.
1. From the main menu, select **Tools > Notifiers**. Then click **Add Notifier**.
1. Select the service you want to use as your notifier, and then fill out the form. For help filling out the form, refer to the configuration section below.
1. Click **Test.** You should receive a notification confirming that the notifier is configured correctly.
1. Click **Add** to complete adding the notifier.
**Result:** Your notifier is added to Rancher.
# Configuration
- [Slack](#slack)
- [Email](#email)
- [PagerDuty](#pagerduty)
- [Webhook](#webhook)
- [WeChat](#wechat)
- [DingTalk](#dingtalk)
- [Microsoft Teams](#microsoft-teams)
### Slack
| Field | Explanation |
|----------|----------------------|
| Name | Enter a **Name** for the notifier. |
| URL | From Slack, create a webhook. For instructions, see the [Slack Documentation](https://get.slack.help/hc/en-us/articles/115005265063-Incoming-WebHooks-for-Slack). Then enter the Slack webhook URL. |
| Default Channel | Enter the name of the channel that you want to send alert notifications in the following format: `#<channelname>`. Both public and private channels are supported. |
| Proxy URL | Proxy for the Slack webhook. |
| Send Resolved Alerts | _Available as of v2.3.0_ Whether to send a follow-up notification if an alert has been resolved (e.g. [Resolved] High CPU Usage) |
**Validation:** Click **Test**. If the test is successful, the Slack channel you're configuring for the notifier outputs **Slack setting validated.**
### Email
| Field | Explanation |
|----------|----------------------|
| Name | Enter a **Name** for the notifier. |
| Default Recipient Address | Enter the email address that you want to receive the notification. |
| Send Resolved Alerts | _Available as of v2.3.0_ Whether to send a follow-up notification if an alert has been resolved (e.g. [Resolved] High CPU Usage) |
SMTP Server Configuration:
| Field | Explanation |
|----------|----------------------|
| Sender | Enter an email address available on your mail server that you want to send the notification. |
| Host | Enter the IP address or hostname for your SMTP server. Example: `smtp.email.com` |
| Port | In the **Port** field, enter the port used for email. Typically, TLS uses `587` and SSL uses `465`. |
| Use TLS | If you're using TLS, make sure **Use TLS** is selected. |
| Username | Username to authenticate with the SMTP server. |
| Password | Password to authenticate with the SMTP server. |
**Validation:** Click **Test**. If the test is successful, Rancher prints **settings validated** and you receive a test notification email.
### PagerDuty
| Field | Explanation |
|----------|----------------------|
| Name | Enter a **Name** for the notifier. |
| Default Integration Key | From PagerDuty, create a Prometheus integration. For instructions, see the [PagerDuty Documentation](https://www.pagerduty.com/docs/guides/prometheus-integration-guide/). Then enter the integration key.
| Service Key | The same as the integration key. For instructions on creating a Prometheus integration, see the [PagerDuty Documentation](https://www.pagerduty.com/docs/guides/prometheus-integration-guide/). Then enter the integration key. |
| Send Resolved Alerts | _Available as of v2.3.0_ Whether to send a follow-up notification if an alert has been resolved (e.g. [Resolved] High CPU Usage) |
**Validation:** Click **Test**. If the test is successful, your PagerDuty endpoint outputs **PagerDuty setting validated.**
### Webhook
| Field | Explanation |
|----------|----------------------|
| Name | Enter a **Name** for the notifier. |
| URL | Using the app of your choice, create a webhook URL. |
| Proxy URL | Proxy for the webhook. |
| Send Resolved Alerts | _Available as of v2.3.0_ Whether to send a follow-up notification if an alert has been resolved (e.g. [Resolved] High CPU Usage) |
**Validation:** Click **Test**. If the test is successful, the URL you're configuring as a notifier outputs **Webhook setting validated.**
### WeChat
_Available as of v2.2.0_
| Field | Explanation |
|----------|----------------------|
| Name | Enter a **Name** for the notifier. |
| Corporation ID | Enter the "EnterpriseID" of your corporation. You can get it fro the [Profile page](https://work.weixin.qq.com/wework_admin/frame#profile). |
| Application Agent ID | From Enterprise WeChat, create an application in the [Application page](https://work.weixin.qq.com/wework_admin/frame#apps), and then enter the "AgentId" of this application. You will also need to enter the application secret. |
| Application Secret | The secret that corresponds to the Application Agent ID. |
| Recipient Type | Party, tag, or user. |
| Default Recipient | The default recipient ID should correspond to the recipient type. It should be the party ID, tag ID or user account that you want to receive the notification. You could get contact information from [Contacts page](https://work.weixin.qq.com/wework_admin/frame#contacts). |
| Proxy URL | If you are using a proxy, enter the proxy URL. |
| Send Resolved Alerts | _Available as of v2.3.0_ Whether to send a follow-up notification if an alert has been resolved (e.g. [Resolved] High CPU Usage) |
**Validation:** Click **Test.** If the test is successful, you should receive an alert message.
### DingTalk
_Available as of v2.4.6_
| Field | Explanation |
|----------|----------------------|
| Name | Enter a **Name** for the notifier. |
| Webhook URL | Enter the DingTalk webhook URL. For help setting up the webhook, refer to the [DingTalk documentation.](https://www.alibabacloud.com/help/doc-detail/52872.htm) |
| Secret | Optional: Enter a secret for the DingTalk webhook. |
| Proxy URL | Optional: Enter a proxy for the DingTalk webhook. |
| Send Resolved Alerts | Whether to send a follow-up notification if an alert has been resolved (e.g. [Resolved] High CPU Usage) |
**Validation:** Click **Test.** If the test is successful, the DingTalk notifier output is **DingTalk setting validated.**
### Microsoft Teams
_Available as of v2.4.6_
| Field | Explanation |
|----------|----------------------|
| Name | Enter a **Name** for the notifier. |
| Webhook URL | Enter the Microsoft Teams webhook URL. For help setting up the webhook, refer to the [Teams Documentation.](https://docs.microsoft.com/en-us/microsoftteams/platform/webhooks-and-connectors/how-to/add-incoming-webhook) |
| Proxy URL | Optional: Enter a proxy for the Teams webhook. |
| Send Resolved Alerts | Whether to send a follow-up notification if an alert has been resolved (e.g. [Resolved] High CPU Usage) |
**Validation:** Click **Test.** If the test is successful, the Teams notifier output is **MicrosoftTeams setting validated.**
# Managing Notifiers
After you set up notifiers, you can manage them. From the **Global** view, open the cluster that you want to manage your notifiers. Select **Tools > Notifiers**. You can:
- **Edit** their settings that you configured during their initial setup.
- **Clone** them, to quickly setup slightly different notifiers.
- **Delete** them when they're no longer necessary.
# Example Payload for a Webhook Alert Notifier
```json
{
"receiver": "c-2a3bc:kube-components-alert",
"status": "firing",
"alerts": [
{
"status": "firing",
"labels": {
"alert_name": "Scheduler is unavailable",
"alert_type": "systemService",
"cluster_name": "mycluster (ID: c-2a3bc)",
"component_name": "scheduler",
"group_id": "c-2a3bc:kube-components-alert",
"logs": "Get http://127.0.0.1:10251/healthz: dial tcp 127.0.0.1:10251: connect: connection refused",
"rule_id": "c-2a3bc:kube-components-alert_scheduler-system-service",
"severity": "critical"
},
"annotations": {},
"startsAt": "2020-01-30T19:18:13.321684733Z",
"endsAt": "0001-01-01T00:00:00Z",
"generatorURL": ""
}
],
"groupLabels": {
"component_name": "scheduler",
"rule_id": "c-2a3bc:kube-components-alert_scheduler-system-service"
},
"commonLabels": {
"alert_name": "Scheduler is unavailable",
"alert_type": "systemService",
"cluster_name": "mycluster (ID: c-2a3bc)"
}
}
```
# What's Next?
After creating a notifier, set up alerts to receive notifications of Rancher system events.
- [Cluster owners]({{<baseurl>}}/rancher/v2.0-v2.4/en/admin-settings/rbac/cluster-project-roles/#cluster-roles) can set up alerts at the [cluster level]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/tools/alerts/).
- [Project owners]({{<baseurl>}}/rancher/v2.0-v2.4/en/admin-settings/rbac/cluster-project-roles/#project-roles) can set up alerts at the [project level]({{<baseurl>}}/rancher/v2.0-v2.4/en/project-admin/tools/alerts/).
versioned_docs/version-2.0-2.4/cluster-admin/tools/opa-gatekeeper/opa-gatekeeper.md
+99
View File
@@ -0,0 +1,99 @@
---
title: OPA Gatekeeper
weight: 17
aliases:
- /rancher/v2.0-v2.4/en/cluster-admin/tools/opa-gatekeeper
- /rancher/v2.0-v2.4/en/opa-gatekeper/Open%20Policy%20Agent
- /rancher/v2.0-v2.4/en/opa-gatekeper
---
_Available as of v2.4.0_
To ensure consistency and compliance, every organization needs the ability to define and enforce policies in its environment in an automated way. [OPA (Open Policy Agent)](https://www.openpolicyagent.org/) is a policy engine that facilitates policy-based control for cloud native environments. Rancher provides the ability to enable OPA Gatekeeper in Kubernetes clusters, and also installs a couple of built-in policy definitions, which are also called constraint templates.
OPA provides a high-level declarative language that lets you specify policy as code and ability to extend simple APIs to offload policy decision-making.
[OPA Gatekeeper](https://github.com/open-policy-agent/gatekeeper) is a project that provides integration between OPA and Kubernetes. OPA Gatekeeper provides:
- An extensible, parameterized policy library.
- Native Kubernetes CRDs for instantiating the policy library, also called “constraints."
- Native Kubernetes CRDs for extending the policy library, also called "constraint templates."
- Audit functionality.
To read more about OPA, please refer to the [official documentation.](https://www.openpolicyagent.org/docs/latest/)
# How the OPA Gatekeeper Integration Works
Kubernetes provides the ability to extend API server functionality via admission controller webhooks, which are invoked whenever a resource is created, updated or deleted. Gatekeeper is installed as a validating webhook and enforces policies defined by Kubernetes custom resource definitions. In addition to the admission control usage, Gatekeeper provides the capability to audit existing resources in Kubernetes clusters and mark current violations of enabled policies.
OPA Gatekeeper is made available via Rancher's Helm system chart, and it is installed in a namespace named `gatekeeper-system.`
# Enabling OPA Gatekeeper in a Cluster
> **Prerequisites:**
>
> - Only administrators and cluster owners can enable OPA Gatekeeper.
> - The dashboard needs to be enabled using the `dashboard` feature flag. For more information, refer to the [section on enabling experimental features.]({{<baseurl>}}/rancher/v2.0-v2.4/en/installation/options/feature-flags/)
1. Navigate to the cluster's **Dashboard** view.
1. On the left side menu, expand the cluster menu and click on **OPA Gatekeeper.**
1. To install Gatekeeper with the default configuration, click on **Enable Gatekeeper (v0.1.0) with defaults.**
1. To change any default configuration, click on **Customize Gatekeeper yaml configuration.**
# Constraint Templates
[Constraint templates](https://github.com/open-policy-agent/gatekeeper#constraint-templates) are Kubernetes custom resources that define the schema and Rego logic of the OPA policy to be applied by Gatekeeper. For more information on the Rego policy language, refer to the [official documentation.](https://www.openpolicyagent.org/docs/latest/policy-language/)
When OPA Gatekeeper is enabled, Rancher installs some templates by default.
To list the constraint templates installed in the cluster, go to the left side menu under OPA Gatekeeper and click on **Templates.**
Rancher also provides the ability to create your own constraint templates by importing YAML definitions.
# Creating and Configuring Constraints
[Constraints](https://github.com/open-policy-agent/gatekeeper#constraints) are Kubernetes custom resources that define the scope of objects to which a specific constraint template applies to. The complete policy is defined by constraint templates and constraints together.
> **Prerequisites:** OPA Gatekeeper must be enabled in the cluster.
To list the constraints installed, go to the left side menu under OPA Gatekeeper, and click on **Constraints.**
New constraints can be created from a constraint template.
Rancher provides the ability to create a constraint by using a convenient form that lets you input the various constraint fields.
The **Edit as yaml** option is also available to configure the the constraint's yaml definition.
### Exempting Rancher's System Namespaces from Constraints
When a constraint is created, ensure that it does not apply to any Rancher or Kubernetes system namespaces. If the system namespaces are not excluded, then it is possible to see many resources under them marked as violations of the constraint.
To limit the scope of the constraint only to user namespaces, always specify these namespaces under the **Match** field of the constraint.
Also, the constraint may interfere with other Rancher functionality and deny system workloads from being deployed. To avoid this, exclude all Rancher-specific namespaces from your constraints.
# Enforcing Constraints in your Cluster
When the **Enforcement Action** is **Deny,** the constraint is immediately enabled and will deny any requests that violate the policy defined. By default, the enforcement value is **Deny.**
When the **Enforcement Action** is **Dryrun,** then any resources that violate the policy are only recorded under the constraint's status field.
To enforce constraints, create a constraint using the form. In the **Enforcement Action** field, choose **Deny.**
# Audit and Violations in your Cluster
OPA Gatekeeper runs a periodic audit to check if any existing resource violates any enforced constraint. The audit-interval (default 300s) can be configured while installing Gatekeeper.
On the Gatekeeper page, any violations of the defined constraints are listed.
Also under **Constraints,** the number of violations of the constraint can be found.
The detail view of each constraint lists information about the resource that violated the constraint.
# Disabling Gatekeeper
1. Navigate to the cluster's Dashboard view
1. On the left side menu, expand the cluster menu and click on **OPA Gatekeeper.**
1. Click the **&#8942; > Disable**.
**Result:** Upon disabling OPA Gatekeeper, all constraint templates and constraints will also be deleted.
versioned_docs/version-2.0-2.4/cluster-admin/tools/tools.md
+71
View File
@@ -0,0 +1,71 @@
---
title: Tools for Logging, Monitoring, and More
weight: 2033
aliases:
- /rancher/v2.0-v2.4/en/tools/notifiers-and-alerts/
---
Rancher contains a variety of tools that aren't included in Kubernetes to assist in your DevOps operations. Rancher can integrate with external services to help your clusters run more efficiently. Tools are divided into following categories:
<!-- TOC -->
- [Logging](#logging)
- [Monitoring](#monitoring)
- [Alerts](#alerts)
- [Notifiers](#notifiers)
- [Istio](#istio)
- [OPA Gatekeeper](#opa-gatekeeper)
- [CIS Scans](#cis-scans)
<!-- /TOC -->
# Logging
Logging is helpful because it allows you to:
- Capture and analyze the state of your cluster
- Look for trends in your environment
- Save your logs to a safe location outside of your cluster
- Stay informed of events like a container crashing, a pod eviction, or a node dying
- More easily debugg and troubleshoot problems
Rancher can integrate with Elasticsearch, splunk, kafka, syslog, and fluentd.
Refer to the logging documentation [here.](./cluster-logging)
# Monitoring
Using Rancher, you can monitor the state and processes of your cluster nodes, Kubernetes components, and software deployments through integration with [Prometheus](https://prometheus.io/), a leading open-source monitoring solution.
For details, refer to [Monitoring.](./cluster-monitoring)
# Alerts
After monitoring is enabled, you can set up alerts and notifiers that provide the mechanism to receive them.
Alerts are rules that trigger notifications. Before you can receive alerts, you must configure one or more notifier in Rancher. The scope for alerts can be set at either the cluster or project level.
For details, refer to [Alerts.](./cluster-alerts)
# Notifiers
Notifiers are services that inform you of alert events. You can configure notifiers to send alert notifications to staff best suited to take corrective action. Notifications can be sent with Slack, email, PagerDuty, WeChat, and webhooks.
For details, refer to [Notifiers.](./notifiers)
# Istio
_Available as of v2.3_
[Istio](https://istio.io/) is an open-source tool that makes it easier for DevOps teams to observe, control, troubleshoot, and secure the traffic within a complex network of microservices.
Refer to the Istio documentation [here.](./istio)
# OPA Gatekeeper
[OPA Gatekeeper](https://github.com/open-policy-agent/gatekeeper) is an open-source project that provides integration between OPA and Kubernetes to provide policy control via admission controller webhooks. For details on how to enable Gatekeeper in Rancher, refer to the [OPA Gatekeeper section.](./opa-gatekeeper)
# CIS Scans
Rancher can run a security scan to check whether Kubernetes is deployed according to security best practices as defined in the CIS Kubernetes Benchmark.
Refer to the CIS scan documentation [here.](./cis-scans)