public/rancher-docs
1
0
Fork 0
mirror of https://github.com/rancher/rancher-docs.git synced 2026-09-26 21:18:04 +00:00
Code Issues Packages Projects Releases Wiki Activity

Remove intermediate /en folder

Browse Source
This commit is contained in:
Billy Tat
2022-08-17 09:14:41 -07:00
parent eb2197be30
commit 506e174643
441 changed files with 4 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/explanations/integrations-in-rancher/cis-scans/configuration-reference.md
+106
View File
@@ -0,0 +1,106 @@
---
title: Configuration
weight: 3
---
This configuration reference is intended to help you manage the custom resources created by the `rancher-cis-benchmark` application. These resources are used for performing CIS scans on a cluster, skipping tests, setting the test profile that will be used during a scan, and other customization.
To configure the custom resources, go to the **Cluster Dashboard** To configure the CIS scans,
1. In the upper left corner, click **☰ > Cluster Management**.
1. On the **Clusters** page, go to the cluster where you want to configure CIS scans and click **Explore**.
1. In the left navigation bar, click **CIS Benchmark**.
### Scans
A scan is created to trigger a CIS scan on the cluster based on the defined profile. A report is created after the scan is completed.
When configuring a scan, you need to define the name of the scan profile that will be used with the `scanProfileName` directive.
An example ClusterScan custom resource is below:
```yaml
apiVersion: cis.cattle.io/v1
kind: ClusterScan
metadata:
name: rke-cis
spec:
scanProfileName: rke-profile-hardened
```
### Profiles
A profile contains the configuration for the CIS scan, which includes the benchmark version to use and any specific tests to skip in that benchmark.
:::caution
By default, a few ClusterScanProfiles are installed as part of the `rancher-cis-benchmark` chart. If a user edits these default benchmarks or profiles, the next chart update will reset them back. So it is advisable for users to not edit the default ClusterScanProfiles.
:::
Users can clone the ClusterScanProfiles to create custom profiles.
Skipped tests are listed under the `skipTests` directive.
When you create a new profile, you will also need to give it a name.
An example `ClusterScanProfile` is below:
```yaml
apiVersion: cis.cattle.io/v1
kind: ClusterScanProfile
metadata:
annotations:
meta.helm.sh/release-name: clusterscan-operator
meta.helm.sh/release-namespace: cis-operator-system
labels:
app.kubernetes.io/managed-by: Helm
name: "<example-profile>"
spec:
benchmarkVersion: cis-1.5
skipTests:
- "1.1.20"
- "1.1.21"
```
### Benchmark Versions
A benchmark version is the name of benchmark to run using `kube-bench`, as well as the valid configuration parameters for that benchmark.
A `ClusterScanBenchmark` defines the CIS `BenchmarkVersion` name and test configurations. The `BenchmarkVersion` name is a parameter provided to the `kube-bench` tool.
By default, a few `BenchmarkVersion` names and test configurations are packaged as part of the CIS scan application. When this feature is enabled, these default BenchmarkVersions will be automatically installed and available for users to create a ClusterScanProfile.
:::caution
If the default BenchmarkVersions are edited, the next chart update will reset them back. Therefore we don't recommend editing the default ClusterScanBenchmarks.
:::
A ClusterScanBenchmark consists of the fields:
- `ClusterProvider`: This is the cluster provider name for which this benchmark is applicable. For example: RKE, EKS, GKE, etc. Leave it empty if this benchmark can be run on any cluster type.
- `MinKubernetesVersion`: Specifies the cluster's minimum kubernetes version necessary to run this benchmark. Leave it empty if there is no dependency on a particular Kubernetes version.
- `MaxKubernetesVersion`: Specifies the cluster's maximum Kubernetes version necessary to run this benchmark. Leave it empty if there is no dependency on a particular k8s version.
An example `ClusterScanBenchmark` is below:
```yaml
apiVersion: cis.cattle.io/v1
kind: ClusterScanBenchmark
metadata:
annotations:
meta.helm.sh/release-name: clusterscan-operator
meta.helm.sh/release-namespace: cis-operator-system
creationTimestamp: "2020-08-28T18:18:07Z"
generation: 1
labels:
app.kubernetes.io/managed-by: Helm
name: cis-1.5
resourceVersion: "203878"
selfLink: /apis/cis.cattle.io/v1/clusterscanbenchmarks/cis-1.5
uid: 309e543e-9102-4091-be91-08d7af7fb7a7
spec:
clusterProvider: ""
minKubernetesVersion: 1.15.0
```
docs/explanations/integrations-in-rancher/cis-scans/custom-benchmark.md
+84
View File
@@ -0,0 +1,84 @@
---
title: Creating a Custom Benchmark Version for Running a Cluster Scan
weight: 4
---
Each Benchmark Version defines a set of test configuration files that define the CIS tests to be run by the <a href="https://github.com/aquasecurity/kube-bench" target="_blank">kube-bench</a> tool.
The `rancher-cis-benchmark` application installs a few default Benchmark Versions which are listed under CIS Benchmark application menu.
But there could be some Kubernetes cluster setups that require custom configurations of the Benchmark tests. For example, the path to the Kubernetes config files or certs might be different than the standard location where the upstream CIS Benchmarks look for them.
It is now possible to create a custom Benchmark Version for running a cluster scan using the `rancher-cis-benchmark` application.
When a cluster scan is run, you need to select a Profile which points to a specific Benchmark Version.
Follow all the steps below to add a custom Benchmark Version and run a scan using it.
1. [Prepare the Custom Benchmark Version ConfigMap](#1-prepare-the-custom-benchmark-version-configmap)
2. [Add a Custom Benchmark Version to a Cluster](#2-add-a-custom-benchmark-version-to-a-cluster)
3. [Create a New Profile for the Custom Benchmark Version](#3-create-a-new-profile-for-the-custom-benchmark-version)
4. [Run a Scan Using the Custom Benchmark Version](#4-run-a-scan-using-the-custom-benchmark-version)
### 1. Prepare the Custom Benchmark Version ConfigMap
To create a custom benchmark version, first you need to create a ConfigMap containing the benchmark version's config files and upload it to your Kubernetes cluster where you want to run the scan.
To prepare a custom benchmark version ConfigMap, suppose we want to add a custom Benchmark Version named `foo`.
1. Create a directory named `foo` and inside this directory, place all the config YAML files that the <a href="https://github.com/aquasecurity/kube-bench" target="_blank">kube-bench</a> tool looks for. For example, here are the config YAML files for a Generic CIS 1.5 Benchmark Version https://github.com/aquasecurity/kube-bench/tree/master/cfg/cis-1.5
1. Place the complete `config.yaml` file, which includes all the components that should be tested.
1. Add the Benchmark version name to the `target_mapping` section of the `config.yaml`:
```yaml
target_mapping:
"foo":
- "master"
- "node"
- "controlplane"
- "etcd"
- "policies"
```
1. Upload this directory to your Kubernetes Cluster by creating a ConfigMap:
```yaml
kubectl create configmap -n <namespace> foo --from-file=<path to directory foo>
```
### 2. Add a Custom Benchmark Version to a Cluster
1. In the upper left corner, click **☰ > Cluster Management**.
1. On the **Clusters** page, go to the cluster where you want to add a custom benchmark and click **Explore**.
1. In the left navigation bar, click **CIS Benchmark > Benchmark Version**.
1. Click **Create**.
1. Enter the **Name** and a description for your custom benchmark version.
1. Choose the cluster provider that your benchmark version applies to.
1. Choose the ConfigMap you have uploaded from the dropdown.
1. Add the minimum and maximum Kubernetes version limits applicable, if any.
1. Click **Create**.
### 3. Create a New Profile for the Custom Benchmark Version
To run a scan using your custom benchmark version, you need to add a new Profile pointing to this benchmark version.
1. In the upper left corner, click **☰ > Cluster Management**.
1. On the **Clusters** page, go to the cluster where you want to add a custom benchmark and click **Explore**.
1. In the left navigation bar, click **CIS Benchmark > Profile**.
1. Click **Create**.
1. Provide a **Name** and description. In this example, we name it `foo-profile`.
1. Choose the Benchmark Version from the dropdown.
1. Click **Create**.
### 4. Run a Scan Using the Custom Benchmark Version
Once the Profile pointing to your custom benchmark version `foo` has been created, you can create a new Scan to run the custom test configs in the Benchmark Version.
To run a scan,
1. In the upper left corner, click **☰ > Cluster Management**.
1. On the **Clusters** page, go to the cluster where you want to add a custom benchmark and click **Explore**.
1. In the left navigation bar, click **CIS Benchmark > Scan**.
1. Click **Create**.
1. Choose the new cluster scan profile.
1. Click **Create**.
**Result:** A report is generated with the scan results. To see the results, click the name of the scan that appears.
docs/explanations/integrations-in-rancher/cis-scans/rbac-for-cis-scans.md
+50
View File
@@ -0,0 +1,50 @@
---
title: Roles-based Access Control
shortTitle: RBAC
weight: 3
---
This section describes the permissions required to use the rancher-cis-benchmark App.
The rancher-cis-benchmark is a cluster-admin only feature by default.
However, the `rancher-cis-benchmark` chart installs these two default `ClusterRoles`:
- cis-admin
- cis-view
In Rancher, only cluster owners and global administrators have `cis-admin` access by default.
Note: If you were using the `cis-edit` role added in Rancher v2.5 setup, it has now been removed since
Rancher v2.5.2 because it essentially is same as `cis-admin`. If you happen to create any clusterrolebindings
for `cis-edit`, please update them to use `cis-admin` ClusterRole instead.
# Cluster-Admin Access
Rancher CIS Scans is a cluster-admin only feature by default.
This means only the Rancher global admins, and the cluster’s cluster-owner can:
- Install/Uninstall the rancher-cis-benchmark App
- See the navigation links for CIS Benchmark CRDs - ClusterScanBenchmarks, ClusterScanProfiles, ClusterScans
- List the default ClusterScanBenchmarks and ClusterScanProfiles
- Create/Edit/Delete new ClusterScanProfiles
- Create/Edit/Delete a new ClusterScan to run the CIS scan on the cluster
- View and Download the ClusterScanReport created after the ClusterScan is complete
# Summary of Default Permissions for Kubernetes Default Roles
The rancher-cis-benchmark creates three `ClusterRoles` and adds the CIS Benchmark CRD access to the following default K8s `ClusterRoles`:
| ClusterRole created by chart | Default K8s ClusterRole | Permissions given with Role
| ------------------------------| ---------------------------| ---------------------------|
| `cis-admin` | `admin`| Ability to CRUD clusterscanbenchmarks, clusterscanprofiles, clusterscans, clusterscanreports CR
| `cis-view` | `view `| Ability to List(R) clusterscanbenchmarks, clusterscanprofiles, clusterscans, clusterscanreports CR
By default only cluster-owner role will have ability to manage and use `rancher-cis-benchmark` feature.
The other Rancher roles (cluster-member, project-owner, project-member) do not have any default permissions to manage and use rancher-cis-benchmark resources.
But if a cluster-owner wants to delegate access to other users, they can do so by creating ClusterRoleBindings between these users and the above CIS ClusterRoles manually.
There is no automatic role aggregation supported for the `rancher-cis-benchmark` ClusterRoles.
docs/explanations/integrations-in-rancher/cis-scans/skipped-and-not-applicable-tests.md
+54
View File
@@ -0,0 +1,54 @@
---
title: Skipped and Not Applicable Tests
weight: 3
---
This section lists the tests that are skipped in the permissive test profile for RKE.
> All the tests that are skipped and not applicable on this page will be counted as Not Applicable in the v2.5 generated report. The skipped test count will only mention the user-defined skipped tests. This allows user-skipped tests to be distinguished from the tests that are skipped by default in the RKE permissive test profile.
# CIS Benchmark v1.5
### CIS Benchmark v1.5 Skipped Tests
| Number | Description | Reason for Skipping |
| ---------- | ------------- | --------- |
| 1.1.12 | Ensure that the etcd data directory ownership is set to etcd:etcd (Automated) | A system service account is required for etcd data directory ownership. Refer to Rancher's hardening guide for more details on how to configure this ownership. |
| 1.2.6 | Ensure that the --kubelet-certificate-authority argument is set as appropriate (Automated) | When generating serving certificates, functionality could break in conjunction with hostname overrides which are required for certain cloud providers. |
| 1.2.16 | Ensure that the admission control plugin PodSecurityPolicy is set (Automated) | Enabling Pod Security Policy can cause applications to unexpectedly fail. |
| 1.2.33 | Ensure that the --encryption-provider-config argument is set as appropriate (Manual) | Enabling encryption changes how data can be recovered as data is encrypted. |
| 1.2.34 | Ensure that encryption providers are appropriately configured (Manual) | Enabling encryption changes how data can be recovered as data is encrypted. |
| 4.2.6 | Ensure that the --protect-kernel-defaults argument is set to true (Automated) | System level configurations are required before provisioning the cluster in order for this argument to be set to true. |
| 4.2.10 | Ensure that the--tls-cert-file and --tls-private-key-file arguments are set as appropriate (Automated) | When generating serving certificates, functionality could break in conjunction with hostname overrides which are required for certain cloud providers. |
| 5.1.5 | Ensure that default service accounts are not actively used. (Automated) | Kubernetes provides default service accounts to be used. |
| 5.2.2 | Minimize the admission of containers wishing to share the host process ID namespace (Automated) | Enabling Pod Security Policy can cause applications to unexpectedly fail. |
| 5.2.3 | Minimize the admission of containers wishing to share the host IPC namespace (Automated) | Enabling Pod Security Policy can cause applications to unexpectedly fail. |
| 5.2.4 | Minimize the admission of containers wishing to share the host network namespace (Automated) | Enabling Pod Security Policy can cause applications to unexpectedly fail. |
| 5.2.5 | Minimize the admission of containers with allowPrivilegeEscalation (Automated) | Enabling Pod Security Policy can cause applications to unexpectedly fail. |
| 5.3.2 | Ensure that all Namespaces have Network Policies defined (Automated) | Enabling Network Policies can prevent certain applications from communicating with each other. |
| 5.6.4 | The default namespace should not be used (Automated) | Kubernetes provides a default namespace. |
### CIS Benchmark v1.5 Not Applicable Tests
| Number | Description | Reason for being not applicable |
| ---------- | ------------- | --------- |
| 1.1.1 | Ensure that the API server pod specification file permissions are set to 644 or more restrictive (Automated) | Clusters provisioned by RKE doesn't require or maintain a configuration file for kube-apiserver. All configuration is passed in as arguments at container run time. |
| 1.1.2 | Ensure that the API server pod specification file ownership is set to root:root (Automated) | Clusters provisioned by RKE doesn't require or maintain a configuration file for kube-apiserver. All configuration is passed in as arguments at container run time. |
| 1.1.3 | Ensure that the controller manager pod specification file permissions are set to 644 or more restrictive (Automated) | Clusters provisioned by RKE doesn't require or maintain a configuration file for controller-manager. All configuration is passed in as arguments at container run time. |
| 1.1.4 | Ensure that the controller manager pod specification file ownership is set to root:root (Automated) | Clusters provisioned by RKE doesn't require or maintain a configuration file for controller-manager. All configuration is passed in as arguments at container run time. |
| 1.1.5 | Ensure that the scheduler pod specification file permissions are set to 644 or more restrictive (Automated) | Clusters provisioned by RKE doesn't require or maintain a configuration file for scheduler. All configuration is passed in as arguments at container run time. |
| 1.1.6 | Ensure that the scheduler pod specification file ownership is set to root:root (Automated) | Clusters provisioned by RKE doesn't require or maintain a configuration file for scheduler. All configuration is passed in as arguments at container run time. |
| 1.1.7 | Ensure that the etcd pod specification file permissions are set to 644 or more restrictive (Automated) | Clusters provisioned by RKE doesn't require or maintain a configuration file for etcd. All configuration is passed in as arguments at container run time. |
| 1.1.8 | Ensure that the etcd pod specification file ownership is set to root:root (Automated) | Clusters provisioned by RKE doesn't require or maintain a configuration file for etcd. All configuration is passed in as arguments at container run time. |
| 1.1.13 | Ensure that the admin.conf file permissions are set to 644 or more restrictive (Automated) | Clusters provisioned by RKE does not store the kubernetes default kubeconfig credentials file on the nodes. |
| 1.1.14 | Ensure that the admin.conf file ownership is set to root:root (Automated) | Clusters provisioned by RKE does not store the kubernetes default kubeconfig credentials file on the nodes. |
| 1.1.15 | Ensure that the scheduler.conf file permissions are set to 644 or more restrictive (Automated) | Clusters provisioned by RKE doesn't require or maintain a configuration file for scheduler. All configuration is passed in as arguments at container run time. |
| 1.1.16 | Ensure that the scheduler.conf file ownership is set to root:root (Automated) | Clusters provisioned by RKE doesn't require or maintain a configuration file for scheduler. All configuration is passed in as arguments at container run time. |
| 1.1.17 | Ensure that the controller-manager.conf file permissions are set to 644 or more restrictive (Automated) | Clusters provisioned by RKE doesn't require or maintain a configuration file for controller-manager. All configuration is passed in as arguments at container run time. |
| 1.1.18 | Ensure that the controller-manager.conf file ownership is set to root:root (Automated) | Clusters provisioned by RKE doesn't require or maintain a configuration file for controller-manager. All configuration is passed in as arguments at container run time. |
| 1.3.6 | Ensure that the RotateKubeletServerCertificate argument is set to true (Automated) | Clusters provisioned by RKE handles certificate rotation directly through RKE. |
| 4.1.1 | Ensure that the kubelet service file permissions are set to 644 or more restrictive (Automated) | Clusters provisioned by RKE doesn’t require or maintain a configuration file for the kubelet service. All configuration is passed in as arguments at container run time. |
| 4.1.2 | Ensure that the kubelet service file ownership is set to root:root (Automated) | Clusters provisioned by RKE doesn’t require or maintain a configuration file for the kubelet service. All configuration is passed in as arguments at container run time. |
| 4.1.9 | Ensure that the kubelet configuration file has permissions set to 644 or more restrictive (Automated) | Clusters provisioned by RKE doesn’t require or maintain a configuration file for the kubelet. All configuration is passed in as arguments at container run time. |
| 4.1.10 | Ensure that the kubelet configuration file ownership is set to root:root (Automated) | Clusters provisioned by RKE doesn’t require or maintain a configuration file for the kubelet. All configuration is passed in as arguments at container run time. |
| 4.2.12 | Ensure that the RotateKubeletServerCertificate argument is set to true (Automated) | Clusters provisioned by RKE handles certificate rotation directly through RKE. |
docs/explanations/integrations-in-rancher/fleet-gitops-at-scale/architecture.md
+9
View File
@@ -0,0 +1,9 @@
---
title: Architecture
weight: 1
---
Fleet can manage deployments from git of raw Kubernetes YAML, Helm charts, or Kustomize or any combination of the three. Regardless of the source, all resources are dynamically turned into Helm charts, and Helm is used as the engine to deploy everything in the cluster. This gives you a high degree of control, consistency, and auditability. Fleet focuses not only on the ability to scale, but to give one a high degree of control and visibility to exactly what is installed on the cluster.
![Architecture](/img/fleet-architecture.svg)
docs/explanations/integrations-in-rancher/fleet-gitops-at-scale/use-fleet-behind-a-proxy.md
+55
View File
@@ -0,0 +1,55 @@
---
title: Using Fleet Behind a Proxy
weight: 3
---
In this section, you'll learn how to enable Fleet in a setup that has a Rancher server with a public IP a Kubernetes cluster that has no public IP, but is configured to use a proxy.
Rancher does not establish connections with registered downstream clusters. The Rancher agent deployed on the downstream cluster must be able to establish the connection with Rancher.
To set up Fleet to work behind a proxy, you will need to set the **Agent Environment Variables** for the downstream cluster. These are cluster-level configuration options.
Through the Rancher UI, you can configure these environment variables for any cluster type, including registered and custom clusters. The variables can be added while editing an existing cluster or while provisioning a new cluster.
For public downstream clusters, it is sufficient to [set the required environment variables in the Rancher UI.](#setting-environment-variables-in-the-rancher-ui)
For private nodes or private clusters, the environment variables need to be set on the nodes themselves. Then the environment variables are configured from the Rancher UI, typically when provisioning a custom cluster or when registering the private cluster. For an example of how to set the environment variables on Ubuntu node in a K3s Kubernetes cluster, see [this section.](#setting-environment-variables-on-private-nodes)
# Required Environment Variables
When adding Fleet agent environment variables for the proxy, replace <PROXY_IP> with your private proxy IP.
| Variable Name | Value |
|------------------|--------|
| `HTTP_PROXY` | http://<PROXY_IP>:8888 |
| `HTTPS_PROXY` | http://<PROXY_IP>:8888
| `NO_PROXY` | 127.0.0.0/8,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,.svc,.cluster.local |
# Setting Environment Variables in the Rancher UI
To add the environment variable to an existing cluster,
1. Click **☰ > Cluster Management**.
1. Go to the cluster where you want to add environment variables and click **⋮ > Edit Config**.
1. Click **Advanced Options**.
1. Click **Add Environment Variable**.
1. Enter the [required environment variables](#required-environment-variables)
1. Click **Save**.
**Result:** The Fleet agent works behind a proxy.
# Setting Environment Variables on Private Nodes
For private nodes and private clusters, the proxy environment variables need to be set on the nodes themselves, as well as configured from the Rancher UI.
This example shows how the environment variables would be set up on an Ubuntu node in a K3s Kubernetes cluster:
```
ssh -o ForwardAgent=yes ubuntu@<public_proxy_ip>
ssh <k3s_ip>
export proxy_private_ip=<private_proxy_ip>
export HTTP_PROXY=http://${proxy_private_ip}:8888
export HTTPS_PROXY=http://${proxy_private_ip}:8888
export NO_PROXY=127.0.0.0/8,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,.svc,.cluster.local
export KUBECONFIG=/etc/rancher/k3s/k3s.yaml
```
docs/explanations/integrations-in-rancher/fleet-gitops-at-scale/windows-support.md
+23
View File
@@ -0,0 +1,23 @@
---
title: Windows Support
weight: 2
---
Prior to Rancher v2.5.6, the `agent` did not have native Windows manifests on downstream clusters with Windows nodes. This would result in a failing `agent` pod for the cluster.
If you are upgrading from an older version of Rancher to v2.5.6+, you can deploy a working `agent` with the following workflow *in the downstream cluster*:
1. Cordon all Windows nodes.
1. Apply the below toleration to the `agent` workload.
1. Uncordon all Windows nodes.
1. Delete all `agent` pods. New pods should be created with the new toleration.
1. Once the `agent` pods are running, and auto-update is enabled for Fleet, they should be updated to a Windows-compatible `agent` version.
```yaml
tolerations:
- effect: NoSchedule
key: cattle.io/os
operator: Equal
value: linux
```
docs/explanations/integrations-in-rancher/harvester.md
+44
View File
@@ -0,0 +1,44 @@
---
title: Harvester Integration
weight: 10
---
Introduced in Rancher v2.6.1, [Harvester](https://docs.harvesterhci.io/) is an open-source hyper-converged infrastructure (HCI) software built on Kubernetes. Harvester installs on bare metal servers and provides integrated virtualization and distributed storage capabilities. Although Harvester operates using Kubernetes, it does not require users to know Kubernetes concepts, making it a more user-friendly application.
---
**_New in v2.6.3_**
Harvester is GA. Please refer to the [Harvester release notes](https://github.com/harvester/harvester/releases) for all updates.
---
### Feature Flag
The Harvester feature flag is used to manage access to the Virtualization Management (VM) page in Rancher where users can navigate directly to Harvester clusters and access the Harvester UI. The Harvester feature flag is enabled by default. Click [here](../../pages-for-subheaders/enable-experimental-features.md) for more information on feature flags in Rancher.
To navigate to the Harvester cluster, click **☰ > Virtualization Management**. From Harvester Clusters page, click one of the clusters listed to go to the single Harvester cluster view.
* If the Harvester feature flag is enabled, Harvester clusters will be filtered out from any pages or apps (such as Fleet and the multi-cluster app) that list Kubernetes clusters.
* If the Harvester feature flag is disabled, and a Harvester cluster is imported, the Harvester cluster will be shown in the Rancher cluster list in the Cluster Management page. Harvester clusters will only be shown on the cluster list when the feature flag is off.
* With the Harvester integration, Harvester clusters can now be imported into Rancher as a cluster type `Harvester`.
* Users may import a Harvester cluster only on the Virtualization Management page. Importing a cluster on the Cluster Management page is not supported, and a warning will advise you to return to the VM page to do so.
### Harvester Node Driver
The [Harvester node driver](https://docs.harvesterhci.io/v0.3/rancher/node-driver/) is marked as `tech preview` on RKE and RKE2 options in Rancher. This will be the case both on the Create page and once the driver is already enabled. The node driver is available whether or not the Harvester feature flag is enabled. Note that the node driver is off by default. Users may create RKE or RKE2 clusters on Harvester only from the Cluster Management page.
Harvester allows `.ISO` images to be uploaded and displayed through the Harvester UI, but this is not supported in the Rancher UI. This is because `.ISO` images usually require additional setup that interferes with a clean deployment (without requiring user intervention), and they are not typically used in cloud environments.
Click [here](../../pages-for-subheaders/about-provisioning-drivers.md#node-drivers) for more information on node drivers in Rancher.
### Limitations
---
**Applicable to Rancher v2.6.1 and v2.6.2 only:**
- Harvester v0.3.0 doesn’t support air-gapped environment installation.
- Harvester v0.3.0 doesn’t support upgrade from v0.2.0 nor upgrade to v1.0.0.
---
docs/explanations/integrations-in-rancher/istio/configuration-options/install-istio-on-rke2-cluster.md
+38
View File
@@ -0,0 +1,38 @@
---
title: Additional Steps for Installing Istio on an RKE2 Cluster
weight: 3
---
When installing or upgrading the Istio Helm chart through **Apps & Marketplace,**
1. If you are installing the chart, click **Customize Helm options before install** and click **Next**.
1. You will see options for configuring the Istio Helm chart. On the **Components** tab, check the box next to **Enabled CNI**.
1. Add a custom overlay file specifying `cniBinDir` and `cniConfDir`. For more information on these options, refer to the [Istio documentation.](https://istio.io/latest/docs/setup/additional-setup/cni/#helm-chart-parameters) An example is below:
```yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
components:
cni:
enabled: true
k8s:
overlays:
- apiVersion: "apps/v1"
kind: "DaemonSet"
name: "istio-cni-node"
patches:
- path: spec.template.spec.containers.[name:install-cni].securityContext.privileged
value: true
values:
cni:
image: rancher/mirrored-istio-install-cni:1.9.3
excludeNamespaces:
- istio-system
- kube-system
logLevel: info
cniBinDir: /opt/cni/bin
cniConfDir: /etc/cni/net.d
```
**Result:** Now you should be able to utilize Istio as desired, including sidecar injection and monitoring via Kiali.
docs/explanations/integrations-in-rancher/istio/configuration-options/pod-security-policies.md
+54
View File
@@ -0,0 +1,54 @@
---
title: Enable Istio with Pod Security Policies
weight: 1
---
If you have restrictive Pod Security Policies enabled, then Istio may not be able to function correctly, because it needs certain permissions in order to install itself and manage pod infrastructure. In this section, we will configure a cluster with PSPs enabled for an Istio install, and also set up the Istio CNI plugin.
The Istio CNI plugin removes the need for each application pod to have a privileged `NET_ADMIN` container. For further information, see the [Istio CNI Plugin docs](https://istio.io/docs/setup/additional-setup/cni). Please note that the [Istio CNI Plugin is in alpha](https://istio.io/about/feature-stages/).
:::note Prerequisites:
- The cluster must be an RKE Kubernetes cluster.
- The cluster must have been created with a default PodSecurityPolicy.
To enable pod security policy support when creating a Kubernetes cluster in the Rancher UI, go to <b>Advanced Options.</b> In the <b>Pod Security Policy Support</b> section, click <b>Enabled.</b> Then select a default pod security policy.
:::
1. [Set the PodSecurityPolicy to unrestricted](#1-set-the-podsecuritypolicy-to-unrestricted)
2. [Enable the CNI](#2-enable-the-cni)
3. [Verify that the CNI is working.](#3-verify-that-the-cni-is-working)
### 1. Set the PodSecurityPolicy to unrestricted
An unrestricted PSP allows Istio to be installed.
Set the PSP to `unrestricted` in the project where is Istio is installed, or the project where you plan to install Istio.
1. Click **☰ > Cluster Management**.
1. Go to the cluster that you created and click **Explore**.
1. Click **Cluster > Projects/Namespaces**.
1. Find the **Project: System** and select the **⋮ > Edit Config**.
1. Change the Pod Security Policy option to be unrestricted, then click **Save**.
### 2. Enable the CNI
When installing or upgrading Istio through **Apps & Marketplace,**
1. Click **Components**.
2. Check the box next to **Enabled CNI**.
3. Finish installing or upgrading Istio.
The CNI can also be enabled by editing the `values.yaml`:
```
istio_cni.enabled: true
```
Istio should install successfully with the CNI enabled in the cluster.
### 3. Verify that the CNI is working
Verify that the CNI is working by deploying a [sample application](https://istio.io/latest/docs/examples/bookinfo/) or deploying one of your own applications.
docs/explanations/integrations-in-rancher/istio/configuration-options/project-network-isolation.md
+22
View File
@@ -0,0 +1,22 @@
---
title: Additional Steps for Project Network Isolation
weight: 4
---
In clusters where:
- You are using the Canal network plugin with Rancher before v2.5.8, or you are using Rancher v2.5.8+ with an any RKE network plug-in that supports the enforcement of Kubernetes network policies, such as Canal or the Cisco ACI plugin
- The Project Network Isolation option is enabled
- You install the Istio Ingress module
The Istio Ingress Gateway pod won't be able to redirect ingress traffic to the workloads by default. This is because all the namespaces will be inaccessible from the namespace where Istio is installed. You have two options.
The first option is to add a new Network Policy in each of the namespaces where you intend to have ingress controlled by Istio. Your policy should include the following lines:
```
- podSelector:
matchLabels:
app: istio-ingressgateway
```
The second option is to move the `istio-system` namespace to the `system` project, which by default is excluded from the network isolation.
docs/explanations/integrations-in-rancher/istio/configuration-options/selectors-and-scrape-configurations.md
+125
View File
@@ -0,0 +1,125 @@
---
title: Selectors and Scrape Configs
weight: 2
---
The Monitoring app sets `prometheus.prometheusSpec.ignoreNamespaceSelectors=false`, which enables monitoring across all namespaces by default.
This ensures you can view traffic, metrics and graphs for resources deployed in a namespace with `istio-injection=enabled` label.
If you would like to limit Prometheus to specific namespaces, set `prometheus.prometheusSpec.ignoreNamespaceSelectors=true`. Once you do this, you will need to add additional configuration to continue to monitor your resources.
- [Limiting Monitoring to Specific Namespaces by Setting ignoreNamespaceSelectors to True](#limiting-monitoring-to-specific-namespaces-by-setting-ignorenamespaceselectors-to-true)
- [Enabling Prometheus to Detect Resources in Other Namespaces](#enabling-prometheus-to-detect-resources-in-other-namespaces)
- [Monitoring Specific Namespaces: Create a Service Monitor or Pod Monitor](#monitoring-specific-namespaces-create-a-service-monitor-or-pod-monitor)
- [Monitoring Across Namespaces: Set ignoreNamespaceSelectors to False](#monitoring-across-namespaces-set-ignorenamespaceselectors-to-false)
### Limiting Monitoring to Specific Namespaces by Setting ignoreNamespaceSelectors to True
To limit monitoring to specific namespaces, you will edit the `ignoreNamespaceSelectors` Helm chart option. You will configure this option when installing or upgrading the Monitoring Helm chart:
1. When installing or upgrading the Monitoring Helm chart, edit the values.yml and set`prometheus.prometheusSpec.ignoreNamespaceSelectors=true`.
1. Complete the install or upgrade.
**Result:** Prometheus will be limited to specific namespaces which means one of the following configurations will need to be set up to continue to view data in various dashboards
### Enabling Prometheus to Detect Resources in Other Namespaces
There are two different ways to enable Prometheus to detect resources in other namespaces when `prometheus.prometheusSpec.ignoreNamespaceSelectors=true`:
- **Monitoring specific namespaces:** Add a Service Monitor or Pod Monitor in the namespace with the targets you want to scrape.
- **Monitoring across namespaces:** Add an `additionalScrapeConfig` to your rancher-monitoring instance to scrape all targets in all namespaces.
### Monitoring Specific Namespaces: Create a Service Monitor or Pod Monitor
This option allows you to define which specific services or pods you would like monitored in a specific namespace.
The usability tradeoff is that you have to create the service monitor or pod monitor per namespace since you cannot monitor across namespaces.
:::note Prerequisite:
Define a ServiceMonitor or PodMonitor for `<your namespace>`. An example ServiceMonitor is provided below.
:::
1. Click **☰ > Cluster Management**.
1. Go to the cluster that you created and click **Explore**.
1. In the top navigation bar, open the kubectl shell.
1. If the ServiceMonitor or PodMonitor file is stored locally in your cluster, in `kubectl create -f <name of service/pod monitor file>.yaml`.
1. If the ServiceMonitor or PodMonitor is not stored locally, run `cat<< EOF | kubectl apply -f -`, paste the file contents into the terminal, then run `EOF` to complete the command.
1. Run `kubectl label namespace <your namespace> istio-injection=enabled` to enable the envoy sidecar injection.
**Result:** `<your namespace>` can be scraped by prometheus.
<figcaption>Example Service Monitor for Istio Proxies</figcaption>
```yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: envoy-stats-monitor
namespace: istio-system
labels:
monitoring: istio-proxies
spec:
selector:
matchExpressions:
- {key: istio-prometheus-ignore, operator: DoesNotExist}
namespaceSelector:
any: true
jobLabel: envoy-stats
endpoints:
- path: /stats/prometheus
targetPort: 15090
interval: 15s
relabelings:
- sourceLabels: [__meta_kubernetes_pod_container_port_name]
action: keep
regex: '.*-envoy-prom'
- action: labeldrop
regex: "__meta_kubernetes_pod_label_(.+)"
- sourceLabels: [__meta_kubernetes_namespace]
action: replace
targetLabel: namespace
- sourceLabels: [__meta_kubernetes_pod_name]
action: replace
targetLabel: pod_name
```
### Monitoring across namespaces: Set ignoreNamespaceSelectors to False
This enables monitoring across namespaces by giving Prometheus additional scrape configurations.
The usability tradeoff is that all of Prometheus' `additionalScrapeConfigs` are maintained in a single Secret. This could make upgrading difficult if monitoring is already deployed with additionalScrapeConfigs before installing Istio.
1. When installing or upgrading the Monitoring Helm chart, edit the values.yml and set the `prometheus.prometheusSpec.additionalScrapeConfigs` array to the **Additional Scrape Config** provided below.
1. Complete the install or upgrade.
**Result:** All namespaces with the `istio-injection=enabled` label will be scraped by prometheus.
<figcaption>Additional Scrape Config</figcaption>
``` yaml
- job_name: 'istio/envoy-stats'
scrape_interval: 15s
metrics_path: /stats/prometheus
kubernetes_sd_configs:
- role: pod
relabel_configs:
- source_labels: [__meta_kubernetes_pod_container_port_name]
action: keep
regex: '.*-envoy-prom'
- source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
action: replace
regex: ([^:]+)(?::\d+)?;(\d+)
replacement: $1:15090
target_label: __address__
- action: labelmap
regex: __meta_kubernetes_pod_label_(.+)
- source_labels: [__meta_kubernetes_namespace]
action: replace
target_label: namespace
- source_labels: [__meta_kubernetes_pod_name]
action: replace
target_label: pod_name
```
docs/explanations/integrations-in-rancher/istio/cpu-and-memory-allocations.md
+64
View File
@@ -0,0 +1,64 @@
---
title: CPU and Memory Allocations
weight: 1
---
This section describes the minimum recommended computing resources for the Istio components in a cluster.
The CPU and memory allocations for each component are [configurable.](#configuring-resource-allocations)
Before enabling Istio, we recommend that you confirm that your Rancher worker nodes have enough CPU and memory to run all of the components of Istio.
:::tip
In larger deployments, it is strongly advised that the infrastructure be placed on dedicated nodes in the cluster by adding a node selector for each Istio component.
:::
The table below shows a summary of the minimum recommended resource requests and limits for the CPU and memory of each core Istio component.
In Kubernetes, the resource request indicates that the workload will not deployed on a node unless the node has at least the specified amount of memory and CPU available. If the workload surpasses the limit for CPU or memory, it can be terminated or evicted from the node. For more information on managing resource limits for containers, refer to the [Kubernetes documentation.](https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/)
| Workload | CPU - Request | Memory - Request | CPU - Limit | Memory - Limit |
|----------------------|---------------|------------|-----------------|-------------------|
| ingress gateway | 100m | 128mi | 2000m | 1024mi |
| egress gateway | 100m | 128mi | 2000m | 1024mi |
| istiod | 500m | 2048mi | No limit | No limit |
| proxy | 10m | 10mi | 2000m | 1024mi |
| **Totals:** | **710m** | **2314Mi** | **6000m** | **3072Mi** |
# Configuring Resource Allocations
You can individually configure the resource allocation for each type of Istio component. This section includes the default resource allocations for each component.
To make it easier to schedule the workloads to a node, a cluster-admin can reduce the CPU and memory resource requests for the component. However, the default CPU and memory allocations are the minimum that we recommend.
You can find more information about Istio configuration in the [official Istio documentation](https://istio.io/).
To configure the resources allocated to an Istio component,
1. Click **☰ > Cluster Management**.
1. Go to the cluster that you created and click **Explore**.
1. In the left navigation bar, click **Apps & Marketplace**.
1. Click **Installed Apps**.
1. Go to the `istio-system` namespace. In one of the Istio workloads, such as `rancher-istio`, click **⋮ > Edit/Upgrade**.
1. Click **Upgrade** to edit the base components via changes to the values.yaml or add an [overlay file](../../../pages-for-subheaders/configuration-options.md#overlay-file). For more information about editing the overlay file, see [this section.](cpu-and-memory-allocations.md#editing-the-overlay-file)
1. Change the CPU or memory allocations, the nodes where each component will be scheduled to, or the node tolerations.
1. Click **Upgrade**. to rollout changes
**Result:** The resource allocations for the Istio components are updated.
### Editing the Overlay File
The overlay file can contain any of the values in the [Istio Operator spec.](https://istio.io/latest/docs/reference/config/istio.operator.v1alpha1/#IstioOperatorSpec) The overlay file included with the Istio application is just one example of a potential configuration of the overlay file.
As long as the file contains `kind: IstioOperator` and the YAML options are valid, the file can be used as an overlay.
In the example overlay file provided with the Istio application, the following section allows you to change Kubernetes resources:
```
# k8s:
# resources:
# requests:
# cpu: 200m
```
docs/explanations/integrations-in-rancher/istio/disable-istio.md
+43
View File
@@ -0,0 +1,43 @@
---
title: Disabling Istio
weight: 4
---
This section describes how to uninstall Istio in a cluster or disable a namespace, or workload.
# Uninstall Istio in a Cluster
To uninstall Istio,
1. Click **☰ > Cluster Management**.
1. Go to the cluster that you created and click **Explore**.
1. In the left navigation bar, click **Apps & Marketplace > Installed Apps**.
1. In the `istio-system` namespace, go to `rancher-istio` and click **⋮ > Delete**.
1. After `rancher-istio` is deleted, you can then select all the remaining apps in the `istio-system` namespace and click **Delete**.
**Result:** The `rancher-istio` app in the cluster gets removed. The Istio sidecar cannot be deployed on any workloads in the cluster.
:::note
You can no longer disable and re-enable your Istio installation. If you would like to save your settings for a future install, view and save individual YAMLs to refer back to / reuse for future installations.
:::
**Troubleshooting Uninstall:** If you didn't follow the uninstall steps, you may encounter a warning during uninstall:
`Error: uninstallation completed with 1 error(s): unable to build kubernetes objects for delete: unable to recognize "": no matches for kind "MonitoringDashboard" in version "monitoring.kiali.io/v1alpha1"`
This could mean a few things. You either selected all the apps in the `istio-system` namespace and deleted them at the same time, or you deleted `rancher-istio` chart dependencies prior to deleting the `rancher-istio` chart. Since the uninstall did not complete properly, you will have resources remaining in the `istio-system` namespace that you will need to manually clean up. Another option to avoid manual clean up is to install `rancher-istio` again, then uninstall it in the correct order.
# Disable Istio in a Namespace
1. Click **☰ > Cluster Management**.
1. Go to the cluster that you created and click **Explore**.
1. Click **Cluster > Projects/Namespaces**.
1. Go to the namespace where you want to enable Istio and click **⋮ > Enable Istio Auto Injection**. Alternately, click the namespace, and then on the namespace detail page, click **⋮ > Enable Istio Auto Injection**.
**Result:** When workloads are deployed in this namespace, they will not have the Istio sidecar.
# Remove the Istio Sidecar from a Workload
Disable Istio in the namespace, then redeploy the workloads with in it. They will be deployed without the Istio sidecar.
docs/explanations/integrations-in-rancher/istio/rbac-for-istio.md
+44
View File
@@ -0,0 +1,44 @@
---
title: Role-based Access Control
weight: 3
---
This section describes the permissions required to access Istio features.
The rancher istio chart installs three `ClusterRoles`
## Cluster-Admin Access
By default, only those with the `cluster-admin` `ClusterRole` can:
- Install istio app in a cluster
- Configure resource allocations for Istio
## Admin and Edit access
By default, only Admin and Edit roles can:
- Enable and disable Istio sidecar auto-injection for namespaces
- Add the Istio sidecar to workloads
- View the traffic metrics and traffic graph for the cluster
- Configure Istio's resources (such as the gateway, destination rules, or virtual services)
## Summary of Default Permissions for Kubernetes Default roles
Istio creates three `ClusterRoles` and adds Istio CRD access to the following default K8s `ClusterRole`:
ClusterRole create by chart | Default K8s ClusterRole | Rancher Role |
------------------------------:| ---------------------------:|---------:|
`istio-admin` | admin| Project Owner |
`istio-edit`| edit | Project Member |
`istio-view` | view | Read-only |
Rancher will continue to use cluster-owner, cluster-member, project-owner, project-member, etc as role names, but will utilize default roles to determine access. For each default K8s `ClusterRole` there are different Istio CRD permissions and K8s actions (Create ( C ), Get ( G ), List ( L ), Watch ( W ), Update ( U ), Patch ( P ), Delete( D ), All ( * )) that can be performed.
|CRDs | Admin | Edit | View
|----------------------------| ------| -----| -----
| <ul><li>`config.istio.io`</li><ul><li>`adapters`</li><li>`attributemanifests`</li><li>`handlers`</li><li>`httpapispecbindings`</li><li>`httpapispecs`</li><li>`instances`</li><li>`quotaspecbindings`</li><li>`quotaspecs`</li><li>`rules`</li><li>`templates`</li></ul></ul>| GLW | GLW | GLW
|<ul><li>`networking.istio.io`</li><ul><li>`destinationrules`</li><li>`envoyfilters`</li><li>`gateways`</li><li>`serviceentries`</li><li>`sidecars`</li><li>`virtualservices`</li><li>`workloadentries`</li></ul></ul>| * | * | GLW
|<ul><li>`security.istio.io`</li><ul><li>`authorizationpolicies`</li><li>`peerauthentications`</li><li>`requestauthentications`</li></ul></ul>| * | * | GLW
docs/explanations/integrations-in-rancher/logging/custom-resource-configuration/flows-and-clusterflows.md
+85
View File
@@ -0,0 +1,85 @@
---
title: Flows and ClusterFlows
weight: 1
---
For the full details on configuring `Flows` and `ClusterFlows`, see the [Banzai Cloud Logging operator documentation.](https://banzaicloud.com/docs/one-eye/logging-operator/configuration/flow/)
- [Configuration](#configuration)
- [YAML Example](#yaml-example)
# Configuration
- [Flows](#flows)
- [Matches](#matches)
- [Filters](#filters)
- [Outputs](#outputs)
- [ClusterFlows](#clusterflows)
# Flows
A `Flow` defines which logs to collect and filter and which output to send the logs to.
The `Flow` is a namespaced resource, which means logs will only be collected from the namespace that the `Flow` is deployed in.
`Flows` can be configured by filling out forms in the Rancher UI.
For more details about the `Flow` custom resource, see [FlowSpec.](https://banzaicloud.com/docs/one-eye/logging-operator/configuration/crds/v1beta1/flow_types/)
### Matches
Match statements are used to select which containers to pull logs from.
You can specify match statements to select or exclude logs according to Kubernetes labels, container and host names. Match statements are evaluated in the order they are defined and processed only until the first matching select or exclude rule applies.
Matches can be configured by filling out the `Flow` or `ClusterFlow` forms in the Rancher UI.
For detailed examples on using the match statement, see the [official documentation on log routing.](https://banzaicloud.com/docs/one-eye/logging-operator/configuration/log-routing/)
### Filters
You can define one or more filters within a `Flow`. Filters can perform various actions on the logs, for example, add additional data, transform the logs, or parse values from the records. The filters in the `Flow` are applied in the order in the definition.
For a list of filters supported by the Banzai Cloud Logging operator, see [this page.](https://banzaicloud.com/docs/one-eye/logging-operator/configuration/plugins/filters/)
Filters need to be configured in YAML.
### Outputs
This `Output` will receive logs from the `Flow`. Because the `Flow` is a namespaced resource, the `Output` must reside in same namespace as the `Flow`.
`Outputs` can be referenced when filling out the `Flow` or `ClusterFlow` forms in the Rancher UI.
# ClusterFlows
Matches, filters and `Outputs` are configured for `ClusterFlows` in the same way that they are configured for `Flows`. The key difference is that the `ClusterFlow` is scoped at the cluster level and can configure log collection across all namespaces.
`ClusterFlows` can be configured by filling out forms in the Rancher UI.
After `ClusterFlow` selects logs from all namespaces in the cluster, logs from the cluster will be collected and logged to the selected `ClusterOutput`.
# YAML Example
The following example `Flow` transforms the log messages from the default namespace and sends them to an S3 `Output`:
```yaml
apiVersion: logging.banzaicloud.io/v1beta1
kind: Flow
metadata:
name: flow-sample
namespace: default
spec:
filters:
- parser:
remove_key_name_field: true
parse:
type: nginx
- tag_normaliser:
format: ${namespace_name}.${pod_name}.${container_name}
localOutputRefs:
- s3-output
match:
- select:
labels:
app: nginx
```
docs/explanations/integrations-in-rancher/logging/custom-resource-configuration/outputs-and-clusteroutputs.md
+306
View File
@@ -0,0 +1,306 @@
---
title: Outputs and ClusterOutputs
weight: 2
---
For the full details on configuring `Outputs` and `ClusterOutputs`, see the [Banzai Cloud Logging operator documentation.](https://banzaicloud.com/docs/one-eye/logging-operator/configuration/output/)
- [Configuration](#configuration)
- [YAML Examples](#yaml-examples)
- [Cluster Output to ElasticSearch](#cluster-output-to-elasticsearch)
- [Output to Splunk](#output-to-splunk)
- [Output to Syslog](#output-to-syslog)
- [Unsupported Outputs](#unsupported-outputs)
# Configuration
- [Outputs](#outputs)
- [ClusterOutputs](#clusteroutputs)
# Outputs
The `Output` resource defines where your `Flows` can send the log messages. `Outputs` are the final stage for a logging `Flow`.
The `Output` is a namespaced resource, which means only a `Flow` within the same namespace can access it.
You can use secrets in these definitions, but they must also be in the same namespace.
`Outputs` can be configured by filling out forms in the Rancher UI.
For the details of `Output` custom resource, see [OutputSpec.](https://banzaicloud.com/docs/one-eye/logging-operator/configuration/crds/v1beta1/output_types/)
The Rancher UI provides forms for configuring the following `Output` types:
- Amazon ElasticSearch
- Azure Storage
- Cloudwatch
- Datadog
- Elasticsearch
- File
- Fluentd
- GCS
- Kafka
- Kinesis Stream
- LogDNA
- LogZ
- Loki
- New Relic
- Splunk
- SumoLogic
- Syslog
The Rancher UI provides forms for configuring the `Output` type, target, and access credentials if applicable.
For example configuration for each logging plugin supported by the logging operator, see the [logging operator documentation.](https://banzaicloud.com/docs/one-eye/logging-operator/configuration/plugins/outputs/)
# ClusterOutputs
`ClusterOutput` defines an `Output` without namespace restrictions. It is only effective when deployed in the same namespace as the logging operator.
`ClusterOutputs` can be configured by filling out forms in the Rancher UI.
For the details of the `ClusterOutput` custom resource, see [ClusterOutput.](https://banzaicloud.com/docs/one-eye/logging-operator/configuration/crds/v1beta1/clusteroutput_types/)
# YAML Examples
Once logging is installed, you can use these examples to help craft your own logging pipeline.
- [Cluster Output to ElasticSearch](#cluster-output-to-elasticsearch)
- [Output to Splunk](#output-to-splunk)
- [Output to Syslog](#output-to-syslog)
- [Unsupported Outputs](#unsupported-outputs)
### Cluster Output to ElasticSearch
Let's say you wanted to send all logs in your cluster to an `elasticsearch` cluster. First, we create a cluster `Output`.
```yaml
apiVersion: logging.banzaicloud.io/v1beta1
kind: ClusterOutput
metadata:
name: "example-es"
namespace: "cattle-logging-system"
spec:
elasticsearch:
host: elasticsearch.example.com
port: 9200
scheme: http
```
We have created this `ClusterOutput`, without elasticsearch configuration, in the same namespace as our operator: `cattle-logging-system.`. Any time we create a `ClusterFlow` or `ClusterOutput`, we have to put it in the `cattle-logging-system` namespace.
Now that we have configured where we want the logs to go, let's configure all logs to go to that `ClusterOutput`.
```yaml
apiVersion: logging.banzaicloud.io/v1beta1
kind: ClusterFlow
metadata:
name: "all-logs"
namespace: "cattle-logging-system"
spec:
globalOutputRefs:
- "example-es"
```
We should now see our configured index with logs in it.
### Output to Splunk
What if we have an application team who only wants logs from a specific namespaces sent to a `splunk` server? For this case, we can use namespaced `Outputs` and `Flows`.
Before we start, let's set up that team's application: `coolapp`.
```yaml
apiVersion: v1
kind: Namespace
metadata:
name: devteam
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: coolapp
namespace: devteam
labels:
app: coolapp
spec:
replicas: 2
selector:
matchLabels:
app: coolapp
template:
metadata:
labels:
app: coolapp
spec:
containers:
- name: generator
image: paynejacob/loggenerator:latest
```
With `coolapp` running, we will follow a similar path as when we created a `ClusterOutput`. However, unlike `ClusterOutputs`, we create our `Output` in our application's namespace.
```yaml
apiVersion: logging.banzaicloud.io/v1beta1
kind: Output
metadata:
name: "devteam-splunk"
namespace: "devteam"
spec:
splunkHec:
hec_host: splunk.example.com
hec_port: 8088
protocol: http
```
Once again, let's feed our `Output` some logs:
```yaml
apiVersion: logging.banzaicloud.io/v1beta1
kind: Flow
metadata:
name: "devteam-logs"
namespace: "devteam"
spec:
localOutputRefs:
- "devteam-splunk"
```
### Output to Syslog
Let's say you wanted to send all logs in your cluster to an `syslog` server. First, we create a `ClusterOutput`:
```yaml
apiVersion: logging.banzaicloud.io/v1beta1
kind: ClusterOutput
metadata:
name: "example-syslog"
namespace: "cattle-logging-system"
spec:
syslog:
buffer:
timekey: 30s
timekey_use_utc: true
timekey_wait: 10s
flush_interval: 5s
format:
type: json
app_name_field: test
host: syslog.example.com
insecure: true
port: 514
transport: tcp
```
Now that we have configured where we want the logs to go, let's configure all logs to go to that `Output`.
```yaml
apiVersion: logging.banzaicloud.io/v1beta1
kind: ClusterFlow
metadata:
name: "all-logs"
namespace: cattle-logging-system
spec:
globalOutputRefs:
- "example-syslog"
```
### Unsupported Outputs
For the final example, we create an `Output` to write logs to a destination that is not supported out of the box:
:::note Note on syslog:
`syslog` is a supported `Output`. However, this example still provides an overview on using unsupported plugins.
:::
```yaml
apiVersion: v1
kind: Secret
metadata:
name: syslog-config
namespace: cattle-logging-system
type: Opaque
stringData:
fluent-bit.conf: |
[INPUT]
Name forward
Port 24224
[OUTPUT]
Name syslog
InstanceName syslog-output
Match *
Addr syslog.example.com
Port 514
Cluster ranchers
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: fluentbit-syslog-forwarder
namespace: cattle-logging-system
labels:
output: syslog
spec:
selector:
matchLabels:
output: syslog
template:
metadata:
labels:
output: syslog
spec:
containers:
- name: fluentbit
image: paynejacob/fluent-bit-out-syslog:latest
ports:
- containerPort: 24224
volumeMounts:
- mountPath: "/fluent-bit/etc/"
name: configuration
volumes:
- name: configuration
secret:
secretName: syslog-config
---
apiVersion: v1
kind: Service
metadata:
name: syslog-forwarder
namespace: cattle-logging-system
spec:
selector:
output: syslog
ports:
- protocol: TCP
port: 24224
targetPort: 24224
---
apiVersion: logging.banzaicloud.io/v1beta1
kind: ClusterFlow
metadata:
name: all-logs
namespace: cattle-logging-system
spec:
globalOutputRefs:
- syslog
---
apiVersion: logging.banzaicloud.io/v1beta1
kind: ClusterOutput
metadata:
name: syslog
namespace: cattle-logging-system
spec:
forward:
servers:
- host: "syslog-forwarder.cattle-logging-system"
require_ack_response: false
ignore_network_errors_at_startup: false
```
Let's break down what is happening here. First, we create a deployment of a container that has the additional `syslog` plugin and accepts logs forwarded from another `fluentd`. Next we create an `Output` configured as a forwarder to our deployment. The deployment `fluentd` will then forward all logs to the configured `syslog` destination.
docs/explanations/integrations-in-rancher/logging/logging-architecture.md
+29
View File
@@ -0,0 +1,29 @@
---
title: Architecture
weight: 1
---
This section summarizes the architecture of the Rancher logging application.
For more details about how the Banzai Cloud Logging operator works, see the [official documentation.](https://banzaicloud.com/docs/one-eye/logging-operator/#architecture)
### How the Banzai Cloud Logging Operator Works
The Logging operator automates the deployment and configuration of a Kubernetes logging pipeline. It deploys and configures a Fluent Bit DaemonSet on every node to collect container and application logs from the node file system.
Fluent Bit queries the Kubernetes API and enriches the logs with metadata about the pods, and transfers both the logs and the metadata to Fluentd. Fluentd receives, filters, and transfers logs to multiple `Outputs`.
The following custom resources are used to define how logs are filtered and sent to their `Outputs`:
- A `Flow` is a namespaced custom resource that uses filters and selectors to route log messages to the appropriate `Outputs`.
- A `ClusterFlow` is used to route cluster-level log messages.
- An `Output` is a namespaced resource that defines where the log messages are sent.
- A `ClusterOutput` defines an `Output` that is available from all `Flows` and `ClusterFlows`.
Each `Flow` must reference an `Output`, and each `ClusterFlow` must reference a `ClusterOutput`.
The following figure from the [Banzai documentation](https://banzaicloud.com/docs/one-eye/logging-operator/#architecture) shows the new logging architecture:
<figcaption>How the Banzai Cloud Logging Operator Works with Fluentd and Fluent Bit</figcaption>
![How the Banzai Cloud Logging Operator Works with Fluentd](/img/banzai-cloud-logging-operator.png)
docs/explanations/integrations-in-rancher/logging/logging-helm-chart-options.md
+103
View File
@@ -0,0 +1,103 @@
---
title: rancher-logging Helm Chart Options
shortTitle: Helm Chart Options
weight: 4
---
- [Enable/Disable Windows Node Logging](#enable-disable-windows-node-logging)
- [Working with a Custom Docker Root Directory](#working-with-a-custom-docker-root-directory)
- [Adding NodeSelector Settings and Tolerations for Custom Taints](#adding-nodeselector-settings-and-tolerations-for-custom-taints)
- [Enabling the Logging Application to Work with SELinux](#enabling-the-logging-application-to-work-with-selinux)
- [Additional Logging Sources](#additional-logging-sources)
- [Systemd Configuration](#systemd-configuration)
### Enable/Disable Windows Node Logging
You can enable or disable Windows node logging by setting `global.cattle.windows.enabled` to either `true` or `false` in the `values.yaml`.
By default, Windows node logging will be enabled if the Cluster Dashboard UI is used to install the logging application on a Windows cluster.
In this scenario, setting `global.cattle.windows.enabled` to `false` will disable Windows node logging on the cluster.
When disabled, logs will still be collected from Linux nodes within the Windows cluster.
:::note
Currently an [issue](https://github.com/rancher/rancher/issues/32325) exists where Windows nodeAgents are not deleted when performing a `helm upgrade` after disabling Windows logging in a Windows cluster. In this scenario, users may need to manually remove the Windows nodeAgents if they are already installed.
:::
### Working with a Custom Docker Root Directory
If using a custom Docker root directory, you can set `global.dockerRootDirectory` in `values.yaml`.
This will ensure that the Logging CRs created will use your specified path rather than the default Docker `data-root` location.
Note that this only affects Linux nodes.
If there are any Windows nodes in the cluster, the change will not be applicable to those nodes.
### Adding NodeSelector Settings and Tolerations for Custom Taints
You can add your own `nodeSelector` settings and add `tolerations` for additional taints by editing the logging Helm chart values. For details, see [this page.](taints-and-tolerations.md)
### Enabling the Logging Application to Work with SELinux
:::note Requirements:
Logging v2 was tested with SELinux on RHEL/CentOS 7 and 8.
:::
[Security-Enhanced Linux (SELinux)](https://en.wikipedia.org/wiki/Security-Enhanced_Linux) is a security enhancement to Linux. After being historically used by government agencies, SELinux is now industry standard and is enabled by default on CentOS 7 and 8.
To use Logging v2 with SELinux, we recommend installing the `rancher-selinux` RPM according to the instructions on [this page.](../../../pages-for-subheaders/selinux-rpm.md#installing-the-rancher-selinux-rpm)
Then, when installing the logging application, configure the chart to be SELinux aware by changing `global.seLinux.enabled` to `true` in the `values.yaml`.
### Additional Logging Sources
By default, Rancher collects logs for [control plane components](https://kubernetes.io/docs/concepts/overview/components/#control-plane-components) and [node components](https://kubernetes.io/docs/concepts/overview/components/#node-components) for all cluster types.
In some cases, Rancher may be able to collect additional logs.
The following table summarizes the sources where additional logs may be collected for each node types:
| Logging Source | Linux Nodes (including in Windows cluster) | Windows Nodes |
| --- | --- | ---|
| RKE | ✓ | ✓ |
| RKE2 | ✓ | |
| K3s | ✓ | |
| AKS | ✓ | |
| EKS | ✓ | |
| GKE | ✓ | |
To enable hosted Kubernetes providers as additional logging sources, enable **Enable enhanced cloud provider logging** option when installing or upgrading the Logging Helm chart.
When enabled, Rancher collects all additional node and control plane logs the provider has made available, which may vary between providers
If you're already using a cloud provider's own logging solution such as AWS CloudWatch or Google Cloud operations suite (formerly Stackdriver), it is not necessary to enable this option as the native solution will have unrestricted access to all logs.
### Systemd Configuration
In Rancher logging, `SystemdLogPath` must be configured for K3s and RKE2 Kubernetes distributions.
K3s and RKE2 Kubernetes distributions log to journald, which is the subsystem of systemd that is used for logging. In order to collect these logs, the `systemdLogPath` needs to be defined. While the `run/log/journal` directory is used by default, some Linux distributions do not default to this path. For example, Ubuntu defaults to `var/log/journal`. To determine your `systemdLogPath` configuration, see steps below.
**Steps for Systemd Configuration:**
* Run `cat /etc/systemd/journald.conf | grep -E ^\#?Storage | cut -d"=" -f2` on one of your nodes.
* If `persistent` is returned, your `systemdLogPath` should be `/var/log/journal`.
* If `volatile` is returned, your `systemdLogPath` should be `/run/log/journal`.
* If `auto` is returned, check if `/var/log/journal` exists.
* If `/var/log/journal` exists, then use `/var/log/journal`.
* If `/var/log/journal` does not exist, then use `/run/log/journal`.
:::note Notes:
If any value not described above is returned, Rancher Logging will not be able to collect control plane logs. To address this issue, you will need to perform the following actions on every control plane node:
* Set `Storage=volatile` in journald.conf.
* Reboot your machine.
* Set `systemdLogPath` to `/run/log/journal`.
:::
docs/explanations/integrations-in-rancher/logging/migrate-to-rancher-v2.5+-logging.md
+202
View File
@@ -0,0 +1,202 @@
---
title: Migrating to Rancher v2.5 Logging
weight: 2
---
Starting in v2.5, the logging feature available within Rancher has been completely overhauled. The [logging operator](https://github.com/banzaicloud/logging-operator) from Banzai Cloud has been adopted; Rancher configures this tooling for use when deploying logging.
Among the many features and changes in the new logging functionality is the removal of project-specific logging configurations. Instead, one now configures logging at the namespace level. Cluster-level logging remains available, but configuration options differ.
- [Installation](#installation)
- [Terminology](#terminology)
- [Cluster Logging](#cluster-logging)
- [Project Logging](#project-logging)
- [Output Configuration](#output-configuration)
- [Elasticsearch](#elasticsearch)
- [Splunk](#splunk)
- [Kafka](#kafka)
- [Fluentd](#fluentd)
- [Syslog](#syslog)
- [Custom Log Fields](#custom-log-fields)
- [System Logging](#system-logging)
# Installation
To install logging in Rancher v2.5+, refer to the [installation instructions](../../../pages-for-subheaders/logging.md#enabling-logging).
### Terminology
In v2.5+, logging configuration in the **Cluster Dashboard**. To configure logging custom resources after the Logging application is installed, go to the left navigation bar and click **Logging**. It is from this menu option that logging for both cluster and namespace is configured.
:::note
Logging is installed on a per-cluster basis. You will need to navigate between clusters to configure logging for each cluster.
:::
There are four key concepts to understand for v2.5+ logging:
1. Outputs
`Outputs` are a configuration resource that determine a destination for collected logs. This is where settings for aggregators such as ElasticSearch, Kafka, etc. are stored. `Outputs` are namespaced resources.
2. Flows
`Flows` are a configuration resource that determine collection, filtering, and destination rules for logs. It is within a flow that one will configure what logs to collect, how to mutate or filter them, and which `Outputs` to send the logs to. `Flows` are namespaced resources, and can connect either to an `Output` in the same namespace, or a `ClusterOutput`.
3. ClusterOutputs
`ClusterOutputs` serve the same functionality as `Outputs`, except they are a cluster-scoped resource. `ClusterOutputs` are necessary when collecting logs cluster-wide, or if you wish to provide an `Output` to all namespaces in your cluster.
4. ClusterFlows
`ClusterFlows` serve the same function as `Flows`, but at the cluster level. They are used to configure log collection for an entire cluster, instead of on a per-namespace level. `ClusterFlows` are also where mutations and filters are defined, same as `Flows` (in functionality).
# Cluster Logging
To configure cluster-wide logging for v2.5+ logging, one needs to set up a `ClusterFlow`. This object defines the source of logs, any transformations or filters to be applied, and finally the `Output` (or `Outputs`) for the logs.
:::note Important:
`ClusterFlows` must be defined within the `cattle-logging-system` namespace. `ClusterFlows` will not work if defined in any other namespace.
:::
In legacy logging, in order to collect logs from across the entire cluster, one only needed to enable cluster-level logging and define the desired `Output`. This basic approach remains in v2.5+ logging. To replicate legacy cluster-level logging, follow these steps:
1. Define a `ClusterOutput` according to the instructions found under [Output Configuration](#output-configuration)
2. Create a `ClusterFlow`, ensuring that it is set to be created in the `cattle-logging-system` namespace
1. Remove all _Include_ and _Exclude_ rules from the `Flow` definition. This ensures that all logs are gathered.
2. You do not need to configure any filters if you do not wish - default behavior does not require their creation
3. Define your cluster `Output` or `Outputs`
This will result in logs from all sources in the cluster (all pods, and all system components) being collected and sent to the `Output` or `Outputs` you defined in the `ClusterFlow`.
# Project Logging
Logging in v2.5+ is not project-aware. This means that in order to collect logs from pods running in project namespaces, you will need to define `Flows` for those namespaces.
To collect logs from a specific namespace, follow these steps:
1. Define an `Output` or `ClusterOutput` according to the instructions found under [Output Configuration](#output-configuration)
2. Create a `Flow`, ensuring that it is set to be created in the namespace in which you want to gather logs.
1. If you wish to define _Include_ or _Exclude_ rules, you may do so. Otherwise, removal of all rules will result in all pods in the target namespace having their logs collected.
2. You do not need to configure any filters if you do not wish - default behavior does not require their creation
3. Define your outputs - these can be either `ClusterOutput` or `Output` objects.
This will result in logs from all sources in the namespace (pods) being collected and sent to the `Output` (or `Outputs`) you defined in your `Flow`.
:::note
To collect logs from a project, repeat the above steps for every namespace within the project. Alternatively, you can label your project workloads with a common label (e.g. `project=my-project`) and use a `ClusterFlow` to collect logs from all pods matching this label.
:::
# Output Configuration
In legacy logging, there are five logging destinations to choose from: Elasticsearch, Splunk, Kafka, Fluentd, and Syslog. With the exception of Syslog, all of these destinations are available in logging v2.5+.
### Elasticsearch
| Legacy Logging | v2.5+ Logging | Notes |
|-----------------------------------------------|-----------------------------------|-----------------------------------------------------------|
| Endpoint | Target -> Host | Make sure to specify Scheme (https/http), as well as Port |
| X-Pack Security -> Username | Access -> User | |
| X-Pack Security -> Password | Access -> Password | Password must now be stored in a secret |
| SSL Configuration -> Client Private Key | SSL -> Client Key | Key must now be stored in a secret |
| SSL Configuration -> Client Certificate | SSL -> Client Cert | Certificate must now be stored in a secret |
| SSL Configuration -> Client Key Password | SSL -> Client Key Pass | Password must now be stored in a secret |
| SSL Configuration -> Enabled SSL Verification | SSL -> Certificate Authority File | Certificate must now be stored in a secret |
In legacy logging, indices were automatically created according to the format in the "Index Patterns" section. In v2.5 logging, default behavior has been changed to logging to a single index. You can still configure index pattern functionality on the `Output` object by editing as YAML and inputting the following values:
```
...
spec:
elasticsearch:
...
logstash_format: true
logstash_prefix: <desired prefix>
logstash_dateformat: "%Y-%m-%d"
```
Replace `<desired prefix>` with the prefix for the indices that will be created. In legacy logging, this defaulted to the name of the cluster.
### Splunk
| Legacy Logging | v2.5+ Logging | Notes |
|------------------------------------------|----------------------------------------|----------------------------------------------------------------------------------------|
| HEC Configuration -> Endpoint | Target -> Host | Protocol (https/http) and port must be defined separately from the host |
| HEC Configuration -> Token | Access -> Token | Token must now be stored as a secret |
| HEC Configuration -> Index | Edit as YAML -> `index` | `index` field must be added as YAML key under `spec.splunkHec` |
| HEC Configuration -> Source | Edit as YAML -> `source` | `source` field must be added as YAML key under `spec.splunkHec` |
| SSL Configuration -> Client Private Key | Edit as YAML -> `client_key` | `client_key` field must be added as YAML key under `spec.splunkHec`. See (1) |
| SSL Configuration -> Client Certificate | Edit as YAML -> `client_cert` | `client_cert` field must be added as YAML key under `spec.splunkHec`. See (1) |
| SSL Configuration -> Client Key Password | _Not Supported_ | Specifying a password for the client private key is not currently supported. |
| SSL Configuration -> SSL Verify | Edit as YAML -> `ca_file` or `ca_path` | `ca_file` or `ca_path` field must be added as YAML key under `spec.splunkHec`. See (2) |
_(1) `client_key` and `client_cert` values must be paths to the key and cert files, respectively. These files must be mounted into the `rancher-logging-fluentd` pod in order to be used._
_(2) Users can configure either `ca_file` (a path to a PEM-encoded CA certificate) or `ca_path` (a path to a directory containing CA certificates in PEM format). These files must be mounted into the `rancher-logging-fluentd` pod in order to be used._
### Kafka
| Legacy Logging | v2.5+ Logging | Notes |
|-----------------------------------------|----------------------------|------------------------------------------------------|
| Kafka Configuration -> Endpoint Type | - | Zookeeper is no longer supported as an endpoint type |
| Kafka Configuration -> Endpoint | Target -> Brokers | Comma-separated list of brokers (host:port) |
| Kafka Configuration -> Topic | Target -> Default Topic | |
| SSL Configuration -> Client Private Key | SSL -> SSL Client Cert | Certificate must be stored as a secret |
| SSL Configuration -> Client Certificate | SSL -> SSL Client Cert Key | Key must be stored as a secret |
| SSL Configuration -> CA Certificate PEM | SSL -> SSL CA Cert | Certificate must be stored as a secret |
| SASL Configuration -> Username | Access -> Username | Username must be stored in a secret |
| SASL Configuration -> Password | Access -> Password | Password must be stored in a secret |
| SASL Configuration -> Scram Mechanism | Access -> Scram Mechanism | Input mechanism as string, e.g. "sha256" or "sha512" |
### Fluentd
As of v2.5.2, it is only possible to add a single Fluentd server using the "Edit as Form" option. To add multiple servers, edit the `Output` as YAML and input multiple servers.
| Legacy Logging | v2.5+ Logging | Notes |
|------------------------------------------|-----------------------------------------------------|----------------------------------------------------------------------|
| Fluentd Configuration -> Endpoint | Target -> Host, Port | Input the host and port separately |
| Fluentd Configuration -> Shared Key | Access -> Shared Key | Shared key must be stored as a secret |
| Fluentd Configuration -> Username | Access -> Username | Username must be stored as a secret |
| Fluentd Configuration -> Password | Access -> Password | Password must be stored as a secret |
| Fluentd Configuration -> Hostname | Edit as YAML -> `host` | `host` field set as YAML key under `spec.forward.servers[n]` |
| Fluentd Configuration -> Weight | Edit as YAML -> `weight` | `weight` field set as YAML key under `spec.forward.servers[n]` |
| SSL Configuration -> Use TLS | - | Do not need to explicitly enable. Define client cert fields instead. |
| SSL Configuration -> Client Private Key | Edit as YAML -> `tls_private_key_path` | Field set as YAML key under `spec.forward`. See (1) |
| SSL Configuration -> Client Certificate | Edit as YAML -> `tls_client_cert_path` | Field set as YAML key under `spec.forward`. See (1) |
| SSL Configuration -> Client Key Password | Edit as YAML -> `tls_client_private_key_passphrase` | Field set as YAML key under `spec.forward`. See (1) |
| SSL Configuration -> SSL Verify | Edit as YAML -> `tls_insecure_mode` | Field set as YAML key under `spec.forward`. Default: `false` |
| SSL Configuration -> CA Certificate PEM | Edit as YAML -> `tls_cert_path` | Field set as YAML key under `spec.forward`. See (1) |
| Enable Gzip Compression | - | No longer supported in v2.5+ logging |
_(1) These values are to be specified as paths to files. Those files must be mounted into the `rancher-logging-fluentd` pod in order to be used._
### Syslog
As of v2.5.2, syslog is not currently supported for `Outputs` using v2.5+ logging.
# Custom Log Fields
In order to add custom log fields, you will need to add the following YAML to your `Flow` configuration:
```
...
spec:
filters:
- record_modifier:
records:
- foo: "bar"
```
(replace `foo: "bar"` with custom log fields you wish to add)
# System Logging
In legacy logging, collecting logs from system components was accomplished by checking a box labeled "Include System Log" when setting up cluster logging. In v2.5+ logging, system logs are gathered in one of two ways:
1. Gather all cluster logs, not specifying any match or exclusion rules. This results in all container logs from the cluster being collected, which includes system logs.
2. Specifically target system logs by adding match rules for system components. Specific match rules depend on the component being collected.
docs/explanations/integrations-in-rancher/logging/rbac-for-logging.md
+25
View File
@@ -0,0 +1,25 @@
---
shortTitle: Role-based Access Control
title: Role-based Access Control for Logging
weight: 3
---
Rancher logging has two roles, `logging-admin` and `logging-view`.
- `logging-admin` gives users full access to namespaced `Flows` and `Outputs`
- `logging-view` allows users to *view* namespaced `Flows` and `Outputs`, and `ClusterFlows` and `ClusterOutputs`
:::note Why choose one role over the other?
Edit access to `ClusterFlow` and `ClusterOutput` resources is powerful. Any user with it has edit access for all logs in the cluster.
:::
In Rancher, the cluster administrator role is the only role with full access to all `rancher-logging` resources. Cluster members are not able to edit or read any logging resources. Project owners and members have the following privileges:
Project Owners | Project Members
--- | ---
able to create namespaced `Flows` and `Outputs` in their projects' namespaces | only able to view the `Flows` and `Outputs` in projects' namespaces
can collect logs from anything in their projects' namespaces | cannot collect any logs in their projects' namespaces
Both project owners and project members require at least *one* namespace in their project to use logging. If they do not, then they may not see the logging button in the top nav dropdown.
docs/explanations/integrations-in-rancher/logging/taints-and-tolerations.md
+66
View File
@@ -0,0 +1,66 @@
---
title: Working with Taints and Tolerations
weight: 6
---
"Tainting" a Kubernetes node causes pods to repel running on that node.
Unless the pods have a `toleration` for that node's taint, they will run on other nodes in the cluster.
[Taints and tolerations](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/) can work in conjunction with the `nodeSelector` [field](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#nodeselector) within the `PodSpec`, which enables the *opposite* effect of a taint.
Using `nodeSelector` gives pods an affinity towards certain nodes.
Both provide choice for the what node(s) the pod will run on.
- [Default Implementation in Rancher's Logging Stack](#default-implementation-in-rancher-s-logging-stack)
- [Adding NodeSelector Settings and Tolerations for Custom Taints](#adding-nodeselector-settings-and-tolerations-for-custom-taints)
### Default Implementation in Rancher's Logging Stack
By default, Rancher taints all Linux nodes with `cattle.io/os=linux`, and does not taint Windows nodes.
The logging stack pods have `tolerations` for this taint, which enables them to run on Linux nodes.
Moreover, most logging stack pods run on Linux only and have a `nodeSelector` added to ensure they run on Linux nodes.
This example Pod YAML file shows a nodeSelector being used with a toleration:
```yaml
apiVersion: v1
kind: Pod
# metadata...
spec:
# containers...
tolerations:
- key: cattle.io/os
operator: "Equal"
value: "linux"
effect: NoSchedule
nodeSelector:
kubernetes.io/os: linux
```
In the above example, we ensure that our pod only runs on Linux nodes, and we add a `toleration` for the taint we have on all of our Linux nodes.
You can do the same with Rancher's existing taints, or with your own custom ones.
### Adding NodeSelector Settings and Tolerations for Custom Taints
If you would like to add your own `nodeSelector` settings, or if you would like to add `tolerations` for additional taints, you can pass the following to the chart's values.
```yaml
tolerations:
# insert tolerations...
nodeSelector:
# insert nodeSelector...
```
These values will add both settings to the `fluentd`, `fluentbit`, and `logging-operator` containers.
Essentially, these are global settings for all pods in the logging stack.
However, if you would like to add tolerations for *only* the `fluentbit` container, you can add the following to the chart's values.
```yaml
fluentbit_tolerations:
# insert tolerations list for fluentbit containers only...
```
docs/explanations/integrations-in-rancher/longhorn.md
+70
View File
@@ -0,0 +1,70 @@
---
title: Longhorn - Cloud native distributed block storage for Kubernetes
shortTitle: Longhorn Storage
weight: 19
---
[Longhorn](https://longhorn.io/) is a lightweight, reliable, and easy-to-use distributed block storage system for Kubernetes.
Longhorn is free, open source software. Originally developed by Rancher Labs, it is now being developed as a sandbox project of the Cloud Native Computing Foundation. It can be installed on any Kubernetes cluster with Helm, with kubectl, or with the Rancher UI. You can learn more about its architecture [here.](https://longhorn.io/docs/latest/concepts/)
With Longhorn, you can:
- Use Longhorn volumes as persistent storage for the distributed stateful applications in your Kubernetes cluster
- Partition your block storage into Longhorn volumes so that you can use Kubernetes volumes with or without a cloud provider
- Replicate block storage across multiple nodes and data centers to increase availability
- Store backup data in external storage such as NFS or AWS S3
- Create cross-cluster disaster recovery volumes so that data from a primary Kubernetes cluster can be quickly recovered from backup in a second Kubernetes cluster
- Schedule recurring snapshots of a volume, and schedule recurring backups to NFS or S3-compatible secondary storage
- Restore volumes from backup
- Upgrade Longhorn without disrupting persistent volumes
<figcaption>Longhorn Dashboard</figcaption>
![Longhorn Dashboard](/img/longhorn-screenshot.png)
### Installing Longhorn with Rancher
1. Fulfill all [Installation Requirements.](https://longhorn.io/docs/latest/deploy/install/#installation-requirements)
1. Go to the cluster where you want to install Longhorn.
1. Click **Apps & Marketplace**.
1. Click **Charts**.
1. Click **Longhorn**.
1. Optional: To customize the initial settings, click **Longhorn Default Settings** and edit the configuration. For help customizing the settings, refer to the [Longhorn documentation.](https://longhorn.io/docs/latest/references/settings/)
1. Click **Install**.
**Result:** Longhorn is deployed in the Kubernetes cluster.
### Accessing Longhorn from the Rancher UI
1. Go to the cluster where Longhorn is installed. In the left navigation menu, click **Longhorn**.
1. On this page, you can edit Kubernetes resources managed by Longhorn. To view the Longhorn UI, click the **Longhorn** button in the **Overview** section.
**Result:** You will be taken to the Longhorn UI, where you can manage your Longhorn volumes and their replicas in the Kubernetes cluster, as well as secondary backups of your Longhorn storage that may exist in another Kubernetes cluster or in S3.
### Uninstalling Longhorn from the Rancher UI
1. Go to the cluster where Longhorn is installed and click **Apps & Marketplace**.
1. Click **Installed Apps**.
1. Go to the `longhorn-system` namespace and check the boxes next to the `longhorn` and `longhorn-crd` apps.
1. Click **Delete,** and confirm **Delete**.
**Result:** Longhorn is uninstalled.
### GitHub Repository
The Longhorn project is available [here.](https://github.com/longhorn/longhorn)
### Documentation
The Longhorn documentation is [here.](https://longhorn.io/docs/)
### Architecture
Longhorn creates a dedicated storage controller for each volume and synchronously replicates the volume across multiple replicas stored on multiple nodes.
The storage controller and replicas are themselves orchestrated using Kubernetes.
You can learn more about its architecture [here.](https://longhorn.io/docs/latest/concepts/)
<figcaption>Longhorn Architecture</figcaption>
![Longhorn Architecture](/img/longhorn-architecture.svg)
docs/explanations/integrations-in-rancher/monitoring-and-alerting/built-in-dashboards.md
+114
View File
@@ -0,0 +1,114 @@
---
title: Built-in Dashboards
weight: 3
---
- [Grafana UI](#grafana-ui)
- [Alertmanager UI](#alertmanager-ui)
- [Prometheus UI](#prometheus-ui)
# Grafana UI
[Grafana](https://grafana.com/grafana/) allows you to query, visualize, alert on and understand your metrics no matter where they are stored. Create, explore, and share dashboards with your team and foster a data driven culture.
To see the default dashboards for time series data visualization, go to the Grafana UI.
### Customizing Grafana
To view and customize the PromQL queries powering the Grafana dashboard, see [this page.](../../../how-to-guides/advanced-user-guides/monitoring-alerting-guides/customize-grafana-dashboard.md)
### Persistent Grafana Dashboards
To create a persistent Grafana dashboard, see [this page.](../../../how-to-guides/advanced-user-guides/monitoring-alerting-guides/create-persistent-grafana-dashboard.md)
### Access to Grafana
For information about role-based access control for Grafana, see [this section.](rbac-for-monitoring.md#role-based-access-control-for-grafana)
# Alertmanager UI
When `rancher-monitoring` is installed, the Prometheus Alertmanager UI is deployed, allowing you to view your alerts and the current Alertmanager configuration.
:::note
This section assumes familiarity with how monitoring components work together. For more information about Alertmanager, see [this section.](how-monitoring-works.md#how-alertmanager-works)
:::
### Accessing the Alertmanager UI
The Alertmanager UI lets you see the most recently fired alerts.
:::note Prerequisite:
The `rancher-monitoring` application must be installed.
:::
To see the Alertmanager UI,
1. In the upper left corner, click **☰ > Cluster Management**.
1. On the **Clusters** page, go to the cluster where you want to see the Alertmanager UI, click **Explore**.
1. In the left navigation bar, click **Monitoring**.
1. Click **Alertmanager**.
**Result:** The Alertmanager UI opens in a new tab. For help with configuration, refer to the [official Alertmanager documentation.](https://prometheus.io/docs/alerting/latest/alertmanager/)
For more information on configuring Alertmanager in Rancher, see [this page.](../../../how-to-guides/advanced-user-guides/monitoring-v2-configuration-guides/advanced-configuration/alertmanager.md)
<figcaption>The Alertmanager UI</figcaption>
![Alertmanager UI](/img/alertmanager-ui.png)
### Viewing Default Alerts
To see alerts that are fired by default, go to the Alertmanager UI and click **Expand all groups**.
# Prometheus UI
By default, the [kube-state-metrics service](https://github.com/kubernetes/kube-state-metrics) provides a wealth of information about CPU and memory utilization to the monitoring application. These metrics cover Kubernetes resources across namespaces. This means that in order to see resource metrics for a service, you don't need to create a new ServiceMonitor for it. Because the data is already in the time series database, you can go to the Prometheus UI and run a PromQL query to get the information. The same query can be used to configure a Grafana dashboard to show a graph of those metrics over time.
To see the Prometheus UI, install `rancher-monitoring`. Then:
1. In the upper left corner, click **☰ > Cluster Management**.
1. On the **Clusters** page, go to the cluster where you want to see the Prometheus UI and click **Explore**.
1. In the left navigation bar, click **Monitoring**.
1. Click **Prometheus Graph**.
<figcaption>Prometheus Graph UI</figcaption>
![Prometheus Graph UI](/img/prometheus-graph-ui.png)
### Viewing the Prometheus Targets
To see what services you are monitoring, you will need to see your targets. Targets are set up by ServiceMonitors and PodMonitors as sources to scrape metrics from. You won't need to directly edit targets, but the Prometheus UI can be useful for giving you an overview of all of the sources of metrics that are being scraped.
To see the Prometheus Targets, install `rancher-monitoring`. Then:
1. In the upper left corner, click **☰ > Cluster Management**.
1. On the **Clusters** page, go to the cluster where you want to see the Prometheus targets and click **Explore**.
1. In the left navigation bar, click **Monitoring**.
1. Click **Prometheus Targets**.
<figcaption>Targets in the Prometheus UI</figcaption>
![Prometheus Targets UI](/img/prometheus-targets-ui.png)
### Viewing the PrometheusRules
When you define a Rule (which is declared within a RuleGroup in a PrometheusRule resource), the [spec of the Rule itself](https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/api.md#rule) contains labels that are used by Alertmanager to figure out which Route should receive a certain Alert.
To see the PrometheusRules, install `rancher-monitoring`. Then:
1. In the upper left corner, click **☰ > Cluster Management**.
1. On the **Clusters** page, go to the cluster where you want to see the visualizations and click **Explore**.
1. In the left navigation bar, click **Monitoring**.
1. Click **Prometheus Rules**.
You can also see the rules in the Prometheus UI:
<figcaption>Rules in the Prometheus UI</figcaption>
![PrometheusRules UI](/img/prometheus-rules-ui.png)
For more information on configuring PrometheusRules in Rancher, see [this page.](../../../how-to-guides/advanced-user-guides/monitoring-v2-configuration-guides/advanced-configuration/prometheusrules.md)
docs/explanations/integrations-in-rancher/monitoring-and-alerting/how-monitoring-works.md
+256
View File
@@ -0,0 +1,256 @@
---
title: How Monitoring Works
weight: 1
---
1. [Architecture Overview](#1-architecture-overview)
2. [How Prometheus Works](#2-how-prometheus-works)
3. [How Alertmanager Works](#3-how-alertmanager-works)
4. [Monitoring V2 Specific Components](#4-monitoring-v2-specific-components)
5. [Scraping and Exposing Metrics](#5-scraping-and-exposing-metrics)
# 1. Architecture Overview
_**The following sections describe how data flows through the Monitoring V2 application:**_
### Prometheus Operator
Prometheus Operator observes ServiceMonitors, PodMonitors, and PrometheusRules being created. When the Prometheus configuration resources are created, Prometheus Operator calls the Prometheus API to sync the new configuration. As the diagram at the end of this section shows, the Prometheus Operator acts as the intermediary between Prometheus and Kubernetes, calling the Prometheus API to synchronize Prometheus with the monitoring-related resources in Kubernetes.
### ServiceMonitors and PodMonitors
ServiceMonitors and PodMonitors declaratively specify targets, such as Services and Pods, that need to be monitored.
- Targets are scraped on a recurring schedule based on the configured Prometheus scrape interval, and the metrics that are scraped are stored into the Prometheus Time Series Database (TSDB).
- In order to perform the scrape, ServiceMonitors and PodMonitors are defined with label selectors that determine which Services or Pods should be scraped and endpoints that determine how the scrape should happen on the given target, e.g., scrape/metrics in TCP 10252, proxying through IP addr x.x.x.x.
- Out of the box, Monitoring V2 comes with certain pre-configured exporters that are deployed based on the type of Kubernetes cluster that it is deployed on. For more information, see [Scraping and Exposing Metrics](#5-scraping-and-exposing-metrics).
### How PushProx Works
- Certain internal Kubernetes components are scraped via a proxy deployed as part of Monitoring V2 called **PushProx**. The Kubernetes components that expose metrics to Prometheus through PushProx are the following:
`kube-controller-manager`, `kube-scheduler`, `etcd`, and `kube-proxy`.
- For each PushProx exporter, we deploy one PushProx client onto all target nodes. For example, a PushProx client is deployed onto all controlplane nodes for kube-controller-manager, all etcd nodes for kube-etcd, and all nodes for kubelet.
- We deploy exactly one PushProx proxy per exporter. The process for exporting metrics is as follows:
1. The PushProx Client establishes an outbound connection with the PushProx Proxy.
1. The client then polls the proxy for scrape requests that have come into the proxy.
1. When the proxy receives a scrape request from Prometheus, the client sees it as a result of the poll.
1. The client scrapes the internal component.
1. The internal component responds by pushing metrics back to the proxy.
<figcaption><br/>Process for Exporting Metrics with PushProx:<br/></figcaption>
![Process for Exporting Metrics with PushProx](/img/pushprox-process.svg)
### PrometheusRules
PrometheusRules allow users to define rules for what metrics or time series database queries should result in alerts being fired. Rules are evaluated on an interval.
- **Recording rules** create a new time series based on existing series that have been collected. They are frequently used to precompute complex queries.
- **Alerting rules** run a particular query and fire an alert from Prometheus if the query evaluates to a non-zero value.
### Alert Routing
Once Prometheus determines that an alert needs to be fired, alerts are forwarded to **Alertmanager**.
- Alerts contain labels that come from the PromQL query itself and additional labels and annotations that can be provided as part of specifying the initial PrometheusRule.
- Before receiving any alerts, Alertmanager will use the **routes** and **receivers** specified in its configuration to form a routing tree on which all incoming alerts are evaluated. Each node of the routing tree can specify additional grouping, labeling, and filtering that needs to happen based on the labels attached to the Prometheus alert. A node on the routing tree (usually a leaf node) can also specify that an alert that reaches it needs to be sent out to a configured Receiver, e.g., Slack, PagerDuty, SMS, etc. Note that Alertmanager will send an alert first to **alertingDriver**, then alertingDriver will send or forward alert to the proper destination.
- Routes and receivers are also stored in the Kubernetes API via the Alertmanager Secret. When the Secret is updated, Alertmanager is also updated automatically. Note that routing occurs via labels only (not via annotations, etc.).
<figcaption>How data flows through the monitoring application:</figcaption>
# 2. How Prometheus Works
### Storing Time Series Data
After collecting metrics from exporters, Prometheus stores the time series in a local on-disk time series database. Prometheus optionally integrates with remote systems, but `rancher-monitoring` uses local storage for the time series database.
Once stored, users can query this TSDB using PromQL, the query language for Prometheus.
PromQL queries can be visualized in one of two ways:
1. By supplying the query in Prometheus's Graph UI, which will show a simple graphical view of the data.
1. By creating a Grafana Dashboard that contains the PromQL query and additional formatting directives that label axes, add units, change colors, use alternative visualizations, etc.
### Defining Rules for Prometheus
Rules define queries that Prometheus needs to execute on a regular `evaluationInterval` to perform certain actions, such as firing an alert (alerting rules) or precomputing a query based on others existing in its TSDB (recording rules). These rules are encoded in PrometheusRules custom resources. When PrometheusRule custom resources are created or updated, the Prometheus Operator observes the change and calls the Prometheus API to synchronize the set of rules that Prometheus is currently evaluating on a regular interval.
A PrometheusRule allows you to define one or more RuleGroups. Each RuleGroup consists of a set of Rule objects that can each represent either an alerting or a recording rule with the following fields:
- The name of the new alert or record
- A PromQL expression for the new alert or record
- Labels that should be attached to the alert or record that identify it (e.g. cluster name or severity)
- Annotations that encode any additional important pieces of information that need to be displayed on the notification for an alert (e.g. summary, description, message, runbook URL, etc.). This field is not required for recording rules.
On evaluating a [rule](https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/api.md#rule), Prometheus will execute the provided PromQL query, add additional provided labels (or annotations - only for alerting rules), and execute the appropriate action for the rule. For example, an Alerting Rule that adds `team: front-end` as a label to the provided PromQL query will append that label to the fired alert, which will allow Alertmanager to forward the alert to the correct Receiver.
### Alerting and Recording Rules
Prometheus doesn't maintain the state of whether alerts are active. It fires alerts repetitively at every evaluation interval, relying on Alertmanager to group and filter the alerts into meaningful notifications.
The `evaluation_interval` constant defines how often Prometheus evaluates its alerting rules against the time series database. Similar to the `scrape_interval`, the `evaluation_interval` also defaults to one minute.
The rules are contained in a set of rule files. Rule files include both alerting rules and recording rules, but only alerting rules result in alerts being fired after their evaluation.
For recording rules, Prometheus runs a query, then stores it as a time series. This synthetic time series is useful for storing the results of an expensive or time-consuming query so that it can be queried more quickly in the future.
Alerting rules are more commonly used. Whenever an alerting rule evaluates to a positive number, Prometheus fires an alert.
The Rule file adds labels and annotations to alerts before firing them, depending on the use case:
- Labels indicate information that identifies the alert and could affect the routing of the alert. For example, if when sending an alert about a certain container, the container ID could be used as a label.
- Annotations denote information that doesn't affect where an alert is routed, for example, a runbook or an error message.
# 3. How Alertmanager Works
The Alertmanager handles alerts sent by client applications such as the Prometheus server. It takes care of the following tasks:
- Deduplicating, grouping, and routing alerts to the correct receiver integration such as email, PagerDuty, or OpsGenie
- Silencing and inhibition of alerts
- Tracking alerts that fire over time
- Sending out the status of whether an alert is currently firing, or if it is resolved
### Alerts Forwarded by alertingDrivers
When alertingDrivers are installed, this creates a `Service` that can be used as the receiver's URL for Teams or SMS, based on the alertingDriver's configuration. The URL in the Receiver points to the alertingDrivers; so the Alertmanager sends alert first to alertingDriver, then alertingDriver forwards or sends alert to the proper destination.
### Routing Alerts to Receivers
Alertmanager coordinates where alerts are sent. It allows you to group alerts based on labels and fire them based on whether certain labels are matched. One top-level route accepts all alerts. From there, Alertmanager continues routing alerts to receivers based on whether they match the conditions of the next route.
While the Rancher UI forms only allow editing a routing tree that is two levels deep, you can configure more deeply nested routing structures by editing the Alertmanager Secret.
### Configuring Multiple Receivers
By editing the forms in the Rancher UI, you can set up a Receiver resource with all the information Alertmanager needs to send alerts to your notification system.
By editing custom YAML in the Alertmanager or Receiver configuration, you can also send alerts to multiple notification systems. For more information, see the section on configuring [Receivers.](../../../reference-guides/monitoring-v2-configuration/receivers.md#configuring-multiple-receivers)
# 4. Monitoring V2 Specific Components
Prometheus Operator introduces a set of [Custom Resource Definitions](https://github.com/prometheus-operator/prometheus-operator#customresourcedefinitions) that allow users to deploy and manage Prometheus and Alertmanager instances by creating and modifying those custom resources on a cluster.
Prometheus Operator will automatically update your Prometheus configuration based on the live state of the resources and configuration options that are edited in the Rancher UI.
### Resources Deployed by Default
By default, a set of resources curated by the [kube-prometheus](https://github.com/prometheus-operator/kube-prometheus) project are deployed onto your cluster as part of installing the Rancher Monitoring Application to set up a basic Monitoring/Alerting stack.
The resources that get deployed onto your cluster to support this solution can be found in the [`rancher-monitoring`](https://github.com/rancher/charts/tree/main/charts/rancher-monitoring) Helm chart, which closely tracks the upstream [kube-prometheus-stack](https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack) Helm chart maintained by the Prometheus community with certain changes tracked in the [CHANGELOG.md](https://github.com/rancher/charts/blob/main/charts/rancher-monitoring/CHANGELOG.md).
### Default Exporters
Monitoring V2 deploys three default exporters that provide additional metrics for Prometheus to store:
1. `node-exporter`: exposes hardware and OS metrics for Linux hosts. For more information on `node-exporter`, refer to the [upstream documentation](https://prometheus.io/docs/guides/node-exporter/).
1. `windows-exporter`: exposes hardware and OS metrics for Windows hosts (only deployed on Windows clusters). For more information on `windows-exporter`, refer to the [upstream documentation](https://github.com/prometheus-community/windows_exporter).
1. `kube-state-metrics`: expose additional metrics that track the state of resources contained in the Kubernetes API (e.g., pods, workloads, etc.). For more information on `kube-state-metrics`, refer to the [upstream documentation](https://github.com/kubernetes/kube-state-metrics/tree/master/docs).
ServiceMonitors and PodMonitors will scrape these exporters, as defined [here](#defining-what-metrics-are-scraped). Prometheus stores these metrics, and you can query the results via either Prometheus's UI or Grafana.
See the [architecture](#1-architecture-overview) section for more information on recording rules, alerting rules, and Alertmanager.
### Components Exposed in the Rancher UI
When the monitoring application is installed, you will be able to edit the following components in the Rancher UI:
| Component | Type of Component | Purpose and Common Use Cases for Editing |
|--------------|------------------------|---------------------------|
| ServiceMonitor | Custom resource | Sets up Kubernetes Services to scrape custom metrics from. Automatically updates the scrape configuration in the Prometheus custom resource. |
| PodMonitor | Custom resource | Sets up Kubernetes Pods to scrape custom metrics from. Automatically updates the scrape configuration in the Prometheus custom resource. |
| Receiver | Configuration block (part of Alertmanager) | Modifies information on where to send an alert (e.g., Slack, PagerDuty, etc.) and any necessary information to send the alert (e.g., TLS certs, proxy URLs, etc.). Automatically updates the Alertmanager custom resource. |
| Route | Configuration block (part of Alertmanager) | Modifies the routing tree that is used to filter, label, and group alerts based on labels and send them to the appropriate Receiver. Automatically updates the Alertmanager custom resource. |
| PrometheusRule | Custom resource | Defines additional queries that need to trigger alerts or define materialized views of existing series that are within Prometheus's TSDB. Automatically updates the Prometheus custom resource. |
### PushProx
PushProx allows Prometheus to scrape metrics across a network boundary, which prevents users from having to expose metrics ports for internal Kubernetes components on each node in a Kubernetes cluster.
Since the metrics for Kubernetes components are generally exposed on the host network of nodes in the cluster, PushProx deploys a DaemonSet of clients that sit on the hostNetwork of each node and make an outbound connection to a single proxy that is sitting on the Kubernetes API. Prometheus can then be configured to proxy scrape requests through the proxy to each client, which allows it to scrape metrics from the internal Kubernetes components without requiring any inbound node ports to be open.
Refer to [Scraping Metrics with PushProx](#scraping-metrics-with-pushprox) for more.
# 5. Scraping and Exposing Metrics
### Defining what Metrics are Scraped
ServiceMonitors and PodMonitors define targets that are intended for Prometheus to scrape. The [Prometheus custom resource](https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/design.md#prometheus) tells Prometheus which ServiceMonitors or PodMonitors it should use to find out where to scrape metrics from.
The Prometheus Operator observes the ServiceMonitors and PodMonitors. When it observes that they are created or updated, it calls the Prometheus API to update the scrape configuration in the Prometheus custom resource and keep it in sync with the scrape configuration in the ServiceMonitors or PodMonitors. This scrape configuration tells Prometheus which endpoints to scrape metrics from and how it will label the metrics from those endpoints.
Prometheus scrapes all of the metrics defined in its scrape configuration at every `scrape_interval`, which is one minute by default.
The scrape configuration can be viewed as part of the Prometheus custom resource that is exposed in the Rancher UI.
### How the Prometheus Operator Sets up Metrics Scraping
The Prometheus Deployment or StatefulSet scrapes metrics, and the configuration of Prometheus is controlled by the Prometheus custom resources. The Prometheus Operator watches for Prometheus and Alertmanager resources, and when they are created, the Prometheus Operator creates a Deployment or StatefulSet for Prometheus or Alertmanager with the user-defined configuration.
When the Prometheus Operator observes ServiceMonitors, PodMonitors, and PrometheusRules being created, it knows that the scrape configuration needs to be updated in Prometheus. It updates Prometheus by first updating the configuration and rules files in the volumes of Prometheus's Deployment or StatefulSet. Then it calls the Prometheus API to sync the new configuration, resulting in the Prometheus Deployment or StatefulSet to be modified in place.
### How Kubernetes Component Metrics are Exposed
Prometheus scrapes metrics from deployments known as [exporters,](https://prometheus.io/docs/instrumenting/exporters/) which export the time series data in a format that Prometheus can ingest. In Prometheus, time series consist of streams of timestamped values belonging to the same metric and the same set of labeled dimensions.
### Scraping Metrics with PushProx
Certain internal Kubernetes components are scraped via a proxy deployed as part of Monitoring V2 called PushProx. For detailed information on PushProx, refer [here](#how-pushprox-works) and to the above [architecture](#1-architecture-overview) section.
### Scraping Metrics
The following Kubernetes components are directly scraped by Prometheus:
- kubelet*
- ingress-nginx**
- coreDns/kubeDns
- kube-api-server
\* You can optionally use `hardenedKubelet.enabled` to use a PushProx, but that is not the default.
** For RKE and RKE2 clusters, ingress-nginx is deployed by default and treated as an internal Kubernetes component.
### Scraping Metrics Based on Kubernetes Distribution
Metrics are scraped differently based on the Kubernetes distribution. For help with terminology, refer [here](#terminology). For details, see the table below:
<figcaption>How Metrics are Exposed to Prometheus</figcaption>
| Kubernetes Component | RKE | RKE2 | KubeADM | K3s |
|-----|-----|-----|-----|-----|
| kube-controller-manager | rkeControllerManager.enabled |rke2ControllerManager.enabled | kubeAdmControllerManager.enabled | k3sServer.enabled |
| kube-scheduler | rkeScheduler.enabled | rke2Scheduler.enabled |kubeAdmScheduler.enabled | k3sServer.enabled |
| etcd | rkeEtcd.enabled | rke2Etcd.enabled | kubeAdmEtcd.enabled | Not available |
| kube-proxy | rkeProxy.enabled | rke2Proxy.enabled | kubeAdmProxy.enabled | k3sServer.enabled |
| kubelet | Collects metrics directly exposed by kubelet | Collects metrics directly exposed by kubelet | Collects metrics directly exposed by kubelet | Collects metrics directly exposed by kubelet |
| ingress-nginx* | Collects metrics directly exposed by kubelet, exposed by rkeIngressNginx.enabled | Collects metrics directly exposed by kubelet, Exposed by rke2IngressNginx.enabled | Not available | Not available |
| coreDns/kubeDns | Collects metrics directly exposed by coreDns/kubeDns | Collects metrics directly exposed by coreDns/kubeDns | Collects metrics directly exposed by coreDns/kubeDns | Collects metrics directly exposed by coreDns/kubeDns |
| kube-api-server | Collects metrics directly exposed by kube-api-server |Collects metrics directly exposed by kube-api-server | Collects metrics directly exposed by kube-appi-server | Collects metrics directly exposed by kube-api-server |
\* For RKE and RKE2 clusters, ingress-nginx is deployed by default and treated as an internal Kubernetes component.
### Terminology
- **kube-scheduler:** The internal Kubernetes component that uses information in the pod spec to decide on which node to run a pod.
- **kube-controller-manager:** The internal Kubernetes component that is responsible for node management (detecting if a node fails), pod replication and endpoint creation.
- **etcd:** The internal Kubernetes component that is the distributed key/value store which Kubernetes uses for persistent storage of all cluster information.
- **kube-proxy:** The internal Kubernetes component that watches the API server for pods/services changes in order to maintain the network up to date.
- **kubelet:** The internal Kubernetes component that watches the API server for pods on a node and makes sure they are running.
- **ingress-nginx:** An Ingress controller for Kubernetes using NGINX as a reverse proxy and load balancer.
- **coreDns/kubeDns:** The internal Kubernetes component responsible for DNS.
- **kube-api-server:** The main internal Kubernetes component that is responsible for exposing APIs for the other master components.
docs/explanations/integrations-in-rancher/monitoring-and-alerting/promql-expressions.md
+428
View File
@@ -0,0 +1,428 @@
---
title: PromQL Expression Reference
weight: 6
---
The PromQL expressions in this doc can be used to configure alerts.
For more information about querying the Prometheus time series database, refer to the official [Prometheus documentation.](https://prometheus.io/docs/prometheus/latest/querying/basics/)
<!-- TOC -->
- [Cluster Metrics](#cluster-metrics)
- [Cluster CPU Utilization](#cluster-cpu-utilization)
- [Cluster Load Average](#cluster-load-average)
- [Cluster Memory Utilization](#cluster-memory-utilization)
- [Cluster Disk Utilization](#cluster-disk-utilization)
- [Cluster Disk I/O](#cluster-disk-i-o)
- [Cluster Network Packets](#cluster-network-packets)
- [Cluster Network I/O](#cluster-network-i-o)
- [Node Metrics](#node-metrics)
- [Node CPU Utilization](#node-cpu-utilization)
- [Node Load Average](#node-load-average)
- [Node Memory Utilization](#node-memory-utilization)
- [Node Disk Utilization](#node-disk-utilization)
- [Node Disk I/O](#node-disk-i-o)
- [Node Network Packets](#node-network-packets)
- [Node Network I/O](#node-network-i-o)
- [Etcd Metrics](#etcd-metrics)
- [Etcd Has a Leader](#etcd-has-a-leader)
- [Number of Times the Leader Changes](#number-of-times-the-leader-changes)
- [Number of Failed Proposals](#number-of-failed-proposals)
- [GRPC Client Traffic](#grpc-client-traffic)
- [Peer Traffic](#peer-traffic)
- [DB Size](#db-size)
- [Active Streams](#active-streams)
- [Raft Proposals](#raft-proposals)
- [RPC Rate](#rpc-rate)
- [Disk Operations](#disk-operations)
- [Disk Sync Duration](#disk-sync-duration)
- [Kubernetes Components Metrics](#kubernetes-components-metrics)
- [API Server Request Latency](#api-server-request-latency)
- [API Server Request Rate](#api-server-request-rate)
- [Scheduling Failed Pods](#scheduling-failed-pods)
- [Controller Manager Queue Depth](#controller-manager-queue-depth)
- [Scheduler E2E Scheduling Latency](#scheduler-e2e-scheduling-latency)
- [Scheduler Preemption Attempts](#scheduler-preemption-attempts)
- [Ingress Controller Connections](#ingress-controller-connections)
- [Ingress Controller Request Process Time](#ingress-controller-request-process-time)
- [Rancher Logging Metrics](#rancher-logging-metrics)
- [Fluentd Buffer Queue Rate](#fluentd-buffer-queue-rate)
- [Fluentd Input Rate](#fluentd-input-rate)
- [Fluentd Output Errors Rate](#fluentd-output-errors-rate)
- [Fluentd Output Rate](#fluentd-output-rate)
- [Workload Metrics](#workload-metrics)
- [Workload CPU Utilization](#workload-cpu-utilization)
- [Workload Memory Utilization](#workload-memory-utilization)
- [Workload Network Packets](#workload-network-packets)
- [Workload Network I/O](#workload-network-i-o)
- [Workload Disk I/O](#workload-disk-i-o)
- [Pod Metrics](#pod-metrics)
- [Pod CPU Utilization](#pod-cpu-utilization)
- [Pod Memory Utilization](#pod-memory-utilization)
- [Pod Network Packets](#pod-network-packets)
- [Pod Network I/O](#pod-network-i-o)
- [Pod Disk I/O](#pod-disk-i-o)
- [Container Metrics](#container-metrics)
- [Container CPU Utilization](#container-cpu-utilization)
- [Container Memory Utilization](#container-memory-utilization)
- [Container Disk I/O](#container-disk-i-o)
<!-- /TOC -->
# Cluster Metrics
### Cluster CPU Utilization
| Catalog | Expression |
| --- | --- |
| Detail | `1 - (avg(irate(node_cpu_seconds_total{mode="idle"}[5m])) by (instance))` |
| Summary | `1 - (avg(irate(node_cpu_seconds_total{mode="idle"}[5m])))` |
### Cluster Load Average
| Catalog | Expression |
| --- | --- |
| Detail | <table><tr><td>load1</td><td>`sum(node_load1) by (instance) / count(node_cpu_seconds_total{mode="system"}) by (instance)`</td></tr><tr><td>load5</td><td>`sum(node_load5) by (instance) / count(node_cpu_seconds_total{mode="system"}) by (instance)`</td></tr><tr><td>load15</td><td>`sum(node_load15) by (instance) / count(node_cpu_seconds_total{mode="system"}) by (instance)`</td></tr></table> |
| Summary | <table><tr><td>load1</td><td>`sum(node_load1) by (instance) / count(node_cpu_seconds_total{mode="system"})`</td></tr><tr><td>load5</td><td>`sum(node_load5) by (instance) / count(node_cpu_seconds_total{mode="system"})`</td></tr><tr><td>load15</td><td>`sum(node_load15) by (instance) / count(node_cpu_seconds_total{mode="system"})`</td></tr></table> |
### Cluster Memory Utilization
| Catalog | Expression |
| --- | --- |
| Detail | `1 - sum(node_memory_MemAvailable_bytes) by (instance) / sum(node_memory_MemTotal_bytes) by (instance)` |
| Summary | `1 - sum(node_memory_MemAvailable_bytes) / sum(node_memory_MemTotal_bytes)` |
### Cluster Disk Utilization
| Catalog | Expression |
| --- | --- |
| Detail | `(sum(node_filesystem_size_bytes{device!="rootfs"}) by (instance) - sum(node_filesystem_free_bytes{device!="rootfs"}) by (instance)) / sum(node_filesystem_size_bytes{device!="rootfs"}) by (instance)` |
| Summary | `(sum(node_filesystem_size_bytes{device!="rootfs"}) - sum(node_filesystem_free_bytes{device!="rootfs"})) / sum(node_filesystem_size_bytes{device!="rootfs"})` |
### Cluster Disk I/O
| Catalog | Expression |
| --- | --- |
| Detail | <table><tr><td>read</td><td>`sum(rate(node_disk_read_bytes_total[5m])) by (instance)`</td></tr><tr><td>written</td><td>`sum(rate(node_disk_written_bytes_total[5m])) by (instance)`</td></tr></table> |
| Summary | <table><tr><td>read</td><td>`sum(rate(node_disk_read_bytes_total[5m]))`</td></tr><tr><td>written</td><td>`sum(rate(node_disk_written_bytes_total[5m]))`</td></tr></table> |
### Cluster Network Packets
| Catalog | Expression |
| --- | --- |
| Detail | <table><tr><td>receive-dropped</td><td><code>sum(rate(node_network_receive_drop_total{device!~"lo &#124; veth.&ast; &#124; docker.&ast; &#124; flannel.&ast; &#124; cali.&ast; &#124; cbr.&ast;"}[5m])) by (instance)</code></td></tr><tr><td>receive-errs</td><td><code>sum(rate(node_network_receive_errs_total{device!~"lo &#124; veth.&ast; &#124; docker.&ast; &#124; flannel.&ast; &#124; cali.&ast; &#124; cbr.&ast;"}[5m])) by (instance)</code></td></tr><tr><td>receive-packets</td><td><code>sum(rate(node_network_receive_packets_total{device!~"lo &#124; veth.&ast; &#124; docker.&ast; &#124; flannel.&ast; &#124; cali.&ast; &#124; cbr.&ast;"}[5m])) by (instance)</code></td></tr><tr><td>transmit-dropped</td><td><code>sum(rate(node_network_transmit_drop_total{device!~"lo &#124; veth.&ast; &#124; docker.&ast; &#124; flannel.&ast; &#124; cali.&ast; &#124; cbr.&ast;"}[5m])) by (instance)</code></td></tr><tr><td>transmit-errs</td><td><code>sum(rate(node_network_transmit_errs_total{device!~"lo &#124; veth.&ast; &#124; docker.&ast; &#124; flannel.&ast; &#124; cali.&ast; &#124; cbr.&ast;"}[5m])) by (instance)</code></td></tr><tr><td>transmit-packets</td><td><code>sum(rate(node_network_transmit_packets_total{device!~"lo &#124; veth.&ast; &#124; docker.&ast; &#124; flannel.&ast; &#124; cali.&ast; &#124; cbr.&ast;"}[5m])) by (instance)</code></td></tr></table> |
| Summary | <table><tr><td>receive-dropped</td><td><code>sum(rate(node_network_receive_drop_total{device!~"lo &#124; veth.&ast; &#124; docker.&ast; &#124; flannel.&ast; &#124; cali.&ast; &#124; cbr.&ast;"}[5m]))</code></td></tr><tr><td>receive-errs</td><td><code>sum(rate(node_network_receive_errs_total{device!~"lo &#124; veth.&ast; &#124; docker.&ast; &#124; flannel.&ast; &#124; cali.&ast; &#124; cbr.&ast;"}[5m]))</code></td></tr><tr><td>receive-packets</td><td><code>sum(rate(node_network_receive_packets_total{device!~"lo &#124; veth.&ast; &#124; docker.&ast; &#124; flannel.&ast; &#124; cali.&ast; &#124; cbr.&ast;"}[5m]))</code></td></tr><tr><td>transmit-dropped</td><td><code>sum(rate(node_network_transmit_drop_total{device!~"lo &#124; veth.&ast; &#124; docker.&ast; &#124; flannel.&ast; &#124; cali.&ast; &#124; cbr.&ast;"}[5m]))</code></td></tr><tr><td>transmit-errs</td><td><code>sum(rate(node_network_transmit_errs_total{device!~"lo &#124; veth.&ast; &#124; docker.&ast; &#124; flannel.&ast; &#124; cali.&ast; &#124; cbr.&ast;"}[5m]))</code></td></tr><tr><td>transmit-packets</td><td><code>sum(rate(node_network_transmit_packets_total{device!~"lo &#124; veth.&ast; &#124; docker.&ast; &#124; flannel.&ast; &#124; cali.&ast; &#124; cbr.&ast;"}[5m]))</code></td></tr></table> |
### Cluster Network I/O
| Catalog | Expression |
| --- | --- |
| Detail | <table><tr><td>receive</td><td><code>sum(rate(node_network_receive_bytes_total{device!~"lo &#124; veth.&ast; &#124; docker.&ast; &#124; flannel.&ast; &#124; cali.&ast; &#124; cbr.&ast;"}[5m])) by (instance)</code></td></tr><tr><td>transmit</td><td><code>sum(rate(node_network_transmit_bytes_total{device!~"lo &#124; veth.&ast; &#124; docker.&ast; &#124; flannel.&ast; &#124; cali.&ast; &#124; cbr.&ast;"}[5m])) by (instance)</code></td></tr></table> |
| Summary | <table><tr><td>receive</td><td><code>sum(rate(node_network_receive_bytes_total{device!~"lo &#124; veth.&ast; &#124; docker.&ast; &#124; flannel.&ast; &#124; cali.&ast; &#124; cbr.&ast;"}[5m]))</code></td></tr><tr><td>transmit</td><td><code>sum(rate(node_network_transmit_bytes_total{device!~"lo &#124; veth.&ast; &#124; docker.&ast; &#124; flannel.&ast; &#124; cali.&ast; &#124; cbr.&ast;"}[5m]))</code></td></tr></table> |
# Node Metrics
### Node CPU Utilization
| Catalog | Expression |
| --- | --- |
| Detail | `avg(irate(node_cpu_seconds_total{mode!="idle", instance=~"$instance"}[5m])) by (mode)` |
| Summary | `1 - (avg(irate(node_cpu_seconds_total{mode="idle", instance=~"$instance"}[5m])))` |
### Node Load Average
| Catalog | Expression |
| --- | --- |
| Detail | <table><tr><td>load1</td><td>`sum(node_load1{instance=~"$instance"}) / count(node_cpu_seconds_total{mode="system",instance=~"$instance"})`</td></tr><tr><td>load5</td><td>`sum(node_load5{instance=~"$instance"}) / count(node_cpu_seconds_total{mode="system",instance=~"$instance"})`</td></tr><tr><td>load15</td><td>`sum(node_load15{instance=~"$instance"}) / count(node_cpu_seconds_total{mode="system",instance=~"$instance"})`</td></tr></table> |
| Summary | <table><tr><td>load1</td><td>`sum(node_load1{instance=~"$instance"}) / count(node_cpu_seconds_total{mode="system",instance=~"$instance"})`</td></tr><tr><td>load5</td><td>`sum(node_load5{instance=~"$instance"}) / count(node_cpu_seconds_total{mode="system",instance=~"$instance"})`</td></tr><tr><td>load15</td><td>`sum(node_load15{instance=~"$instance"}) / count(node_cpu_seconds_total{mode="system",instance=~"$instance"})`</td></tr></table> |
### Node Memory Utilization
| Catalog | Expression |
| --- | --- |
| Detail | `1 - sum(node_memory_MemAvailable_bytes{instance=~"$instance"}) / sum(node_memory_MemTotal_bytes{instance=~"$instance"})` |
| Summary | `1 - sum(node_memory_MemAvailable_bytes{instance=~"$instance"}) / sum(node_memory_MemTotal_bytes{instance=~"$instance"}) ` |
### Node Disk Utilization
| Catalog | Expression |
| --- | --- |
| Detail | `(sum(node_filesystem_size_bytes{device!="rootfs",instance=~"$instance"}) by (device) - sum(node_filesystem_free_bytes{device!="rootfs",instance=~"$instance"}) by (device)) / sum(node_filesystem_size_bytes{device!="rootfs",instance=~"$instance"}) by (device)` |
| Summary | `(sum(node_filesystem_size_bytes{device!="rootfs",instance=~"$instance"}) - sum(node_filesystem_free_bytes{device!="rootfs",instance=~"$instance"})) / sum(node_filesystem_size_bytes{device!="rootfs",instance=~"$instance"})` |
### Node Disk I/O
| Catalog | Expression |
| --- | --- |
| Detail | <table><tr><td>read</td><td>`sum(rate(node_disk_read_bytes_total{instance=~"$instance"}[5m]))`</td></tr><tr><td>written</td><td>`sum(rate(node_disk_written_bytes_total{instance=~"$instance"}[5m]))`</td></tr></table> |
| Summary | <table><tr><td>read</td><td>`sum(rate(node_disk_read_bytes_total{instance=~"$instance"}[5m]))`</td></tr><tr><td>written</td><td>`sum(rate(node_disk_written_bytes_total{instance=~"$instance"}[5m]))`</td></tr></table> |
### Node Network Packets
| Catalog | Expression |
| --- | --- |
| Detail | <table><tr><td>receive-dropped</td><td><code>sum(rate(node_network_receive_drop_total{device!~"lo &#124; veth.&ast; &#124; docker.&ast; &#124; flannel.&ast; &#124; cali.&ast; &#124; cbr.&ast;",instance=~"$instance"}[5m])) by (device)</code></td></tr><tr><td>receive-errs</td><td><code>sum(rate(node_network_receive_errs_total{device!~"lo &#124; veth.&ast; &#124; docker.&ast; &#124; flannel.&ast; &#124; cali.&ast; &#124; cbr.&ast;",instance=~"$instance"}[5m])) by (device)</code></td></tr><tr><td>receive-packets</td><td><code>sum(rate(node_network_receive_packets_total{device!~"lo &#124; veth.&ast; &#124; docker.&ast; &#124; flannel.&ast; &#124; cali.&ast; &#124; cbr.&ast;",instance=~"$instance"}[5m])) by (device)</code></td></tr><tr><td>transmit-dropped</td><td><code>sum(rate(node_network_transmit_drop_total{device!~"lo &#124; veth.&ast; &#124; docker.&ast; &#124; flannel.&ast; &#124; cali.&ast; &#124; cbr.&ast;",instance=~"$instance"}[5m])) by (device)</code></td></tr><tr><td>transmit-errs</td><td><code>sum(rate(node_network_transmit_errs_total{device!~"lo &#124; veth.&ast; &#124; docker.&ast; &#124; flannel.&ast; &#124; cali.&ast; &#124; cbr.&ast;",instance=~"$instance"}[5m])) by (device)</code></td></tr><tr><td>transmit-packets</td><td><code>sum(rate(node_network_transmit_packets_total{device!~"lo &#124; veth.&ast; &#124; docker.&ast; &#124; flannel.&ast; &#124; cali.&ast; &#124; cbr.&ast;",instance=~"$instance"}[5m])) by (device)</code></td></tr></table> |
| Summary | <table><tr><td>receive-dropped</td><td><code>sum(rate(node_network_receive_drop_total{device!~"lo &#124; veth.&ast; &#124; docker.&ast; &#124; flannel.&ast; &#124; cali.&ast; &#124; cbr.&ast;",instance=~"$instance"}[5m]))</code></td></tr><tr><td>receive-errs</td><td><code>sum(rate(node_network_receive_errs_total{device!~"lo &#124; veth.&ast; &#124; docker.&ast; &#124; flannel.&ast; &#124; cali.&ast; &#124; cbr.&ast;",instance=~"$instance"}[5m]))</code></td></tr><tr><td>receive-packets</td><td><code>sum(rate(node_network_receive_packets_total{device!~"lo &#124; veth.&ast; &#124; docker.&ast; &#124; flannel.&ast; &#124; cali.&ast; &#124; cbr.&ast;",instance=~"$instance"}[5m]))</code></td></tr><tr><td>transmit-dropped</td><td><code>sum(rate(node_network_transmit_drop_total{device!~"lo &#124; veth.&ast; &#124; docker.&ast; &#124; flannel.&ast; &#124; cali.&ast; &#124; cbr.&ast;",instance=~"$instance"}[5m]))</code></td></tr><tr><td>transmit-errs</td><td><code>sum(rate(node_network_transmit_errs_total{device!~"lo &#124; veth.&ast; &#124; docker.&ast; &#124; flannel.&ast; &#124; cali.&ast; &#124; cbr.&ast;",instance=~"$instance"}[5m]))</code></td></tr><tr><td>transmit-packets</td><td><code>sum(rate(node_network_transmit_packets_total{device!~"lo &#124; veth.&ast; &#124; docker.&ast; &#124; flannel.&ast; &#124; cali.&ast; &#124; cbr.&ast;",instance=~"$instance"}[5m]))</code></td></tr></table> |
### Node Network I/O
| Catalog | Expression |
| --- | --- |
| Detail | <table><tr><td>receive</td><td><code>sum(rate(node_network_receive_bytes_total{device!~"lo &#124; veth.&ast; &#124; docker.&ast; &#124; flannel.&ast; &#124; cali.&ast; &#124; cbr.&ast;",instance=~"$instance"}[5m])) by (device)</code></td></tr><tr><td>transmit</td><td><code>sum(rate(node_network_transmit_bytes_total{device!~"lo &#124; veth.&ast; &#124; docker.&ast; &#124; flannel.&ast; &#124; cali.&ast; &#124; cbr.&ast;",instance=~"$instance"}[5m])) by (device)</code></td></tr></table> |
| Summary | <table><tr><td>receive</td><td><code>sum(rate(node_network_receive_bytes_total{device!~"lo &#124; veth.&ast; &#124; docker.&ast; &#124; flannel.&ast; &#124; cali.&ast; &#124; cbr.&ast;",instance=~"$instance"}[5m]))</code></td></tr><tr><td>transmit</td><td><code>sum(rate(node_network_transmit_bytes_total{device!~"lo &#124; veth.&ast; &#124; docker.&ast; &#124; flannel.&ast; &#124; cali.&ast; &#124; cbr.&ast;",instance=~"$instance"}[5m]))</code></td></tr></table> |
# Etcd Metrics
### Etcd Has a Leader
`max(etcd_server_has_leader)`
### Number of Times the Leader Changes
`max(etcd_server_leader_changes_seen_total)`
### Number of Failed Proposals
`sum(etcd_server_proposals_failed_total)`
### GRPC Client Traffic
| Catalog | Expression |
| --- | --- |
| Detail | <table><tr><td>in</td><td>`sum(rate(etcd_network_client_grpc_received_bytes_total[5m])) by (instance)`</td></tr><tr><td>out</td><td>`sum(rate(etcd_network_client_grpc_sent_bytes_total[5m])) by (instance)`</td></tr></table> |
| Summary | <table><tr><td>in</td><td>`sum(rate(etcd_network_client_grpc_received_bytes_total[5m]))`</td></tr><tr><td>out</td><td>`sum(rate(etcd_network_client_grpc_sent_bytes_total[5m]))`</td></tr></table> |
### Peer Traffic
| Catalog | Expression |
| --- | --- |
| Detail | <table><tr><td>in</td><td>`sum(rate(etcd_network_peer_received_bytes_total[5m])) by (instance)`</td></tr><tr><td>out</td><td>`sum(rate(etcd_network_peer_sent_bytes_total[5m])) by (instance)`</td></tr></table> |
| Summary | <table><tr><td>in</td><td>`sum(rate(etcd_network_peer_received_bytes_total[5m]))`</td></tr><tr><td>out</td><td>`sum(rate(etcd_network_peer_sent_bytes_total[5m]))`</td></tr></table> |
### DB Size
| Catalog | Expression |
| --- | --- |
| Detail | `sum(etcd_debugging_mvcc_db_total_size_in_bytes) by (instance)` |
| Summary | `sum(etcd_debugging_mvcc_db_total_size_in_bytes)` |
### Active Streams
| Catalog | Expression |
| --- | --- |
| Detail | <table><tr><td>lease-watch</td><td>`sum(grpc_server_started_total{grpc_service="etcdserverpb.Lease",grpc_type="bidi_stream"}) by (instance) - sum(grpc_server_handled_total{grpc_service="etcdserverpb.Lease",grpc_type="bidi_stream"}) by (instance)`</td></tr><tr><td>watch</td><td>`sum(grpc_server_started_total{grpc_service="etcdserverpb.Watch",grpc_type="bidi_stream"}) by (instance) - sum(grpc_server_handled_total{grpc_service="etcdserverpb.Watch",grpc_type="bidi_stream"}) by (instance)`</td></tr></table> |
| Summary | <table><tr><td>lease-watch</td><td>`sum(grpc_server_started_total{grpc_service="etcdserverpb.Lease",grpc_type="bidi_stream"}) - sum(grpc_server_handled_total{grpc_service="etcdserverpb.Lease",grpc_type="bidi_stream"})`</td></tr><tr><td>watch</td><td>`sum(grpc_server_started_total{grpc_service="etcdserverpb.Watch",grpc_type="bidi_stream"}) - sum(grpc_server_handled_total{grpc_service="etcdserverpb.Watch",grpc_type="bidi_stream"})`</td></tr></table> |
### Raft Proposals
| Catalog | Expression |
| --- | --- |
| Detail | <table><tr><td>applied</td><td>`sum(increase(etcd_server_proposals_applied_total[5m])) by (instance)`</td></tr><tr><td>committed</td><td>`sum(increase(etcd_server_proposals_committed_total[5m])) by (instance)`</td></tr><tr><td>pending</td><td>`sum(increase(etcd_server_proposals_pending[5m])) by (instance)`</td></tr><tr><td>failed</td><td>`sum(increase(etcd_server_proposals_failed_total[5m])) by (instance)`</td></tr></table> |
| Summary | <table><tr><td>applied</td><td>`sum(increase(etcd_server_proposals_applied_total[5m]))`</td></tr><tr><td>committed</td><td>`sum(increase(etcd_server_proposals_committed_total[5m]))`</td></tr><tr><td>pending</td><td>`sum(increase(etcd_server_proposals_pending[5m]))`</td></tr><tr><td>failed</td><td>`sum(increase(etcd_server_proposals_failed_total[5m]))`</td></tr></table> |
### RPC Rate
| Catalog | Expression |
| --- | --- |
| Detail | <table><tr><td>total</td><td>`sum(rate(grpc_server_started_total{grpc_type="unary"}[5m])) by (instance)`</td></tr><tr><td>fail</td><td>`sum(rate(grpc_server_handled_total{grpc_type="unary",grpc_code!="OK"}[5m])) by (instance)`</td></tr></table> |
| Summary | <table><tr><td>total</td><td>`sum(rate(grpc_server_started_total{grpc_type="unary"}[5m]))`</td></tr><tr><td>fail</td><td>`sum(rate(grpc_server_handled_total{grpc_type="unary",grpc_code!="OK"}[5m]))`</td></tr></table> |
### Disk Operations
| Catalog | Expression |
| --- | --- |
| Detail | <table><tr><td>commit-called-by-backend</td><td>`sum(rate(etcd_disk_backend_commit_duration_seconds_sum[1m])) by (instance)`</td></tr><tr><td>fsync-called-by-wal</td><td>`sum(rate(etcd_disk_wal_fsync_duration_seconds_sum[1m])) by (instance)`</td></tr></table> |
| Summary | <table><tr><td>commit-called-by-backend</td><td>`sum(rate(etcd_disk_backend_commit_duration_seconds_sum[1m]))`</td></tr><tr><td>fsync-called-by-wal</td><td>`sum(rate(etcd_disk_wal_fsync_duration_seconds_sum[1m]))`</td></tr></table> |
### Disk Sync Duration
| Catalog | Expression |
| --- | --- |
| Detail | <table><tr><td>wal</td><td>`histogram_quantile(0.99, sum(rate(etcd_disk_wal_fsync_duration_seconds_bucket[5m])) by (instance, le))`</td></tr><tr><td>db</td><td>`histogram_quantile(0.99, sum(rate(etcd_disk_backend_commit_duration_seconds_bucket[5m])) by (instance, le))`</td></tr></table> |
| Summary | <table><tr><td>wal</td><td>`sum(histogram_quantile(0.99, sum(rate(etcd_disk_wal_fsync_duration_seconds_bucket[5m])) by (instance, le)))`</td></tr><tr><td>db</td><td>`sum(histogram_quantile(0.99, sum(rate(etcd_disk_backend_commit_duration_seconds_bucket[5m])) by (instance, le)))`</td></tr></table> |
# Kubernetes Components Metrics
### API Server Request Latency
| Catalog | Expression |
| --- | --- |
| Detail | `avg(apiserver_request_latencies_sum / apiserver_request_latencies_count) by (instance, verb) /1e+06` |
| Summary | `avg(apiserver_request_latencies_sum / apiserver_request_latencies_count) by (instance) /1e+06` |
### API Server Request Rate
| Catalog | Expression |
| --- | --- |
| Detail | `sum(rate(apiserver_request_count[5m])) by (instance, code)` |
| Summary | `sum(rate(apiserver_request_count[5m])) by (instance)` |
### Scheduling Failed Pods
| Catalog | Expression |
| --- | --- |
| Detail | `sum(kube_pod_status_scheduled{condition="false"})` |
| Summary | `sum(kube_pod_status_scheduled{condition="false"})` |
### Controller Manager Queue Depth
| Catalog | Expression |
| --- | --- |
| Detail | <table><tr><td>volumes</td><td>`sum(volumes_depth) by instance`</td></tr><tr><td>deployment</td><td>`sum(deployment_depth) by instance`</td></tr><tr><td>replicaset</td><td>`sum(replicaset_depth) by instance`</td></tr><tr><td>service</td><td>`sum(service_depth) by instance`</td></tr><tr><td>serviceaccount</td><td>`sum(serviceaccount_depth) by instance`</td></tr><tr><td>endpoint</td><td>`sum(endpoint_depth) by instance`</td></tr><tr><td>daemonset</td><td>`sum(daemonset_depth) by instance`</td></tr><tr><td>statefulset</td><td>`sum(statefulset_depth) by instance`</td></tr><tr><td>replicationmanager</td><td>`sum(replicationmanager_depth) by instance`</td></tr></table> |
| Summary | <table><tr><td>volumes</td><td>`sum(volumes_depth)`</td></tr><tr><td>deployment</td><td>`sum(deployment_depth)`</td></tr><tr><td>replicaset</td><td>`sum(replicaset_depth)`</td></tr><tr><td>service</td><td>`sum(service_depth)`</td></tr><tr><td>serviceaccount</td><td>`sum(serviceaccount_depth)`</td></tr><tr><td>endpoint</td><td>`sum(endpoint_depth)`</td></tr><tr><td>daemonset</td><td>`sum(daemonset_depth)`</td></tr><tr><td>statefulset</td><td>`sum(statefulset_depth)`</td></tr><tr><td>replicationmanager</td><td>`sum(replicationmanager_depth)`</td></tr></table> |
### Scheduler E2E Scheduling Latency
| Catalog | Expression |
| --- | --- |
| Detail | `histogram_quantile(0.99, sum(scheduler_e2e_scheduling_latency_microseconds_bucket) by (le, instance)) / 1e+06` |
| Summary | `sum(histogram_quantile(0.99, sum(scheduler_e2e_scheduling_latency_microseconds_bucket) by (le, instance)) / 1e+06)` |
### Scheduler Preemption Attempts
| Catalog | Expression |
| --- | --- |
| Detail | `sum(rate(scheduler_total_preemption_attempts[5m])) by (instance)` |
| Summary | `sum(rate(scheduler_total_preemption_attempts[5m]))` |
### Ingress Controller Connections
| Catalog | Expression |
| --- | --- |
| Detail | <table><tr><td>reading</td><td>`sum(nginx_ingress_controller_nginx_process_connections{state="reading"}) by (instance)`</td></tr><tr><td>waiting</td><td>`sum(nginx_ingress_controller_nginx_process_connections{state="waiting"}) by (instance)`</td></tr><tr><td>writing</td><td>`sum(nginx_ingress_controller_nginx_process_connections{state="writing"}) by (instance)`</td></tr><tr><td>accepted</td><td>`sum(ceil(increase(nginx_ingress_controller_nginx_process_connections_total{state="accepted"}[5m]))) by (instance)`</td></tr><tr><td>active</td><td>`sum(ceil(increase(nginx_ingress_controller_nginx_process_connections_total{state="active"}[5m]))) by (instance)`</td></tr><tr><td>handled</td><td>`sum(ceil(increase(nginx_ingress_controller_nginx_process_connections_total{state="handled"}[5m]))) by (instance)`</td></tr></table> |
| Summary | <table><tr><td>reading</td><td>`sum(nginx_ingress_controller_nginx_process_connections{state="reading"})`</td></tr><tr><td>waiting</td><td>`sum(nginx_ingress_controller_nginx_process_connections{state="waiting"})`</td></tr><tr><td>writing</td><td>`sum(nginx_ingress_controller_nginx_process_connections{state="writing"})`</td></tr><tr><td>accepted</td><td>`sum(ceil(increase(nginx_ingress_controller_nginx_process_connections_total{state="accepted"}[5m])))`</td></tr><tr><td>active</td><td>`sum(ceil(increase(nginx_ingress_controller_nginx_process_connections_total{state="active"}[5m])))`</td></tr><tr><td>handled</td><td>`sum(ceil(increase(nginx_ingress_controller_nginx_process_connections_total{state="handled"}[5m])))`</td></tr></table> |
### Ingress Controller Request Process Time
| Catalog | Expression |
| --- | --- |
| Detail | `topk(10, histogram_quantile(0.95,sum by (le, host, path)(rate(nginx_ingress_controller_request_duration_seconds_bucket{host!="_"}[5m]))))` |
| Summary | `topk(10, histogram_quantile(0.95,sum by (le, host)(rate(nginx_ingress_controller_request_duration_seconds_bucket{host!="_"}[5m]))))` |
# Rancher Logging Metrics
### Fluentd Buffer Queue Rate
| Catalog | Expression |
| --- | --- |
| Detail | `sum(rate(fluentd_output_status_buffer_queue_length[5m])) by (instance)` |
| Summary | `sum(rate(fluentd_output_status_buffer_queue_length[5m]))` |
### Fluentd Input Rate
| Catalog | Expression |
| --- | --- |
| Detail | `sum(rate(fluentd_input_status_num_records_total[5m])) by (instance)` |
| Summary | `sum(rate(fluentd_input_status_num_records_total[5m]))` |
### Fluentd Output Errors Rate
| Catalog | Expression |
| --- | --- |
| Detail | `sum(rate(fluentd_output_status_num_errors[5m])) by (type)` |
| Summary | `sum(rate(fluentd_output_status_num_errors[5m]))` |
### Fluentd Output Rate
| Catalog | Expression |
| --- | --- |
| Detail | `sum(rate(fluentd_output_status_num_records_total[5m])) by (instance)` |
| Summary | `sum(rate(fluentd_output_status_num_records_total[5m]))` |
# Workload Metrics
### Workload CPU Utilization
| Catalog | Expression |
| --- | --- |
| Detail | <table><tr><td>cfs throttled seconds</td><td>`sum(rate(container_cpu_cfs_throttled_seconds_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m])) by (pod_name)`</td></tr><tr><td>user seconds</td><td>`sum(rate(container_cpu_user_seconds_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m])) by (pod_name)`</td></tr><tr><td>system seconds</td><td>`sum(rate(container_cpu_system_seconds_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m])) by (pod_name)`</td></tr><tr><td>usage seconds</td><td>`sum(rate(container_cpu_usage_seconds_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m])) by (pod_name)`</td></tr></table> |
| Summary | <table><tr><td>cfs throttled seconds</td><td>`sum(rate(container_cpu_cfs_throttled_seconds_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m]))`</td></tr><tr><td>user seconds</td><td>`sum(rate(container_cpu_user_seconds_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m]))`</td></tr><tr><td>system seconds</td><td>`sum(rate(container_cpu_system_seconds_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m]))`</td></tr><tr><td>usage seconds</td><td>`sum(rate(container_cpu_usage_seconds_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m]))`</td></tr></table> |
### Workload Memory Utilization
| Catalog | Expression |
| --- | --- |
| Detail | `sum(container_memory_working_set_bytes{namespace="$namespace",pod_name=~"$podName", container_name!=""}) by (pod_name)` |
| Summary | `sum(container_memory_working_set_bytes{namespace="$namespace",pod_name=~"$podName", container_name!=""})` |
### Workload Network Packets
| Catalog | Expression |
| --- | --- |
| Detail | <table><tr><td>receive-packets</td><td>`sum(rate(container_network_receive_packets_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m])) by (pod_name)`</td></tr><tr><td>receive-dropped</td><td>`sum(rate(container_network_receive_packets_dropped_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m])) by (pod_name)`</td></tr><tr><td>receive-errors</td><td>`sum(rate(container_network_receive_errors_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m])) by (pod_name)`</td></tr><tr><td>transmit-packets</td><td>`sum(rate(container_network_transmit_packets_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m])) by (pod_name)`</td></tr><tr><td>transmit-dropped</td><td>`sum(rate(container_network_transmit_packets_dropped_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m])) by (pod_name)`</td></tr><tr><td>transmit-errors</td><td>`sum(rate(container_network_transmit_errors_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m])) by (pod_name)`</td></tr></table> |
| Summary | <table><tr><td>receive-packets</td><td>`sum(rate(container_network_receive_packets_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m]))`</td></tr><tr><td>receive-dropped</td><td>`sum(rate(container_network_receive_packets_dropped_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m]))`</td></tr><tr><td>receive-errors</td><td>`sum(rate(container_network_receive_errors_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m]))`</td></tr><tr><td>transmit-packets</td><td>`sum(rate(container_network_transmit_packets_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m]))`</td></tr><tr><td>transmit-dropped</td><td>`sum(rate(container_network_transmit_packets_dropped_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m]))`</td></tr><tr><td>transmit-errors</td><td>`sum(rate(container_network_transmit_errors_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m]))`</td></tr></table> |
### Workload Network I/O
| Catalog | Expression |
| --- | --- |
| Detail | <table><tr><td>receive</td><td>`sum(rate(container_network_receive_bytes_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m])) by (pod_name)`</td></tr><tr><td>transmit</td><td>`sum(rate(container_network_transmit_bytes_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m])) by (pod_name)`</td></tr></table> |
| Summary | <table><tr><td>receive</td><td>`sum(rate(container_network_receive_bytes_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m]))`</td></tr><tr><td>transmit</td><td>`sum(rate(container_network_transmit_bytes_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m]))`</td></tr></table> |
### Workload Disk I/O
| Catalog | Expression |
| --- | --- |
| Detail | <table><tr><td>read</td><td>`sum(rate(container_fs_reads_bytes_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m])) by (pod_name)`</td></tr><tr><td>write</td><td>`sum(rate(container_fs_writes_bytes_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m])) by (pod_name)`</td></tr></table> |
| Summary | <table><tr><td>read</td><td>`sum(rate(container_fs_reads_bytes_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m]))`</td></tr><tr><td>write</td><td>`sum(rate(container_fs_writes_bytes_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m]))`</td></tr></table> |
# Pod Metrics
### Pod CPU Utilization
| Catalog | Expression |
| --- | --- |
| Detail | <table><tr><td>cfs throttled seconds</td><td>`sum(rate(container_cpu_cfs_throttled_seconds_total{container_name!="POD",namespace="$namespace",pod_name="$podName", container_name!=""}[5m])) by (container_name)`</td></tr><tr><td>usage seconds</td><td>`sum(rate(container_cpu_usage_seconds_total{container_name!="POD",namespace="$namespace",pod_name="$podName", container_name!=""}[5m])) by (container_name)`</td></tr><tr><td>system seconds</td><td>`sum(rate(container_cpu_system_seconds_total{container_name!="POD",namespace="$namespace",pod_name="$podName", container_name!=""}[5m])) by (container_name)`</td></tr><tr><td>user seconds</td><td>`sum(rate(container_cpu_user_seconds_total{container_name!="POD",namespace="$namespace",pod_name="$podName", container_name!=""}[5m])) by (container_name)`</td></tr></table> |
| Summary | <table><tr><td>cfs throttled seconds</td><td>`sum(rate(container_cpu_cfs_throttled_seconds_total{container_name!="POD",namespace="$namespace",pod_name="$podName", container_name!=""}[5m]))`</td></tr><tr><td>usage seconds</td><td>`sum(rate(container_cpu_usage_seconds_total{container_name!="POD",namespace="$namespace",pod_name="$podName", container_name!=""}[5m]))`</td></tr><tr><td>system seconds</td><td>`sum(rate(container_cpu_system_seconds_total{container_name!="POD",namespace="$namespace",pod_name="$podName", container_name!=""}[5m]))`</td></tr><tr><td>user seconds</td><td>`sum(rate(container_cpu_user_seconds_total{container_name!="POD",namespace="$namespace",pod_name="$podName", container_name!=""}[5m]))`</td></tr></table> |
### Pod Memory Utilization
| Catalog | Expression |
| --- | --- |
| Detail | `sum(container_memory_working_set_bytes{container_name!="POD",namespace="$namespace",pod_name="$podName",container_name!=""}) by (container_name)` |
| Summary | `sum(container_memory_working_set_bytes{container_name!="POD",namespace="$namespace",pod_name="$podName",container_name!=""})` |
### Pod Network Packets
| Catalog | Expression |
| --- | --- |
| Detail | <table><tr><td>receive-packets</td><td>`sum(rate(container_network_receive_packets_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`</td></tr><tr><td>receive-dropped</td><td>`sum(rate(container_network_receive_packets_dropped_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`</td></tr><tr><td>receive-errors</td><td>`sum(rate(container_network_receive_errors_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`</td></tr><tr><td>transmit-packets</td><td>`sum(rate(container_network_transmit_packets_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`</td></tr><tr><td>transmit-dropped</td><td>`sum(rate(container_network_transmit_packets_dropped_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`</td></tr><tr><td>transmit-errors</td><td>`sum(rate(container_network_transmit_errors_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`</td></tr></table> |
| Summary | <table><tr><td>receive-packets</td><td>`sum(rate(container_network_receive_packets_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`</td></tr><tr><td>receive-dropped</td><td>`sum(rate(container_network_receive_packets_dropped_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`</td></tr><tr><td>receive-errors</td><td>`sum(rate(container_network_receive_errors_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`</td></tr><tr><td>transmit-packets</td><td>`sum(rate(container_network_transmit_packets_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`</td></tr><tr><td>transmit-dropped</td><td>`sum(rate(container_network_transmit_packets_dropped_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`</td></tr><tr><td>transmit-errors</td><td>`sum(rate(container_network_transmit_errors_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`</td></tr></table> |
### Pod Network I/O
| Catalog | Expression |
| --- | --- |
| Detail | <table><tr><td>receive</td><td>`sum(rate(container_network_receive_bytes_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`</td></tr><tr><td>transmit</td><td>`sum(rate(container_network_transmit_bytes_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`</td></tr></table> |
| Summary | <table><tr><td>receive</td><td>`sum(rate(container_network_receive_bytes_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`</td></tr><tr><td>transmit</td><td>`sum(rate(container_network_transmit_bytes_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`</td></tr></table> |
### Pod Disk I/O
| Catalog | Expression |
| --- | --- |
| Detail | <table><tr><td>read</td><td>`sum(rate(container_fs_reads_bytes_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m])) by (container_name)`</td></tr><tr><td>write</td><td>`sum(rate(container_fs_writes_bytes_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m])) by (container_name)`</td></tr></table> |
| Summary | <table><tr><td>read</td><td>`sum(rate(container_fs_reads_bytes_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`</td></tr><tr><td>write</td><td>`sum(rate(container_fs_writes_bytes_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`</td></tr></table> |
# Container Metrics
### Container CPU Utilization
| Catalog | Expression |
| --- | --- |
| cfs throttled seconds | `sum(rate(container_cpu_cfs_throttled_seconds_total{namespace="$namespace",pod_name="$podName",container_name="$containerName"}[5m]))` |
| usage seconds | `sum(rate(container_cpu_usage_seconds_total{namespace="$namespace",pod_name="$podName",container_name="$containerName"}[5m]))` |
| system seconds | `sum(rate(container_cpu_system_seconds_total{namespace="$namespace",pod_name="$podName",container_name="$containerName"}[5m]))` |
| user seconds | `sum(rate(container_cpu_user_seconds_total{namespace="$namespace",pod_name="$podName",container_name="$containerName"}[5m]))` |
### Container Memory Utilization
`sum(container_memory_working_set_bytes{namespace="$namespace",pod_name="$podName",container_name="$containerName"})`
### Container Disk I/O
| Catalog | Expression |
| --- | --- |
| read | `sum(rate(container_fs_reads_bytes_total{namespace="$namespace",pod_name="$podName",container_name="$containerName"}[5m]))` |
| write | `sum(rate(container_fs_writes_bytes_total{namespace="$namespace",pod_name="$podName",container_name="$containerName"}[5m]))` |
docs/explanations/integrations-in-rancher/monitoring-and-alerting/rbac-for-monitoring.md
+181
View File
@@ -0,0 +1,181 @@
---
title: Role-based Access Control
shortTitle: RBAC
weight: 2
---
This section describes the expectations for RBAC for Rancher Monitoring.
- [Cluster Admins](#cluster-admins)
- [Users with Kubernetes ClusterRole-based Permissions](#users-with-kubernetes-clusterrole-based-permissions)
- [Users with Kubernetes Admin/Edit Permissions](#users-with-kubernetes-admin-edit-permissions)
- [Users with Kubernetes View Permissions](#users-with-kubernetes-view-permissions)
- [Additional Monitoring Roles](#additional-monitoring-roles)
- [Additional Monitoring ClusterRoles](#additional-monitoring-clusterroles)
- [Users with Rancher Based Permissions](#users-with-rancher-based-permissions)
- [Differences in 2.5.x](#differences-in-2-5-x)
- [Assigning Additional Access](#assigning-additional-access)
- [Role-based Access Control for Grafana](#role-based-access-control-for-grafana)
# Cluster Admins
By default, only those with the cluster-admin `ClusterRole` should be able to:
- Install the `rancher-monitoring` App onto a cluster and all other relevant configuration performed on the chart deploy
- e.g. whether default dashboards are created, what exporters are deployed onto the cluster to collect metrics, etc.
- Create / modify / delete Prometheus deployments in the cluster via Prometheus CRs
- Create / modify / delete Alertmanager deployments in the cluster via Alertmanager CRs
- Persist new Grafana dashboards or datasources via creating ConfigMaps in the appropriate namespace
- Expose certain Prometheus metrics to the k8s Custom Metrics API for HPA via a Secret in the `cattle-monitoring-system` namespace
# Users with Kubernetes ClusterRole-based Permissions
The `rancher-monitoring` chart installs the following three `ClusterRoles`. By default, they aggregate into the corresponding k8s `ClusterRoles`:
| ClusterRole | Aggregates To Default K8s ClusterRole |
| ------------------------------| ---------------------------|
| `monitoring-admin` | `admin`|
| `monitoring-edit` | `edit` |
| `monitoring-view` | `view ` |
These `ClusterRoles` provide different levels of access to the Monitoring CRDs based on the actions that can be performed:
| CRDs (monitoring.coreos.com) | Admin | Edit | View |
| ------------------------------| ---------------------------| ---------------------------| ---------------------------|
| <ul><li>`prometheuses`</li><li>`alertmanagers`</li></ul>| Get, List, Watch | Get, List, Watch | Get, List, Watch |
| <ul><li>`servicemonitors`</li><li>`podmonitors`</li><li>`prometheusrules`</li></ul>| * | * | Get, List, Watch |
On a high level, the following permissions are assigned by default as a result.
### Users with Kubernetes Admin/Edit Permissions
Only those with the the cluster-admin, admin or edit `ClusterRole` should be able to:
- Modify the scrape configuration of Prometheus deployments via ServiceMonitor and PodMonitor CRs
- Modify the alerting / recording rules of a Prometheus deployment via PrometheusRules CRs
### Users with Kubernetes View Permissions
Only those with who have some Kubernetes `ClusterRole` should be able to:
- View the configuration of Prometheuses that are deployed within the cluster
- View the configuration of Alertmanagers that are deployed within the cluster
- View the scrape configuration of Prometheus deployments via ServiceMonitor and PodMonitor CRs
- View the alerting/recording rules of a Prometheus deployment via PrometheusRules CRs
### Additional Monitoring Roles
Monitoring also creates additional `Roles` that are not assigned to users by default but are created within the cluster. They can be bound to a namespace by deploying a `RoleBinding` that references it. To define a `RoleBinding` with `kubectl` instead of through Rancher, click [here](#assigning-roles-and-clusterroles-with-kubectl).
Admins should use these roles to provide more fine-grained access to users:
| Role | Purpose |
| ------------------------------| ---------------------------|
| monitoring-config-admin | Allow admins to assign roles to users to be able to view / modify Secrets and ConfigMaps within the cattle-monitoring-system namespace. Modifying Secrets / ConfigMaps in this namespace could allow users to alter the cluster's Alertmanager configuration, Prometheus Adapter configuration, additional Grafana datasources, TLS secrets, etc. |
| monitoring-config-edit | Allow admins to assign roles to users to be able to view / modify Secrets and ConfigMaps within the cattle-monitoring-system namespace. Modifying Secrets / ConfigMaps in this namespace could allow users to alter the cluster's Alertmanager configuration, Prometheus Adapter configuration, additional Grafana datasources, TLS secrets, etc. |
| monitoring-config-view | Allow admins to assign roles to users to be able to view Secrets and ConfigMaps within the cattle-monitoring-system namespace. Viewing Secrets / ConfigMaps in this namespace could allow users to observe the cluster's Alertmanager configuration, Prometheus Adapter configuration, additional Grafana datasources, TLS secrets, etc. |
| monitoring-dashboard-admin | Allow admins to assign roles to users to be able to edit / view ConfigMaps within the cattle-dashboards namespace. ConfigMaps in this namespace will correspond to Grafana Dashboards that are persisted onto the cluster. |
| monitoring-dashboard-edit | Allow admins to assign roles to users to be able to edit / view ConfigMaps within the cattle-dashboards namespace. ConfigMaps in this namespace will correspond to Grafana Dashboards that are persisted onto the cluster. |
| monitoring-dashboard-view | Allow admins to assign roles to users to be able to view ConfigMaps within the cattle-dashboards namespace. ConfigMaps in this namespace will correspond to Grafana Dashboards that are persisted onto the cluster. |
### Additional Monitoring ClusterRoles
Monitoring also creates additional `ClusterRoles` that are not assigned to users by default but are created within the cluster. They are not aggregated by default but can be bound to a namespace by deploying a `RoleBinding` or `ClusterRoleBinding` that references it. To define a `RoleBinding` with `kubectl` instead of through Rancher, click [here](#assigning-roles-and-clusterroles-with-kubectl).
| Role | Purpose |
| ------------------------------| ---------------------------|
| monitoring-ui-view | <a id="monitoring-ui-view"></a>_Available as of Monitoring v2 14.5.100+_ Provides read-only access to external Monitoring UIs by giving a user permission to list the Prometheus, Alertmanager, and Grafana endpoints and make GET requests to Prometheus, Grafana, and Alertmanager UIs through the Rancher proxy. |
### Assigning Roles and ClusterRoles with kubectl
An alternative method to using Rancher to attach a `Role` or `ClusterRole` to a user or group is by defining bindings in YAML files that you create. You must first configure the `RoleBinding` with the YAML file, then you apply the config changes by running the `kubectl apply` command.
* **Roles**: Below is an example of a YAML file to help you configure `RoleBindings` in Kubernetes. You will need to fill in the name below, and name is case-sensitive.
```
# monitoring-config-view-role-binding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: monitoring-config-view
namespace: cattle-monitoring-system
roleRef:
kind: Role
name: monitoring-config-view
apiGroup: rbac.authorization.k8s.io
subjects:
- kind: User
name: u-b4qkhsnliz # this can be found via `kubectl get users -A`
apiGroup: rbac.authorization.k8s.io
```
* **kubectl**: Below is an example of a `kubectl` command used to apply the binding you've created in the YAML file. As noted, you will need to fill in your YAML filename accordingly.
* **`kubectl apply -f monitoring-config-view-role-binding.yaml`
# Users with Rancher Based Permissions
The relationship between the default roles deployed by Rancher (i.e. cluster-owner, cluster-member, project-owner, project-member), the default Kubernetes roles, and the roles deployed by the rancher-monitoring chart are detailed in the table below:
<figcaption>Default Rancher Permissions and Corresponding Kubernetes ClusterRoles</figcaption>
| Rancher Role | Kubernetes Role | Monitoring ClusterRole / Role | ClusterRoleBinding or RoleBinding? |
| --------- | --------- | --------- | --------- |
| cluster-owner | cluster-admin | N/A | ClusterRoleBinding |
| cluster-member | admin | monitoring-admin | ClusterRoleBinding |
| project-owner | admin | monitoring-admin | RoleBinding within Project namespace |
| project-member | edit | monitoring-edit | RoleBinding within Project namespace |
In addition to these default Roles, the following additional Rancher project roles can be applied to members of your Cluster to provide additional access to Monitoring. These Rancher Roles will be tied to ClusterRoles deployed by the Monitoring chart:
<figcaption>Non-default Rancher Permissions and Corresponding Kubernetes ClusterRoles</figcaption>
| Rancher Role | Kubernetes ClusterRole | Available In Rancher From | Available in Monitoring v2 From |
|--------------------------|-------------------------------|-------|------|
| View Monitoring* | [monitoring-ui-view](#monitoring-ui-view) | 2.4.8+ | 9.4.204+ |
\* A User bound to the **View Monitoring** Rancher Role only has permissions to access external Monitoring UIs if provided links to those UIs. In order to access the Monitoring Pane to get those links, the User must be a Project Member of at least one Project.
### Differences in 2.5.x
Users with the project-member or project-owners roles assigned will not be given access to either Prometheus or Grafana in Rancher 2.5.x since we only create Grafana or Prometheus on a cluster-level.
In addition, while project owners will still be only able to add ServiceMonitors / PodMonitors that scrape resources within their project's namespace by default, PrometheusRules are not scoped to a single namespace / project. Therefore, any alert rules or recording rules created by project-owners within their project namespace will be applied across the entire cluster, although they will be unable to view / edit / delete any rules that were created outside the project's namespace.
### Assigning Additional Access
If cluster-admins would like to provide additional admin/edit access to users outside of the roles offered by the rancher-monitoring chart, the following table identifies the potential impact:
|CRDs (monitoring.coreos.com) | Can it cause impact outside of a namespace / project? | Impact |
|----------------------------| ------| ----------------------------|
| `prometheuses`| Yes, this resource can scrape metrics from any targets across the entire cluster (unless the Operator itself is otherwise configured). | User will be able to define the configuration of new cluster-level Prometheus deployments that should be created in the cluster. |
| `alertmanagers`| No | User will be able to define the configuration of new cluster-level Alertmanager deployments that should be created in the cluster. Note: if you just want to allow users to configure settings like Routes and Receivers, you should just provide access to the Alertmanager Config Secret instead. |
| <ul><li>`servicemonitors`</li><li>`podmonitors`</li></ul>| No, not by default; this is configurable via `ignoreNamespaceSelectors` on the Prometheus CR. | User will be able to set up scrapes by Prometheus on endpoints exposed by Services / Pods within the namespace they are given this permission in. |
| `prometheusrules`| Yes, PrometheusRules are cluster-scoped. | User will be able to define alert or recording rules on Prometheus based on any series collected across the entire cluster. |
| k8s Resources | Namespace | Can it cause impact outside of a namespace / project? | Impact |
|----------------------------| ------| ------| ----------------------------|
| <ul><li>`secrets`</li><li>`configmaps`</li></ul>| `cattle-monitoring-system` | Yes, Configs and Secrets in this namespace can impact the entire monitoring / alerting pipeline. | User will be able to create or edit Secrets / ConfigMaps such as the Alertmanager Config, Prometheus Adapter Config, TLS secrets, additional Grafana datasources, etc. This can have broad impact on all cluster monitoring / alerting. |
| <ul><li>`secrets`</li><li>`configmaps`</li></ul>| `cattle-dashboards` | Yes, Configs and Secrets in this namespace can create dashboards that make queries on all metrics collected at a cluster-level. | User will be able to create Secrets / ConfigMaps that persist new Grafana Dashboards only. |
# Role-based Access Control for Grafana
Rancher allows any users who are authenticated by Kubernetes and have access the Grafana service deployed by the Rancher Monitoring chart to access Grafana via the Rancher Dashboard UI. By default, all users who are able to access Grafana are given the [Viewer](https://grafana.com/docs/grafana/latest/permissions/organization_roles/#viewer-role) role, which allows them to view any of the default dashboards deployed by Rancher.
However, users can choose to log in to Grafana as an [Admin](https://grafana.com/docs/grafana/latest/permissions/organization_roles/#admin-role) if necessary. The default Admin username and password for the Grafana instance will be `admin`/`prom-operator`, but alternative credentials can also be supplied on deploying or upgrading the chart.
To see the Grafana UI, install `rancher-monitoring`. Then:
1. In the upper left corner, click **☰ > Cluster Management**.
1. On the **Clusters** page, go to the cluster where you want to see the visualizations and click **Explore**.
1. In the left navigation bar, click **Monitoring**.
1. Click **Grafana**.
<figcaption>Cluster Compute Resources Dashboard in Grafana</figcaption>
![Cluster Compute Resources Dashboard in Grafana](/img/cluster-compute-resources-dashboard.png)
<figcaption>Default Dashboards in Grafana</figcaption>
![Default Dashboards in Grafana](/img/grafana-default-dashboard.png)
docs/explanations/integrations-in-rancher/monitoring-and-alerting/windows-support.md
+62
View File
@@ -0,0 +1,62 @@
---
title: Windows Cluster Support for Monitoring V2
shortTitle: Windows Support
weight: 5
---
_Available as of v2.5.8_
Starting at Monitoring V2 14.5.100 (used by default in Rancher 2.5.8), Monitoring V2 can now be deployed on a Windows cluster and will scrape metrics from Windows nodes using [prometheus-community/windows_exporter](https://github.com/prometheus-community/windows_exporter) (previously named `wmi_exporter`).
- [Comparison to Monitoring V1](#comparison-to-monitoring-v1)
- [Cluster Requirements](#cluster-requirements)
- [Upgrading Existing Clusters to wins v0.1.0](#upgrading-existing-clusters-to-wins-v0-1-0)
# Comparison to Monitoring V1
Unlike Monitoring V1 for Windows, metrics collected by `windows_exporter` will be labeled as `windows_` instead of `wmi_` in accordance to a naming change from upstream from `wmi_exporter` to `windows_exporter`.
In addition, Monitoring V2 for Windows will no longer require users to keep port 9796 open on Windows hosts since the host metrics will published directly onto a port exposed on the windows-exporter Pod. This feature was powered by recent changes made by `wins` v0.1.0 to support publishing ports exposed on the hostNetwork on Pods that use wins to run a privileged Windows binary as a host process.
# Cluster Requirements
Monitoring V2 for Windows can only scrape metrics from Windows hosts that have a minimum `wins` version of v0.1.0. To be able to fully deploy Monitoring V2 for Windows, all of your hosts must meet this requirement.
If you provision a fresh RKE1 cluster in Rancher 2.5.8, your cluster should already meet this requirement.
### Upgrading Existing Clusters to wins v0.1.0
If the cluster was provisioned before Rancher 2.5.8 (even if the current Rancher version is 2.5.8), you will not be able to successfully deploy Monitoring V2 for Windows until you upgrade the wins version on each host to at least v0.1.0.
To facilitate this upgrade, Rancher 2.5.8 has released a brand new Helm chart called `rancher-wins-upgrader`.
> **Prerequisite:** Make sure Monitoring V1 for Windows is uninstalled.
1. Deploy `rancher-wins-upgrader` with the following override:
```yaml
# Masquerading bootstraps the wins-upgrader installation via
# a previously whitelisted process path since the normal install path,
# c:\etc\rancher\wins\wins-upgrade.exe is not normally whitelisted.
# In this case, we are using the previously whitelisted process
# path used by Monitoring V1.
masquerade:
enabled: true
as: c:\\etc\wmi-exporter\wmi-exporter.exe
```
:::note Note for Non-Default Windows Prefix Path:
- If you set up the RKE cluster with a `cluster.yml` that has a non-default `win_prefix_path`, you will need to update the `masquerade.as` field with your prefix path in place of `c:\\`.
- For example, if you have `win_prefix_path: 'c:\host\opt\'`, then you will need to set `as: c:\host\opt\etc\wmi-exporter\wmi-exporter.exe`.
:::
2. Once all your hosts have been successfully upgraded, please ensure that you deploy the Helm chart once again with default values to avoid conflicts with the following settings:
```yaml
masquerade:
enabled: false
```
**Result:** The hosts are ready for Monitoring V2 to be installed. You may choose to uninstall the `rancher-wins-upgrader` chart or keep it in your cluster to facilitate future upgrades.
For more information on how it can be used, please see the [README.md](https://github.com/rancher/wins/blob/master/charts/rancher-wins-upgrader/README.md) of the chart.
docs/explanations/integrations-in-rancher/neuvector.md
+124
View File
@@ -0,0 +1,124 @@
---
title: NeuVector Integration
weight: 22
---
### NeuVector Integration in Rancher
New in Rancher v2.6.5, [NeuVector 5.x](https://open-docs.neuvector.com/) is an open-source container-centric security platform that is now integrated into Rancher. NeuVector offers real-time compliance, visibility, and protection for critical applications and data during runtime. NeuVector provides a firewall, container process/file system monitoring, security auditing with CIS benchmarks, and vulnerability scanning. For more information on Rancher security, please see the [security documentation](../../pages-for-subheaders/rancher-security.md).
NeuVector can be enabled through a Helm chart that may be installed either through **Apps & Marketplace** or through the **Cluster Tools** button in the Rancher UI. Once the Helm chart is installed, users can easily [deploy and manage NeuVector clusters within Rancher](https://open-docs.neuvector.com/deploying/rancher#deploy-and-manage-neuvector-through-rancher-apps-marketplace).
### Installing NeuVector with Rancher
The Harvester Helm Chart is used to manage access to the NeuVector UI in Rancher where users can navigate directly to deploy and manage their NeuVector clusters.
**To navigate to and install the NeuVector chart through Apps & Marketplace:**
1. Click **☰ > Cluster Management**.
1. On the Clusters page, go to the cluster where you want to deploy NeuVector, and click **Explore**.
1. Go to **Apps & Marketplace > Charts**, and install **NeuVector** from the chart repo.
1. Different cluster types require different container runtimes. When configuring Helm chart values, go to the **Container Runtime** section, and select your runtime in accordance with the cluster type. Finally, click **Install** again.
Some examples are as follows:
- RKE1: `docker`
- K3s and RKE2: `k3scontainerd`
- AKS: `containerd` for v1.19 and up
- EKS: `docker` for v1.22 and below; `containerd` for v1.23 and up
- GKE: `containerd` (see the [Google docs](https://cloud.google.com/kubernetes-engine/docs/concepts/using-containerd) for more)
:::note
Only one container runtime engine may be selected at a time during installation.
:::
**To navigate to and install the NeuVector chart through Cluster Tools:**
1. Click **☰ > Cluster Management**.
1. On the Clusters page, go to the cluster where you want to deploy NeuVector, and click **Explore**.
1. Click on **Cluster Tools** at the bottom of the left navigation bar.
1. Repeat step 4 above to select your container runtime accordingly, then click **Install** again.
### Accessing NeuVector from the Rancher UI
1. Navigate to the cluster explorer of the cluster where NeuVector is installed. In the left navigation bar, click **NeuVector**.
1. Click the external link to go to the NeuVector UI. Once the link is selected, users must accept the `END USER LICENSE AGREEMENT` to access the NeuVector UI.
### Uninstalling NeuVector from the Rancher UI
**To uninstall from Apps & Marketplace:**
1. Click **☰ > Cluster Management**.
1. Under **Apps & Marketplace**, click **Installed Apps**.
1. Under `cattle-neuvector-system`, select both the NeuVector app (and the associated CRD if desired), then click **Delete**.
**To uninstall from Cluster Tools:**
1. Click **☰ > Cluster Management**.
1. Click on **Cluster Tools** at the bottom-left of the screen, then click on the trash can icon under the NeuVector chart. Select `Delete the CRD associated with this app` if desired, then click **Delete**.
### GitHub Repository
The NeuVector project is available [here](https://github.com/neuvector/neuvector).
### Documentation
The NeuVector documentation is [here](https://open-docs.neuvector.com/).
### Architecture
The NeuVector security solution contains four types of security containers: Controllers, Enforcers, Managers, and Scanners. A special container called an All-in-One is also provided to combine the Controller, Enforcer, and Manager functions all in one container, primarily for Docker-native deployments. There is also an Updater which, when run, will update the CVE database.
- **Controller:** Manages the NeuVector Enforcer container; provides REST APIs for the management console.
- **Enforcer:** Enforces security policies.
- **Manager:** Provides a web-UI and CLI console to manage the NeuVector platform.
- **All-in-One:** Includes the Controller, Enforcer, and Manager.
- **Scanner:** Performs the vulnerability and compliance scanning for images, containers, and nodes.
- **Updater:** Updates the CVE database for Neuvector (when run); redeploys scanner pods.
<figcaption>**NeuVector Security Containers:**</figcaption>
![NeuVector Security Containers](/img/neuvector-security-containers.png)
<figcaption>**NeuVector Architecture:**</figcaption>
![NeuVector Architecture](/img/neuvector-architecture.png)
To learn more about NeuVector's architecture, please refer [here](https://open-docs.neuvector.com/basics/overview#architecture).
### CPU and Memory Allocations
Below are the minimum recommended computing resources for the NeuVector chart installation in a default deployment. Note that the resource limit is not set.
| Container | CPU - Request | Memory - Request |
|------------|--------|---------|
| Controller | 3 (1GB 1vCPU needed per controller) | *
| Enforcer | On all nodes (500MB .5vCPU) | 1GB
| Manager | 1 (500MB .5vCPU) | *
| Scanner | 3 (100MB .5vCPU) | *
\* Minimum 1GB of memory total required for Controller, Manager, and Scanner containers combined.
### Support Limitations
* Only admins and cluster owners are currently supported.
* Fleet multi-cluster deployment is not supported.
* NeuVector is not supported on a Windows cluster.
* NeuVector installation is not supported on hardened clusters.
* NeuVector installation is not supported on SELinux clusters.
* NeuVector installation is not supported on clusters in an air-gapped environment.
### Other Limitations
* Currently, NeuVector feature chart installation fails when a NeuVector partner chart already exists. To work around this issue, uninstall the NeuVector partner chart and reinstall the NeuVector feature chart.
* Sometimes when the controllers are not ready, the NeuVector UI is not accessible from the Rancher UI. During this time, controllers will try to restart, and it takes a few minutes for the controllers to be active.
* Container runtime is not auto-detected for different cluster types when installing the NeuVector chart. To work around this, you can specify the runtime manually.
docs/explanations/integrations-in-rancher/opa-gatekeeper.md
+112
View File
@@ -0,0 +1,112 @@
---
title: OPA Gatekeeper
weight: 16
---
To ensure consistency and compliance, every organization needs the ability to define and enforce policies in its environment in an automated way. [OPA (Open Policy Agent)](https://www.openpolicyagent.org/) is a policy engine that facilitates policy-based control for cloud native environments. Rancher provides the ability to enable OPA Gatekeeper in Kubernetes clusters, and also installs a couple of built-in policy definitions, which are also called constraint templates.
OPA provides a high-level declarative language that lets you specify policy as code and ability to extend simple APIs to offload policy decision-making.
[OPA Gatekeeper](https://github.com/open-policy-agent/gatekeeper) is a project that provides integration between OPA and Kubernetes. OPA Gatekeeper provides:
- An extensible, parameterized policy library.
- Native Kubernetes CRDs for instantiating the policy library, also called “constraints."
- Native Kubernetes CRDs for extending the policy library, also called "constraint templates."
- Audit functionality.
To read more about OPA, please refer to the [official documentation.](https://www.openpolicyagent.org/docs/latest/)
# How the OPA Gatekeeper Integration Works
Kubernetes provides the ability to extend API server functionality via admission controller webhooks, which are invoked whenever a resource is created, updated or deleted. Gatekeeper is installed as a validating webhook and enforces policies defined by Kubernetes custom resource definitions. In addition to the admission control usage, Gatekeeper provides the capability to audit existing resources in Kubernetes clusters and mark current violations of enabled policies.
OPA Gatekeeper is made available via Rancher's Helm system chart, and it is installed in a namespace named `gatekeeper-system.`
# Enabling OPA Gatekeeper in a Cluster
:::note
In Rancher v2.5, the OPA Gatekeeper application was improved. The Rancher v2.4 feature can't be upgraded to the new version in Rancher v2.5. If you installed OPA Gatekeeper in Rancher v2.4, you will need to uninstall OPA Gatekeeper and its CRDs from the old UI, then reinstall it in Rancher v2.5. To uninstall the CRDs run the following command in the kubectl console `kubectl delete crd configs.config.gatekeeper.sh constrainttemplates.templates.gatekeeper.sh`.
:::
:::note Prerequisite:
Only administrators and cluster owners can enable OPA Gatekeeper.
:::
The OPA Gatekeeper Helm chart can be installed from **Apps & Marketplace**.
### Enabling OPA Gatekeeper
1. In the upper left corner, click **☰ > Cluster Management**.
1. In the **Clusters** page, go to the cluster where you want to enable OPA Gatekeeper and click **Explore**.
1. In the left navigation bar, click **Apps & Marketplace**.
1. Click **Charts** and click **OPA Gatekeeper**.
1. Click **Install**.
**Result:** OPA Gatekeeper is deployed in your Kubernetes cluster.
# Constraint Templates
[Constraint templates](https://github.com/open-policy-agent/gatekeeper#constraint-templates) are Kubernetes custom resources that define the schema and Rego logic of the OPA policy to be applied by Gatekeeper. For more information on the Rego policy language, refer to the [official documentation.](https://www.openpolicyagent.org/docs/latest/policy-language/)
When OPA Gatekeeper is enabled, Rancher installs some templates by default.
To list the constraint templates installed in the cluster, go to the left side menu under OPA Gatekeeper and click on **Templates**.
Rancher also provides the ability to create your own constraint templates by importing YAML definitions.
# Creating and Configuring Constraints
[Constraints](https://github.com/open-policy-agent/gatekeeper#constraints) are Kubernetes custom resources that define the scope of objects to which a specific constraint template applies to. The complete policy is defined by constraint templates and constraints together.
:::note Prerequisite:
OPA Gatekeeper must be enabled in the cluster.
:::
To list the constraints installed, go to the left side menu under OPA Gatekeeper, and click on **Constraints**.
New constraints can be created from a constraint template.
Rancher provides the ability to create a constraint by using a convenient form that lets you input the various constraint fields.
The **Edit as yaml** option is also available to configure the the constraint's yaml definition.
### Exempting Rancher's System Namespaces from Constraints
When a constraint is created, ensure that it does not apply to any Rancher or Kubernetes system namespaces. If the system namespaces are not excluded, then it is possible to see many resources under them marked as violations of the constraint.
To limit the scope of the constraint only to user namespaces, always specify these namespaces under the **Match** field of the constraint.
Also, the constraint may interfere with other Rancher functionality and deny system workloads from being deployed. To avoid this, exclude all Rancher-specific namespaces from your constraints.
# Enforcing Constraints in your Cluster
When the **Enforcement Action** is **Deny,** the constraint is immediately enabled and will deny any requests that violate the policy defined. By default, the enforcement value is **Deny**.
When the **Enforcement Action** is **Dryrun,** then any resources that violate the policy are only recorded under the constraint's status field.
To enforce constraints, create a constraint using the form. In the **Enforcement Action** field, choose **Deny**.
# Audit and Violations in your Cluster
OPA Gatekeeper runs a periodic audit to check if any existing resource violates any enforced constraint. The audit-interval (default 300s) can be configured while installing Gatekeeper.
On the Gatekeeper page, any violations of the defined constraints are listed.
Also under **Constraints,** the number of violations of the constraint can be found.
The detail view of each constraint lists information about the resource that violated the constraint.
# Disabling Gatekeeper
1. Navigate to the cluster's Dashboard view
1. On the left side menu, expand the cluster menu and click on **OPA Gatekeeper**.
1. Click the **⋮ > Disable**.
**Result:** Upon disabling OPA Gatekeeper, all constraint templates and constraints will also be deleted.