mirror of
https://github.com/rancher/rancher-docs.git
synced 2026-09-26 13:08:11 +00:00
Remove unneeded intermediate folders
This commit is contained in:
Billy Tat
@@ -0,0 +1,225 @@
|
||||
---
|
||||
title: Backing up a Cluster
|
||||
weight: 2045
|
||||
---
|
||||
|
||||
import Tabs from '@theme/Tabs';
|
||||
import TabItem from '@theme/TabItem';
|
||||
|
||||
_Available as of v2.2.0_
|
||||
|
||||
In the Rancher UI, etcd backup and recovery for [Rancher launched Kubernetes clusters]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-provisioning/rke-clusters/) can be easily performed.
|
||||
|
||||
Rancher recommends configuring recurrent `etcd` snapshots for all production clusters. Additionally, one-time snapshots can easily be taken as well.
|
||||
|
||||
Snapshots of the etcd database are taken and saved either [locally onto the etcd nodes](#local-backup-target) or to a [S3 compatible target](#s3-backup-target). The advantages of configuring S3 is that if all etcd nodes are lost, your snapshot is saved remotely and can be used to restore the cluster.
|
||||
|
||||
This section covers the following topics:
|
||||
|
||||
- [How snapshots work](#how-snapshots-work)
|
||||
- [Configuring recurring snapshots](#configuring-recurring-snapshots)
|
||||
- [One-time snapshots](#one-time-snapshots)
|
||||
- [Snapshot backup targets](#snapshot-backup-targets)
|
||||
- [Local backup target](#local-backup-target)
|
||||
- [S3 backup target](#s3-backup-target)
|
||||
- [Using a custom CA certificate for S3](#using-a-custom-ca-certificate-for-s3)
|
||||
- [IAM Support for storing snapshots in S3](#iam-support-for-storing-snapshots-in-s3)
|
||||
- [Viewing available snapshots](#viewing-available-snapshots)
|
||||
- [Safe timestamps](#safe-timestamps)
|
||||
- [Enabling snapshot features for clusters created before Rancher v2.2.0](#enabling-snapshot-features-for-clusters-created-before-rancher-v2-2-0)
|
||||
|
||||
# How Snapshots Work
|
||||
|
||||
<Tabs>
|
||||
<TabItem value="Rancher v2.4.0+">
|
||||
|
||||
### Snapshot Components
|
||||
|
||||
When Rancher creates a snapshot, it includes three components:
|
||||
|
||||
- The cluster data in etcd
|
||||
- The Kubernetes version
|
||||
- The cluster configuration in the form of the `cluster.yml`
|
||||
|
||||
Because the Kubernetes version is now included in the snapshot, it is possible to restore a cluster to a prior Kubernetes version.
|
||||
|
||||
The multiple components of the snapshot allow you to select from the following options if you need to restore a cluster from a snapshot:
|
||||
|
||||
- **Restore just the etcd contents:** This restore is similar to restoring to snapshots in Rancher before v2.4.0.
|
||||
- **Restore etcd and Kubernetes version:** This option should be used if a Kubernetes upgrade is the reason that your cluster is failing, and you haven't made any cluster configuration changes.
|
||||
- **Restore etcd, Kubernetes versions and cluster configuration:** This option should be used if you changed both the Kubernetes version and cluster configuration when upgrading.
|
||||
|
||||
It's always recommended to take a new snapshot before any upgrades.
|
||||
|
||||
### Generating the Snapshot from etcd Nodes
|
||||
|
||||
For each etcd node in the cluster, the etcd cluster health is checked. If the node reports that the etcd cluster is healthy, a snapshot is created from it and optionally uploaded to S3.
|
||||
|
||||
The snapshot is stored in `/opt/rke/etcd-snapshots`. If the directory is configured on the nodes as a shared mount, it will be overwritten. On S3, the snapshot will always be from the last node that uploads it, as all etcd nodes upload it and the last will remain.
|
||||
|
||||
In the case when multiple etcd nodes exist, any created snapshot is created after the cluster has been health checked, so it can be considered a valid snapshot of the data in the etcd cluster.
|
||||
|
||||
### Snapshot Naming Conventions
|
||||
|
||||
The name of the snapshot is auto-generated. The `--name` option can be used to override the name of the snapshot when creating one-time snapshots with the RKE CLI.
|
||||
|
||||
When Rancher creates a snapshot of an RKE cluster, the snapshot name is based on the type (whether the snapshot is manual or recurring) and the target (whether the snapshot is saved locally or uploaded to S3). The naming convention is as follows:
|
||||
|
||||
- `m` stands for manual
|
||||
- `r` stands for recurring
|
||||
- `l` stands for local
|
||||
- `s` stands for S3
|
||||
|
||||
Some example snapshot names are:
|
||||
|
||||
- c-9dmxz-rl-8b2cx
|
||||
- c-9dmxz-ml-kr56m
|
||||
- c-9dmxz-ms-t6bjb
|
||||
- c-9dmxz-rs-8gxc8
|
||||
|
||||
### How Restoring from a Snapshot Works
|
||||
|
||||
On restore, the following process is used:
|
||||
|
||||
1. The snapshot is retrieved from S3, if S3 is configured.
|
||||
2. The snapshot is unzipped (if zipped).
|
||||
3. One of the etcd nodes in the cluster serves that snapshot file to the other nodes.
|
||||
4. The other etcd nodes download the snapshot and validate the checksum so that they all use the same snapshot for the restore.
|
||||
5. The cluster is restored and post-restore actions will be done in the cluster.
|
||||
|
||||
</TabItem>
|
||||
<TabItem value="Rancher before v2.4.0">
|
||||
|
||||
When Rancher creates a snapshot, only the etcd data is included in the snapshot.
|
||||
|
||||
Because the Kubernetes version is not included in the snapshot, there is no option to restore a cluster to a different Kubernetes version.
|
||||
|
||||
It's always recommended to take a new snapshot before any upgrades.
|
||||
|
||||
### Generating the Snapshot from etcd Nodes
|
||||
|
||||
For each etcd node in the cluster, the etcd cluster health is checked. If the node reports that the etcd cluster is healthy, a snapshot is created from it and optionally uploaded to S3.
|
||||
|
||||
The snapshot is stored in `/opt/rke/etcd-snapshots`. If the directory is configured on the nodes as a shared mount, it will be overwritten. On S3, the snapshot will always be from the last node that uploads it, as all etcd nodes upload it and the last will remain.
|
||||
|
||||
In the case when multiple etcd nodes exist, any created snapshot is created after the cluster has been health checked, so it can be considered a valid snapshot of the data in the etcd cluster.
|
||||
|
||||
### Snapshot Naming Conventions
|
||||
|
||||
The name of the snapshot is auto-generated. The `--name` option can be used to override the name of the snapshot when creating one-time snapshots with the RKE CLI.
|
||||
|
||||
When Rancher creates a snapshot of an RKE cluster, the snapshot name is based on the type (whether the snapshot is manual or recurring) and the target (whether the snapshot is saved locally or uploaded to S3). The naming convention is as follows:
|
||||
|
||||
- `m` stands for manual
|
||||
- `r` stands for recurring
|
||||
- `l` stands for local
|
||||
- `s` stands for S3
|
||||
|
||||
Some example snapshot names are:
|
||||
|
||||
- c-9dmxz-rl-8b2cx
|
||||
- c-9dmxz-ml-kr56m
|
||||
- c-9dmxz-ms-t6bjb
|
||||
- c-9dmxz-rs-8gxc8
|
||||
|
||||
### How Restoring from a Snapshot Works
|
||||
|
||||
On restore, the following process is used:
|
||||
|
||||
1. The snapshot is retrieved from S3, if S3 is configured.
|
||||
2. The snapshot is unzipped (if zipped).
|
||||
3. One of the etcd nodes in the cluster serves that snapshot file to the other nodes.
|
||||
4. The other etcd nodes download the snapshot and validate the checksum so that they all use the same snapshot for the restore.
|
||||
5. The cluster is restored and post-restore actions will be done in the cluster.
|
||||
|
||||
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
|
||||
# Configuring Recurring Snapshots
|
||||
|
||||
Select how often you want recurring snapshots to be taken as well as how many snapshots to keep. The amount of time is measured in hours. With timestamped snapshots, the user has the ability to do a point-in-time recovery.
|
||||
|
||||
By default, [Rancher launched Kubernetes clusters]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-provisioning/rke-clusters/) are configured to take recurring snapshots (saved to local disk). To protect against local disk failure, using the [S3 Target](#s3-backup-target) or replicating the path on disk is advised.
|
||||
|
||||
During cluster provisioning or editing the cluster, the configuration for snapshots can be found in the advanced section for **Cluster Options**. Click on **Show advanced options**.
|
||||
|
||||
In the **Advanced Cluster Options** section, there are several options available to configure:
|
||||
|
||||
| Option | Description | Default Value|
|
||||
| --- | ---| --- |
|
||||
| etcd Snapshot Backup Target | Select where you want the snapshots to be saved. Options are either local or in S3 | local|
|
||||
|Recurring etcd Snapshot Enabled| Enable/Disable recurring snapshots | Yes|
|
||||
| Recurring etcd Snapshot Creation Period | Time in hours between recurring snapshots| 12 hours |
|
||||
| Recurring etcd Snapshot Retention Count | Number of snapshots to retain| 6 |
|
||||
|
||||
# One-Time Snapshots
|
||||
|
||||
In addition to recurring snapshots, you may want to take a "one-time" snapshot. For example, before upgrading the Kubernetes version of a cluster it's best to backup the state of the cluster to protect against upgrade failure.
|
||||
|
||||
1. In the **Global** view, navigate to the cluster that you want to take a one-time snapshot.
|
||||
|
||||
2. Click the **⋮ > Snapshot Now**.
|
||||
|
||||
**Result:** Based on your [snapshot backup target](#snapshot-backup-targets), a one-time snapshot will be taken and saved in the selected backup target.
|
||||
|
||||
# Snapshot Backup Targets
|
||||
|
||||
Rancher supports two different backup targets:
|
||||
|
||||
* [Local Target](#local-backup-target)
|
||||
* [S3 Target](#s3-backup-target)
|
||||
|
||||
### Local Backup Target
|
||||
|
||||
By default, the `local` backup target is selected. The benefits of this option is that there is no external configuration. Snapshots are automatically saved locally to the etcd nodes in the [Rancher launched Kubernetes clusters]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-provisioning/rke-clusters/) in `/opt/rke/etcd-snapshots`. All recurring snapshots are taken at configured intervals. The downside of using the `local` backup target is that if there is a total disaster and _all_ etcd nodes are lost, there is no ability to restore the cluster.
|
||||
|
||||
### S3 Backup Target
|
||||
|
||||
The `S3` backup target allows users to configure a S3 compatible backend to store the snapshots. The primary benefit of this option is that if the cluster loses all the etcd nodes, the cluster can still be restored as the snapshots are stored externally. Rancher recommends external targets like `S3` backup, however its configuration requirements do require additional effort that should be considered.
|
||||
|
||||
| Option | Description | Required|
|
||||
|---|---|---|
|
||||
|S3 Bucket Name| S3 bucket name where backups will be stored| *|
|
||||
|S3 Region|S3 region for the backup bucket| |
|
||||
|S3 Region Endpoint|S3 regions endpoint for the backup bucket|* |
|
||||
|S3 Access Key|S3 access key with permission to access the backup bucket|*|
|
||||
|S3 Secret Key|S3 secret key with permission to access the backup bucket|*|
|
||||
| Custom CA Certificate | A custom certificate used to access private S3 backends _Available as of v2.2.5_ ||
|
||||
|
||||
### Using a custom CA certificate for S3
|
||||
|
||||
_Available as of v2.2.5_
|
||||
|
||||
The backup snapshot can be stored on a custom `S3` backup like [minio](https://min.io/). If the S3 back end uses a self-signed or custom certificate, provide a custom certificate using the `Custom CA Certificate` option to connect to the S3 backend.
|
||||
|
||||
### IAM Support for Storing Snapshots in S3
|
||||
|
||||
The `S3` backup target supports using IAM authentication to AWS API in addition to using API credentials. An IAM role gives temporary permissions that an application can use when making API calls to S3 storage. To use IAM authentication, the following requirements must be met:
|
||||
|
||||
- The cluster etcd nodes must have an instance role that has read/write access to the designated backup bucket.
|
||||
- The cluster etcd nodes must have network access to the specified S3 endpoint.
|
||||
- The Rancher Server worker node(s) must have an instance role that has read/write to the designated backup bucket.
|
||||
- The Rancher Server worker node(s) must have network access to the specified S3 endpoint.
|
||||
|
||||
To give an application access to S3, refer to the AWS documentation on [Using an IAM Role to Grant Permissions to Applications Running on Amazon EC2 Instances.](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html)
|
||||
|
||||
# Viewing Available Snapshots
|
||||
|
||||
The list of all available snapshots for the cluster is available in the Rancher UI.
|
||||
|
||||
1. In the **Global** view, navigate to the cluster that you want to view snapshots.
|
||||
|
||||
2. Click **Tools > Snapshots** from the navigation bar to view the list of saved snapshots. These snapshots include a timestamp of when they were created.
|
||||
|
||||
# Safe Timestamps
|
||||
|
||||
_Available as of v2.3.0_
|
||||
|
||||
As of v2.2.6, snapshot files are timestamped to simplify processing the files using external tools and scripts, but in some S3 compatible backends, these timestamps were unusable. As of Rancher v2.3.0, the option `safe_timestamp` is added to support compatible file names. When this flag is set to `true`, all special characters in the snapshot filename timestamp are replaced.
|
||||
|
||||
This option is not available directly in the UI, and is only available through the `Edit as Yaml` interface.
|
||||
|
||||
# Enabling Snapshot Features for Clusters Created Before Rancher v2.2.0
|
||||
|
||||
If you have any Rancher launched Kubernetes clusters that were created before v2.2.0, after upgrading Rancher, you must [edit the cluster]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/editing-clusters/) and _save_ it, in order to enable the updated snapshot features. Even if you were already creating snapshots before v2.2.0, you must do this step as the older snapshots will not be available to use to [back up and restore etcd through the UI]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/restoring-etcd/).
|
||||
+87
@@ -0,0 +1,87 @@
|
||||
---
|
||||
title: Certificate Rotation
|
||||
weight: 2040
|
||||
---
|
||||
|
||||
> **Warning:** Rotating Kubernetes certificates may result in your cluster being temporarily unavailable as components are restarted. For production environments, it's recommended to perform this action during a maintenance window.
|
||||
|
||||
By default, Kubernetes clusters require certificates and Rancher launched Kubernetes clusters automatically generate certificates for the Kubernetes components. Rotating these certificates is important before the certificates expire as well as if a certificate is compromised. After the certificates are rotated, the Kubernetes components are automatically restarted.
|
||||
|
||||
Certificates can be rotated for the following services:
|
||||
|
||||
- etcd
|
||||
- kubelet
|
||||
- kube-apiserver
|
||||
- kube-proxy
|
||||
- kube-scheduler
|
||||
- kube-controller-manager
|
||||
|
||||
|
||||
### Certificate Rotation in Rancher v2.2.x
|
||||
|
||||
_Available as of v2.2.0_
|
||||
|
||||
Rancher launched Kubernetes clusters have the ability to rotate the auto-generated certificates through the UI.
|
||||
|
||||
1. In the **Global** view, navigate to the cluster that you want to rotate certificates.
|
||||
|
||||
2. Select the **⋮ > Rotate Certificates**.
|
||||
|
||||
3. Select which certificates that you want to rotate.
|
||||
|
||||
* Rotate all Service certificates (keep the same CA)
|
||||
* Rotate an individual service and choose one of the services from the drop down menu
|
||||
|
||||
4. Click **Save**.
|
||||
|
||||
**Results:** The selected certificates will be rotated and the related services will be restarted to start using the new certificate.
|
||||
|
||||
> **Note:** Even though the RKE CLI can use custom certificates for the Kubernetes cluster components, Rancher currently doesn't allow the ability to upload these in Rancher Launched Kubernetes clusters.
|
||||
|
||||
|
||||
### Certificate Rotation in Rancher v2.1.x and v2.0.x
|
||||
|
||||
_Available as of v2.0.14 and v2.1.9_
|
||||
|
||||
Rancher launched Kubernetes clusters have the ability to rotate the auto-generated certificates through the API.
|
||||
|
||||
1. In the **Global** view, navigate to the cluster that you want to rotate certificates.
|
||||
|
||||
2. Select the **⋮ > View in API**.
|
||||
|
||||
3. Click on **RotateCertificates**.
|
||||
|
||||
4. Click on **Show Request**.
|
||||
|
||||
5. Click on **Send Request**.
|
||||
|
||||
**Results:** All Kubernetes certificates will be rotated.
|
||||
|
||||
### Rotating Expired Certificates After Upgrading Older Rancher Versions
|
||||
|
||||
If you are upgrading from Rancher v2.0.13 or earlier, or v2.1.8 or earlier, and your clusters have expired certificates, some manual steps are required to complete the certificate rotation.
|
||||
|
||||
1. For the `controlplane` and `etcd` nodes, log in to each corresponding host and check if the certificate `kube-apiserver-requestheader-ca.pem` is in the following directory:
|
||||
|
||||
```
|
||||
cd /etc/kubernetes/.tmp
|
||||
```
|
||||
|
||||
If the certificate is not in the directory, perform the following commands:
|
||||
|
||||
```
|
||||
cp kube-ca.pem kube-apiserver-requestheader-ca.pem
|
||||
cp kube-ca-key.pem kube-apiserver-requestheader-ca-key.pem
|
||||
cp kube-apiserver.pem kube-apiserver-proxy-client.pem
|
||||
cp kube-apiserver-key.pem kube-apiserver-proxy-client-key.pem
|
||||
```
|
||||
|
||||
If the `.tmp` directory does not exist, you can copy the entire SSL certificate to `.tmp`:
|
||||
|
||||
```
|
||||
cp -r /etc/kubernetes/ssl /etc/kubernetes/.tmp
|
||||
```
|
||||
|
||||
1. Rotate the certificates. For Rancher v2.0.x and v2.1.x, use the [Rancher API.](#certificate-rotation-in-rancher-v2-1-x-and-v2-0-x) For Rancher 2.2.x, [use the UI.](#certificate-rotation-in-rancher-v2-2-x)
|
||||
|
||||
1. After the command is finished, check if the `worker` nodes are Active. If not, log in to each `worker` node and restart the kubelet and proxy.
|
||||
+284
@@ -0,0 +1,284 @@
|
||||
---
|
||||
title: Removing Kubernetes Components from Nodes
|
||||
description: Learn about cluster cleanup when removing nodes from your Rancher-launched Kubernetes cluster. What is removed, how to do it manually
|
||||
weight: 2055
|
||||
---
|
||||
|
||||
import Tabs from '@theme/Tabs';
|
||||
import TabItem from '@theme/TabItem';
|
||||
|
||||
This section describes how to disconnect a node from a Rancher-launched Kubernetes cluster and remove all of the Kubernetes components from the node. This process allows you to use the node for other purposes.
|
||||
|
||||
When you use Rancher to install Kubernetes on new nodes in an infrastructure provider, resources (containers/virtual network interfaces) and configuration items (certificates/configuration files) are created.
|
||||
|
||||
When removing nodes from your Rancher launched Kubernetes cluster (provided that they are in `Active` state), those resources are automatically cleaned, and the only action needed is to restart the node. When a node has become unreachable and the automatic cleanup process cannot be used, we describe the steps that need to be executed before the node can be added to a cluster again.
|
||||
|
||||
## What Gets Removed?
|
||||
|
||||
When cleaning nodes provisioned using Rancher, the following components are deleted based on the type of cluster node you're removing.
|
||||
|
||||
| Removed Component | [Nodes Hosted by Infrastructure Provider][1] | [Custom Nodes][2] | [Hosted Cluster][3] | [Imported Nodes][4] |
|
||||
| ------------------------------------------------------------------------------ | --------------- | ----------------- | ------------------- | ------------------- |
|
||||
| The Rancher deployment namespace (`cattle-system` by default) | ✓ | ✓ | ✓ | ✓ |
|
||||
| `serviceAccount`, `clusterRoles`, and `clusterRoleBindings` labeled by Rancher | ✓ | ✓ | ✓ | ✓ |
|
||||
| Labels, Annotations, and Finalizers | ✓ | ✓ | ✓ | ✓ |
|
||||
| Rancher Deployment | ✓ | ✓ | ✓ | |
|
||||
| Machines, clusters, projects, and user custom resource definitions (CRDs) | ✓ | ✓ | ✓ | |
|
||||
| All resources create under the `management.cattle.io` API Group | ✓ | ✓ | ✓ | |
|
||||
| All CRDs created by Rancher v2.x | ✓ | ✓ | ✓ | |
|
||||
|
||||
[1]: {{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-provisioning/rke-clusters/node-pools/
|
||||
[2]: {{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-provisioning/rke-clusters/custom-nodes/
|
||||
[3]: {{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-provisioning/hosted-kubernetes-clusters/
|
||||
[4]: {{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-provisioning/imported-clusters/
|
||||
|
||||
## Removing a Node from a Cluster by Rancher UI
|
||||
|
||||
When the node is in `Active` state, removing the node from a cluster will trigger a process to clean up the node. Please restart the node after the automatic cleanup process is done to make sure any non-persistent data is properly removed.
|
||||
|
||||
**To restart a node:**
|
||||
|
||||
```
|
||||
# using reboot
|
||||
$ sudo reboot
|
||||
|
||||
# using shutdown
|
||||
$ sudo shutdown -r now
|
||||
```
|
||||
|
||||
## Removing Rancher Components from a Cluster Manually
|
||||
|
||||
When a node is unreachable and removed from the cluster, the automatic cleaning process can't be triggered because the node is unreachable. Please follow the steps below to manually remove the Rancher components.
|
||||
|
||||
>**Warning:** The commands listed below will remove data from the node. Make sure you have created a backup of files you want to keep before executing any of the commands as data will be lost.
|
||||
|
||||
### Removing Rancher Components from Imported Clusters
|
||||
|
||||
For imported clusters, the process for removing Rancher is a little different. You have the option of simply deleting the cluster in the Rancher UI, or your can run a script that removes Rancher components from the nodes. Both options make the same deletions.
|
||||
|
||||
After the imported cluster is detached from Rancher, the cluster's workloads will be unaffected and you can access the cluster using the same methods that you did before the cluster was imported into Rancher.
|
||||
|
||||
<Tabs>
|
||||
<TabItem value="By UI / API">
|
||||
|
||||
>**Warning:** This process will remove data from your cluster. Make sure you have created a backup of files you want to keep before executing the command, as data will be lost.
|
||||
|
||||
After you initiate the removal of an imported cluster using the Rancher UI (or API), the following events occur.
|
||||
|
||||
1. Rancher creates a `serviceAccount` that it uses to remove the Rancher components from the cluster. This account is assigned the [clusterRole](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#role-and-clusterrole) and [clusterRoleBinding](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#rolebinding-and-clusterrolebinding) permissions, which are required to remove the Rancher components.
|
||||
|
||||
1. Using the `serviceAccount`, Rancher schedules and runs a [job](https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/) that cleans the Rancher components off of the cluster. This job also references the `serviceAccount` and its roles as dependencies, so the job deletes them before its completion.
|
||||
|
||||
1. Rancher is removed from the cluster. However, the cluster persists, running the native version of Kubernetes.
|
||||
|
||||
**Result:** All components listed for imported clusters in [What Gets Removed?](#what-gets-removed) are deleted.
|
||||
|
||||
</TabItem>
|
||||
<TabItem value="By Script">
|
||||
|
||||
Rather than cleaning imported cluster nodes using the Rancher UI, you can run a script instead. This functionality is available since `v2.1.0`.
|
||||
|
||||
>**Prerequisite:**
|
||||
>
|
||||
>Install [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/).
|
||||
|
||||
1. Open a web browser, navigate to [GitHub](https://github.com/rancher/rancher/blob/master/cleanup/user-cluster.sh), and download `user-cluster.sh`.
|
||||
|
||||
1. Make the script executable by running the following command from the same directory as `user-cluster.sh`:
|
||||
|
||||
```
|
||||
chmod +x user-cluster.sh
|
||||
```
|
||||
|
||||
1. **Air Gap Environments Only:** Open `user-cluster.sh` and replace `yaml_url` with the URL in `user-cluster.yml`.
|
||||
|
||||
If you don't have an air gap environment, skip this step.
|
||||
|
||||
1. From the same directory, run the script and provide the `rancher/rancher-agent` image version which should be equal to the version of Rancher used to manage the cluster. (`<RANCHER_VERSION>`):
|
||||
|
||||
>**Tip:**
|
||||
>
|
||||
>Add the `-dry-run` flag to preview the script's outcome without making changes.
|
||||
```
|
||||
./user-cluster.sh rancher/rancher-agent:<RANCHER_VERSION>
|
||||
```
|
||||
|
||||
**Result:** The script runs. All components listed for imported clusters in [What Gets Removed?](#what-gets-removed) are deleted.
|
||||
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
|
||||
### Windows Nodes
|
||||
|
||||
To clean up a Windows node, you can run a cleanup script located in `c:\etc\rancher`. The script deletes Kubernetes generated resources and the execution binary. It also drops the firewall rules and network settings.
|
||||
|
||||
To run the script, you can use this command in the PowerShell:
|
||||
|
||||
```
|
||||
pushd c:\etc\rancher
|
||||
.\cleanup.ps1
|
||||
popd
|
||||
```
|
||||
|
||||
**Result:** The node is reset and can be re-added to a Kubernetes cluster.
|
||||
|
||||
### Docker Containers, Images, and Volumes
|
||||
|
||||
Based on what role you assigned to the node, there are Kubernetes components in containers, containers belonging to overlay networking, DNS, ingress controller and Rancher agent. (and pods you created that have been scheduled to this node)
|
||||
|
||||
**To clean all Docker containers, images and volumes:**
|
||||
|
||||
```
|
||||
docker rm -f $(docker ps -qa)
|
||||
docker rmi -f $(docker images -q)
|
||||
docker volume rm $(docker volume ls -q)
|
||||
```
|
||||
|
||||
### Mounts
|
||||
|
||||
Kubernetes components and secrets leave behind mounts on the system that need to be unmounted.
|
||||
|
||||
Mounts |
|
||||
--------|
|
||||
`/var/lib/kubelet/pods/XXX` (miscellaneous mounts) |
|
||||
`/var/lib/kubelet` |
|
||||
`/var/lib/rancher` |
|
||||
|
||||
**To unmount all mounts:**
|
||||
|
||||
```
|
||||
for mount in $(mount | grep tmpfs | grep '/var/lib/kubelet' | awk '{ print $3 }') /var/lib/kubelet /var/lib/rancher; do umount $mount; done
|
||||
```
|
||||
|
||||
### Directories and Files
|
||||
|
||||
The following directories are used when adding a node to a cluster, and should be removed. You can remove a directory using `rm -rf /directory_name`.
|
||||
|
||||
>**Note:** Depending on the role you assigned to the node, some of the directories will or won't be present on the node.
|
||||
|
||||
Directories |
|
||||
--------|
|
||||
`/etc/ceph` |
|
||||
`/etc/cni` |
|
||||
`/etc/kubernetes` |
|
||||
`/opt/cni` |
|
||||
`/opt/rke` |
|
||||
`/run/secrets/kubernetes.io` |
|
||||
`/run/calico` |
|
||||
`/run/flannel` |
|
||||
`/var/lib/calico` |
|
||||
`/var/lib/etcd` |
|
||||
`/var/lib/cni` |
|
||||
`/var/lib/kubelet` |
|
||||
`/var/lib/rancher/rke/log` |
|
||||
`/var/log/containers` |
|
||||
`/var/log/kube-audit` |
|
||||
`/var/log/pods` |
|
||||
`/var/run/calico` |
|
||||
|
||||
**To clean the directories:**
|
||||
|
||||
```
|
||||
rm -rf /etc/ceph \
|
||||
/etc/cni \
|
||||
/etc/kubernetes \
|
||||
/opt/cni \
|
||||
/opt/rke \
|
||||
/run/secrets/kubernetes.io \
|
||||
/run/calico \
|
||||
/run/flannel \
|
||||
/var/lib/calico \
|
||||
/var/lib/etcd \
|
||||
/var/lib/cni \
|
||||
/var/lib/kubelet \
|
||||
/var/lib/rancher/rke/log \
|
||||
/var/log/containers \
|
||||
/var/log/kube-audit \
|
||||
/var/log/pods \
|
||||
/var/run/calico
|
||||
```
|
||||
|
||||
### Network Interfaces and Iptables
|
||||
|
||||
The remaining two components that are changed/configured are (virtual) network interfaces and iptables rules. Both are non-persistent to the node, meaning that they will be cleared after a restart of the node. To remove these components, a restart is recommended.
|
||||
|
||||
**To restart a node:**
|
||||
|
||||
```
|
||||
# using reboot
|
||||
$ sudo reboot
|
||||
|
||||
# using shutdown
|
||||
$ sudo shutdown -r now
|
||||
```
|
||||
|
||||
If you want to know more on (virtual) network interfaces or iptables rules, please see the specific subjects below.
|
||||
|
||||
### Network Interfaces
|
||||
|
||||
>**Note:** Depending on the network provider configured for the cluster the node was part of, some of the interfaces will or won't be present on the node.
|
||||
|
||||
Interfaces |
|
||||
--------|
|
||||
`flannel.1` |
|
||||
`cni0` |
|
||||
`tunl0` |
|
||||
`caliXXXXXXXXXXX` (random interface names) |
|
||||
`vethXXXXXXXX` (random interface names) |
|
||||
|
||||
**To list all interfaces:**
|
||||
|
||||
```
|
||||
# Using ip
|
||||
ip address show
|
||||
|
||||
# Using ifconfig
|
||||
ifconfig -a
|
||||
```
|
||||
|
||||
**To remove an interface:**
|
||||
|
||||
```
|
||||
ip link delete interface_name
|
||||
```
|
||||
|
||||
### Iptables
|
||||
|
||||
>**Note:** Depending on the network provider configured for the cluster the node was part of, some of the chains will or won't be present on the node.
|
||||
|
||||
Iptables rules are used to route traffic from and to containers. The created rules are not persistent, so restarting the node will restore iptables to its original state.
|
||||
|
||||
Chains |
|
||||
--------|
|
||||
`cali-failsafe-in` |
|
||||
`cali-failsafe-out` |
|
||||
`cali-fip-dnat` |
|
||||
`cali-fip-snat` |
|
||||
`cali-from-hep-forward` |
|
||||
`cali-from-host-endpoint` |
|
||||
`cali-from-wl-dispatch` |
|
||||
`cali-fw-caliXXXXXXXXXXX` (random chain names) |
|
||||
`cali-nat-outgoing` |
|
||||
`cali-pri-kns.NAMESPACE` (chain per namespace) |
|
||||
`cali-pro-kns.NAMESPACE` (chain per namespace) |
|
||||
`cali-to-hep-forward` |
|
||||
`cali-to-host-endpoint` |
|
||||
`cali-to-wl-dispatch` |
|
||||
`cali-tw-caliXXXXXXXXXXX` (random chain names) |
|
||||
`cali-wl-to-host` |
|
||||
`KUBE-EXTERNAL-SERVICES` |
|
||||
`KUBE-FIREWALL` |
|
||||
`KUBE-MARK-DROP` |
|
||||
`KUBE-MARK-MASQ` |
|
||||
`KUBE-NODEPORTS` |
|
||||
`KUBE-SEP-XXXXXXXXXXXXXXXX` (random chain names) |
|
||||
`KUBE-SERVICES` |
|
||||
`KUBE-SVC-XXXXXXXXXXXXXXXX` (random chain names) |
|
||||
|
||||
**To list all iptables rules:**
|
||||
|
||||
```
|
||||
iptables -L -t nat
|
||||
iptables -L -t mangle
|
||||
iptables -L
|
||||
```
|
||||
@@ -0,0 +1,101 @@
|
||||
---
|
||||
title: Cloning Clusters
|
||||
weight: 2035
|
||||
aliases:
|
||||
- /rancher/v2.0-v2.4/en/cluster-provisioning/cloning-clusters/
|
||||
---
|
||||
|
||||
If you have a cluster in Rancher that you want to use as a template for creating similar clusters, you can use Rancher CLI to clone the cluster's configuration, edit it, and then use it to quickly launch the cloned cluster.
|
||||
|
||||
Duplication of imported clusters is not supported.
|
||||
|
||||
| Cluster Type | Cloneable? |
|
||||
|----------------------------------|---------------|
|
||||
| [Nodes Hosted by Infrastructure Provider]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-provisioning/rke-clusters/node-pools/) | ✓ |
|
||||
| [Hosted Kubernetes Providers]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-provisioning/hosted-kubernetes-clusters/) | ✓ |
|
||||
| [Custom Cluster]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-provisioning/rke-clusters/custom-nodes) | ✓ |
|
||||
| [Imported Cluster]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-provisioning/imported-clusters/) | |
|
||||
|
||||
> **Warning:** During the process of duplicating a cluster, you will edit a config file full of cluster settings. However, we recommend editing only values explicitly listed in this document, as cluster duplication is designed for simple cluster copying, _not_ wide scale configuration changes. Editing other values may invalidate the config file, which will lead to cluster deployment failure.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
Download and install [Rancher CLI]({{<baseurl>}}/rancher/v2.0-v2.4/en/cli). Remember to [create an API bearer token]({{<baseurl>}}/rancher/v2.0-v2.4/en/user-settings/api-keys) if necessary.
|
||||
|
||||
|
||||
## 1. Export Cluster Config
|
||||
|
||||
Begin by using Rancher CLI to export the configuration for the cluster that you want to clone.
|
||||
|
||||
1. Open Terminal and change your directory to the location of the Rancher CLI binary, `rancher`.
|
||||
|
||||
1. Enter the following command to list the clusters managed by Rancher.
|
||||
|
||||
|
||||
./rancher cluster ls
|
||||
|
||||
|
||||
1. Find the cluster that you want to clone, and copy either its resource `ID` or `NAME` to your clipboard. From this point on, we'll refer to the resource `ID` or `NAME` as `<RESOURCE_ID>`, which is used as a placeholder in the next step.
|
||||
|
||||
1. Enter the following command to export the configuration for your cluster.
|
||||
|
||||
|
||||
./rancher clusters export <RESOURCE_ID>
|
||||
|
||||
|
||||
**Step Result:** The YAML for a cloned cluster prints to Terminal.
|
||||
|
||||
1. Copy the YAML to your clipboard and paste it in a new file. Save the file as `cluster-template.yml` (or any other name, as long as it has a `.yml` extension).
|
||||
|
||||
## 2. Modify Cluster Config
|
||||
|
||||
Use your favorite text editor to modify the cluster configuration in `cluster-template.yml` for your cloned cluster.
|
||||
|
||||
> **Note:** As of Rancher v2.3.0, cluster configuration directives must be nested under the `rancher_kubernetes_engine_config` directive in `cluster.yml`. For more information, refer to the section on [the config file structure in Rancher v2.3.0+.]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-provisioning/rke-clusters/options/#config-file-structure-in-rancher-v2-3-0)
|
||||
|
||||
1. Open `cluster-template.yml` (or whatever you named your config) in your favorite text editor.
|
||||
|
||||
>**Warning:** Only edit the cluster config values explicitly called out below. Many of the values listed in this file are used to provision your cloned cluster, and editing their values may break the provisioning process.
|
||||
|
||||
|
||||
1. As depicted in the example below, at the `<CLUSTER_NAME>` placeholder, replace your original cluster's name with a unique name (`<CLUSTER_NAME>`). If your cloned cluster has a duplicate name, the cluster will not provision successfully.
|
||||
|
||||
```yml
|
||||
Version: v3
|
||||
clusters:
|
||||
<CLUSTER_NAME>: # ENTER UNIQUE NAME
|
||||
dockerRootDir: /var/lib/docker
|
||||
enableNetworkPolicy: false
|
||||
rancherKubernetesEngineConfig:
|
||||
addonJobTimeout: 30
|
||||
authentication:
|
||||
strategy: x509
|
||||
authorization: {}
|
||||
bastionHost: {}
|
||||
cloudProvider: {}
|
||||
ignoreDockerVersion: true
|
||||
```
|
||||
|
||||
1. For each `nodePools` section, replace the original nodepool name with a unique name at the `<NODEPOOL_NAME>` placeholder. If your cloned cluster has a duplicate nodepool name, the cluster will not provision successfully.
|
||||
|
||||
```yml
|
||||
nodePools:
|
||||
<NODEPOOL_NAME>:
|
||||
clusterId: do
|
||||
controlPlane: true
|
||||
etcd: true
|
||||
hostnamePrefix: mark-do
|
||||
nodeTemplateId: do
|
||||
quantity: 1
|
||||
worker: true
|
||||
```
|
||||
|
||||
1. When you're done, save and close the configuration.
|
||||
|
||||
## 3. Launch Cloned Cluster
|
||||
|
||||
Move `cluster-template.yml` into the same directory as the Rancher CLI binary. Then run this command:
|
||||
|
||||
./rancher up --file cluster-template.yml
|
||||
|
||||
**Result:** Your cloned cluster begins provisioning. Enter `./rancher cluster ls` to confirm. You can also log into the Rancher UI and open the **Global** view to watch your provisioning cluster's progress.
|
||||
@@ -0,0 +1,48 @@
|
||||
---
|
||||
title: How the Authorized Cluster Endpoint Works
|
||||
weight: 2015
|
||||
---
|
||||
|
||||
This section describes how the kubectl CLI, the kubeconfig file, and the authorized cluster endpoint work together to allow you to access a downstream Kubernetes cluster directly, without authenticating through the Rancher server. It is intended to provide background information and context to the instructions for [how to set up kubectl to directly access a cluster.](../kubectl/#authenticating-directly-with-a-downstream-cluster)
|
||||
|
||||
### About the kubeconfig File
|
||||
|
||||
The _kubeconfig file_ is a file used to configure access to Kubernetes when used in conjunction with the kubectl command line tool (or other clients).
|
||||
|
||||
This kubeconfig file and its contents are specific to the cluster you are viewing. It can be downloaded from the cluster view in Rancher. You will need a separate kubeconfig file for each cluster that you have access to in Rancher.
|
||||
|
||||
After you download the kubeconfig file, you will be able to use the kubeconfig file and its Kubernetes [contexts](https://kubernetes.io/docs/reference/kubectl/cheatsheet/#kubectl-context-and-configuration) to access your downstream cluster.
|
||||
|
||||
_Available as of v2.4.6_
|
||||
|
||||
If admins have [enforced TTL on kubeconfig tokens]({{<baseurl>}}/rancher/v2.0-v2.4/en/api/api-tokens/#setting-ttl-on-kubeconfig-tokens), the kubeconfig file requires [rancher cli](../cli) to be present in your PATH.
|
||||
|
||||
|
||||
### Two Authentication Methods for RKE Clusters
|
||||
|
||||
If the cluster is not an [RKE cluster,]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-provisioning/rke-clusters/) the kubeconfig file allows you to access the cluster in only one way: it lets you be authenticated with the Rancher server, then Rancher allows you to run kubectl commands on the cluster.
|
||||
|
||||
For RKE clusters, the kubeconfig file allows you to be authenticated in two ways:
|
||||
|
||||
- **Through the Rancher server authentication proxy:** Rancher's authentication proxy validates your identity, then connects you to the downstream cluster that you want to access.
|
||||
- **Directly with the downstream cluster's API server:** RKE clusters have an authorized cluster endpoint enabled by default. This endpoint allows you to access your downstream Kubernetes cluster with the kubectl CLI and a kubeconfig file, and it is enabled by default for RKE clusters. In this scenario, the downstream cluster's Kubernetes API server authenticates you by calling a webhook (the `kube-api-auth` microservice) that Rancher set up.
|
||||
|
||||
This second method, the capability to connect directly to the cluster's Kubernetes API server, is important because it lets you access your downstream cluster if you can't connect to Rancher.
|
||||
|
||||
To use the authorized cluster endpoint, you will need to configure kubectl to use the extra kubectl context in the kubeconfig file that Rancher generates for you when the RKE cluster is created. This file can be downloaded from the cluster view in the Rancher UI, and the instructions for configuring kubectl are on [this page.](../kubectl/#authenticating-directly-with-a-downstream-cluster)
|
||||
|
||||
These methods of communicating with downstream Kubernetes clusters are also explained in the [architecture page]({{<baseurl>}}/rancher/v2.0-v2.4/en/overview/architecture/#communicating-with-downstream-user-clusters) in the larger context of explaining how Rancher works and how Rancher communicates with downstream clusters.
|
||||
|
||||
### About the kube-api-auth Authentication Webhook
|
||||
|
||||
The `kube-api-auth` microservice is deployed to provide the user authentication functionality for the [authorized cluster endpoint,]({{<baseurl>}}/rancher/v2.0-v2.4/en/overview/architecture/#4-authorized-cluster-endpoint) which is only available for [RKE clusters.]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-provisioning/rke-clusters/) When you access the user cluster using `kubectl`, the cluster's Kubernetes API server authenticates you by using the `kube-api-auth` service as a webhook.
|
||||
|
||||
During cluster provisioning, the file `/etc/kubernetes/kube-api-authn-webhook.yaml` is deployed and `kube-apiserver` is configured with `--authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml`. This configures the `kube-apiserver` to query `http://127.0.0.1:6440/v1/authenticate` to determine authentication for bearer tokens.
|
||||
|
||||
The scheduling rules for `kube-api-auth` are listed below:
|
||||
|
||||
_Applies to v2.3.0 and higher_
|
||||
|
||||
| Component | nodeAffinity nodeSelectorTerms | nodeSelector | Tolerations |
|
||||
| -------------------- | ------------------------------------------ | ------------ | ------------------------------------------------------------------------------ |
|
||||
| kube-api-auth | `beta.kubernetes.io/os:NotIn:windows`<br/>`node-role.kubernetes.io/controlplane:In:"true"` | none | `operator:Exists` |
|
||||
@@ -0,0 +1,32 @@
|
||||
---
|
||||
title: Cluster Access
|
||||
weight: 1
|
||||
---
|
||||
|
||||
This section is about what tools can be used to access clusters managed by Rancher.
|
||||
|
||||
For information on how to give users permission to access a cluster, see the section on [adding users to clusters.]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/cluster-access/cluster-members/)
|
||||
|
||||
For more information on roles-based access control, see [this section.]({{<baseurl>}}/rancher/v2.0-v2.4/en/admin-settings/rbac/)
|
||||
|
||||
For information on how to set up an authentication system, see [this section.]({{<baseurl>}}/rancher/v2.0-v2.4/en/admin-settings/authentication/)
|
||||
|
||||
|
||||
### Rancher UI
|
||||
|
||||
Rancher provides an intuitive user interface for interacting with your clusters. All options available in the UI use the Rancher API. Therefore any action possible in the UI is also possible in the Rancher CLI or Rancher API.
|
||||
|
||||
### kubectl
|
||||
|
||||
You can use the Kubernetes command-line tool, [kubectl](https://kubernetes.io/docs/reference/kubectl/overview/), to manage your clusters. You have two options for using kubectl:
|
||||
|
||||
- **Rancher kubectl shell:** Interact with your clusters by launching a kubectl shell available in the Rancher UI. This option requires no configuration actions on your part. For more information, see [Accessing Clusters with kubectl Shell]({{<baseurl>}}/rancher/v2.0-v2.4/en/k8s-in-rancher/kubectl/).
|
||||
- **Terminal remote connection:** You can also interact with your clusters by installing [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/) on your local desktop and then copying the cluster's kubeconfig file to your local `~/.kube/config` directory. For more information, see [Accessing Clusters with kubectl and a kubeconfig File](./kubectl/).
|
||||
|
||||
### Rancher CLI
|
||||
|
||||
You can control your clusters by downloading Rancher's own command-line interface, [Rancher CLI]({{<baseurl>}}/rancher/v2.0-v2.4/en/cli/). This CLI tool can interact directly with different clusters and projects or pass them `kubectl` commands.
|
||||
|
||||
### Rancher API
|
||||
|
||||
Finally, you can interact with your clusters over the Rancher API. Before you use the API, you must obtain an [API key]({{<baseurl>}}/rancher/v2.0-v2.4/en/user-settings/api-keys/). To view the different resource fields and actions for an API object, open the API UI, which can be accessed by clicking on **View in API** for any Rancher UI object.
|
||||
+57
@@ -0,0 +1,57 @@
|
||||
---
|
||||
title: Adding Users to Clusters
|
||||
weight: 2020
|
||||
aliases:
|
||||
- /rancher/v2.0-v2.4/en/tasks/clusters/adding-managing-cluster-members/
|
||||
- /rancher/v2.0-v2.4/en/k8s-in-rancher/cluster-members/
|
||||
- /rancher/v2.0-v2.4/en/cluster-admin/cluster-members
|
||||
---
|
||||
|
||||
If you want to provide a user with access and permissions to _all_ projects, nodes, and resources within a cluster, assign the user a cluster membership.
|
||||
|
||||
>**Tip:** Want to provide a user with access to a _specific_ project within a cluster? See [Adding Project Members]({{<baseurl>}}/rancher/v2.0-v2.4/en/k8s-in-rancher/projects-and-namespaces/project-members/) instead.
|
||||
|
||||
There are two contexts where you can add cluster members:
|
||||
|
||||
- Adding Members to a New Cluster
|
||||
|
||||
You can add members to a cluster as you create it (recommended if possible).
|
||||
|
||||
- [Adding Members to an Existing Cluster](#editing-cluster-membership)
|
||||
|
||||
You can always add members to a cluster after a cluster is provisioned.
|
||||
|
||||
## Editing Cluster Membership
|
||||
|
||||
Cluster administrators can edit the membership for a cluster, controlling which Rancher users can access the cluster and what features they can use.
|
||||
|
||||
1. From the **Global** view, open the cluster that you want to add members to.
|
||||
|
||||
2. From the main menu, select **Members**. Then click **Add Member**.
|
||||
|
||||
3. Search for the user or group that you want to add to the cluster.
|
||||
|
||||
If external authentication is configured:
|
||||
|
||||
- Rancher returns users from your [external authentication]({{<baseurl>}}/rancher/v2.0-v2.4/en/admin-settings/authentication/) source as you type.
|
||||
|
||||
>**Using AD but can't find your users?**
|
||||
>There may be an issue with your search attribute configuration. See [Configuring Active Directory Authentication: Step 5]({{<baseurl>}}/rancher/v2.0-v2.4/en/admin-settings/authentication/ad/).
|
||||
|
||||
- A drop-down allows you to add groups instead of individual users. The drop-down only lists groups that you, the logged in user, are part of.
|
||||
|
||||
>**Note:** If you are logged in as a local user, external users do not display in your search results. For more information, see [External Authentication Configuration and Principal Users]({{<baseurl>}}/rancher/v2.0-v2.4/en/admin-settings/authentication/#external-authentication-configuration-and-principal-users).
|
||||
|
||||
4. Assign the user or group **Cluster** roles.
|
||||
|
||||
[What are Cluster Roles?]({{<baseurl>}}/rancher/v2.0-v2.4/en/admin-settings/rbac/cluster-project-roles/)
|
||||
|
||||
>**Tip:** For Custom Roles, you can modify the list of individual roles available for assignment.
|
||||
>
|
||||
> - To add roles to the list, [Add a Custom Role]({{<baseurl>}}/rancher/v2.0-v2.4/en/admin-settings/rbac/default-custom-roles/).
|
||||
> - To remove roles from the list, [Lock/Unlock Roles]({{<baseurl>}}/rancher/v2.0-v2.4/en/admin-settings/rbac/locked-roles).
|
||||
|
||||
**Result:** The chosen users are added to the cluster.
|
||||
|
||||
- To revoke cluster membership, select the user and click **Delete**. This action deletes membership, not the user.
|
||||
- To modify a user's roles in the cluster, delete them from the cluster, and then re-add them with modified roles.
|
||||
@@ -0,0 +1,109 @@
|
||||
---
|
||||
title: "Access a Cluster with Kubectl and kubeconfig"
|
||||
description: "Learn how you can access and manage your Kubernetes clusters using kubectl with kubectl Shell or with kubectl CLI and kubeconfig file. A kubeconfig file is used to configure access to Kubernetes. When you create a cluster with Rancher, it automatically creates a kubeconfig for your cluster."
|
||||
weight: 2010
|
||||
aliases:
|
||||
- /rancher/v2.0-v2.4/en/k8s-in-rancher/kubectl/
|
||||
- /rancher/v2.0-v2.4/en/cluster-admin/kubectl
|
||||
- /rancher/v2.0-v2.4/en/concepts/clusters/kubeconfig-files/
|
||||
- /rancher/v2.0-v2.4/en/k8s-in-rancher/kubeconfig/
|
||||
- /rancher/2.x/en/cluster-admin/kubeconfig
|
||||
---
|
||||
|
||||
This section describes how to manipulate your downstream Kubernetes cluster with kubectl from the Rancher UI or from your workstation.
|
||||
|
||||
For more information on using kubectl, see [Kubernetes Documentation: Overview of kubectl](https://kubernetes.io/docs/reference/kubectl/overview/).
|
||||
|
||||
- [Accessing clusters with kubectl shell in the Rancher UI](#accessing-clusters-with-kubectl-shell-in-the-rancher-ui)
|
||||
- [Accessing clusters with kubectl from your workstation](#accessing-clusters-with-kubectl-from-your-workstation)
|
||||
- [Note on Resources created using kubectl](#note-on-resources-created-using-kubectl)
|
||||
- [Authenticating Directly with a Downstream Cluster](#authenticating-directly-with-a-downstream-cluster)
|
||||
- [Connecting Directly to Clusters with FQDN Defined](#connecting-directly-to-clusters-with-fqdn-defined)
|
||||
- [Connecting Directly to Clusters without FQDN Defined](#connecting-directly-to-clusters-without-fqdn-defined)
|
||||
|
||||
|
||||
### Accessing Clusters with kubectl Shell in the Rancher UI
|
||||
|
||||
You can access and manage your clusters by logging into Rancher and opening the kubectl shell in the UI. No further configuration necessary.
|
||||
|
||||
1. From the **Global** view, open the cluster that you want to access with kubectl.
|
||||
|
||||
2. Click **Launch kubectl**. Use the window that opens to interact with your Kubernetes cluster.
|
||||
|
||||
### Accessing Clusters with kubectl from Your Workstation
|
||||
|
||||
This section describes how to download your cluster's kubeconfig file, launch kubectl from your workstation, and access your downstream cluster.
|
||||
|
||||
This alternative method of accessing the cluster allows you to authenticate with Rancher and manage your cluster without using the Rancher UI.
|
||||
|
||||
> **Prerequisites:** These instructions assume that you have already created a Kubernetes cluster, and that kubectl is installed on your workstation. For help installing kubectl, refer to the official [Kubernetes documentation.](https://kubernetes.io/docs/tasks/tools/install-kubectl/)
|
||||
|
||||
1. Log into Rancher. From the **Global** view, open the cluster that you want to access with kubectl.
|
||||
1. Click **Kubeconfig File**.
|
||||
1. Copy the contents displayed to your clipboard.
|
||||
1. Paste the contents into a new file on your local computer. Move the file to `~/.kube/config`. Note: The default location that kubectl uses for the kubeconfig file is `~/.kube/config`, but you can use any directory and specify it using the `--kubeconfig` flag, as in this command:
|
||||
```
|
||||
kubectl --kubeconfig /custom/path/kube.config get pods
|
||||
```
|
||||
1. From your workstation, launch kubectl. Use it to interact with your kubernetes cluster.
|
||||
|
||||
|
||||
### Note on Resources Created Using kubectl
|
||||
|
||||
Rancher will discover and show resources created by `kubectl`. However, these resources might not have all the necessary annotations on discovery. If an operation (for instance, scaling the workload) is done to the resource using the Rancher UI/API, this may trigger recreation of the resources due to the missing annotations. This should only happen the first time an operation is done to the discovered resource.
|
||||
|
||||
# Authenticating Directly with a Downstream Cluster
|
||||
|
||||
This section intended to help you set up an alternative method to access an [RKE cluster.]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-provisioning/rke-clusters)
|
||||
|
||||
This method is only available for RKE clusters that have the [authorized cluster endpoint]({{<baseurl>}}/rancher/v2.0-v2.4/en/overview/architecture/#4-authorized-cluster-endpoint) enabled. When Rancher creates this RKE cluster, it generates a kubeconfig file that includes additional kubectl context(s) for accessing your cluster. This additional context allows you to use kubectl to authenticate with the downstream cluster without authenticating through Rancher. For a longer explanation of how the authorized cluster endpoint works, refer to [this page.](../ace)
|
||||
|
||||
We recommend that as a best practice, you should set up this method to access your RKE cluster, so that just in case you can’t connect to Rancher, you can still access the cluster.
|
||||
|
||||
> **Prerequisites:** The following steps assume that you have created a Kubernetes cluster and followed the steps to [connect to your cluster with kubectl from your workstation.](#accessing-clusters-with-kubectl-from-your-workstation)
|
||||
|
||||
To find the name of the context(s) in your downloaded kubeconfig file, run:
|
||||
|
||||
```
|
||||
kubectl config get-contexts --kubeconfig /custom/path/kube.config
|
||||
CURRENT NAME CLUSTER AUTHINFO NAMESPACE
|
||||
* my-cluster my-cluster user-46tmn
|
||||
my-cluster-controlplane-1 my-cluster-controlplane-1 user-46tmn
|
||||
```
|
||||
|
||||
In this example, when you use `kubectl` with the first context, `my-cluster`, you will be authenticated through the Rancher server.
|
||||
|
||||
With the second context, `my-cluster-controlplane-1`, you would authenticate with the authorized cluster endpoint, communicating with an downstream RKE cluster directly.
|
||||
|
||||
We recommend using a load balancer with the authorized cluster endpoint. For details, refer to the [recommended architecture section.]({{<baseurl>}}/rancher/v2.0-v2.4/en/overview/architecture-recommendations/#architecture-for-an-authorized-cluster-endpoint)
|
||||
|
||||
Now that you have the name of the context needed to authenticate directly with the cluster, you can pass the name of the context in as an option when running kubectl commands. The commands will differ depending on whether your cluster has an FQDN defined. Examples are provided in the sections below.
|
||||
|
||||
When `kubectl` works normally, it confirms that you can access your cluster while bypassing Rancher's authentication proxy.
|
||||
|
||||
### Connecting Directly to Clusters with FQDN Defined
|
||||
|
||||
If an FQDN is defined for the cluster, a single context referencing the FQDN will be created. The context will be named `<CLUSTER_NAME>-fqdn`. When you want to use `kubectl` to access this cluster without Rancher, you will need to use this context.
|
||||
|
||||
Assuming the kubeconfig file is located at `~/.kube/config`:
|
||||
|
||||
```
|
||||
kubectl --context <CLUSTER_NAME>-fqdn get nodes
|
||||
```
|
||||
Directly referencing the location of the kubeconfig file:
|
||||
```
|
||||
kubectl --kubeconfig /custom/path/kube.config --context <CLUSTER_NAME>-fqdn get pods
|
||||
```
|
||||
|
||||
### Connecting Directly to Clusters without FQDN Defined
|
||||
|
||||
If there is no FQDN defined for the cluster, extra contexts will be created referencing the IP address of each node in the control plane. Each context will be named `<CLUSTER_NAME>-<NODE_NAME>`. When you want to use `kubectl` to access this cluster without Rancher, you will need to use this context.
|
||||
|
||||
Assuming the kubeconfig file is located at `~/.kube/config`:
|
||||
```
|
||||
kubectl --context <CLUSTER_NAME>-<NODE_NAME> get nodes
|
||||
```
|
||||
Directly referencing the location of the kubeconfig file:
|
||||
```
|
||||
kubectl --kubeconfig /custom/path/kube.config --context <CLUSTER_NAME>-<NODE_NAME> get pods
|
||||
```
|
||||
@@ -0,0 +1,39 @@
|
||||
---
|
||||
title: Cluster Administration
|
||||
weight: 8
|
||||
---
|
||||
|
||||
After you provision a cluster in Rancher, you can begin using powerful Kubernetes features to deploy and scale your containerized applications in development, testing, or production environments.
|
||||
|
||||
This page covers the following topics:
|
||||
|
||||
- [Switching between clusters](#switching-between-clusters)
|
||||
- [Managing clusters in Rancher](#managing-clusters-in-rancher)
|
||||
- [Configuring tools](#configuring-tools)
|
||||
|
||||
> This section assumes a basic familiarity with Docker and Kubernetes. For a brief explanation of how Kubernetes components work together, refer to the [concepts]({{<baseurl>}}/rancher/v2.0-v2.4/en/overview/concepts) page.
|
||||
|
||||
## Switching between Clusters
|
||||
|
||||
To switch between clusters, use the drop-down available in the navigation bar.
|
||||
|
||||
Alternatively, you can switch between projects and clusters directly in the navigation bar. Open the **Global** view and select **Clusters** from the main menu. Then select the name of the cluster you want to open.
|
||||
|
||||
## Managing Clusters in Rancher
|
||||
|
||||
After clusters have been [provisioned into Rancher]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-provisioning/), [cluster owners]({{<baseurl>}}/rancher/v2.0-v2.4/en/admin-settings/rbac/cluster-project-roles/#cluster-roles) will need to manage these clusters. There are many different options of how to manage your cluster.
|
||||
|
||||
{{% include file="/rancher/v2.0-v2.4/en/cluster-provisioning/cluster-capabilities-table</summary>
|
||||
|
||||
## Configuring Tools
|
||||
|
||||
Rancher contains a variety of tools that aren't included in Kubernetes to assist in your DevOps operations. Rancher can integrate with external services to help your clusters run more efficiently. Tools are divided into following categories:
|
||||
|
||||
- Alerts
|
||||
- Notifiers
|
||||
- Logging
|
||||
- Monitoring
|
||||
- Istio Service Mesh
|
||||
- OPA Gatekeeper
|
||||
|
||||
For more information, see [Tools]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/tools/)
|
||||
@@ -0,0 +1,580 @@
|
||||
---
|
||||
title: Cluster Autoscaler with AWS EC2 Auto Scaling Groups
|
||||
weight: 1
|
||||
---
|
||||
|
||||
This guide will show you how to install and use [Kubernetes cluster-autoscaler](https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/) on Rancher custom clusters using AWS EC2 Auto Scaling Groups.
|
||||
|
||||
We are going to install a Rancher RKE custom cluster with a fixed number of nodes with the etcd and controlplane roles, and a variable nodes with the worker role, managed by `cluster-autoscaler`.
|
||||
|
||||
- [Prerequisites](#prerequisites)
|
||||
- [1. Create a Custom Cluster](#1-create-a-custom-cluster)
|
||||
- [2. Configure the Cloud Provider](#2-configure-the-cloud-provider)
|
||||
- [3. Deploy Nodes](#3-deploy-nodes)
|
||||
- [4. Install cluster-autoscaler](#4-install-cluster-autoscaler)
|
||||
- [Parameters](#parameters)
|
||||
- [Deployment](#deployment)
|
||||
- [Testing](#testing)
|
||||
- [Generating Load](#generating-load)
|
||||
- [Checking Scale](#checking-scale)
|
||||
|
||||
# Prerequisites
|
||||
|
||||
These elements are required to follow this guide:
|
||||
|
||||
* The Rancher server is up and running
|
||||
* You have an AWS EC2 user with proper permissions to create virtual machines, auto scaling groups, and IAM profiles and roles
|
||||
|
||||
### 1. Create a Custom Cluster
|
||||
|
||||
On Rancher server, we should create a custom k8s cluster v1.18.x. Be sure that cloud_provider name is set to `amazonec2`. Once cluster is created we need to get:
|
||||
|
||||
* clusterID: `c-xxxxx` will be used on EC2 `kubernetes.io/cluster/<clusterID>` instance tag
|
||||
* clusterName: will be used on EC2 `k8s.io/cluster-autoscaler/<clusterName>` instance tag
|
||||
* nodeCommand: will be added on EC2 instance user_data to include new nodes on cluster
|
||||
|
||||
```sh
|
||||
sudo docker run -d --restart=unless-stopped --net=host -v /etc/kubernetes:/etc/kubernetes -v /var/run:/var/run rancher/rancher-agent:<RANCHER_VERSION> --server https://<RANCHER_URL> --token <RANCHER_TOKEN> --ca-checksum <RANCHER_CHECKSUM> <roles>
|
||||
```
|
||||
|
||||
### 2. Configure the Cloud Provider
|
||||
|
||||
On AWS EC2, we should create a few objects to configure our system. We've defined three distinct groups and IAM profiles to configure on AWS.
|
||||
|
||||
1. Autoscaling group: Nodes that will be part of the EC2 Auto Scaling Group (ASG). The ASG will be used by `cluster-autoscaler` to scale up and down.
|
||||
* IAM profile: Required by k8s nodes where cluster-autoscaler will be running. It is recommended for Kubernetes master nodes. This profile is called `K8sAutoscalerProfile`.
|
||||
|
||||
```json
|
||||
{
|
||||
"Version": "2012-10-17",
|
||||
"Statement": [
|
||||
{
|
||||
"Effect": "Allow",
|
||||
"Action": [
|
||||
"autoscaling:DescribeAutoScalingGroups",
|
||||
"autoscaling:DescribeAutoScalingInstances",
|
||||
"autoscaling:DescribeLaunchConfigurations",
|
||||
"autoscaling:SetDesiredCapacity",
|
||||
"autoscaling:TerminateInstanceInAutoScalingGroup",
|
||||
"autoscaling:DescribeTags",
|
||||
"autoscaling:DescribeLaunchConfigurations",
|
||||
"ec2:DescribeLaunchTemplateVersions"
|
||||
],
|
||||
"Resource": [
|
||||
"*"
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
2. Master group: Nodes that will be part of the Kubernetes etcd and/or control planes. This will be out of the ASG.
|
||||
* IAM profile: Required by the Kubernetes cloud_provider integration. Optionally, `AWS_ACCESS_KEY` and `AWS_SECRET_KEY` can be used instead [using-aws-credentials.](https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/aws/README.md#using-aws-credentials) This profile is called `K8sMasterProfile`.
|
||||
|
||||
```json
|
||||
{
|
||||
"Version": "2012-10-17",
|
||||
"Statement": [
|
||||
{
|
||||
"Effect": "Allow",
|
||||
"Action": [
|
||||
"autoscaling:DescribeAutoScalingGroups",
|
||||
"autoscaling:DescribeLaunchConfigurations",
|
||||
"autoscaling:DescribeTags",
|
||||
"ec2:DescribeInstances",
|
||||
"ec2:DescribeRegions",
|
||||
"ec2:DescribeRouteTables",
|
||||
"ec2:DescribeSecurityGroups",
|
||||
"ec2:DescribeSubnets",
|
||||
"ec2:DescribeVolumes",
|
||||
"ec2:CreateSecurityGroup",
|
||||
"ec2:CreateTags",
|
||||
"ec2:CreateVolume",
|
||||
"ec2:ModifyInstanceAttribute",
|
||||
"ec2:ModifyVolume",
|
||||
"ec2:AttachVolume",
|
||||
"ec2:AuthorizeSecurityGroupIngress",
|
||||
"ec2:CreateRoute",
|
||||
"ec2:DeleteRoute",
|
||||
"ec2:DeleteSecurityGroup",
|
||||
"ec2:DeleteVolume",
|
||||
"ec2:DetachVolume",
|
||||
"ec2:RevokeSecurityGroupIngress",
|
||||
"ec2:DescribeVpcs",
|
||||
"elasticloadbalancing:AddTags",
|
||||
"elasticloadbalancing:AttachLoadBalancerToSubnets",
|
||||
"elasticloadbalancing:ApplySecurityGroupsToLoadBalancer",
|
||||
"elasticloadbalancing:CreateLoadBalancer",
|
||||
"elasticloadbalancing:CreateLoadBalancerPolicy",
|
||||
"elasticloadbalancing:CreateLoadBalancerListeners",
|
||||
"elasticloadbalancing:ConfigureHealthCheck",
|
||||
"elasticloadbalancing:DeleteLoadBalancer",
|
||||
"elasticloadbalancing:DeleteLoadBalancerListeners",
|
||||
"elasticloadbalancing:DescribeLoadBalancers",
|
||||
"elasticloadbalancing:DescribeLoadBalancerAttributes",
|
||||
"elasticloadbalancing:DetachLoadBalancerFromSubnets",
|
||||
"elasticloadbalancing:DeregisterInstancesFromLoadBalancer",
|
||||
"elasticloadbalancing:ModifyLoadBalancerAttributes",
|
||||
"elasticloadbalancing:RegisterInstancesWithLoadBalancer",
|
||||
"elasticloadbalancing:SetLoadBalancerPoliciesForBackendServer",
|
||||
"elasticloadbalancing:AddTags",
|
||||
"elasticloadbalancing:CreateListener",
|
||||
"elasticloadbalancing:CreateTargetGroup",
|
||||
"elasticloadbalancing:DeleteListener",
|
||||
"elasticloadbalancing:DeleteTargetGroup",
|
||||
"elasticloadbalancing:DescribeListeners",
|
||||
"elasticloadbalancing:DescribeLoadBalancerPolicies",
|
||||
"elasticloadbalancing:DescribeTargetGroups",
|
||||
"elasticloadbalancing:DescribeTargetHealth",
|
||||
"elasticloadbalancing:ModifyListener",
|
||||
"elasticloadbalancing:ModifyTargetGroup",
|
||||
"elasticloadbalancing:RegisterTargets",
|
||||
"elasticloadbalancing:SetLoadBalancerPoliciesOfListener",
|
||||
"iam:CreateServiceLinkedRole",
|
||||
"ecr:GetAuthorizationToken",
|
||||
"ecr:BatchCheckLayerAvailability",
|
||||
"ecr:GetDownloadUrlForLayer",
|
||||
"ecr:GetRepositoryPolicy",
|
||||
"ecr:DescribeRepositories",
|
||||
"ecr:ListImages",
|
||||
"ecr:BatchGetImage",
|
||||
"kms:DescribeKey"
|
||||
],
|
||||
"Resource": [
|
||||
"*"
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
* IAM role: `K8sMasterRole: [K8sMasterProfile,K8sAutoscalerProfile]`
|
||||
* Security group: `K8sMasterSg` More info at[RKE ports (custom nodes tab)]({{<baseurl>}}/rancher/v2.0-v2.4/en/installation/requirements/ports/#downstream-kubernetes-cluster-nodes)
|
||||
* Tags:
|
||||
`kubernetes.io/cluster/<clusterID>: owned`
|
||||
* User data: `K8sMasterUserData` Ubuntu 18.04(ami-0e11cbb34015ff725), installs docker and add etcd+controlplane node to the k8s cluster
|
||||
|
||||
```sh
|
||||
#!/bin/bash -x
|
||||
|
||||
cat <<EOF > /etc/sysctl.d/90-kubelet.conf
|
||||
vm.overcommit_memory = 1
|
||||
vm.panic_on_oom = 0
|
||||
kernel.panic = 10
|
||||
kernel.panic_on_oops = 1
|
||||
kernel.keys.root_maxkeys = 1000000
|
||||
kernel.keys.root_maxbytes = 25000000
|
||||
EOF
|
||||
sysctl -p /etc/sysctl.d/90-kubelet.conf
|
||||
|
||||
curl -sL https://releases.rancher.com/install-docker/19.03.sh | sh
|
||||
sudo usermod -aG docker ubuntu
|
||||
|
||||
TOKEN=$(curl -s -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600")
|
||||
PRIVATE_IP=$(curl -H "X-aws-ec2-metadata-token: ${TOKEN}" -s http://169.254.169.254/latest/meta-data/local-ipv4)
|
||||
PUBLIC_IP=$(curl -H "X-aws-ec2-metadata-token: ${TOKEN}" -s http://169.254.169.254/latest/meta-data/public-ipv4)
|
||||
K8S_ROLES="--etcd --controlplane"
|
||||
|
||||
sudo docker run -d --restart=unless-stopped --net=host -v /etc/kubernetes:/etc/kubernetes -v /var/run:/var/run rancher/rancher-agent:<RANCHER_VERSION> --server https://<RANCHER_URL> --token <RANCHER_TOKEN> --ca-checksum <RANCHER_CA_CHECKSUM> --address ${PUBLIC_IP} --internal-address ${PRIVATE_IP} ${K8S_ROLES}
|
||||
```
|
||||
|
||||
3. Worker group: Nodes that will be part of the k8s worker plane. Worker nodes will be scaled by cluster-autoscaler using the ASG.
|
||||
* IAM profile: Provides cloud_provider worker integration.
|
||||
This profile is called `K8sWorkerProfile`.
|
||||
|
||||
```json
|
||||
{
|
||||
"Version": "2012-10-17",
|
||||
"Statement": [
|
||||
{
|
||||
"Effect": "Allow",
|
||||
"Action": [
|
||||
"ec2:DescribeInstances",
|
||||
"ec2:DescribeRegions",
|
||||
"ecr:GetAuthorizationToken",
|
||||
"ecr:BatchCheckLayerAvailability",
|
||||
"ecr:GetDownloadUrlForLayer",
|
||||
"ecr:GetRepositoryPolicy",
|
||||
"ecr:DescribeRepositories",
|
||||
"ecr:ListImages",
|
||||
"ecr:BatchGetImage"
|
||||
],
|
||||
"Resource": "*"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
* IAM role: `K8sWorkerRole: [K8sWorkerProfile]`
|
||||
* Security group: `K8sWorkerSg` More info at [RKE ports (custom nodes tab)]({{<baseurl>}}/rancher/v2.0-v2.4/en/installation/requirements/ports/#downstream-kubernetes-cluster-nodes)
|
||||
* Tags:
|
||||
* `kubernetes.io/cluster/<clusterID>: owned`
|
||||
* `k8s.io/cluster-autoscaler/<clusterName>: true`
|
||||
* `k8s.io/cluster-autoscaler/enabled: true`
|
||||
* User data: `K8sWorkerUserData` Ubuntu 18.04(ami-0e11cbb34015ff725), installs docker and add worker node to the k8s cluster
|
||||
|
||||
```sh
|
||||
#!/bin/bash -x
|
||||
|
||||
cat <<EOF > /etc/sysctl.d/90-kubelet.conf
|
||||
vm.overcommit_memory = 1
|
||||
vm.panic_on_oom = 0
|
||||
kernel.panic = 10
|
||||
kernel.panic_on_oops = 1
|
||||
kernel.keys.root_maxkeys = 1000000
|
||||
kernel.keys.root_maxbytes = 25000000
|
||||
EOF
|
||||
sysctl -p /etc/sysctl.d/90-kubelet.conf
|
||||
|
||||
curl -sL https://releases.rancher.com/install-docker/19.03.sh | sh
|
||||
sudo usermod -aG docker ubuntu
|
||||
|
||||
TOKEN=$(curl -s -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600")
|
||||
PRIVATE_IP=$(curl -H "X-aws-ec2-metadata-token: ${TOKEN}" -s http://169.254.169.254/latest/meta-data/local-ipv4)
|
||||
PUBLIC_IP=$(curl -H "X-aws-ec2-metadata-token: ${TOKEN}" -s http://169.254.169.254/latest/meta-data/public-ipv4)
|
||||
K8S_ROLES="--worker"
|
||||
|
||||
sudo docker run -d --restart=unless-stopped --net=host -v /etc/kubernetes:/etc/kubernetes -v /var/run:/var/run rancher/rancher-agent:<RANCHER_VERSION> --server https://<RANCHER_URL> --token <RANCHER_TOKEN> --ca-checksum <RANCHER_CA_CHECKCSUM> --address ${PUBLIC_IP} --internal-address ${PRIVATE_IP} ${K8S_ROLES}
|
||||
```
|
||||
|
||||
More info is at [RKE clusters on AWS]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-provisioning/rke-clusters/cloud-providers/amazon/) and [Cluster Autoscaler on AWS.](https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/aws/README.md)
|
||||
|
||||
### 3. Deploy Nodes
|
||||
|
||||
Once we've configured AWS, let's create VMs to bootstrap our cluster:
|
||||
|
||||
* master (etcd+controlplane): Depending your needs, deploy three master instances with proper size. More info is at [the recommendations for production-ready clusters.]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-provisioning/production/)
|
||||
* IAM role: `K8sMasterRole`
|
||||
* Security group: `K8sMasterSg`
|
||||
* Tags:
|
||||
* `kubernetes.io/cluster/<clusterID>: owned`
|
||||
* User data: `K8sMasterUserData`
|
||||
|
||||
* worker: Define an ASG on EC2 with the following settings:
|
||||
* Name: `K8sWorkerAsg`
|
||||
* IAM role: `K8sWorkerRole`
|
||||
* Security group: `K8sWorkerSg`
|
||||
* Tags:
|
||||
* `kubernetes.io/cluster/<clusterID>: owned`
|
||||
* `k8s.io/cluster-autoscaler/<clusterName>: true`
|
||||
* `k8s.io/cluster-autoscaler/enabled: true`
|
||||
* User data: `K8sWorkerUserData`
|
||||
* Instances:
|
||||
* minimum: 2
|
||||
* desired: 2
|
||||
* maximum: 10
|
||||
|
||||
Once the VMs are deployed, you should have a Rancher custom cluster up and running with three master and two worker nodes.
|
||||
|
||||
### 4. Install Cluster-autoscaler
|
||||
|
||||
At this point, we should have rancher cluster up and running. We are going to install cluster-autoscaler on master nodes and `kube-system` namespace, following cluster-autoscaler recommendation.
|
||||
|
||||
#### Parameters
|
||||
|
||||
This table shows cluster-autoscaler parameters for fine tuning:
|
||||
|
||||
| Parameter | Default | Description |
|
||||
|---|---|---|
|
||||
|cluster-name|-|Autoscaled cluster name, if available|
|
||||
|address|:8085|The address to expose Prometheus metrics|
|
||||
|kubernetes|-|Kubernetes master location. Leave blank for default|
|
||||
|kubeconfig|-|Path to kubeconfig file with authorization and master location information|
|
||||
|cloud-config|-|The path to the cloud provider configuration file. Empty string for no configuration file|
|
||||
|namespace|"kube-system"|Namespace in which cluster-autoscaler run|
|
||||
|scale-down-enabled|true|Should CA scale down the cluster|
|
||||
|scale-down-delay-after-add|"10m"|How long after scale up that scale down evaluation resumes|
|
||||
|scale-down-delay-after-delete|0|How long after node deletion that scale down evaluation resumes, defaults to scanInterval|
|
||||
|scale-down-delay-after-failure|"3m"|How long after scale down failure that scale down evaluation resumes|
|
||||
|scale-down-unneeded-time|"10m"|How long a node should be unneeded before it is eligible for scale down|
|
||||
|scale-down-unready-time|"20m"|How long an unready node should be unneeded before it is eligible for scale down|
|
||||
|scale-down-utilization-threshold|0.5|Sum of cpu or memory of all pods running on the node divided by node's corresponding allocatable resource, below which a node can be considered for scale down|
|
||||
|scale-down-gpu-utilization-threshold|0.5|Sum of gpu requests of all pods running on the node divided by node's allocatable resource, below which a node can be considered for scale down|
|
||||
|scale-down-non-empty-candidates-count|30|Maximum number of non empty nodes considered in one iteration as candidates for scale down with drain|
|
||||
|scale-down-candidates-pool-ratio|0.1|A ratio of nodes that are considered as additional non empty candidates for scale down when some candidates from previous iteration are no longer valid|
|
||||
|scale-down-candidates-pool-min-count|50|Minimum number of nodes that are considered as additional non empty candidates for scale down when some candidates from previous iteration are no longer valid|
|
||||
|node-deletion-delay-timeout|"2m"|Maximum time CA waits for removing delay-deletion.cluster-autoscaler.kubernetes.io/ annotations before deleting the node|
|
||||
|scan-interval|"10s"|How often cluster is reevaluated for scale up or down|
|
||||
|max-nodes-total|0|Maximum number of nodes in all node groups. Cluster autoscaler will not grow the cluster beyond this number|
|
||||
|cores-total|"0:320000"|Minimum and maximum number of cores in cluster, in the format `<min>:<max>.` Cluster autoscaler will not scale the cluster beyond these numbers|
|
||||
|memory-total|"0:6400000"|Minimum and maximum number of gigabytes of memory in cluster, in the format `<min>:<max>.` Cluster autoscaler will not scale the cluster beyond these numbers|
|
||||
cloud-provider|-|Cloud provider type|
|
||||
|max-bulk-soft-taint-count|10|Maximum number of nodes that can be tainted/untainted PreferNoSchedule at the same time. Set to 0 to turn off such tainting|
|
||||
|max-bulk-soft-taint-time|"3s"|Maximum duration of tainting/untainting nodes as PreferNoSchedule at the same time|
|
||||
|max-empty-bulk-delete|10|Maximum number of empty nodes that can be deleted at the same time|
|
||||
|max-graceful-termination-sec|600|Maximum number of seconds CA waits for pod termination when trying to scale down a node|
|
||||
|max-total-unready-percentage|45|Maximum percentage of unready nodes in the cluster. After this is exceeded, CA halts operations|
|
||||
|ok-total-unready-count|3|Number of allowed unready nodes, irrespective of max-total-unready-percentage|
|
||||
|scale-up-from-zero|true|Should CA scale up when there 0 ready nodes|
|
||||
|max-node-provision-time|"15m"|Maximum time CA waits for node to be provisioned|
|
||||
|nodes|-|sets min,max size and other configuration data for a node group in a format accepted by cloud provider. Can be used multiple times. Format: `<min>:<max>:<other...>`|
|
||||
|node-group-auto-discovery|-|One or more definition(s) of node group auto-discovery. A definition is expressed `<name of discoverer>:[<key>[=<value>]]`|
|
||||
|estimator|-|"binpacking"|Type of resource estimator to be used in scale up. Available values: ["binpacking"]|
|
||||
|expander|"random"|Type of node group expander to be used in scale up. Available values: `["random","most-pods","least-waste","price","priority"]`|
|
||||
|ignore-daemonsets-utilization|false|Should CA ignore DaemonSet pods when calculating resource utilization for scaling down|
|
||||
|ignore-mirror-pods-utilization|false|Should CA ignore Mirror pods when calculating resource utilization for scaling down|
|
||||
|write-status-configmap|true|Should CA write status information to a configmap|
|
||||
|max-inactivity|"10m"|Maximum time from last recorded autoscaler activity before automatic restart|
|
||||
|max-failing-time|"15m"|Maximum time from last recorded successful autoscaler run before automatic restart|
|
||||
|balance-similar-node-groups|false|Detect similar node groups and balance the number of nodes between them|
|
||||
|node-autoprovisioning-enabled|false|Should CA autoprovision node groups when needed|
|
||||
|max-autoprovisioned-node-group-count|15|The maximum number of autoprovisioned groups in the cluster|
|
||||
|unremovable-node-recheck-timeout|"5m"|The timeout before we check again a node that couldn't be removed before|
|
||||
|expendable-pods-priority-cutoff|-10|Pods with priority below cutoff will be expendable. They can be killed without any consideration during scale down and they don't cause scale up. Pods with null priority (PodPriority disabled) are non expendable|
|
||||
|regional|false|Cluster is regional|
|
||||
|new-pod-scale-up-delay|"0s"|Pods less than this old will not be considered for scale-up|
|
||||
|ignore-taint|-|Specifies a taint to ignore in node templates when considering to scale a node group|
|
||||
|balancing-ignore-label|-|Specifies a label to ignore in addition to the basic and cloud-provider set of labels when comparing if two node groups are similar|
|
||||
|aws-use-static-instance-list|false|Should CA fetch instance types in runtime or use a static list. AWS only|
|
||||
|profiling|false|Is debug/pprof endpoint enabled|
|
||||
|
||||
#### Deployment
|
||||
|
||||
Based on [cluster-autoscaler-run-on-master.yaml](https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/aws/examples/cluster-autoscaler-run-on-master.yaml) example, we've created our own `cluster-autoscaler-deployment.yaml` to use preferred [auto-discovery setup](https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler/cloudprovider/aws#auto-discovery-setup), updating tolerations, nodeSelector, image version and command config:
|
||||
|
||||
|
||||
```yml
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: ServiceAccount
|
||||
metadata:
|
||||
labels:
|
||||
k8s-addon: cluster-autoscaler.addons.k8s.io
|
||||
k8s-app: cluster-autoscaler
|
||||
name: cluster-autoscaler
|
||||
namespace: kube-system
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: ClusterRole
|
||||
metadata:
|
||||
name: cluster-autoscaler
|
||||
labels:
|
||||
k8s-addon: cluster-autoscaler.addons.k8s.io
|
||||
k8s-app: cluster-autoscaler
|
||||
rules:
|
||||
- apiGroups: [""]
|
||||
resources: ["events", "endpoints"]
|
||||
verbs: ["create", "patch"]
|
||||
- apiGroups: [""]
|
||||
resources: ["pods/eviction"]
|
||||
verbs: ["create"]
|
||||
- apiGroups: [""]
|
||||
resources: ["pods/status"]
|
||||
verbs: ["update"]
|
||||
- apiGroups: [""]
|
||||
resources: ["endpoints"]
|
||||
resourceNames: ["cluster-autoscaler"]
|
||||
verbs: ["get", "update"]
|
||||
- apiGroups: [""]
|
||||
resources: ["nodes"]
|
||||
verbs: ["watch", "list", "get", "update"]
|
||||
- apiGroups: [""]
|
||||
resources:
|
||||
- "pods"
|
||||
- "services"
|
||||
- "replicationcontrollers"
|
||||
- "persistentvolumeclaims"
|
||||
- "persistentvolumes"
|
||||
verbs: ["watch", "list", "get"]
|
||||
- apiGroups: ["extensions"]
|
||||
resources: ["replicasets", "daemonsets"]
|
||||
verbs: ["watch", "list", "get"]
|
||||
- apiGroups: ["policy"]
|
||||
resources: ["poddisruptionbudgets"]
|
||||
verbs: ["watch", "list"]
|
||||
- apiGroups: ["apps"]
|
||||
resources: ["statefulsets", "replicasets", "daemonsets"]
|
||||
verbs: ["watch", "list", "get"]
|
||||
- apiGroups: ["storage.k8s.io"]
|
||||
resources: ["storageclasses", "csinodes"]
|
||||
verbs: ["watch", "list", "get"]
|
||||
- apiGroups: ["batch", "extensions"]
|
||||
resources: ["jobs"]
|
||||
verbs: ["get", "list", "watch", "patch"]
|
||||
- apiGroups: ["coordination.k8s.io"]
|
||||
resources: ["leases"]
|
||||
verbs: ["create"]
|
||||
- apiGroups: ["coordination.k8s.io"]
|
||||
resourceNames: ["cluster-autoscaler"]
|
||||
resources: ["leases"]
|
||||
verbs: ["get", "update"]
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: Role
|
||||
metadata:
|
||||
name: cluster-autoscaler
|
||||
namespace: kube-system
|
||||
labels:
|
||||
k8s-addon: cluster-autoscaler.addons.k8s.io
|
||||
k8s-app: cluster-autoscaler
|
||||
rules:
|
||||
- apiGroups: [""]
|
||||
resources: ["configmaps"]
|
||||
verbs: ["create","list","watch"]
|
||||
- apiGroups: [""]
|
||||
resources: ["configmaps"]
|
||||
resourceNames: ["cluster-autoscaler-status", "cluster-autoscaler-priority-expander"]
|
||||
verbs: ["delete", "get", "update", "watch"]
|
||||
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: ClusterRoleBinding
|
||||
metadata:
|
||||
name: cluster-autoscaler
|
||||
labels:
|
||||
k8s-addon: cluster-autoscaler.addons.k8s.io
|
||||
k8s-app: cluster-autoscaler
|
||||
roleRef:
|
||||
apiGroup: rbac.authorization.k8s.io
|
||||
kind: ClusterRole
|
||||
name: cluster-autoscaler
|
||||
subjects:
|
||||
- kind: ServiceAccount
|
||||
name: cluster-autoscaler
|
||||
namespace: kube-system
|
||||
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: RoleBinding
|
||||
metadata:
|
||||
name: cluster-autoscaler
|
||||
namespace: kube-system
|
||||
labels:
|
||||
k8s-addon: cluster-autoscaler.addons.k8s.io
|
||||
k8s-app: cluster-autoscaler
|
||||
roleRef:
|
||||
apiGroup: rbac.authorization.k8s.io
|
||||
kind: Role
|
||||
name: cluster-autoscaler
|
||||
subjects:
|
||||
- kind: ServiceAccount
|
||||
name: cluster-autoscaler
|
||||
namespace: kube-system
|
||||
|
||||
---
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: cluster-autoscaler
|
||||
namespace: kube-system
|
||||
labels:
|
||||
app: cluster-autoscaler
|
||||
spec:
|
||||
replicas: 1
|
||||
selector:
|
||||
matchLabels:
|
||||
app: cluster-autoscaler
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: cluster-autoscaler
|
||||
annotations:
|
||||
prometheus.io/scrape: 'true'
|
||||
prometheus.io/port: '8085'
|
||||
spec:
|
||||
serviceAccountName: cluster-autoscaler
|
||||
tolerations:
|
||||
- effect: NoSchedule
|
||||
operator: "Equal"
|
||||
value: "true"
|
||||
key: node-role.kubernetes.io/controlplane
|
||||
nodeSelector:
|
||||
node-role.kubernetes.io/controlplane: "true"
|
||||
containers:
|
||||
- image: eu.gcr.io/k8s-artifacts-prod/autoscaling/cluster-autoscaler:v1.18.1
|
||||
name: cluster-autoscaler
|
||||
resources:
|
||||
limits:
|
||||
cpu: 100m
|
||||
memory: 300Mi
|
||||
requests:
|
||||
cpu: 100m
|
||||
memory: 300Mi
|
||||
command:
|
||||
- ./cluster-autoscaler
|
||||
- --v=4
|
||||
- --stderrthreshold=info
|
||||
- --cloud-provider=aws
|
||||
- --skip-nodes-with-local-storage=false
|
||||
- --expander=least-waste
|
||||
- --node-group-auto-discovery=asg:tag=k8s.io/cluster-autoscaler/enabled,k8s.io/cluster-autoscaler/<clusterName>
|
||||
volumeMounts:
|
||||
- name: ssl-certs
|
||||
mountPath: /etc/ssl/certs/ca-certificates.crt
|
||||
readOnly: true
|
||||
imagePullPolicy: "Always"
|
||||
volumes:
|
||||
- name: ssl-certs
|
||||
hostPath:
|
||||
path: "/etc/ssl/certs/ca-certificates.crt"
|
||||
|
||||
```
|
||||
|
||||
Once the manifest file is prepared, deploy it in the Kubernetes cluster (Rancher UI can be used instead):
|
||||
|
||||
```sh
|
||||
kubectl -n kube-system apply -f cluster-autoscaler-deployment.yaml
|
||||
```
|
||||
|
||||
**Note:** Cluster-autoscaler deployment can also be set up using [manual configuration](https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler/cloudprovider/aws#manual-configuration)
|
||||
|
||||
# Testing
|
||||
|
||||
At this point, we should have a cluster-scaler up and running in our Rancher custom cluster. Cluster-scale should manage `K8sWorkerAsg` ASG to scale up and down between 2 and 10 nodes, when one of the following conditions is true:
|
||||
|
||||
* There are pods that failed to run in the cluster due to insufficient resources. In this case, the cluster is scaled up.
|
||||
* There are nodes in the cluster that have been underutilized for an extended period of time and their pods can be placed on other existing nodes. In this case, the cluster is scaled down.
|
||||
|
||||
### Generating Load
|
||||
|
||||
We've prepared a `test-deployment.yaml` just to generate load on the Kubernetes cluster and see if cluster-autoscaler is working properly. The test deployment is requesting 1000m CPU and 1024Mi memory by three replicas. Adjust the requested resources and/or replica to be sure you exhaust the Kubernetes cluster resources:
|
||||
|
||||
```yaml
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
labels:
|
||||
app: hello-world
|
||||
name: hello-world
|
||||
spec:
|
||||
replicas: 3
|
||||
selector:
|
||||
matchLabels:
|
||||
app: hello-world
|
||||
strategy:
|
||||
rollingUpdate:
|
||||
maxSurge: 1
|
||||
maxUnavailable: 0
|
||||
type: RollingUpdate
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: hello-world
|
||||
spec:
|
||||
containers:
|
||||
- image: rancher/hello-world
|
||||
imagePullPolicy: Always
|
||||
name: hello-world
|
||||
ports:
|
||||
- containerPort: 80
|
||||
protocol: TCP
|
||||
resources:
|
||||
limits:
|
||||
cpu: 1000m
|
||||
memory: 1024Mi
|
||||
requests:
|
||||
cpu: 1000m
|
||||
memory: 1024Mi
|
||||
```
|
||||
|
||||
Once the test deployment is prepared, deploy it in the Kubernetes cluster default namespace (Rancher UI can be used instead):
|
||||
|
||||
```
|
||||
kubectl -n default apply -f test-deployment.yaml
|
||||
```
|
||||
|
||||
### Checking Scale
|
||||
|
||||
Once the Kubernetes resources got exhausted, cluster-autoscaler should scale up worker nodes where pods failed to be scheduled. It should scale up until up until all pods became scheduled. You should see the new nodes on the ASG and on the Kubernetes cluster. Check the logs on the `kube-system` cluster-autoscaler pod.
|
||||
|
||||
Once scale up is checked, let check for scale down. To do it, reduce the replica number on the test deployment until you release enough Kubernetes cluster resources to scale down. You should see nodes disappear on the ASG and on the Kubernetes cluster. Check the logs on the `kube-system` cluster-autoscaler pod.
|
||||
@@ -0,0 +1,25 @@
|
||||
---
|
||||
title: Cluster Autoscaler
|
||||
weight: 1
|
||||
---
|
||||
|
||||
In this section, you'll learn how to install and use the [Kubernetes cluster-autoscaler](https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/) on Rancher custom clusters using AWS EC2 Auto Scaling Groups.
|
||||
|
||||
The cluster autoscaler is a tool that automatically adjusts the size of the Kubernetes cluster when one of the following conditions is true:
|
||||
|
||||
* There are pods that failed to run in the cluster due to insufficient resources.
|
||||
* There are nodes in the cluster that have been underutilized for an extended period of time and their pods can be placed on other existing nodes.
|
||||
|
||||
To prevent your pod from being evicted, set a `priorityClassName: system-cluster-critical` property on your pod spec.
|
||||
|
||||
Cluster Autoscaler is designed to run on Kubernetes master nodes. It can run in the `kube-system` namespace. Cluster Autoscaler doesn't scale down nodes with non-mirrored `kube-system` pods running on them.
|
||||
|
||||
It's possible to run a customized deployment of Cluster Autoscaler on worker nodes, but extra care needs to be taken to ensure that Cluster Autoscaler remains up and running.
|
||||
|
||||
# Cloud Providers
|
||||
|
||||
Cluster Autoscaler provides support to distinct cloud providers. For more information, go to [cluster-autoscaler supported cloud providers.](https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler#deployment)
|
||||
|
||||
### Setting up Cluster Autoscaler on Amazon Cloud Provider
|
||||
|
||||
For details on running the cluster autoscaler on Amazon cloud provider, refer to [this page.]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/cluster-autoscaler/amazon)
|
||||
@@ -0,0 +1,68 @@
|
||||
---
|
||||
title: Cluster Configuration
|
||||
weight: 2025
|
||||
---
|
||||
|
||||
After you provision a Kubernetes cluster using Rancher, you can still edit options and settings for the cluster.
|
||||
|
||||
For information on editing cluster membership, go to [this page.]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/cluster-access/cluster-members)
|
||||
|
||||
- [Cluster Management Capabilities by Cluster Type](#cluster-management-capabilities-by-cluster-type)
|
||||
- [Editing Clusters in the Rancher UI](#editing-clusters-in-the-rancher-ui)
|
||||
- [Editing Clusters with YAML](#editing-clusters-with-yaml)
|
||||
- [Updating ingress-nginx](#updating-ingress-nginx)
|
||||
|
||||
### Cluster Management Capabilities by Cluster Type
|
||||
|
||||
The options and settings available for an existing cluster change based on the method that you used to provision it. For example, only clusters [provisioned by RKE]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-provisioning/rke-clusters/) have **Cluster Options** available for editing.
|
||||
|
||||
The following table summarizes the options and settings available for each cluster type:
|
||||
|
||||
{{% include file="/rancher/v2.0-v2.4/en/cluster-provisioning/cluster-capabilities-table</summary>
|
||||
|
||||
### Editing Clusters in the Rancher UI
|
||||
|
||||
To edit your cluster, open the **Global** view, make sure the **Clusters** tab is selected, and then select **⋮ > Edit** for the cluster that you want to edit.
|
||||
|
||||
In [clusters launched by RKE]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-provisioning/rke-clusters/), you can edit any of the remaining options that follow.
|
||||
|
||||
Note that these options are not available for imported clusters or hosted Kubernetes clusters.
|
||||
|
||||
Option | Description |
|
||||
---------|----------|
|
||||
Kubernetes Version | The version of Kubernetes installed on each cluster node. For more detail, see [Upgrading Kubernetes]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/upgrading-kubernetes). |
|
||||
Network Provider | The \container networking interface (CNI) that powers networking for your cluster.<br/><br/>**Note:** You can only choose this option while provisioning your cluster. It cannot be edited later. |
|
||||
Project Network Isolation | As of Rancher v2.0.7, if you're using the Canal network provider, you can choose whether to enable or disable inter-project communication. |
|
||||
Nginx Ingress | If you want to publish your applications in a high-availability configuration, and you're hosting your nodes with a cloud-provider that doesn't have a native load-balancing feature, enable this option to use Nginx ingress within the cluster. |
|
||||
Metrics Server Monitoring | Each cloud provider capable of launching a cluster using RKE can collect metrics and monitor for your cluster nodes. Enable this option to view your node metrics from your cloud provider's portal. |
|
||||
Pod Security Policy Support | Enables [pod security policies]({{<baseurl>}}/rancher/v2.0-v2.4/en/admin-settings/pod-security-policies/) for the cluster. After enabling this option, choose a policy using the **Default Pod Security Policy** drop-down. |
|
||||
Docker version on nodes | Configures whether nodes are allowed to run versions of Docker that Rancher doesn't officially support. If you choose to require a [supported Docker version]({{<baseurl>}}/rancher/v2.0-v2.4/en/installation/options/rke-add-on/layer-7-lb/), Rancher will stop pods from running on nodes that don't have a supported Docker version installed. |
|
||||
Docker Root Directory | The directory on your cluster nodes where you've installed Docker. If you install Docker on your nodes to a non-default directory, update this path. |
|
||||
Default Pod Security Policy | If you enable **Pod Security Policy Support**, use this drop-down to choose the pod security policy that's applied to the cluster. |
|
||||
Cloud Provider | If you're using a cloud provider to host cluster nodes launched by RKE, enable [this option]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-provisioning/rke-clusters/options/cloud-providers/) so that you can use the cloud provider's native features. If you want to store persistent data for your cloud-hosted cluster, this option is required. |
|
||||
|
||||
### Editing Clusters with YAML
|
||||
|
||||
Instead of using the Rancher UI to choose Kubernetes options for the cluster, advanced users can create an RKE config file. Using a config file allows you to set any of the options available in an RKE installation, except for system_images configuration, by specifying them in YAML.
|
||||
|
||||
- To edit an RKE config file directly from the Rancher UI, click **Edit as YAML**.
|
||||
- To read from an existing RKE file, click **Read from File**.
|
||||
|
||||
|
||||
|
||||
For an example of RKE config file syntax, see the [RKE documentation]({{<baseurl>}}/rke/latest/en/example-yamls/).
|
||||
|
||||
For the complete reference of configurable options for RKE Kubernetes clusters in YAML, see the [RKE documentation.]({{<baseurl>}}/rke/latest/en/config-options/)
|
||||
|
||||
In Rancher v2.0.0-v2.2.x, the config file is identical to the [cluster config file for the Rancher Kubernetes Engine]({{<baseurl>}}/rke/latest/en/config-options/), which is the tool Rancher uses to provision clusters. In Rancher v2.3.0, the RKE information is still included in the config file, but it is separated from other options, so that the RKE cluster config options are nested under the `rancher_kubernetes_engine_config` directive. For more information, see the [cluster configuration reference.]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-provisioning/rke-clusters/options)
|
||||
|
||||
>**Note:** In Rancher v2.0.5 and v2.0.6, the names of services in the Config File (YAML) should contain underscores only: `kube_api` and `kube_controller`.
|
||||
|
||||
|
||||
|
||||
|
||||
### Updating ingress-nginx
|
||||
|
||||
Clusters that were created before Kubernetes 1.16 will have an `ingress-nginx` `updateStrategy` of `OnDelete`. Clusters that were created with Kubernetes 1.16 or newer will have `RollingUpdate`.
|
||||
|
||||
If the `updateStrategy` of `ingress-nginx` is `OnDelete`, you will need to delete these pods to get the correct version for your deployment.
|
||||
@@ -0,0 +1,232 @@
|
||||
---
|
||||
title: Nodes and Node Pools
|
||||
weight: 2030
|
||||
---
|
||||
|
||||
import Tabs from '@theme/Tabs';
|
||||
import TabItem from '@theme/TabItem';
|
||||
|
||||
After you launch a Kubernetes cluster in Rancher, you can manage individual nodes from the cluster's **Node** tab. Depending on the [option used]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-provisioning/) to provision the cluster, there are different node options available.
|
||||
|
||||
> If you want to manage the _cluster_ and not individual nodes, see [Editing Clusters]({{< baseurl >}}/rancher/v2.0-v2.4/en/cluster-admin/editing-clusters/#editing-clusters-with-yaml).
|
||||
|
||||
This section covers the following topics:
|
||||
|
||||
- [Node options available for each cluster creation option](#node-options-available-for-each-cluster-creation-option)
|
||||
- [Nodes hosted by an infrastructure provider](#nodes-hosted-by-an-infrastructure-provider)
|
||||
- [Nodes provisioned by hosted Kubernetes providers](#nodes-provisioned-by-hosted-kubernetes-providers)
|
||||
- [Imported nodes](#imported-nodes)
|
||||
- [Managing and editing individual nodes](#managing-and-editing-individual-nodes)
|
||||
- [Viewing a node in the Rancher API](#viewing-a-node-in-the-rancher-api)
|
||||
- [Deleting a node](#deleting-a-node)
|
||||
- [Scaling nodes](#scaling-nodes)
|
||||
- [SSH into a node hosted by an infrastructure provider](#ssh-into-a-node-hosted-by-an-infrastructure-provider)
|
||||
- [Cordoning a node](#cordoning-a-node)
|
||||
- [Draining a node](#draining-a-node)
|
||||
- [Aggressive and safe draining options](#aggressive-and-safe-draining-options)
|
||||
- [Grace period](#grace-period)
|
||||
- [Timeout](#timeout)
|
||||
- [Drained and cordoned state](#drained-and-cordoned-state)
|
||||
- [Labeling a node to be ignored by Rancher](#labeling-a-node-to-be-ignored-by-rancher)
|
||||
|
||||
# Node Options Available for Each Cluster Creation Option
|
||||
|
||||
The following table lists which node options are available for each type of cluster in Rancher. Click the links in the **Option** column for more detailed information about each feature.
|
||||
|
||||
| Option | [Nodes Hosted by an Infrastructure Provider][1] | [Custom Node][2] | [Hosted Cluster][3] | [Imported Nodes][4] | Description |
|
||||
| ------------------------------------------------ | ------------------------------------------------ | ---------------- | ------------------- | ------------------- | ------------------------------------------------------------------ |
|
||||
| [Cordon](#cordoning-a-node) | ✓ | ✓ | ✓ | | Marks the node as unschedulable. |
|
||||
| [Drain](#draining-a-node) | ✓ | ✓ | ✓ | | Marks the node as unschedulable _and_ evicts all pods. |
|
||||
| [Edit](#managing-and-editing-individual-nodes) | ✓ | ✓ | ✓ | | Enter a custom name, description, label, or taints for a node. |
|
||||
| [View API](#viewing-a-node-in-the-rancher-api) | ✓ | ✓ | ✓ | | View API data. |
|
||||
| [Delete](#deleting-a-node) | ✓ | ✓ | | | Deletes defective nodes from the cluster. |
|
||||
| [Download Keys](#ssh-into-a-node-hosted-by-an-infrastructure-provider) | ✓ | | | | Download SSH key for in order to SSH into the node. |
|
||||
| [Node Scaling](#scaling-nodes) | ✓ | | | | Scale the number of nodes in the node pool up or down. |
|
||||
|
||||
[1]: {{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-provisioning/rke-clusters/node-pools/
|
||||
[2]: {{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-provisioning/rke-clusters/custom-nodes/
|
||||
[3]: {{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-provisioning/hosted-kubernetes-clusters/
|
||||
[4]: {{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-provisioning/imported-clusters/
|
||||
|
||||
### Nodes Hosted by an Infrastructure Provider
|
||||
|
||||
Node pools are available when you provision Rancher-launched Kubernetes clusters on nodes that are [hosted in an infrastructure provider.]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-provisioning/rke-clusters/node-pools/)
|
||||
|
||||
Clusters provisioned using [one of the node pool options]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-provisioning/rke-clusters/node-pools/#node-pools) can be scaled up or down if the node pool is edited.
|
||||
|
||||
A node pool can also automatically maintain the node scale that's set during the initial cluster provisioning if [node auto-replace is enabled.]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-provisioning/rke-clusters/node-pools/#about-node-auto-replace) This scale determines the number of active nodes that Rancher maintains for the cluster.
|
||||
|
||||
Rancher uses [node templates]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-provisioning/rke-clusters/node-pools/#node-templates) to replace nodes in the node pool. Each node template uses cloud provider credentials to allow Rancher to set up the node in the infrastructure provider.
|
||||
|
||||
### Nodes Provisioned by Hosted Kubernetes Providers
|
||||
|
||||
Options for managing nodes [hosted by a Kubernetes provider]({{<baseurl >}}/rancher/v2.0-v2.4/en/cluster-provisioning/hosted-kubernetes-clusters/) are somewhat limited in Rancher. Rather than using the Rancher UI to make edits such as scaling the number of nodes up or down, edit the cluster directly.
|
||||
|
||||
### Imported Nodes
|
||||
|
||||
Although you can deploy workloads to an [imported cluster]({{< baseurl >}}/rancher/v2.0-v2.4/en/cluster-provisioning/imported-clusters/) using Rancher, you cannot manage individual cluster nodes. All management of imported cluster nodes must take place outside of Rancher.
|
||||
|
||||
# Managing and Editing Individual Nodes
|
||||
|
||||
Editing a node lets you:
|
||||
|
||||
* Change its name
|
||||
* Change its description
|
||||
* Add [labels](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/)
|
||||
* Add/Remove [taints](https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/)
|
||||
|
||||
To manage individual nodes, browse to the cluster that you want to manage and then select **Nodes** from the main menu. You can open the options menu for a node by clicking its **⋮** icon (**...**).
|
||||
|
||||
# Viewing a Node in the Rancher API
|
||||
|
||||
Select this option to view the node's [API endpoints]({{< baseurl >}}/rancher/v2.0-v2.4/en/api/).
|
||||
|
||||
# Deleting a Node
|
||||
|
||||
Use **Delete** to remove defective nodes from the cloud provider.
|
||||
|
||||
When you the delete a defective node, Rancher can automatically replace it with an identically provisioned node if the node is in a node pool and [node auto-replace is enabled.]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-provisioning/rke-clusters/node-pools/#about-node-auto-replace)
|
||||
|
||||
>**Tip:** If your cluster is hosted by an infrastructure provider, and you want to scale your cluster down instead of deleting a defective node, [scale down](#scaling-nodes) rather than delete.
|
||||
|
||||
# Scaling Nodes
|
||||
|
||||
For nodes hosted by an infrastructure provider, you can scale the number of nodes in each [node pool]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-provisioning/rke-clusters/node-pools/#node-pools) by using the scale controls. This option isn't available for other cluster types.
|
||||
|
||||
# SSH into a Node Hosted by an Infrastructure Provider
|
||||
|
||||
For [nodes hosted by an infrastructure provider]({{< baseurl >}}/rancher/v2.0-v2.4/en/cluster-provisioning/rke-clusters/node-pools/), you have the option of downloading its SSH key so that you can connect to it remotely from your desktop.
|
||||
|
||||
1. From the cluster hosted by an infrastructure provider, select **Nodes** from the main menu.
|
||||
|
||||
1. Find the node that you want to remote into. Select **⋮ > Download Keys**.
|
||||
|
||||
**Step Result:** A ZIP file containing files used for SSH is downloaded.
|
||||
|
||||
1. Extract the ZIP file to any location.
|
||||
|
||||
1. Open Terminal. Change your location to the extracted ZIP file.
|
||||
|
||||
1. Enter the following command:
|
||||
|
||||
```
|
||||
ssh -i id_rsa root@<IP_OF_HOST>
|
||||
```
|
||||
|
||||
# Cordoning a Node
|
||||
|
||||
_Cordoning_ a node marks it as unschedulable. This feature is useful for performing short tasks on the node during small maintenance windows, like reboots, upgrades, or decommissions. When you're done, power back on and make the node schedulable again by uncordoning it.
|
||||
|
||||
# Draining a Node
|
||||
|
||||
_Draining_ is the process of first cordoning the node, and then evicting all its pods. This feature is useful for performing node maintenance (like kernel upgrades or hardware maintenance). It prevents new pods from deploying to the node while redistributing existing pods so that users don't experience service interruption.
|
||||
|
||||
- For pods with a replica set, the pod is replaced by a new pod that will be scheduled to a new node. Additionally, if the pod is part of a service, then clients will automatically be redirected to the new pod.
|
||||
|
||||
- For pods with no replica set, you need to bring up a new copy of the pod, and assuming it is not part of a service, redirect clients to it.
|
||||
|
||||
You can drain nodes that are in either a `cordoned` or `active` state. When you drain a node, the node is cordoned, the nodes are evaluated for conditions they must meet to be drained, and then (if it meets the conditions) the node evicts its pods.
|
||||
|
||||
However, you can override the conditions draining when you initiate the drain. You're also given an opportunity to set a grace period and timeout value.
|
||||
|
||||
### Aggressive and Safe Draining Options
|
||||
|
||||
The node draining options are different based on your version of Rancher.
|
||||
|
||||
<Tabs>
|
||||
<TabItem value="Rancher v2.2.x+">
|
||||
|
||||
There are two drain modes: aggressive and safe.
|
||||
|
||||
- **Aggressive Mode**
|
||||
|
||||
In this mode, pods won't get rescheduled to a new node, even if they do not have a controller. Kubernetes expects you to have your own logic that handles the deletion of these pods.
|
||||
|
||||
Kubernetes also expects the implementation to decide what to do with pods using emptyDir. If a pod uses emptyDir to store local data, you might not be able to safely delete it, since the data in the emptyDir will be deleted once the pod is removed from the node. Choosing aggressive mode will delete these pods.
|
||||
|
||||
- **Safe Mode**
|
||||
|
||||
If a node has standalone pods or ephemeral data it will be cordoned but not drained.
|
||||
|
||||
</TabItem>
|
||||
<TabItem value="Rancher before v2.2.x">
|
||||
|
||||
The following list describes each drain option:
|
||||
|
||||
- **Even if there are pods not managed by a ReplicationController, ReplicaSet, Job, DaemonSet or StatefulSet**
|
||||
|
||||
These types of pods won't get rescheduled to a new node, since they do not have a controller. Kubernetes expects you to have your own logic that handles the deletion of these pods. Kubernetes forces you to choose this option (which will delete/evict these pods) or drain won't proceed.
|
||||
|
||||
- **Even if there are DaemonSet-managed pods**
|
||||
|
||||
Similar to above, if you have any daemonsets, drain would proceed only if this option is selected. Even when this option is on, pods won't be deleted since they'll immediately be replaced. On startup, Rancher currently has a few daemonsets running by default in the system, so this option is turned on by default.
|
||||
|
||||
- **Even if there are pods using emptyDir**
|
||||
|
||||
If a pod uses emptyDir to store local data, you might not be able to safely delete it, since the data in the emptyDir will be deleted once the pod is removed from the node. Similar to the first option, Kubernetes expects the implementation to decide what to do with these pods. Choosing this option will delete these pods.
|
||||
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
|
||||
### Grace Period
|
||||
|
||||
The timeout given to each pod for cleaning things up, so they will have chance to exit gracefully. For example, when pods might need to finish any outstanding requests, roll back transactions or save state to some external storage. If negative, the default value specified in the pod will be used.
|
||||
|
||||
### Timeout
|
||||
|
||||
The amount of time drain should continue to wait before giving up.
|
||||
|
||||
>**Kubernetes Known Issue:** The [timeout setting](https://github.com/kubernetes/kubernetes/pull/64378) was not enforced while draining a node before Kubernetes 1.12.
|
||||
|
||||
### Drained and Cordoned State
|
||||
|
||||
If there's any error related to user input, the node enters a `cordoned` state because the drain failed. You can either correct the input and attempt to drain the node again, or you can abort by uncordoning the node.
|
||||
|
||||
If the drain continues without error, the node enters a `draining` state. You'll have the option to stop the drain when the node is in this state, which will stop the drain process and change the node's state to `cordoned`.
|
||||
|
||||
Once drain successfully completes, the node will be in a state of `drained`. You can then power off or delete the node.
|
||||
|
||||
>**Want to know more about cordon and drain?** See the [Kubernetes documentation](https://kubernetes.io/docs/tasks/administer-cluster/cluster-management/#maintenance-on-a-node).
|
||||
|
||||
# Labeling a Node to be Ignored by Rancher
|
||||
|
||||
_Available as of 2.3.3_
|
||||
|
||||
Some solutions, such as F5's BIG-IP integration, may require creating a node that is never registered to a cluster.
|
||||
|
||||
Since the node will never finish registering, it will always be shown as unhealthy in the Rancher UI.
|
||||
|
||||
In that case, you may want to label the node to be ignored by Rancher so that Rancher only shows nodes as unhealthy when they are actually failing.
|
||||
|
||||
You can label nodes to be ignored by using a setting in the Rancher UI, or by using `kubectl`.
|
||||
|
||||
> **Note:** There is an [open issue](https://github.com/rancher/rancher/issues/24172) in which nodes labeled to be ignored can get stuck in an updating state.
|
||||
|
||||
### Labeling Nodes to be Ignored with the Rancher UI
|
||||
|
||||
To add a node that is ignored by Rancher,
|
||||
|
||||
1. From the **Global** view, click the **Settings** tab.
|
||||
1. Go to the `ignore-node-name` setting and click **⋮ > Edit.**
|
||||
1. Enter a name that Rancher will use to ignore nodes. All nodes with this name will be ignored.
|
||||
1. Click **Save.**
|
||||
|
||||
**Result:** Rancher will not wait to register nodes with this name. In the UI, the node will displayed with a grayed-out status. The node is still part of the cluster and can be listed with `kubectl`.
|
||||
|
||||
If the setting is changed afterward, the ignored nodes will continue to be hidden.
|
||||
|
||||
### Labeling Nodes to be Ignored with kubectl
|
||||
|
||||
To add a node that will be ignored by Rancher, use `kubectl` to create a node that has the following label:
|
||||
|
||||
```
|
||||
cattle.rancher.io/node-status: ignore
|
||||
```
|
||||
|
||||
**Result:** If you add the node to a cluster, Rancher will not attempt to sync with this node. The node can still be part of the cluster and can be listed with `kubectl`.
|
||||
|
||||
If the label is added before the node is added to the cluster, the node will not be shown in the Rancher UI.
|
||||
|
||||
If the label is added after the node is added to a Rancher cluster, the node will not be removed from the UI.
|
||||
|
||||
If you delete the node from the Rancher server using the Rancher UI or API, the node will not be removed from the cluster if the `nodeName` is listed in the Rancher settings under `ignore-node-name`.
|
||||
+30
@@ -0,0 +1,30 @@
|
||||
---
|
||||
title: Adding a Pod Security Policy
|
||||
weight: 80
|
||||
---
|
||||
|
||||
> **Prerequisite:** The options below are available only for clusters that are [launched using RKE.]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-provisioning/rke-clusters/)
|
||||
|
||||
When your cluster is running pods with security-sensitive configurations, assign it a [pod security policy]({{<baseurl>}}/rancher/v2.0-v2.4/en/admin-settings/pod-security-policies/), which is a set of rules that monitors the conditions and settings in your pods. If a pod doesn't meet the rules specified in your policy, the policy stops it from running.
|
||||
|
||||
You can assign a pod security policy when you provision a cluster. However, if you need to relax or restrict security for your pods later, you can update the policy while editing your cluster.
|
||||
|
||||
1. From the **Global** view, find the cluster to which you want to apply a pod security policy. Select **⋮ > Edit**.
|
||||
|
||||
2. Expand **Cluster Options**.
|
||||
|
||||
3. From **Pod Security Policy Support**, select **Enabled**.
|
||||
|
||||
>**Note:** This option is only available for clusters [provisioned by RKE]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-provisioning/rke-clusters/).
|
||||
|
||||
4. From the **Default Pod Security Policy** drop-down, select the policy you want to apply to the cluster.
|
||||
|
||||
Rancher ships with [policies]({{<baseurl>}}/rancher/v2.0-v2.4/en/admin-settings/pod-security-policies/#default-pod-security-policies) of `restricted` and `unrestricted`, although you can [create custom policies]({{<baseurl>}}/rancher/v2.0-v2.4/en/admin-settings/pod-security-policies/#default-pod-security-policies) as well.
|
||||
|
||||
5. Click **Save**.
|
||||
|
||||
**Result:** The pod security policy is applied to the cluster and any projects within the cluster.
|
||||
|
||||
>**Note:** Workloads already running before assignment of a pod security policy are grandfathered in. Even if they don't meet your pod security policy, workloads running before assignment of the policy continue to run.
|
||||
>
|
||||
>To check if a running workload passes your pod security policy, clone or upgrade it.
|
||||
+206
@@ -0,0 +1,206 @@
|
||||
---
|
||||
title: Projects and Kubernetes Namespaces with Rancher
|
||||
description: Rancher Projects ease the administrative burden of your cluster and support multi-tenancy. Learn to create projects and divide projects into Kubernetes namespaces
|
||||
weight: 2032
|
||||
aliases:
|
||||
- /rancher/v2.0-v2.4/en/concepts/projects/
|
||||
- /rancher/v2.0-v2.4/en/tasks/projects/
|
||||
- /rancher/v2.0-v2.4/en/tasks/projects/create-project/
|
||||
- /rancher/v2.0-v2.4/en/tasks/projects/create-project/
|
||||
---
|
||||
|
||||
A namespace is a Kubernetes concept that allows a virtual cluster within a cluster, which is useful for dividing the cluster into separate "virtual clusters" that each have their own access control and resource quotas.
|
||||
|
||||
A project is a group of namespaces, and it is a concept introduced by Rancher. Projects allow you to manage multiple namespaces as a group and perform Kubernetes operations in them. You can use projects to support multi-tenancy, so that a team can access a project within a cluster without having access to other projects in the same cluster.
|
||||
|
||||
This section describes how projects and namespaces work with Rancher. It covers the following topics:
|
||||
|
||||
- [About namespaces](#about-namespaces)
|
||||
- [About projects](#about-projects)
|
||||
- [The cluster's default project](#the-cluster-s-default-project)
|
||||
- [The system project](#the-system-project)
|
||||
- [Project authorization](#project-authorization)
|
||||
- [Pod security policies](#pod-security-policies)
|
||||
- [Creating projects](#creating-projects)
|
||||
- [Switching between clusters and projects](#switching-between-clusters-and-projects)
|
||||
|
||||
# About Namespaces
|
||||
|
||||
A namespace is a concept introduced by Kubernetes. According to the [official Kubernetes documentation on namespaces,](https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/)
|
||||
|
||||
> Kubernetes supports multiple virtual clusters backed by the same physical cluster. These virtual clusters are called namespaces. [...] Namespaces are intended for use in environments with many users spread across multiple teams, or projects. For clusters with a few to tens of users, you should not need to create or think about namespaces at all.
|
||||
|
||||
Namespaces provide the following functionality:
|
||||
|
||||
- **Providing a scope for names:** Names of resources need to be unique within a namespace, but not across namespaces. Namespaces can not be nested inside one another and each Kubernetes resource can only be in one namespace.
|
||||
- **Resource quotas:** Namespaces provide a way to divide cluster resources between multiple users.
|
||||
|
||||
You can assign resources at the project level so that each namespace in the project can use them. You can also bypass this inheritance by assigning resources explicitly to a namespace.
|
||||
|
||||
You can assign the following resources directly to namespaces:
|
||||
|
||||
- [Workloads]({{<baseurl>}}/rancher/v2.0-v2.4/en/k8s-in-rancher/workloads/)
|
||||
- [Load Balancers/Ingress]({{<baseurl>}}/rancher/v2.0-v2.4/en/k8s-in-rancher/load-balancers-and-ingress/)
|
||||
- [Service Discovery Records]({{<baseurl>}}/rancher/v2.0-v2.4/en/k8s-in-rancher/service-discovery/)
|
||||
- [Persistent Volume Claims]({{<baseurl>}}/rancher/v2.0-v2.4/en/k8s-in-rancher/volumes-and-storage/persistent-volume-claims/)
|
||||
- [Certificates]({{<baseurl>}}/rancher/v2.0-v2.4/en/k8s-in-rancher/certificates/)
|
||||
- [ConfigMaps]({{<baseurl>}}/rancher/v2.0-v2.4/en/k8s-in-rancher/configmaps/)
|
||||
- [Registries]({{<baseurl>}}/rancher/v2.0-v2.4/en/k8s-in-rancher/registries/)
|
||||
- [Secrets]({{<baseurl>}}/rancher/v2.0-v2.4/en/k8s-in-rancher/secrets/)
|
||||
|
||||
To manage permissions in a vanilla Kubernetes cluster, cluster admins configure role-based access policies for each namespace. With Rancher, user permissions are assigned on the project level instead, and permissions are automatically inherited by any namespace owned by the particular project.
|
||||
|
||||
For more information on creating and moving namespaces, see [Namespaces]({{<baseurl>}}/rancher/v2.0-v2.4/en/project-admin/namespaces/).
|
||||
|
||||
### Role-based access control issues with namespaces and kubectl
|
||||
|
||||
Because projects are a concept introduced by Rancher, kubectl does not have the capability to restrict the creation of namespaces to a project the creator has access to.
|
||||
|
||||
This means that when standard users with project-scoped permissions create a namespaces with `kubectl`, it may be unusable because `kubectl` doesn't require the new namespace to be scoped within a certain project.
|
||||
|
||||
If your permissions are restricted to the project level, it is better to [create a namespace through Rancher]({{<baseurl>}}/rancher/v2.0-v2.4/en/project-admin/namespaces/) to ensure that you will have permission to access the namespace.
|
||||
|
||||
If a standard user is a project owner, the user will be able to create namespaces within that project. The Rancher UI will prevent that user from creating namespaces outside the scope of the projects they have access to.
|
||||
|
||||
# About Projects
|
||||
|
||||
In terms of hierarchy:
|
||||
|
||||
- Clusters contain projects
|
||||
- Projects contain namespaces
|
||||
|
||||
You can use projects to support multi-tenancy, so that a team can access a project within a cluster without having access to other projects in the same cluster.
|
||||
|
||||
In the base version of Kubernetes, features like role-based access rights or cluster resources are assigned to individual namespaces. A project allows you to save time by giving an individual or a team access to multiple namespaces simultaneously.
|
||||
|
||||
You can use projects to perform actions such as:
|
||||
|
||||
- Assign users to a group of namespaces (i.e., [project membership]({{<baseurl>}}/rancher/v2.0-v2.4/en/k8s-in-rancher/projects-and-namespaces/project-members)).
|
||||
- Assign users specific roles in a project. A role can be owner, member, read-only, or [custom]({{<baseurl>}}/rancher/v2.0-v2.4/en/admin-settings/rbac/default-custom-roles/).
|
||||
- Assign resources to the project.
|
||||
- Assign Pod Security Policies.
|
||||
|
||||
When you create a cluster, two projects are automatically created within it:
|
||||
|
||||
- [Default Project](#the-cluster-s-default-project)
|
||||
- [System Project](#the-system-project)
|
||||
|
||||
### The Cluster's Default Project
|
||||
|
||||
When you provision a cluster with Rancher, it automatically creates a `default` project for the cluster. This is a project you can use to get started with your cluster, but you can always delete it and replace it with projects that have more descriptive names.
|
||||
|
||||
If you don't have a need for more than the default namespace, you also do not need more than the **Default** project in Rancher.
|
||||
|
||||
If you require another level of organization beyond the **Default** project, you can create more projects in Rancher to isolate namespaces, applications and resources.
|
||||
|
||||
### The System Project
|
||||
|
||||
_Available as of v2.0.7_
|
||||
|
||||
When troubleshooting, you can view the `system` project to check if important namespaces in the Kubernetes system are working properly. This easily accessible project saves you from troubleshooting individual system namespace containers.
|
||||
|
||||
To open it, open the **Global** menu, and then select the `system` project for your cluster.
|
||||
|
||||
The `system` project:
|
||||
|
||||
- Is automatically created when you provision a cluster.
|
||||
- Lists all namespaces that exist in `v3/settings/system-namespaces`, if they exist.
|
||||
- Allows you to add more namespaces or move its namespaces to other projects.
|
||||
- Cannot be deleted because it's required for cluster operations.
|
||||
|
||||
>**Note:** In clusters where both:
|
||||
>
|
||||
> - The Canal network plug-in is in use.
|
||||
> - The Project Network Isolation option is enabled.
|
||||
>
|
||||
>The `system` project overrides the Project Network Isolation option so that it can communicate with other projects, collect logs, and check health.
|
||||
|
||||
# Project Authorization
|
||||
|
||||
Standard users are only authorized for project access in two situations:
|
||||
|
||||
- An administrator, cluster owner or cluster member explicitly adds the standard user to the project's **Members** tab.
|
||||
- Standard users can access projects that they create themselves.
|
||||
|
||||
# Pod Security Policies
|
||||
|
||||
Rancher extends Kubernetes to allow the application of [Pod Security Policies](https://kubernetes.io/docs/concepts/policy/pod-security-policy/) at the [project level]({{<baseurl>}}/rancher/v2.0-v2.4/en/project-admin/pod-security-policies) in addition to the [cluster level.](../pod-security-policy) However, as a best practice, we recommend applying Pod Security Policies at the cluster level.
|
||||
|
||||
# Creating Projects
|
||||
|
||||
This section describes how to create a new project with a name and with optional pod security policy, members, and resource quotas.
|
||||
|
||||
1. [Name a new project.](#1-name-a-new-project)
|
||||
2. [Optional: Select a pod security policy.](#2-optional-select-a-pod-security-policy)
|
||||
3. [Recommended: Add project members.](#3-recommended-add-project-members)
|
||||
4. [Optional: Add resource quotas.](#4-optional-add-resource-quotas)
|
||||
|
||||
### 1. Name a New Project
|
||||
|
||||
1. From the **Global** view, choose **Clusters** from the main menu. From the **Clusters** page, open the cluster from which you want to create a project.
|
||||
|
||||
1. From the main menu, choose **Projects/Namespaces**. Then click **Add Project**.
|
||||
|
||||
1. Enter a **Project Name**.
|
||||
|
||||
### 2. Optional: Select a Pod Security Policy
|
||||
|
||||
This option is only available if you've already created a Pod Security Policy. For instruction, see [Creating Pod Security Policies]({{<baseurl>}}/rancher/v2.0-v2.4/en/admin-settings/pod-security-policies/).
|
||||
|
||||
Assigning a PSP to a project will:
|
||||
|
||||
- Override the cluster's default PSP.
|
||||
- Apply the PSP to the project.
|
||||
- Apply the PSP to any namespaces you add to the project later.
|
||||
|
||||
### 3. Recommended: Add Project Members
|
||||
|
||||
Use the **Members** section to provide other users with project access and roles.
|
||||
|
||||
By default, your user is added as the project `Owner`.
|
||||
|
||||
>**Notes on Permissions:**
|
||||
>
|
||||
>- Users assigned the `Owner` or `Member` role for a project automatically inherit the `namespace creation` role. However, this role is a [Kubernetes ClusterRole](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#role-and-clusterrole), meaning its scope extends to all projects in the cluster. Therefore, users explicitly assigned the `Owner` or `Member` role for a project can create namespaces in other projects they're assigned to, even with only the `Read Only` role assigned.
|
||||
>
|
||||
>- By default, the Rancher role of `project-member` inherits from the `Kubernetes-edit` role, and the `project-owner` role inherits from the `Kubernetes-admin` role. As such, both `project-member` and `project-owner` roles will allow for namespace management, including the ability to create and delete namespaces.
|
||||
>
|
||||
>- Choose `Custom` to create a custom role on the fly: [Custom Project Roles]({{<baseurl>}}/rancher/v2.0-v2.4/en/admin-settings/rbac/cluster-project-roles/#custom-project-roles).
|
||||
|
||||
To add members:
|
||||
|
||||
1. Click **Add Member**.
|
||||
1. From the **Name** combo box, search for a user or group that you want to assign project access. Note: You can only search for groups if external authentication is enabled.
|
||||
1. From the **Role** drop-down, choose a role. For more information, refer to the [documentation on project roles.]({{<baseurl>}}/rancher/v2.0-v2.4/en/admin-settings/rbac/cluster-project-roles/)
|
||||
|
||||
### 4. Optional: Add Resource Quotas
|
||||
|
||||
_Available as of v2.1.0_
|
||||
|
||||
Resource quotas limit the resources that a project (and its namespaces) can consume. For more information, see [Resource Quotas]({{<baseurl>}}/rancher/v2.0-v2.4/en/k8s-in-rancher/projects-and-namespaces/resource-quotas).
|
||||
|
||||
To add a resource quota,
|
||||
|
||||
1. Click **Add Quota**.
|
||||
1. Select a Resource Type. For more information, see [Resource Quotas.]({{<baseurl>}}/rancher/v2.0-v2.4/en/k8s-in-rancher/projects-and-namespaces/resource-quotas/).
|
||||
1. Enter values for the **Project Limit** and the **Namespace Default Limit**.
|
||||
1. **Optional:** Specify **Container Default Resource Limit**, which will be applied to every container started in the project. The parameter is recommended if you have CPU or Memory limits set by the Resource Quota. It can be overridden on per an individual namespace or a container level. For more information, see [Container Default Resource Limit]({{<baseurl>}}/rancher/v2.0-v2.4/en/project-admin/resource-quotas/) Note: This option is available as of v2.2.0.
|
||||
1. Click **Create**.
|
||||
|
||||
**Result:** Your project is created. You can view it from the cluster's **Projects/Namespaces** view.
|
||||
|
||||
| Field | Description |
|
||||
| ----------------------- | -------------------------------------------------------------------------------------------------------- |
|
||||
| Project Limit | The overall resource limit for the project. |
|
||||
| Namespace Default Limit | The default resource limit available for each namespace. This limit is propagated to each namespace in the project when created. The combined limit of all project namespaces shouldn't exceed the project limit. |
|
||||
|
||||
# Switching between Clusters and Projects
|
||||
|
||||
To switch between clusters and projects, use the **Global** drop-down available in the main menu.
|
||||
|
||||
|
||||
|
||||
Alternatively, you can switch between projects and clusters using the main menu.
|
||||
|
||||
- To switch between clusters, open the **Global** view and select **Clusters** from the main menu. Then open a cluster.
|
||||
- To switch between projects, open a cluster, and then select **Projects/Namespaces** from the main menu. Select the link for the project that you want to open.
|
||||
@@ -0,0 +1,116 @@
|
||||
---
|
||||
title: Restoring a Cluster from Backup
|
||||
weight: 2050
|
||||
---
|
||||
|
||||
import Tabs from '@theme/Tabs';
|
||||
import TabItem from '@theme/TabItem';
|
||||
|
||||
_Available as of v2.2.0_
|
||||
|
||||
etcd backup and recovery for [Rancher launched Kubernetes clusters]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-provisioning/rke-clusters/) can be easily performed. Snapshots of the etcd database are taken and saved either locally onto the etcd nodes or to a S3 compatible target. The advantages of configuring S3 is that if all etcd nodes are lost, your snapshot is saved remotely and can be used to restore the cluster.
|
||||
|
||||
Rancher recommends enabling the [ability to set up recurring snapshots of etcd]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/backing-up-etcd/#configuring-recurring-snapshots), but [one-time snapshots]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/backing-up-etcd/#one-time-snapshots) can easily be taken as well. Rancher allows restore from [saved snapshots](#restoring-a-cluster-from-a-snapshot) or if you don't have any snapshots, you can still [restore etcd](#recovering-etcd-without-a-snapshot).
|
||||
|
||||
As of Rancher v2.4.0, clusters can also be restored to a prior Kubernetes version and cluster configuration.
|
||||
|
||||
This section covers the following topics:
|
||||
|
||||
- [Viewing Available Snapshots](#viewing-available-snapshots)
|
||||
- [Restoring a Cluster from a Snapshot](#restoring-a-cluster-from-a-snapshot)
|
||||
- [Recovering etcd without a Snapshot](#recovering-etcd-without-a-snapshot)
|
||||
- [Enabling snapshot features for clusters created before Rancher v2.2.0](#enabling-snapshot-features-for-clusters-created-before-rancher-v2-2-0)
|
||||
|
||||
## Viewing Available Snapshots
|
||||
|
||||
The list of all available snapshots for the cluster is available.
|
||||
|
||||
1. In the **Global** view, navigate to the cluster that you want to view snapshots.
|
||||
|
||||
2. Click **Tools > Snapshots** from the navigation bar to view the list of saved snapshots. These snapshots include a timestamp of when they were created.
|
||||
|
||||
## Restoring a Cluster from a Snapshot
|
||||
|
||||
If your Kubernetes cluster is broken, you can restore the cluster from a snapshot.
|
||||
|
||||
Restores changed in Rancher v2.4.0.
|
||||
|
||||
<Tabs>
|
||||
<TabItem value="Rancher v2.4.0+">
|
||||
|
||||
Snapshots are composed of the cluster data in etcd, the Kubernetes version, and the cluster configuration in the `cluster.yml.` These components allow you to select from the following options when restoring a cluster from a snapshot:
|
||||
|
||||
- **Restore just the etcd contents:** This restore is similar to restoring to snapshots in Rancher before v2.4.0.
|
||||
- **Restore etcd and Kubernetes version:** This option should be used if a Kubernetes upgrade is the reason that your cluster is failing, and you haven't made any cluster configuration changes.
|
||||
- **Restore etcd, Kubernetes versions and cluster configuration:** This option should be used if you changed both the Kubernetes version and cluster configuration when upgrading.
|
||||
|
||||
When rolling back to a prior Kubernetes version, the [upgrade strategy options]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/upgrading-kubernetes/#configuring-the-upgrade-strategy) are ignored. Worker nodes are not cordoned or drained before being reverted to the older Kubernetes version, so that an unhealthy cluster can be more quickly restored to a healthy state.
|
||||
|
||||
> **Prerequisite:** To restore snapshots from S3, the cluster needs to be configured to [take recurring snapshots on S3.]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/backing-up-etcd/#configuring-recurring-snapshots)
|
||||
|
||||
1. In the **Global** view, navigate to the cluster that you want to restore from a snapshots.
|
||||
|
||||
2. Click the **⋮ > Restore Snapshot**.
|
||||
|
||||
3. Select the snapshot that you want to use for restoring your cluster from the dropdown of available snapshots.
|
||||
|
||||
4. In the **Restoration Type** field, choose one of the restore options described above.
|
||||
|
||||
5. Click **Save**.
|
||||
|
||||
**Result:** The cluster will go into `updating` state and the process of restoring the `etcd` nodes from the snapshot will start. The cluster is restored when it returns to an `active` state.
|
||||
|
||||
</TabItem>
|
||||
<TabItem value="Rancher before v2.4.0">
|
||||
|
||||
> **Prerequisites:**
|
||||
>
|
||||
> - Make sure your etcd nodes are healthy. If you are restoring a cluster with unavailable etcd nodes, it's recommended that all etcd nodes are removed from Rancher before attempting to restore. For clusters in which Rancher used node pools to provision [nodes in an infrastructure provider]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-provisioning/rke-clusters/node-pools/), new etcd nodes will automatically be created. For [custom clusters]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-provisioning/rke-clusters/custom-nodes/), please ensure that you add new etcd nodes to the cluster.
|
||||
> - To restore snapshots from S3, the cluster needs to be configured to [take recurring snapshots on S3.]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/backing-up-etcd/#configuring-recurring-snapshots)
|
||||
|
||||
1. In the **Global** view, navigate to the cluster that you want to restore from a snapshot.
|
||||
|
||||
2. Click the **⋮ > Restore Snapshot**.
|
||||
|
||||
3. Select the snapshot that you want to use for restoring your cluster from the dropdown of available snapshots.
|
||||
|
||||
4. Click **Save**.
|
||||
|
||||
**Result:** The cluster will go into `updating` state and the process of restoring the `etcd` nodes from the snapshot will start. The cluster is restored when it returns to an `active` state.
|
||||
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
|
||||
## Recovering etcd without a Snapshot
|
||||
|
||||
If the group of etcd nodes loses quorum, the Kubernetes cluster will report a failure because no operations, e.g. deploying workloads, can be executed in the Kubernetes cluster. The cluster should have three etcd nodes to prevent a loss of quorum. If you want to recover your set of etcd nodes, follow these instructions:
|
||||
|
||||
1. Keep only one etcd node in the cluster by removing all other etcd nodes.
|
||||
|
||||
2. On the single remaining etcd node, run the following command:
|
||||
|
||||
```
|
||||
$ docker run --rm -v /var/run/docker.sock:/var/run/docker.sock assaflavie/runlike etcd
|
||||
```
|
||||
|
||||
This command outputs the running command for etcd, save this command to use later.
|
||||
|
||||
3. Stop the etcd container that you launched in the previous step and rename it to `etcd-old`.
|
||||
|
||||
```
|
||||
$ docker stop etcd
|
||||
$ docker rename etcd etcd-old
|
||||
```
|
||||
|
||||
4. Take the saved command from Step 2 and revise it:
|
||||
|
||||
- If you originally had more than 1 etcd node, then you need to change `--initial-cluster` to only contain the node that remains.
|
||||
- Add `--force-new-cluster` to the end of the command.
|
||||
|
||||
5. Run the revised command.
|
||||
|
||||
6. After the single nodes is up and running, Rancher recommends adding additional etcd nodes to your cluster. If you have a [custom cluster]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-provisioning/rke-clusters/custom-nodes) and you want to reuse an old node, you are required to [clean up the nodes]({{<baseurl>}}/rancher/v2.0-v2.4/en/faq/cleaning-cluster-nodes/) before attempting to add them back into a cluster.
|
||||
|
||||
# Enabling Snapshot Features for Clusters Created Before Rancher v2.2.0
|
||||
|
||||
If you have any Rancher launched Kubernetes clusters that were created before v2.2.0, after upgrading Rancher, you must [edit the cluster]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/editing-clusters/) and _save_ it, in order to enable the updated snapshot features. Even if you were already creating snapshots before v2.2.0, you must do this step as the older snapshots will not be available to use to [back up and restore etcd through the UI]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/restoring-etcd/).
|
||||
@@ -0,0 +1,156 @@
|
||||
---
|
||||
title: CIS Scans
|
||||
weight: 18
|
||||
aliases:
|
||||
- /rancher/v2.0-v2.4/en/cis-scans/legacy
|
||||
- /rancher/v2.0-v2.4/en/cis-scans
|
||||
- /rancher/v2.x/en/cis-scans/v2.4/
|
||||
---
|
||||
|
||||
_Available as of v2.4.0_
|
||||
|
||||
- [Prerequisites](#prerequisites)
|
||||
- [Running a scan](#running-a-scan)
|
||||
- [Scheduling recurring scans](#scheduling-recurring-scans)
|
||||
- [Skipping tests](#skipping-tests)
|
||||
- [Setting alerts](#setting-alerts)
|
||||
- [Deleting a report](#deleting-a-report)
|
||||
- [Downloading a report](#downloading-a-report)
|
||||
- [List of skipped and not applicable tests](#list-of-skipped-and-not-applicable-tests)
|
||||
|
||||
# Prerequisites
|
||||
|
||||
To run security scans on a cluster and access the generated reports, you must be an [Administrator]({{<baseurl>}}/rancher/v2.0-v2.4/en/admin-settings/rbac/global-permissions/) or [Cluster Owner.]({{<baseurl>}}/rancher/v2.0-v2.4/en/admin-settings/rbac/cluster-project-roles/)
|
||||
|
||||
Rancher can only run security scans on clusters that were created with RKE, which includes custom clusters and clusters that Rancher created in an infrastructure provider such as Amazon EC2 or GCE. Imported clusters and clusters in hosted Kubernetes providers can't be scanned by Rancher.
|
||||
|
||||
The security scan cannot run in a cluster that has Windows nodes.
|
||||
|
||||
You will only be able to see the CIS scan reports for clusters that you have access to.
|
||||
|
||||
# Running a Scan
|
||||
|
||||
1. From the cluster view in Rancher, click **Tools > CIS Scans.**
|
||||
1. Click **Run Scan.**
|
||||
1. Choose a CIS scan profile.
|
||||
|
||||
**Result:** A report is generated and displayed in the **CIS Scans** page. To see details of the report, click the report's name.
|
||||
|
||||
# Scheduling Recurring Scans
|
||||
|
||||
Recurring scans can be scheduled to run on any RKE Kubernetes cluster.
|
||||
|
||||
To enable recurring scans, edit the advanced options in the cluster configuration during cluster creation or after the cluster has been created.
|
||||
|
||||
To schedule scans for an existing cluster:
|
||||
|
||||
1. Go to the cluster view in Rancher.
|
||||
1. Click **Tools > CIS Scans.**
|
||||
1. Click **Add Schedule.** This takes you to the section of the cluster editing page that is applicable to configuring a schedule for CIS scans. (This section can also be reached by going to the cluster view, clicking **⋮ > Edit,** and going to the **Advanced Options.**)
|
||||
1. In the **CIS Scan Enabled** field, click **Yes.**
|
||||
1. In the **CIS Scan Profile** field, choose a **Permissive** or **Hardened** profile. The corresponding CIS Benchmark version is included in the profile name. Note: Any skipped tests [defined in a separate ConfigMap](#skipping-tests) will be skipped regardless of whether a **Permissive** or **Hardened** profile is selected. When selecting the the permissive profile, you should see which tests were skipped by Rancher (tests that are skipped by default for RKE clusters) and which tests were skipped by a Rancher user. In the hardened test profile, the only skipped tests will be skipped by users.
|
||||
1. In the **CIS Scan Interval (cron)** job, enter a [cron expression](https://en.wikipedia.org/wiki/Cron#CRON_expression) to define how often the cluster will be scanned.
|
||||
1. In the **CIS Scan Report Retention** field, enter the number of past reports that should be kept.
|
||||
|
||||
**Result:** The security scan will run and generate reports at the scheduled intervals.
|
||||
|
||||
The test schedule can be configured in the `cluster.yml`:
|
||||
|
||||
```yaml
|
||||
scheduled_cluster_scan:
|
||||
enabled: true
|
||||
scan_config:
|
||||
cis_scan_config:
|
||||
override_benchmark_version: rke-cis-1.4
|
||||
profile: permissive
|
||||
schedule_config:
|
||||
cron_schedule: 0 0 * * *
|
||||
retention: 24
|
||||
```
|
||||
|
||||
|
||||
# Skipping Tests
|
||||
|
||||
You can define a set of tests that will be skipped by the CIS scan when the next report is generated.
|
||||
|
||||
These tests will be skipped for subsequent CIS scans, including both manually triggered and scheduled scans, and the tests will be skipped with any profile.
|
||||
|
||||
The skipped tests will be listed alongside the test profile name in the cluster configuration options when a test profile is selected for a recurring cluster scan. The skipped tests will also be shown every time a scan is triggered manually from the Rancher UI by clicking **Run Scan.** The display of skipped tests allows you to know ahead of time which tests will be run in each scan.
|
||||
|
||||
To skip tests, you will need to define them in a Kubernetes ConfigMap resource. Each skipped CIS scan test is listed in the ConfigMap alongside the version of the CIS benchmark that the test belongs to.
|
||||
|
||||
To skip tests by editing a ConfigMap resource,
|
||||
|
||||
1. Create a `security-scan` namespace.
|
||||
1. Create a ConfigMap named `security-scan-cfg`.
|
||||
1. Enter the skip information under the key `config.json` in the following format:
|
||||
|
||||
```json
|
||||
{
|
||||
"skip": {
|
||||
"rke-cis-1.4": [
|
||||
"1.1.1",
|
||||
"1.2.2"
|
||||
]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
In the example above, the CIS benchmark version is specified alongside the tests to be skipped for that version.
|
||||
|
||||
**Result:** These tests will be skipped on subsequent scans that use the defined CIS Benchmark version.
|
||||
|
||||
# Setting Alerts
|
||||
|
||||
Rancher provides a set of alerts for cluster scans. which are not configured to have notifiers by default:
|
||||
|
||||
- A manual cluster scan was completed
|
||||
- A manual cluster scan has failures
|
||||
- A scheduled cluster scan was completed
|
||||
- A scheduled cluster scan has failures
|
||||
|
||||
> **Prerequisite:** You need to configure a [notifier]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/tools/notifiers/) before configuring, sending, or receiving alerts.
|
||||
|
||||
To activate an existing alert for a CIS scan result,
|
||||
|
||||
1. From the cluster view in Rancher, click **Tools > Alerts.**
|
||||
1. Go to the section called **A set of alerts for cluster scans.**
|
||||
1. Go to the alert you want to activate and click **⋮ > Activate.**
|
||||
1. Go to the alert rule group **A set of alerts for cluster scans** and click **⋮ > Edit.**
|
||||
1. Scroll down to the **Alert** section. In the **To** field, select the notifier that you would like to use for sending alert notifications.
|
||||
1. Optional: To limit the frequency of the notifications, click on **Show advanced options** and configure the time interval of the alerts.
|
||||
1. Click **Save.**
|
||||
|
||||
**Result:** The notifications will be triggered when the a scan is run on a cluster and the active alerts have satisfied conditions.
|
||||
|
||||
To create a new alert,
|
||||
|
||||
1. Go to the cluster view and click **Tools > CIS Scans.**
|
||||
1. Click **Add Alert.**
|
||||
1. Fill out the form.
|
||||
1. Enter a name for the alert.
|
||||
1. In the **Is** field, set the alert to be triggered when a scan is completed or when a scan has a failure.
|
||||
1. In the **Send a** field, set the alert as a **Critical,** **Warning,** or **Info** alert level.
|
||||
1. Choose a [notifier]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/tools/notifiers/) for the alert.
|
||||
|
||||
**Result:** The alert is created and activated. The notifications will be triggered when the a scan is run on a cluster and the active alerts have satisfied conditions.
|
||||
|
||||
For more information about alerts, refer to [this page.]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/tools/alerts/)
|
||||
|
||||
# Deleting a Report
|
||||
|
||||
1. From the cluster view in Rancher, click **Tools > CIS Scans.**
|
||||
1. Go to the report that should be deleted.
|
||||
1. Click the **⋮ > Delete.**
|
||||
1. Click **Delete.**
|
||||
|
||||
# Downloading a Report
|
||||
|
||||
1. From the cluster view in Rancher, click **Tools > CIS Scans.**
|
||||
1. Go to the report that you want to download. Click **⋮ > Download.**
|
||||
|
||||
**Result:** The report is downloaded in CSV format.
|
||||
|
||||
# List of Skipped and Not Applicable Tests
|
||||
|
||||
For a list of skipped and not applicable tests, refer to <a href="{{<baseurl>}}/rancher/v2.0-v2.4/en/cis-scans/legacy/skipped-tests" target="_blank">this page.</a>
|
||||
+110
@@ -0,0 +1,110 @@
|
||||
---
|
||||
title: Skipped and Not Applicable Tests
|
||||
weight: 1
|
||||
aliases:
|
||||
- /rancher/v2.0-v2.4/en/cis-scans/legacy/skipped-tests
|
||||
- /rancher/v2.0-v2.4/en/cis-scans/v2.4/skipped-tests
|
||||
- /rancher/v2.0-v2.4/en/cis-scans/skipped-tests
|
||||
- /rancher/v2.x/en/cis-scans/v2.4/skipped-tests/
|
||||
---
|
||||
|
||||
This section lists the tests that are skipped in the permissive test profile for RKE.
|
||||
|
||||
All the tests that are skipped and not applicable on this page will be counted as Not Applicable in the generated report. The skipped test count will only mention the user-defined skipped tests. This allows user-skipped tests to be distinguished from the tests that are skipped by default in the RKE permissive test profile.
|
||||
|
||||
- [CIS Benchmark v1.5](#cis-benchmark-v1-5)
|
||||
- [CIS Benchmark v1.4](#cis-benchmark-v1-4)
|
||||
|
||||
# CIS Benchmark v1.5
|
||||
|
||||
### CIS Benchmark v1.5 Skipped Tests
|
||||
|
||||
| Number | Description | Reason for Skipping |
|
||||
| ---------- | ------------- | --------- |
|
||||
| 1.1.12 | Ensure that the etcd data directory ownership is set to etcd:etcd (Scored) | A system service account is required for etcd data directory ownership. Refer to Rancher's hardening guide for more details on how to configure this ownership. |
|
||||
| 1.2.6 | Ensure that the --kubelet-certificate-authority argument is set as appropriate (Scored) | When generating serving certificates, functionality could break in conjunction with hostname overrides which are required for certain cloud providers. |
|
||||
| 1.2.16 | Ensure that the admission control plugin PodSecurityPolicy is set (Scored) | Enabling Pod Security Policy can cause applications to unexpectedly fail. |
|
||||
| 1.2.33 | Ensure that the --encryption-provider-config argument is set as appropriate (Not Scored) | Enabling encryption changes how data can be recovered as data is encrypted. |
|
||||
| 1.2.34 | Ensure that encryption providers are appropriately configured (Not Scored) | Enabling encryption changes how data can be recovered as data is encrypted. |
|
||||
| 4.2.6 | Ensure that the --protect-kernel-defaults argument is set to true (Scored) | System level configurations are required before provisioning the cluster in order for this argument to be set to true. |
|
||||
| 4.2.10 | Ensure that the--tls-cert-file and --tls-private-key-file arguments are set as appropriate (Scored) | When generating serving certificates, functionality could break in conjunction with hostname overrides which are required for certain cloud providers. |
|
||||
| 5.1.5 | Ensure that default service accounts are not actively used. (Scored) | Kubernetes provides default service accounts to be used. |
|
||||
| 5.2.2 | Minimize the admission of containers wishing to share the host process ID namespace (Scored) | Enabling Pod Security Policy can cause applications to unexpectedly fail. |
|
||||
| 5.2.3 | Minimize the admission of containers wishing to share the host IPC namespace (Scored) | Enabling Pod Security Policy can cause applications to unexpectedly fail. |
|
||||
| 5.2.4 | Minimize the admission of containers wishing to share the host network namespace (Scored) | Enabling Pod Security Policy can cause applications to unexpectedly fail. |
|
||||
| 5.2.5 | Minimize the admission of containers with allowPrivilegeEscalation (Scored) | Enabling Pod Security Policy can cause applications to unexpectedly fail. |
|
||||
| 5.3.2 | Ensure that all Namespaces have Network Policies defined (Scored) | Enabling Network Policies can prevent certain applications from communicating with each other. |
|
||||
| 5.6.4 | The default namespace should not be used (Scored) | Kubernetes provides a default namespace. |
|
||||
|
||||
### CIS Benchmark v1.5 Not Applicable Tests
|
||||
|
||||
| Number | Description | Reason for being not applicable |
|
||||
| ---------- | ------------- | --------- |
|
||||
| 1.1.1 | Ensure that the API server pod specification file permissions are set to 644 or more restrictive (Scored) | Clusters provisioned by RKE doesn't require or maintain a configuration file for kube-apiserver. All configuration is passed in as arguments at container run time. |
|
||||
| 1.1.2 | Ensure that the API server pod specification file ownership is set to root:root (Scored) | Clusters provisioned by RKE doesn't require or maintain a configuration file for kube-apiserver. All configuration is passed in as arguments at container run time. |
|
||||
| 1.1.3 | Ensure that the controller manager pod specification file permissions are set to 644 or more restrictive (Scored) | Clusters provisioned by RKE doesn't require or maintain a configuration file for controller-manager. All configuration is passed in as arguments at container run time. |
|
||||
| 1.1.4 | Ensure that the controller manager pod specification file ownership is set to root:root (Scored) | Clusters provisioned by RKE doesn't require or maintain a configuration file for controller-manager. All configuration is passed in as arguments at container run time. |
|
||||
| 1.1.5 | Ensure that the scheduler pod specification file permissions are set to 644 or more restrictive (Scored) | Clusters provisioned by RKE doesn't require or maintain a configuration file for scheduler. All configuration is passed in as arguments at container run time. |
|
||||
| 1.1.6 | Ensure that the scheduler pod specification file ownership is set to root:root (Scored) | Clusters provisioned by RKE doesn't require or maintain a configuration file for scheduler. All configuration is passed in as arguments at container run time. |
|
||||
| 1.1.7 | Ensure that the etcd pod specification file permissions are set to 644 or more restrictive (Scored) | Clusters provisioned by RKE doesn't require or maintain a configuration file for etcd. All configuration is passed in as arguments at container run time. |
|
||||
| 1.1.8 | Ensure that the etcd pod specification file ownership is set to root:root (Scored) | Clusters provisioned by RKE doesn't require or maintain a configuration file for etcd. All configuration is passed in as arguments at container run time. |
|
||||
| 1.1.13 | Ensure that the admin.conf file permissions are set to 644 or more restrictive (Scored) | Clusters provisioned by RKE does not store the kubernetes default kubeconfig credentials file on the nodes. |
|
||||
| 1.1.14 | Ensure that the admin.conf file ownership is set to root:root (Scored) | Clusters provisioned by RKE does not store the kubernetes default kubeconfig credentials file on the nodes. |
|
||||
| 1.1.15 | Ensure that the scheduler.conf file permissions are set to 644 or more restrictive (Scored) | Clusters provisioned by RKE doesn't require or maintain a configuration file for scheduler. All configuration is passed in as arguments at container run time. |
|
||||
| 1.1.16 | Ensure that the scheduler.conf file ownership is set to root:root (Scored) | Clusters provisioned by RKE doesn't require or maintain a configuration file for scheduler. All configuration is passed in as arguments at container run time. |
|
||||
| 1.1.17 | Ensure that the controller-manager.conf file permissions are set to 644 or more restrictive (Scored) | Clusters provisioned by RKE doesn't require or maintain a configuration file for controller-manager. All configuration is passed in as arguments at container run time. |
|
||||
| 1.1.18 | Ensure that the controller-manager.conf file ownership is set to root:root (Scored) | Clusters provisioned by RKE doesn't require or maintain a configuration file for controller-manager. All configuration is passed in as arguments at container run time. |
|
||||
| 1.3.6 | Ensure that the RotateKubeletServerCertificate argument is set to true (Scored) | Clusters provisioned by RKE handles certificate rotation directly through RKE. |
|
||||
| 4.1.1 | Ensure that the kubelet service file permissions are set to 644 or more restrictive (Scored) | Clusters provisioned by RKE doesn’t require or maintain a configuration file for the kubelet service. All configuration is passed in as arguments at container run time. |
|
||||
| 4.1.2 | Ensure that the kubelet service file ownership is set to root:root (Scored) | Clusters provisioned by RKE doesn’t require or maintain a configuration file for the kubelet service. All configuration is passed in as arguments at container run time. |
|
||||
| 4.1.9 | Ensure that the kubelet configuration file has permissions set to 644 or more restrictive (Scored) | Clusters provisioned by RKE doesn’t require or maintain a configuration file for the kubelet. All configuration is passed in as arguments at container run time. |
|
||||
| 4.1.10 | Ensure that the kubelet configuration file ownership is set to root:root (Scored) | Clusters provisioned by RKE doesn’t require or maintain a configuration file for the kubelet. All configuration is passed in as arguments at container run time. |
|
||||
| 4.2.12 | Ensure that the RotateKubeletServerCertificate argument is set to true (Scored) | Clusters provisioned by RKE handles certificate rotation directly through RKE. |
|
||||
|
||||
# CIS Benchmark v1.4
|
||||
|
||||
The skipped and not applicable tests for CIS Benchmark v1.4 are as follows:
|
||||
|
||||
### CIS Benchmark v1.4 Skipped Tests
|
||||
|
||||
Number | Description | Reason for Skipping
|
||||
---|---|---
|
||||
1.1.11 | "Ensure that the admission control plugin AlwaysPullImages is set (Scored)" | Enabling AlwaysPullImages can use significant bandwidth.
|
||||
1.1.21 | "Ensure that the --kubelet-certificate-authority argument is set as appropriate (Scored)" | When generating serving certificates, functionality could break in conjunction with hostname overrides which are required for certain cloud providers.
|
||||
1.1.24 | "Ensure that the admission control plugin PodSecurityPolicy is set (Scored)" | Enabling Pod Security Policy can cause applications to unexpectedly fail.
|
||||
1.1.34 | "Ensure that the --encryption-provider-config argument is set as appropriate (Scored)" | Enabling encryption changes how data can be recovered as data is encrypted.
|
||||
1.1.35 | "Ensure that the encryption provider is set to aescbc (Scored)" | Enabling encryption changes how data can be recovered as data is encrypted.
|
||||
1.1.36 | "Ensure that the admission control plugin EventRateLimit is set (Scored)" | EventRateLimit needs to be tuned depending on the cluster.
|
||||
1.2.2 | "Ensure that the --address argument is set to 127.0.0.1 (Scored)" | Adding this argument prevents Rancher's monitoring tool to collect metrics on the scheduler.
|
||||
1.3.7 | "Ensure that the --address argument is set to 127.0.0.1 (Scored)" | Adding this argument prevents Rancher's monitoring tool to collect metrics on the controller manager.
|
||||
1.4.12 | "Ensure that the etcd data directory ownership is set to etcd:etcd (Scored)" | A system service account is required for etcd data directory ownership. Refer to Rancher's hardening guide for more details on how to configure this ownership.
|
||||
1.7.2 | "Do not admit containers wishing to share the host process ID namespace (Scored)" | Enabling Pod Security Policy can cause applications to unexpectedly fail.
|
||||
1.7.3 | "Do not admit containers wishing to share the host IPC namespace (Scored)" | Enabling Pod Security Policy can cause applications to unexpectedly fail.
|
||||
1.7.4 | "Do not admit containers wishing to share the host network namespace (Scored)" | Enabling Pod Security Policy can cause applications to unexpectedly fail.
|
||||
1.7.5 | " Do not admit containers with allowPrivilegeEscalation (Scored)" | Enabling Pod Security Policy can cause applications to unexpectedly fail.
|
||||
2.1.6 | "Ensure that the --protect-kernel-defaults argument is set to true (Scored)" | System level configurations are required before provisioning the cluster in order for this argument to be set to true.
|
||||
2.1.10 | "Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate (Scored)" | When generating serving certificates, functionality could break in conjunction with hostname overrides which are required for certain cloud providers.
|
||||
|
||||
### CIS Benchmark v1.4 Not Applicable Tests
|
||||
|
||||
Number | Description | Reason for being not applicable
|
||||
---|---|---
|
||||
1.1.9 | "Ensure that the --repair-malformed-updates argument is set to false (Scored)" | The argument --repair-malformed-updates has been removed as of Kubernetes version 1.14
|
||||
1.3.6 | "Ensure that the RotateKubeletServerCertificate argument is set to true" | Cluster provisioned by RKE handles certificate rotation directly through RKE.
|
||||
1.4.1 | "Ensure that the API server pod specification file permissions are set to 644 or more restrictive (Scored)" | Cluster provisioned by RKE doesn't require or maintain a configuration file for kube-apiserver.
|
||||
1.4.2 | "Ensure that the API server pod specification file ownership is set to root:root (Scored)" | Cluster provisioned by RKE doesn't require or maintain a configuration file for kube-apiserver.
|
||||
1.4.3 | "Ensure that the controller manager pod specification file permissions are set to 644 or more restrictive (Scored)" | Cluster provisioned by RKE doesn't require or maintain a configuration file for controller-manager.
|
||||
1.4.4 | "Ensure that the controller manager pod specification file ownership is set to root:root (Scored)" | Cluster provisioned by RKE doesn't require or maintain a configuration file for controller-manager.
|
||||
1.4.5 | "Ensure that the scheduler pod specification file permissions are set to 644 or more restrictive (Scored)" | Cluster provisioned by RKE doesn't require or maintain a configuration file for scheduler.
|
||||
1.4.6 | "Ensure that the scheduler pod specification file ownership is set to root:root (Scored)" | Cluster provisioned by RKE doesn't require or maintain a configuration file for scheduler.
|
||||
1.4.7 | "Ensure that the etcd pod specification file permissions are set to 644 or more restrictive (Scored)" | Cluster provisioned by RKE doesn't require or maintain a configuration file for etcd.
|
||||
1.4.8 | "Ensure that the etcd pod specification file ownership is set to root:root (Scored)" | Cluster provisioned by RKE doesn't require or maintain a configuration file for etcd.
|
||||
1.4.13 | "Ensure that the admin.conf file permissions are set to 644 or more restrictive (Scored)" | Cluster provisioned by RKE does not store the kubernetes default kubeconfig credentials file on the nodes.
|
||||
1.4.14 | "Ensure that the admin.conf file ownership is set to root:root (Scored)" | Cluster provisioned by RKE does not store the kubernetes default kubeconfig credentials file on the nodes.
|
||||
2.1.8 | "Ensure that the --hostname-override argument is not set (Scored)" | Clusters provisioned by RKE clusters and most cloud providers require hostnames.
|
||||
2.1.12 | "Ensure that the --rotate-certificates argument is not set to false (Scored)" | Cluster provisioned by RKE handles certificate rotation directly through RKE.
|
||||
2.1.13 | "Ensure that the RotateKubeletServerCertificate argument is set to true (Scored)" | Cluster provisioned by RKE handles certificate rotation directly through RKE.
|
||||
2.2.3 | "Ensure that the kubelet service file permissions are set to 644 or more restrictive (Scored)" | Cluster provisioned by RKE doesn’t require or maintain a configuration file for the kubelet service.
|
||||
2.2.4 | "Ensure that the kubelet service file ownership is set to root:root (Scored)" | Cluster provisioned by RKE doesn’t require or maintain a configuration file for the kubelet service.
|
||||
2.2.9 | "Ensure that the kubelet configuration file ownership is set to root:root (Scored)" | RKE doesn’t require or maintain a configuration file for the kubelet.
|
||||
2.2.10 | "Ensure that the kubelet configuration file has permissions set to 644 or more restrictive (Scored)" | RKE doesn’t require or maintain a configuration file for the kubelet.
|
||||
@@ -0,0 +1,347 @@
|
||||
---
|
||||
title: Cluster Alerts
|
||||
shortTitle: Alerts
|
||||
weight: 2
|
||||
aliases:
|
||||
- /rancher/v2.0-v2.4/en/cluster-admin/tools/alerts
|
||||
- /rancher/v2.0-v2.4/en/monitoring-alerting/legacy/alerts/cluster-alerts
|
||||
- /rancher/v2.0-v2.4/en/monitoring-alerting/v2.0.x-v2.4.x/cluster-alerts
|
||||
- /rancher/v2.x/en/monitoring-alerting/v2.0.x-v2.4.x/cluster-alerts/
|
||||
---
|
||||
|
||||
To keep your clusters and applications healthy and driving your organizational productivity forward, you need to stay informed of events occurring in your clusters and projects, both planned and unplanned. When an event occurs, your alert is triggered, and you are sent a notification. You can then, if necessary, follow up with corrective actions.
|
||||
|
||||
This section covers the following topics:
|
||||
|
||||
- [About Alerts](#about-alerts)
|
||||
- [Alert Event Examples](#alert-event-examples)
|
||||
- [Alerts Triggered by Prometheus Queries](#alerts-triggered-by-prometheus-queries)
|
||||
- [Urgency Levels](#urgency-levels)
|
||||
- [Scope of Alerts](#scope-of-alerts)
|
||||
- [Managing Cluster Alerts](#managing-cluster-alerts)
|
||||
- [Adding Cluster Alerts](#adding-cluster-alerts)
|
||||
- [Cluster Alert Configuration](#cluster-alert-configuration)
|
||||
- [System Service Alerts](#system-service-alerts)
|
||||
- [Resource Event Alerts](#resource-event-alerts)
|
||||
- [Node Alerts](#node-alerts)
|
||||
- [Node Selector Alerts](#node-selector-alerts)
|
||||
- [CIS Scan Alerts](#cis-scan-alerts)
|
||||
- [Metric Expression Alerts](#metric-expression-alerts)
|
||||
|
||||
# About Alerts
|
||||
|
||||
Notifiers and alerts are built on top of the [Prometheus Alertmanager](https://prometheus.io/docs/alerting/alertmanager/). Leveraging these tools, Rancher can notify [cluster owners]({{<baseurl>}}/rancher/v2.0-v2.4/en/admin-settings/rbac/cluster-project-roles/#cluster-roles) and [project owners]({{<baseurl>}}/rancher/v2.0-v2.4/en/admin-settings/rbac/cluster-project-roles/#project-roles) of events they need to address.
|
||||
|
||||
Before you can receive alerts, you must configure one or more notifier in Rancher.
|
||||
|
||||
When you create a cluster, some alert rules are predefined. You can receive these alerts if you configure a [notifier]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/tools/notifiers) for them.
|
||||
|
||||
For details about what triggers the predefined alerts, refer to the [documentation on default alerts.]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/tools/alerts/default-alerts)
|
||||
|
||||
### Alert Event Examples
|
||||
|
||||
Some examples of alert events are:
|
||||
|
||||
- A Kubernetes master component entering an unhealthy state.
|
||||
- A node or workload error occurring.
|
||||
- A scheduled deployment taking place as planned.
|
||||
- A node's hardware resources becoming overstressed.
|
||||
|
||||
### Alerts Triggered by Prometheus Queries
|
||||
|
||||
When you edit an alert rule, you will have the opportunity to configure the alert to be triggered based on a Prometheus expression. For examples of expressions, refer to [this page.]({{<baseurl>}}/rancher/v2.0-v2.4/en/monitoring-alerting/v2.0.x-v2.4.x/cluster-monitoring/expression/)
|
||||
|
||||
Monitoring must be [enabled]({{<baseurl>}}/rancher/v2.0-v2.4/en/monitoring-alerting/v2.0.x-v2.4.x/cluster-monitoring/) before you can trigger alerts with custom Prometheus queries or expressions.
|
||||
|
||||
### Urgency Levels
|
||||
|
||||
You can set an urgency level for each alert. This urgency appears in the notification you receive, helping you to prioritize your response actions. For example, if you have an alert configured to inform you of a routine deployment, no action is required. These alerts can be assigned a low priority level. However, if a deployment fails, it can critically impact your organization, and you need to react quickly. Assign these alerts a high priority level.
|
||||
|
||||
### Scope of Alerts
|
||||
|
||||
The scope for alerts can be set at either the cluster level or [project level]({{<baseurl>}}/rancher/v2.0-v2.4/en/project-admin/tools/alerts/).
|
||||
|
||||
At the cluster level, Rancher monitors components in your Kubernetes cluster, and sends you alerts related to:
|
||||
|
||||
- The state of your nodes.
|
||||
- The system services that manage your Kubernetes cluster.
|
||||
- The resource events from specific system services.
|
||||
- The Prometheus expression cross the thresholds
|
||||
|
||||
### Managing Cluster Alerts
|
||||
|
||||
After you set up cluster alerts, you can manage each alert object. To manage alerts, browse to the cluster containing the alerts, and then select **Tools > Alerts** that you want to manage. You can:
|
||||
|
||||
- Deactivate/Reactive alerts
|
||||
- Edit alert settings
|
||||
- Delete unnecessary alerts
|
||||
- Mute firing alerts
|
||||
- Unmute muted alerts
|
||||
|
||||
# Adding Cluster Alerts
|
||||
|
||||
As a [cluster owner]({{<baseurl>}}/rancher/v2.0-v2.4/en/admin-settings/rbac/cluster-project-roles/#cluster-roles), you can configure Rancher to send you alerts for cluster events.
|
||||
|
||||
>**Prerequisite:** Before you can receive cluster alerts, you must [add a notifier]({{<baseurl>}}/rancher/v2.0-v2.4/en/monitoring-alerting/legacy/notifiers/).
|
||||
|
||||
1. From the **Global** view, navigate to the cluster that you want to configure cluster alerts for. Select **Tools > Alerts**. Then click **Add Alert Group**.
|
||||
1. Enter a **Name** for the alert that describes its purpose, you could group alert rules for the different purpose.
|
||||
1. Based on the type of alert you want to create, refer to the [cluster alert configuration section.](#cluster-alert-configuration)
|
||||
1. Continue adding more **Alert Rule** to the group.
|
||||
1. Finally, choose the [notifiers]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/tools/notifiers/) to send the alerts to.
|
||||
|
||||
- You can set up multiple notifiers.
|
||||
- You can change notifier recipients on the fly.
|
||||
1. Click **Create.**
|
||||
|
||||
**Result:** Your alert is configured. A notification is sent when the alert is triggered.
|
||||
|
||||
|
||||
# Cluster Alert Configuration
|
||||
|
||||
- [System Service Alerts](#system-service-alerts)
|
||||
- [Resource Event Alerts](#resource-event-alerts)
|
||||
- [Node Alerts](#node-alerts)
|
||||
- [Node Selector Alerts](#node-selector-alerts)
|
||||
- [CIS Scan Alerts](#cis-scan-alerts)
|
||||
- [Metric Expression Alerts](#metric-expression-alerts)
|
||||
|
||||
# System Service Alerts
|
||||
|
||||
This alert type monitor for events that affect one of the Kubernetes master components, regardless of the node it occurs on.
|
||||
|
||||
Each of the below sections corresponds to a part of the alert rule configuration section in the Rancher UI.
|
||||
|
||||
### When a
|
||||
|
||||
Select the **System Services** option, and then select an option from the dropdown:
|
||||
|
||||
- [controller-manager](https://kubernetes.io/docs/concepts/overview/components/#kube-controller-manager)
|
||||
- [etcd](https://kubernetes.io/docs/concepts/overview/components/#etcd)
|
||||
- [scheduler](https://kubernetes.io/docs/concepts/overview/components/#kube-scheduler)
|
||||
|
||||
### Is
|
||||
|
||||
The alert will be triggered when the selected Kubernetes master component is unhealthy.
|
||||
|
||||
### Send a
|
||||
|
||||
Select the urgency level of the alert. The options are:
|
||||
|
||||
- **Critical**: Most urgent
|
||||
- **Warning**: Normal urgency
|
||||
- **Info**: Least urgent
|
||||
|
||||
Select the urgency level based on the importance of the service and how many nodes fill the role within your cluster. For example, if you're making an alert for the `etcd` service, select **Critical**. If you're making an alert for redundant schedulers, **Warning** is more appropriate.
|
||||
|
||||
### Advanced Options
|
||||
|
||||
By default, the below options will apply to all alert rules within the group. You can disable these advanced options when configuring a specific rule.
|
||||
|
||||
- **Group Wait Time**: How long to wait to buffer alerts of the same group before sending initially, default to 30 seconds.
|
||||
- **Group Interval Time**: How long to wait before sending an alert that has been added to a group which contains already fired alerts, default to 30 seconds.
|
||||
- **Repeat Wait Time**: How long to wait before re-sending a given alert that has already been sent, default to 1 hour.
|
||||
|
||||
# Resource Event Alerts
|
||||
|
||||
This alert type monitors for specific events that are thrown from a resource type.
|
||||
|
||||
Each of the below sections corresponds to a part of the alert rule configuration section in the Rancher UI.
|
||||
|
||||
### When a
|
||||
|
||||
Choose the type of resource event that triggers an alert. The options are:
|
||||
|
||||
- **Normal**: triggers an alert when any standard resource event occurs.
|
||||
- **Warning**: triggers an alert when unexpected resource events occur.
|
||||
|
||||
Select a resource type from the **Choose a Resource** drop-down that you want to trigger an alert.
|
||||
|
||||
- [DaemonSet](https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/)
|
||||
- [Deployment](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/)
|
||||
- [Node](https://kubernetes.io/docs/concepts/architecture/nodes/)
|
||||
- [Pod](https://kubernetes.io/docs/concepts/workloads/pods/pod/)
|
||||
- [StatefulSet](https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/)
|
||||
|
||||
### Send a
|
||||
|
||||
Select the urgency level of the alert.
|
||||
|
||||
- **Critical**: Most urgent
|
||||
- **Warning**: Normal urgency
|
||||
- **Info**: Least urgent
|
||||
|
||||
Select the urgency level of the alert by considering factors such as how often the event occurs or its importance. For example:
|
||||
|
||||
- If you set a normal alert for pods, you're likely to receive alerts often, and individual pods usually self-heal, so select an urgency of **Info**.
|
||||
- If you set a warning alert for StatefulSets, it's very likely to impact operations, so select an urgency of **Critical**.
|
||||
|
||||
### Advanced Options
|
||||
|
||||
By default, the below options will apply to all alert rules within the group. You can disable these advanced options when configuring a specific rule.
|
||||
|
||||
- **Group Wait Time**: How long to wait to buffer alerts of the same group before sending initially, default to 30 seconds.
|
||||
- **Group Interval Time**: How long to wait before sending an alert that has been added to a group which contains already fired alerts, default to 30 seconds.
|
||||
- **Repeat Wait Time**: How long to wait before re-sending a given alert that has already been sent, default to 1 hour.
|
||||
|
||||
# Node Alerts
|
||||
|
||||
This alert type monitors for events that occur on a specific node.
|
||||
|
||||
Each of the below sections corresponds to a part of the alert rule configuration section in the Rancher UI.
|
||||
|
||||
### When a
|
||||
|
||||
Select the **Node** option, and then make a selection from the **Choose a Node** drop-down.
|
||||
|
||||
### Is
|
||||
|
||||
Choose an event to trigger the alert.
|
||||
|
||||
- **Not Ready**: Sends you an alert when the node is unresponsive.
|
||||
- **CPU usage over**: Sends you an alert when the node raises above an entered percentage of its processing allocation.
|
||||
- **Mem usage over**: Sends you an alert when the node raises above an entered percentage of its memory allocation.
|
||||
|
||||
### Send a
|
||||
|
||||
Select the urgency level of the alert.
|
||||
|
||||
- **Critical**: Most urgent
|
||||
- **Warning**: Normal urgency
|
||||
- **Info**: Least urgent
|
||||
|
||||
Select the urgency level of the alert based on its impact on operations. For example, an alert triggered when a node's CPU raises above 60% deems an urgency of **Info**, but a node that is **Not Ready** deems an urgency of **Critical**.
|
||||
|
||||
### Advanced Options
|
||||
|
||||
By default, the below options will apply to all alert rules within the group. You can disable these advanced options when configuring a specific rule.
|
||||
|
||||
- **Group Wait Time**: How long to wait to buffer alerts of the same group before sending initially, default to 30 seconds.
|
||||
- **Group Interval Time**: How long to wait before sending an alert that has been added to a group which contains already fired alerts, default to 30 seconds.
|
||||
- **Repeat Wait Time**: How long to wait before re-sending a given alert that has already been sent, default to 1 hour.
|
||||
|
||||
# Node Selector Alerts
|
||||
|
||||
This alert type monitors for events that occur on any node on marked with a label. For more information, see the Kubernetes documentation for [Labels](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/).
|
||||
|
||||
Each of the below sections corresponds to a part of the alert rule configuration section in the Rancher UI.
|
||||
|
||||
### When a
|
||||
|
||||
Select the **Node Selector** option, and then click **Add Selector** to enter a key value pair for a label. This label should be applied to one or more of your nodes. Add as many selectors as you'd like.
|
||||
|
||||
### Is
|
||||
|
||||
Choose an event to trigger the alert.
|
||||
|
||||
- **Not Ready**: Sends you an alert when selected nodes are unresponsive.
|
||||
- **CPU usage over**: Sends you an alert when selected nodes raise above an entered percentage of processing allocation.
|
||||
- **Mem usage over**: Sends you an alert when selected nodes raise above an entered percentage of memory allocation.
|
||||
|
||||
### Send a
|
||||
|
||||
Select the urgency level of the alert.
|
||||
|
||||
- **Critical**: Most urgent
|
||||
- **Warning**: Normal urgency
|
||||
- **Info**: Least urgent
|
||||
|
||||
Select the urgency level of the alert based on its impact on operations. For example, an alert triggered when a node's CPU raises above 60% deems an urgency of **Info**, but a node that is **Not Ready** deems an urgency of **Critical**.
|
||||
|
||||
### Advanced Options
|
||||
|
||||
By default, the below options will apply to all alert rules within the group. You can disable these advanced options when configuring a specific rule.
|
||||
|
||||
- **Group Wait Time**: How long to wait to buffer alerts of the same group before sending initially, default to 30 seconds.
|
||||
- **Group Interval Time**: How long to wait before sending an alert that has been added to a group which contains already fired alerts, default to 30 seconds.
|
||||
- **Repeat Wait Time**: How long to wait before re-sending a given alert that has already been sent, default to 1 hour.
|
||||
|
||||
# CIS Scan Alerts
|
||||
_Available as of v2.4.0_
|
||||
|
||||
This alert type is triggered based on the results of a CIS scan.
|
||||
|
||||
Each of the below sections corresponds to a part of the alert rule configuration section in the Rancher UI.
|
||||
|
||||
### When a
|
||||
|
||||
Select **CIS Scan.**
|
||||
|
||||
### Is
|
||||
|
||||
Choose an event to trigger the alert:
|
||||
|
||||
- Completed Scan
|
||||
- Has Failure
|
||||
|
||||
### Send a
|
||||
|
||||
Select the urgency level of the alert.
|
||||
|
||||
- **Critical**: Most urgent
|
||||
- **Warning**: Normal urgency
|
||||
- **Info**: Least urgent
|
||||
|
||||
Select the urgency level of the alert based on its impact on operations. For example, an alert triggered when a node's CPU raises above 60% deems an urgency of **Info**, but a node that is **Not Ready** deems an urgency of **Critical**.
|
||||
|
||||
### Advanced Options
|
||||
|
||||
By default, the below options will apply to all alert rules within the group. You can disable these advanced options when configuring a specific rule.
|
||||
|
||||
- **Group Wait Time**: How long to wait to buffer alerts of the same group before sending initially, default to 30 seconds.
|
||||
- **Group Interval Time**: How long to wait before sending an alert that has been added to a group which contains already fired alerts, default to 30 seconds.
|
||||
- **Repeat Wait Time**: How long to wait before re-sending a given alert that has already been sent, default to 1 hour.
|
||||
|
||||
# Metric Expression Alerts
|
||||
|
||||
This alert type monitors for the overload from Prometheus expression querying, it would be available after you enable monitoring.
|
||||
|
||||
Each of the below sections corresponds to a part of the alert rule configuration section in the Rancher UI.
|
||||
|
||||
### When a
|
||||
|
||||
Input or select an **Expression**, the dropdown shows the original metrics from Prometheus, including:
|
||||
|
||||
- [**Node**](https://github.com/prometheus/node_exporter)
|
||||
- [**Container**](https://github.com/google/cadvisor)
|
||||
- [**ETCD**](https://etcd.io/docs/v3.4.0/op-guide/monitoring/)
|
||||
- [**Kubernetes Components**](https://github.com/kubernetes/metrics)
|
||||
- [**Kubernetes Resources**](https://github.com/kubernetes/kube-state-metrics)
|
||||
- [**Fluentd**](https://docs.fluentd.org/v1.0/articles/monitoring-prometheus) (supported by [Logging]({{<baseurl>}}/rancher/v2.0-v2.4//en/cluster-admin/tools/logging))
|
||||
- [**Cluster Level Grafana**](http://docs.grafana.org/administration/metrics/)
|
||||
- **Cluster Level Prometheus**
|
||||
|
||||
### Is
|
||||
|
||||
Choose a comparison:
|
||||
|
||||
- **Equal**: Trigger alert when expression value equal to the threshold.
|
||||
- **Not Equal**: Trigger alert when expression value not equal to the threshold.
|
||||
- **Greater Than**: Trigger alert when expression value greater than to threshold.
|
||||
- **Less Than**: Trigger alert when expression value equal or less than the threshold.
|
||||
- **Greater or Equal**: Trigger alert when expression value greater to equal to the threshold.
|
||||
- **Less or Equal**: Trigger alert when expression value less or equal to the threshold.
|
||||
|
||||
If applicable, choose a comparison value or a threshold for the alert to be triggered.
|
||||
|
||||
### For
|
||||
|
||||
Select a duration for a trigger alert when the expression value crosses the threshold longer than the configured duration.
|
||||
|
||||
### Send a
|
||||
|
||||
Select the urgency level of the alert.
|
||||
|
||||
- **Critical**: Most urgent
|
||||
- **Warning**: Normal urgency
|
||||
- **Info**: Least urgent
|
||||
|
||||
Select the urgency level of the alert based on its impact on operations. For example, an alert triggered when a node's load expression ```sum(node_load5) / count(node_cpu_seconds_total{mode="system"})``` raises above 0.6 deems an urgency of **Info**, but 1 deems an urgency of **Critical**.
|
||||
|
||||
### Advanced Options
|
||||
|
||||
By default, the below options will apply to all alert rules within the group. You can disable these advanced options when configuring a specific rule.
|
||||
|
||||
- **Group Wait Time**: How long to wait to buffer alerts of the same group before sending initially, default to 30 seconds.
|
||||
- **Group Interval Time**: How long to wait before sending an alert that has been added to a group which contains already fired alerts, default to 30 seconds.
|
||||
- **Repeat Wait Time**: How long to wait before re-sending a given alert that has already been sent, default to 1 hour.
|
||||
+60
@@ -0,0 +1,60 @@
|
||||
---
|
||||
title: Default Alerts for Cluster Monitoring
|
||||
weight: 1
|
||||
aliases:
|
||||
- /rancher/v2.0-v2.4/en/cluster-admin/tools/alerts/default-alerts
|
||||
- /rancher/v2.0-v2.4/en/monitoring-alerting/legacy/alerts/cluster-alerts/default-alerts
|
||||
- /rancher/v2.0-v2.4/en/monitoring-alerting/v2.0.x-v2.4.x/cluster-alerts/default-alerts
|
||||
---
|
||||
|
||||
When you create a cluster, some alert rules are predefined. These alerts notify you about signs that the cluster could be unhealthy. You can receive these alerts if you configure a [notifier]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/tools/notifiers) for them.
|
||||
|
||||
Several of the alerts use Prometheus expressions as the metric that triggers the alert. For more information on how expressions work, you can refer to the Rancher [documentation about Prometheus expressions]({{<baseurl>}}/rancher/v2.0-v2.4/en/monitoring-alerting/legacy/monitoring/cluster-monitoring/expression/) or the Prometheus [documentation about querying metrics](https://prometheus.io/docs/prometheus/latest/querying/basics/).
|
||||
|
||||
# Alerts for etcd
|
||||
Etcd is the key-value store that contains the state of the Kubernetes cluster. Rancher provides default alerts if the built-in monitoring detects a potential problem with etcd. You don't have to enable monitoring to receive these alerts.
|
||||
|
||||
A leader is the node that handles all client requests that need cluster consensus. For more information, you can refer to this [explanation of how etcd works.](https://rancher.com/blog/2019/2019-01-29-what-is-etcd/#how-does-etcd-work)
|
||||
|
||||
The leader of the cluster can change in response to certain events. It is normal for the leader to change, but too many changes can indicate a problem with the network or a high CPU load. With longer latencies, the default etcd configuration may cause frequent heartbeat timeouts, which trigger a new leader election.
|
||||
|
||||
| Alert | Explanation |
|
||||
|-------|-------------|
|
||||
| A high number of leader changes within the etcd cluster are happening | A warning alert is triggered when the leader changes more than three times in one hour. |
|
||||
| Database usage close to the quota 500M | A warning alert is triggered when the size of etcd exceeds 500M.|
|
||||
| Etcd is unavailable | A critical alert is triggered when etcd becomes unavailable. |
|
||||
| Etcd member has no leader | A critical alert is triggered when the etcd cluster does not have a leader for at least three minutes. |
|
||||
|
||||
|
||||
# Alerts for Kubernetes Components
|
||||
Rancher provides alerts when core Kubernetes system components become unhealthy.
|
||||
|
||||
Controllers update Kubernetes resources based on changes in etcd. The [controller manager](https://kubernetes.io/docs/reference/command-line-tools-reference/kube-controller-manager/) monitors the cluster desired state through the Kubernetes API server and makes the necessary changes to the current state to reach the desired state.
|
||||
|
||||
The [scheduler](https://kubernetes.io/docs/reference/command-line-tools-reference/kube-scheduler/) service is a core component of Kubernetes. It is responsible for scheduling cluster workloads to nodes, based on various configurations, metrics, resource requirements and workload-specific requirements.
|
||||
|
||||
| Alert | Explanation |
|
||||
|-------|-------------|
|
||||
| Controller Manager is unavailable | A critical warning is triggered when the cluster’s controller-manager becomes unavailable. |
|
||||
| Scheduler is unavailable | A critical warning is triggered when the cluster’s scheduler becomes unavailable. |
|
||||
|
||||
|
||||
# Alerts for Events
|
||||
Kubernetes events are objects that provide insight into what is happening inside a cluster, such as what decisions were made by the scheduler or why some pods were evicted from the node. In the Rancher UI, from the project view, you can see events for each workload.
|
||||
|
||||
| Alert | Explanation |
|
||||
|-------|-------------|
|
||||
| Get warning deployment event | A warning alert is triggered when a warning event happens on a deployment. |
|
||||
|
||||
|
||||
# Alerts for Nodes
|
||||
Alerts can be triggered based on node metrics. Each computing resource in a Kubernetes cluster is called a node. Nodes can be either bare-metal servers or virtual machines.
|
||||
|
||||
| Alert | Explanation |
|
||||
|-------|-------------|
|
||||
| High CPU load | A warning alert is triggered if the node uses more than 100 percent of the node’s available CPU seconds for at least three minutes. |
|
||||
| High node memory utilization | A warning alert is triggered if the node uses more than 80 percent of its available memory for at least three minutes. |
|
||||
| Node disk is running full within 24 hours | A critical alert is triggered if the disk space on the node is expected to run out in the next 24 hours based on the disk growth over the last 6 hours. |
|
||||
|
||||
# Project-level Alerts
|
||||
When you enable monitoring for the project, some project-level alerts are provided. For details, refer to the [section on project-level alerts.]({{<baseurl>}}/rancher/v2.0-v2.4/en/project-admin/tools/alerts/)
|
||||
+128
@@ -0,0 +1,128 @@
|
||||
---
|
||||
title: Cluster Logging
|
||||
shortTitle: Logging
|
||||
description: Rancher integrates with popular logging services. Learn the requirements and benefits of integrating with logging services, and enable logging on your cluster.
|
||||
metaDescription: "Rancher integrates with popular logging services. Learn the requirements and benefits of integrating with logging services, and enable logging on your cluster."
|
||||
weight: 3
|
||||
aliases:
|
||||
- /rancher/v2.0-v2.4/en/tasks/logging/
|
||||
- /rancher/v2.0-v2.4/en/cluster-admin/tools/logging
|
||||
- /rancher/v2.0-v2.4/en/logging/legacy/cluster-logging
|
||||
- /rancher/v2.0-v2.4/en/logging/v2.0.x-v2.4.x/cluster-logging/
|
||||
- /rancher/v2.x/en/logging/v2.0.x-v2.4.x/
|
||||
- /rancher/v2.x/en/logging/v2.0.x-v2.4.x/cluster-logging/
|
||||
---
|
||||
|
||||
Logging is helpful because it allows you to:
|
||||
|
||||
- Capture and analyze the state of your cluster
|
||||
- Look for trends in your environment
|
||||
- Save your logs to a safe location outside of your cluster
|
||||
- Stay informed of events like a container crashing, a pod eviction, or a node dying
|
||||
- More easily debug and troubleshoot problems
|
||||
|
||||
Rancher supports integration with the following services:
|
||||
|
||||
- Elasticsearch
|
||||
- Splunk
|
||||
- Kafka
|
||||
- Syslog
|
||||
- Fluentd
|
||||
|
||||
This section covers the following topics:
|
||||
|
||||
- [How logging integrations work](#how-logging-integrations-work)
|
||||
- [Requirements](#requirements)
|
||||
- [Logging scope](#logging-scope)
|
||||
- [Enabling cluster logging](#enabling-cluster-logging)
|
||||
|
||||
# How Logging Integrations Work
|
||||
|
||||
Rancher can integrate with popular external services used for event streams, telemetry, or search. These services can log errors and warnings in your Kubernetes infrastructure to a stream.
|
||||
|
||||
These services collect container log events, which are saved to the `/var/log/containers` directory on each of your nodes. The service collects both standard and error events. You can then log into your services to review the events collected, leveraging each service's unique features.
|
||||
|
||||
When configuring Rancher to integrate with these services, you'll have to point Rancher toward the service's endpoint and provide authentication information.
|
||||
|
||||
Additionally, you'll have the opportunity to enter key-value pairs to filter the log events collected. The service will only collect events for containers marked with your configured key-value pairs.
|
||||
|
||||
>**Note:** You can only configure one logging service per cluster or per project.
|
||||
|
||||
# Requirements
|
||||
|
||||
The Docker daemon on each node in the cluster should be [configured](https://docs.docker.com/config/containers/logging/configure/) with the (default) log-driver: `json-file`. You can check the log-driver by running the following command:
|
||||
|
||||
```
|
||||
$ docker info | grep 'Logging Driver'
|
||||
Logging Driver: json-file
|
||||
```
|
||||
|
||||
# Logging Scope
|
||||
|
||||
You can configure logging at either cluster level or project level.
|
||||
|
||||
- Cluster logging writes logs for every pod in the cluster, i.e. in all the projects. For [RKE clusters]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-provisioning/rke-clusters), it also writes logs for all the Kubernetes system components.
|
||||
- [Project logging]({{<baseurl>}}/rancher/v2.0-v2.4/en/project-admin/tools/logging/) writes logs for every pod in that particular project.
|
||||
|
||||
Logs that are sent to your logging service are from the following locations:
|
||||
|
||||
- Pod logs stored at `/var/log/containers`.
|
||||
- Kubernetes system components logs stored at `/var/lib/rancher/rke/log/`.
|
||||
|
||||
# Enabling Cluster Logging
|
||||
|
||||
As an [administrator]({{<baseurl>}}/rancher/v2.0-v2.4/en/admin-settings/rbac/global-permissions/) or [cluster owner]({{<baseurl>}}/rancher/v2.0-v2.4/en/admin-settings/rbac/cluster-project-roles/#cluster-roles), you can configure Rancher to send Kubernetes logs to a logging service.
|
||||
|
||||
1. From the **Global** view, navigate to the cluster that you want to configure cluster logging.
|
||||
|
||||
1. Select **Tools > Logging** in the navigation bar.
|
||||
|
||||
1. Select a logging service and enter the configuration. Refer to the specific service for detailed configuration. Rancher supports integration with the following services:
|
||||
|
||||
- [Elasticsearch]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/tools/logging/elasticsearch/)
|
||||
- [Splunk]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/tools/logging/splunk/)
|
||||
- [Kafka]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/tools/logging/kafka/)
|
||||
- [Syslog]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/tools/logging/syslog/)
|
||||
- [Fluentd]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/tools/logging/fluentd/)
|
||||
|
||||
1. (Optional) Instead of using the UI to configure the logging services, you can enter custom advanced configurations by clicking on **Edit as File**, which is located above the logging targets. This link is only visible after you select a logging service.
|
||||
|
||||
- With the file editor, enter raw fluentd configuration for any logging service. Refer to the documentation for each logging service on how to setup the output configuration.
|
||||
|
||||
- [Elasticsearch Documentation](https://github.com/uken/fluent-plugin-elasticsearch)
|
||||
- [Splunk Documentation](https://github.com/fluent/fluent-plugin-splunk)
|
||||
- [Kafka Documentation](https://github.com/fluent/fluent-plugin-kafka)
|
||||
- [Syslog Documentation](https://github.com/dlackty/fluent-plugin-remote_syslog)
|
||||
- [Fluentd Documentation](https://docs.fluentd.org/v1.0/articles/out_forward)
|
||||
|
||||
- If the logging service is using TLS, you also need to complete the **SSL Configuration** form.
|
||||
1. Provide the **Client Private Key** and **Client Certificate**. You can either copy and paste them or upload them by using the **Read from a file** button.
|
||||
|
||||
- You can use either a self-signed certificate or one provided by a certificate authority.
|
||||
|
||||
- You can generate a self-signed certificate using an openssl command. For example:
|
||||
|
||||
```
|
||||
openssl req -x509 -newkey rsa:2048 -keyout myservice.key -out myservice.cert -days 365 -nodes -subj "/CN=myservice.example.com"
|
||||
```
|
||||
2. If you are using a self-signed certificate, provide the **CA Certificate PEM**.
|
||||
|
||||
1. (Optional) Complete the **Additional Logging Configuration** form.
|
||||
|
||||
1. **Optional:** Use the **Add Field** button to add custom log fields to your logging configuration. These fields are key value pairs (such as `foo=bar`) that you can use to filter the logs from another system.
|
||||
|
||||
1. Enter a **Flush Interval**. This value determines how often [Fluentd](https://www.fluentd.org/) flushes data to the logging server. Intervals are measured in seconds.
|
||||
|
||||
1. **Include System Log**. The logs from pods in system project and RKE components will be sent to the target. Uncheck it to exclude the system logs.
|
||||
|
||||
1. Click **Test**. Rancher sends a test log to the service.
|
||||
|
||||
> **Note:** This button is replaced with _Dry Run_ if you are using the custom configuration editor. In this case, Rancher calls the fluentd dry run command to validate the configuration.
|
||||
|
||||
1. Click **Save**.
|
||||
|
||||
**Result:** Rancher is now configured to send logs to the selected service. Log into the logging service so that you can start viewing the logs.
|
||||
|
||||
## Related Links
|
||||
|
||||
[Logging Architecture](https://kubernetes.io/docs/concepts/cluster-administration/logging/)
|
||||
+47
@@ -0,0 +1,47 @@
|
||||
---
|
||||
title: Elasticsearch
|
||||
weight: 200
|
||||
aliases:
|
||||
- /rancher/v2.0-v2.4/en/tools/logging/elasticsearch/
|
||||
- /rancher/v2.0-v2.4/en/cluster-admin/tools/logging/elasticsearch
|
||||
- /rancher/v2.0-v2.4/en/logging/legacy/cluster-logging/elasticsearch
|
||||
- /rancher/v2.0-v2.4/en/logging/v2.0.x-v2.4.x/cluster-logging/elasticsearch
|
||||
- /rancher/v2.x/en/logging/v2.0.x-v2.4.x/cluster-logging/elasticsearch/
|
||||
---
|
||||
|
||||
If your organization uses [Elasticsearch](https://www.elastic.co/), either on premise or in the cloud, you can configure Rancher to send it Kubernetes logs. Afterwards, you can log into your Elasticsearch deployment to view logs.
|
||||
|
||||
>**Prerequisites:** Configure an [Elasticsearch deployment](https://www.elastic.co/guide/en/cloud/saas-release/ec-create-deployment.html).
|
||||
|
||||
## Elasticsearch Deployment Configuration
|
||||
|
||||
1. In the **Endpoint** field, enter the IP address and port of your Elasticsearch instance. You can find this information from the dashboard of your Elasticsearch deployment.
|
||||
|
||||
* Elasticsearch usually uses port `9200` for HTTP and `9243` for HTTPS.
|
||||
|
||||
1. If you are using [X-Pack Security](https://www.elastic.co/guide/en/x-pack/current/xpack-introduction.html), enter your Elasticsearch **Username** and **Password** for authentication.
|
||||
|
||||
1. Enter an [Index Pattern](https://www.elastic.co/guide/en/kibana/current/index-patterns.html).
|
||||
|
||||
## SSL Configuration
|
||||
|
||||
If your instance of Elasticsearch uses SSL, your **Endpoint** will need to begin with `https://`. With the correct endpoint, the **SSL Configuration** form is enabled and ready to be completed.
|
||||
|
||||
1. Provide the **Client Private Key** and **Client Certificate**. You can either copy and paste them or upload them by using the **Read from a file** button.
|
||||
|
||||
- You can use either a self-signed certificate or one provided by a certificate authority.
|
||||
|
||||
- You can generate a self-signed certificate using an openssl command. For example:
|
||||
|
||||
```
|
||||
openssl req -x509 -newkey rsa:2048 -keyout myservice.key -out myservice.cert -days 365 -nodes -subj "/CN=myservice.example.com"
|
||||
```
|
||||
|
||||
1. Enter your **Client Key Password**.
|
||||
|
||||
1. Enter your **SSL Version**. The default version is `TLSv1_2`.
|
||||
|
||||
1. Select whether or not you want to verify your SSL.
|
||||
|
||||
* If you are using a self-signed certificate, select **Enabled - Input trusted server certificate**, provide the **CA Certificate PEM**. You can copy and paste the certificate or upload it using the **Read from a file** button.
|
||||
* If you are using a certificate from a certificate authority, select **Enabled - Input trusted server certificate**. You do not need to provide a **CA Certificate PEM**.
|
||||
@@ -0,0 +1,39 @@
|
||||
---
|
||||
title: Fluentd
|
||||
weight: 600
|
||||
aliases:
|
||||
- /rancher/v2.0-v2.4/en/cluster-admin/tools/logging/fluentd
|
||||
- /rancher/v2.0-v2.4/en/logging/legacy/cluster-logging/fluentd
|
||||
- /rancher/v2.0-v2.4/en/logging/v2.0.x-v2.4.x/cluster-logging/fluentd
|
||||
- /rancher/v2.x/en/logging/v2.0.x-v2.4.x/cluster-logging/fluentd/
|
||||
---
|
||||
|
||||
If your organization uses [Fluentd](https://www.fluentd.org/), you can configure Rancher to send it Kubernetes logs. Afterwards, you can log into your Fluentd server to view logs.
|
||||
|
||||
>**Prerequisites:** Configure Fluentd input forward to receive the event stream.
|
||||
>
|
||||
>See [Fluentd Documentation](https://docs.fluentd.org/v1.0/articles/in_forward) for details.
|
||||
|
||||
## Fluentd Configuration
|
||||
|
||||
You can add multiple Fluentd Servers. If you want to add additional Fluentd servers, click **Add Fluentd Server**. For each Fluentd server, complete the configuration information:
|
||||
|
||||
1. In the **Endpoint** field, enter the address and port of your Fluentd instance, e.g. `http://Fluentd-server:24224`.
|
||||
|
||||
1. Enter the **Shared Key** if your Fluentd Server is using a shared key for authentication.
|
||||
|
||||
1. Enter the **Username** and **Password** if your Fluentd Server is using username and password for authentication.
|
||||
|
||||
1. **Optional:** Enter the **Hostname** of the Fluentd server.
|
||||
|
||||
1. Enter the load balancing **Weight** of the Fluentd server. If the weight of one server is 20 and the other server is 30, events will be sent in a 2:3 ratio. If you do not enter a weight, the default weight is 60.
|
||||
|
||||
1. If this server is a standby server, check **Use as Standby Only**. Standby servers are used when all other servers are not available.
|
||||
|
||||
After adding all the Fluentd servers, you have the option to select **Enable Gzip Compression**. By default, this is enabled because the transferred payload size will be reduced.
|
||||
|
||||
## SSL Configuration
|
||||
|
||||
If your Fluentd servers are using TLS, you need to select **Use TLS**. If you are using a self-signed certificate, provide the **CA Certificate PEM**. You can copy and paste the certificate or upload it using the **Read from a file** button.
|
||||
|
||||
>**Note:** Fluentd does not support self-signed certificates when client authentication is enabled.
|
||||
@@ -0,0 +1,47 @@
|
||||
---
|
||||
title: Kafka
|
||||
weight: 400
|
||||
aliases:
|
||||
- /rancher/v2.0-v2.4/en/tools/logging/kafka/
|
||||
- /rancher/v2.0-v2.4/en/cluster-admin/tools/logging/kafka
|
||||
- /rancher/v2.0-v2.4/en/logging/legacy/cluster-logging/kafka
|
||||
- /rancher/v2.0-v2.4/en/logging/v2.0.x-v2.4.x/cluster-logging/kafka
|
||||
- /rancher/v2.x/en/logging/v2.0.x-v2.4.x/cluster-logging/kafka/
|
||||
---
|
||||
|
||||
If your organization uses [Kafka](https://kafka.apache.org/), you can configure Rancher to send it Kubernetes logs. Afterwards, you can log into your Kafka server to view logs.
|
||||
|
||||
>**Prerequisite:** You must have a Kafka server configured.
|
||||
|
||||
## Kafka Server Configuration
|
||||
|
||||
1. Select the type of **Endpoint** your Kafka server is using:
|
||||
|
||||
* **Zookeeper**: Enter the IP address and port. By default, Zookeeper uses port `2181`. Please note that a Zookeeper endpoint cannot enable TLS.
|
||||
* **Broker**: Click on **Add Endpoint**. For each Kafka broker, enter the IP address and port. By default, Kafka brokers use port `9092`.
|
||||
|
||||
1. In the **Topic** field, enter the name of a Kafka [topic](https://kafka.apache.org/documentation/#basic_ops_add_topic) that your Kubernetes cluster submits logs to.
|
||||
|
||||
## **Broker** Endpoint Type
|
||||
|
||||
### SSL Configuration
|
||||
|
||||
If your Kafka cluster is using SSL for the **Broker**, you need to complete the **SSL Configuration** form.
|
||||
|
||||
1. Provide the **Client Private Key** and **Client Certificate**. You can either copy and paste them or upload them by using the **Read from a file** button.
|
||||
|
||||
1. Provide the **CA Certificate PEM**. You can either copy and paste the certificate or upload it using the **Read from a file** button.
|
||||
|
||||
>**Note:** Kafka does not support self-signed certificates when client authentication is enabled.
|
||||
|
||||
### SASL configuration
|
||||
|
||||
If your Kafka cluster is using [SASL authentication](https://kafka.apache.org/documentation/#security_sasl) for the Broker, you need to complete the **SASL Configuration** form.
|
||||
|
||||
1. Enter the SASL **Username** and **Password**.
|
||||
|
||||
1. Select the **SASL Type** that your Kafka cluster is using.
|
||||
|
||||
* If your Kafka is using **Plain**, please ensure your Kafka cluster is using SSL.
|
||||
|
||||
* If your Kafka is using **Scram**, you need to select which **Scram Mechanism** Kafka is using.
|
||||
@@ -0,0 +1,81 @@
|
||||
---
|
||||
title: Splunk
|
||||
weight: 300
|
||||
aliases:
|
||||
- /rancher/v2.0-v2.4/en/tasks/logging/splunk/
|
||||
- /rancher/v2.0-v2.4/en/tools/logging/splunk/
|
||||
- /rancher/v2.0-v2.4/en/cluster-admin/tools/logging/splunk
|
||||
- /rancher/v2.0-v2.4/en/logging/legacy/cluster-logging/splunk
|
||||
- /rancher/v2.0-v2.4/en/logging/v2.0.x-v2.4.x/cluster-logging/splunk
|
||||
- /rancher/v2.x/en/logging/v2.0.x-v2.4.x/cluster-logging/splunk/
|
||||
- /rancher/v2.x/en/cluster-admin/tools/logging/splunk
|
||||
---
|
||||
|
||||
If your organization uses [Splunk](https://www.splunk.com/), you can configure Rancher to send it Kubernetes logs. Afterwards, you can log into your Splunk server to view logs.
|
||||
|
||||
>**Prerequisites:**
|
||||
>
|
||||
>- Configure HTTP event collection for your Splunk Server (Splunk Enterprise or Splunk Cloud).
|
||||
>- Either create a new token or copy an existing token.
|
||||
>
|
||||
>For more information, see [Splunk Documentation](http://docs.splunk.com/Documentation/Splunk/7.1.2/Data/UsetheHTTPEventCollector#About_Event_Collector_tokens).
|
||||
|
||||
## Splunk Configuration
|
||||
|
||||
1. In the **Endpoint** field, enter the IP address and port for you Splunk instance (i.e. `http://splunk-server:8088`)
|
||||
|
||||
* Splunk usually uses port `8088`. If you're using Splunk Cloud, you'll need to work with [Splunk support](https://www.splunk.com/en_us/support-and-services.html) to get an endpoint URL.
|
||||
|
||||
1. Enter the **Token** you obtained while completing the prerequisites (i.e., when you created a token in Splunk).
|
||||
|
||||
1. In the **Source** field, enter the name of the token as entered in Splunk.
|
||||
|
||||
1. **Optional:** Provide one or more [index](http://docs.splunk.com/Documentation/Splunk/7.1.2/Indexer/Aboutindexesandindexers) that's allowed for your token.
|
||||
|
||||
## SSL Configuration
|
||||
|
||||
If your instance of Splunk uses SSL, your **Endpoint** will need to begin with `https://`. With the correct endpoint, the **SSL Configuration** form is enabled and ready to be completed.
|
||||
|
||||
1. Provide the **Client Private Key** and **Client Certificate**. You can either copy and paste them or upload them by using the **Read from a file** button.
|
||||
|
||||
- You can use either a self-signed certificate or one provided by a certificate authority.
|
||||
|
||||
- You can generate a self-signed certificate using an openssl command. For example:
|
||||
|
||||
```
|
||||
openssl req -x509 -newkey rsa:2048 -keyout myservice.key -out myservice.cert -days 365 -nodes -subj "/CN=myservice.example.com"
|
||||
```
|
||||
|
||||
1. Enter your **Client Key Password**.
|
||||
|
||||
1. Select whether or not you want to verify your SSL.
|
||||
|
||||
* If you are using a self-signed certificate, select **Enabled - Input trusted server certificate**, provide the **CA Certificate PEM**. You can copy and paste the certificate or upload it using the **Read from a file** button.
|
||||
* If you are using a certificate from a certificate authority, select **Enabled - Input trusted server certificate**. You do not need to provide a **CA Certificate PEM**.
|
||||
|
||||
## Viewing Logs
|
||||
|
||||
1. Log into your Splunk server.
|
||||
|
||||
1. Click on **Search & Reporting**. The number of **Indexed Events** listed should be increasing.
|
||||
|
||||
1. Click on Data Summary and select the Sources tab.
|
||||
|
||||
|
||||
1. To view the actual logs, click on the source that you declared earlier.
|
||||
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
You can use curl to see if **HEC** is listening for HTTP event data.
|
||||
|
||||
```
|
||||
$ curl http://splunk-server:8088/services/collector/event \
|
||||
-H 'Authorization: Splunk 8da70994-b1b0-4a79-b154-bfaae8f93432' \
|
||||
-d '{"event": "hello world"}'
|
||||
```
|
||||
|
||||
If Splunk is configured correctly, you should receive **json** data returning `success code 0`. You should be able
|
||||
to send logging data to HEC.
|
||||
|
||||
If you received an error, check your configuration in Splunk and Rancher.
|
||||
@@ -0,0 +1,47 @@
|
||||
---
|
||||
title: Syslog
|
||||
weight: 500
|
||||
aliases:
|
||||
- /rancher/v2.0-v2.4/en/tools/logging/syslog/
|
||||
- /rancher/v2.0-v2.4/en/cluster-admin/tools/logging/syslog
|
||||
- /rancher/v2.0-v2.4/en/logging/legacy/cluster-logging/syslog
|
||||
- /rancher/v2.0-v2.4/en/logging/v2.0.x-v2.4.x/cluster-logging/syslog
|
||||
- /rancher/v2.x/en/logging/v2.0.x-v2.4.x/cluster-logging/syslog/
|
||||
---
|
||||
|
||||
If your organization uses [Syslog](https://tools.ietf.org/html/rfc5424), you can configure Rancher to send it Kubernetes logs. Afterwards, you can log into your Syslog server to view logs.
|
||||
|
||||
>**Prerequisite:** You must have a Syslog server configured.
|
||||
|
||||
If you are using rsyslog, please make sure your rsyslog authentication mode is `x509/name`.
|
||||
|
||||
## Syslog Server Configuration
|
||||
|
||||
1. In the **Endpoint** field, enter the IP address and port for your Syslog server. Additionally, in the dropdown, select the protocol that your Syslog server uses.
|
||||
|
||||
1. In the **Program** field, enter the name of the application sending logs to your Syslog server, e.g. `Rancher`.
|
||||
|
||||
1. If you are using a cloud logging service, e.g. [Sumologic](https://www.sumologic.com/), enter a **Token** that authenticates with your Syslog server. You will need to create this token in the cloud logging service.
|
||||
|
||||
1. Select a **Log Severity** for events that are logged to the Syslog server. For more information on each severity level, see the [Syslog protocol documentation](https://tools.ietf.org/html/rfc5424#page-11).
|
||||
|
||||
- By specifying a **Log Severity** does not mean that will act as a filtering mechanism for logs. To do that you should use a parser on the Syslog server.
|
||||
|
||||
## Encryption Configuration
|
||||
|
||||
If your Syslog server is using **TCP** protocol and uses TLS, you need to select **Use TLS** and complete the **Encryption Configuration** form.
|
||||
|
||||
1. Provide the **Client Private Key** and **Client Certificate**. You can either copy and paste them or upload them by using the **Read from a file** button.
|
||||
|
||||
- You can use either a self-signed certificate or one provided by a certificate authority.
|
||||
|
||||
- You can generate a self-signed certificate using an openssl command. For example:
|
||||
|
||||
```
|
||||
openssl req -x509 -newkey rsa:2048 -keyout myservice.key -out myservice.cert -days 365 -nodes -subj "/CN=myservice.example.com"
|
||||
```
|
||||
|
||||
1. Select whether or not you want to verify your SSL.
|
||||
|
||||
* If you are using a self-signed certificate, select **Enabled - Input trusted server certificate**, provide the **CA Certificate PEM**. You can copy and paste the certificate or upload it using the **Read from a file** button.
|
||||
* If you are using a certificate from a certificate authority, select **Enabled - Input trusted server certificate**. You do not need to provide a **CA Certificate PEM**.
|
||||
+119
@@ -0,0 +1,119 @@
|
||||
---
|
||||
title: Cluster Metrics
|
||||
weight: 3
|
||||
aliases:
|
||||
- /rancher/v2.0-v2.4/en/project-admin/tools/monitoring/cluster-metrics
|
||||
- /rancher/v2.0-v2.4/en/cluster-admin/tools/monitoring/cluster-metrics
|
||||
- /rancher/v2.0-v2.4/en/monitoring-alerting/legacy/monitoring/cluster-monitoring/cluster-metrics
|
||||
- /rancher/v2.0-v2.4/en/monitoring-alerting/v2.0.x-v2.4.x/cluster-monitoring/cluster-metrics
|
||||
- /rancher/v2.x/en/monitoring-alerting/v2.0.x-v2.4.x/cluster-monitoring/cluster-metrics/
|
||||
---
|
||||
|
||||
_Available as of v2.2.0_
|
||||
|
||||
Cluster metrics display the hardware utilization for all nodes in your cluster, regardless of its role. They give you a global monitoring insight into the cluster.
|
||||
|
||||
Some of the biggest metrics to look out for:
|
||||
|
||||
- **CPU Utilization**
|
||||
|
||||
High load either indicates that your cluster is running efficiently or that you're running out of CPU resources.
|
||||
|
||||
- **Disk Utilization**
|
||||
|
||||
Be on the lookout for increased read and write rates on nodes nearing their disk capacity. This advice is especially true for etcd nodes, as running out of storage on an etcd node leads to cluster failure.
|
||||
|
||||
- **Memory Utilization**
|
||||
|
||||
Deltas in memory utilization usually indicate a memory leak.
|
||||
|
||||
- **Load Average**
|
||||
|
||||
Generally, you want your load average to match your number of logical CPUs for the cluster. For example, if your cluster has 8 logical CPUs, the ideal load average would be 8 as well. If you load average is well under the number of logical CPUs for the cluster, you may want to reduce cluster resources. On the other hand, if your average is over 8, your cluster may need more resources.
|
||||
|
||||
## Finding Node Metrics
|
||||
|
||||
1. From the **Global** view, navigate to the cluster that you want to view metrics.
|
||||
|
||||
1. Select **Nodes** in the navigation bar.
|
||||
|
||||
1. Select a specific node and click on its name.
|
||||
|
||||
1. Click on **Node Metrics**.
|
||||
|
||||
[_Get expressions for Cluster Metrics_]({{<baseurl>}}/rancher/v2.0-v2.4/en/monitoring-alerting/v2.0.x-v2.4.x/cluster-monitoring/expression/#cluster-metrics)
|
||||
|
||||
### Etcd Metrics
|
||||
|
||||
>**Note:** Only supported for [Rancher launched Kubernetes clusters]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-provisioning/rke-clusters/).
|
||||
|
||||
Etcd metrics display the operations of the etcd database on each of your cluster nodes. After establishing a baseline of normal etcd operational metrics, observe them for abnormal deltas between metric refreshes, which indicate potential issues with etcd. Always address etcd issues immediately!
|
||||
|
||||
You should also pay attention to the text at the top of the etcd metrics, which displays leader election statistics. This text indicates if etcd currently has a leader, which is the etcd instance that coordinates the other etcd instances in your cluster. A large increase in leader changes implies etcd is unstable. If you notice a change in leader election statistics, you should investigate them for issues.
|
||||
|
||||
Some of the biggest metrics to look out for:
|
||||
|
||||
- **Etcd has a leader**
|
||||
|
||||
etcd is usually deployed on multiple nodes and elects a leader to coordinate its operations. If etcd does not have a leader, its operations are not being coordinated.
|
||||
|
||||
- **Number of leader changes**
|
||||
|
||||
If this statistic suddenly grows, it usually indicates network communication issues that constantly force the cluster to elect a new leader.
|
||||
|
||||
[_Get expressions for Etcd Metrics_]({{<baseurl>}}/rancher/v2.0-v2.4/en/monitoring-alerting/v2.0.x-v2.4.x/cluster-monitoring/expression/#etcd-metrics)
|
||||
|
||||
### Kubernetes Components Metrics
|
||||
|
||||
Kubernetes components metrics display data about the cluster's individual Kubernetes components. Primarily, it displays information about connections and latency for each component: the API server, controller manager, scheduler, and ingress controller.
|
||||
|
||||
>**Note:** The metrics for the controller manager, scheduler and ingress controller are only supported for [Rancher launched Kubernetes clusters]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-provisioning/rke-clusters/).
|
||||
|
||||
When analyzing Kubernetes component metrics, don't be concerned about any single standalone metric in the charts and graphs that display. Rather, you should establish a baseline for metrics considered normal following a period of observation, e.g. the range of values that your components usually operate within and are considered normal. After you establish this baseline, be on the lookout for large deltas in the charts and graphs, as these big changes usually indicate a problem that you need to investigate.
|
||||
|
||||
Some of the more important component metrics to monitor are:
|
||||
|
||||
- **API Server Request Latency**
|
||||
|
||||
Increasing API response times indicate there's a generalized problem that requires investigation.
|
||||
|
||||
- **API Server Request Rate**
|
||||
|
||||
Rising API request rates usually coincide with increased API response times. Increased request rates also indicate a generalized problem requiring investigation.
|
||||
|
||||
- **Scheduler Preemption Attempts**
|
||||
|
||||
If you see a spike in scheduler preemptions, it's an indication that you're running out of hardware resources, as Kubernetes is recognizing it doesn't have enough resources to run all your pods and is prioritizing the more important ones.
|
||||
|
||||
- **Scheduling Failed Pods**
|
||||
|
||||
Failed pods can have a variety of causes, such as unbound persistent volume claims, exhausted hardware resources, non-responsive nodes, etc.
|
||||
|
||||
- **Ingress Controller Request Process Time**
|
||||
|
||||
How fast ingress is routing connections to your cluster services.
|
||||
|
||||
[_Get expressions for Kubernetes Component Metrics_]({{<baseurl>}}/rancher/v2.0-v2.4/en/monitoring-alerting/legacy/monitoring/cluster-monitoring/expression/#kubernetes-components-metrics)
|
||||
|
||||
## Rancher Logging Metrics
|
||||
|
||||
Although the Dashboard for a cluster primarily displays data sourced from Prometheus, it also displays information for cluster logging, provided that you have [configured Rancher to use a logging service]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/tools/logging/).
|
||||
|
||||
[_Get expressions for Rancher Logging Metrics_]({{<baseurl>}}/rancher/v2.0-v2.4/en/monitoring-alerting/v2.0.x-v2.4.x/cluster-monitoring/expression/#rancher-logging-metrics)
|
||||
|
||||
## Finding Workload Metrics
|
||||
|
||||
Workload metrics display the hardware utilization for a Kubernetes workload. You can also view metrics for [deployments](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/), [stateful sets](https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/) and so on.
|
||||
|
||||
1. From the **Global** view, navigate to the project that you want to view workload metrics.
|
||||
|
||||
1. From the main navigation bar, choose **Resources > Workloads.** In versions before v2.3.0, choose **Workloads** on the main navigation bar.
|
||||
|
||||
1. Select a specific workload and click on its name.
|
||||
|
||||
1. In the **Pods** section, select a specific pod and click on its name.
|
||||
|
||||
- **View the Pod Metrics:** Click on **Pod Metrics**.
|
||||
- **View the Container Metrics:** In the **Containers** section, select a specific container and click on its name. Click on **Container Metrics**.
|
||||
|
||||
[_Get expressions for Workload Metrics_]({{<baseurl>}}/rancher/v2.0-v2.4/en/monitoring-alerting/v2.0.x-v2.4.x/cluster-monitoring/expression/#workload-metrics)
|
||||
+124
@@ -0,0 +1,124 @@
|
||||
---
|
||||
title: Integrating Rancher and Prometheus for Cluster Monitoring
|
||||
shortTitle: Monitoring
|
||||
description: Prometheus lets you view metrics from your different Rancher and Kubernetes objects. Learn about the scope of monitoring and how to enable cluster monitoring
|
||||
weight: 1
|
||||
aliases:
|
||||
- /rancher/v2.0-v2.4/en/project-admin/tools/monitoring
|
||||
- /rancher/v2.0-v2.4/en/monitoring-alerting/legacy/monitoring/cluster-monitoring
|
||||
- /rancher/v2.0-v2.4/en/monitoring-alerting/v2.0.x-v2.4.x/monitoring/cluster-monitoring
|
||||
- /rancher/v2.0-v2.4/en/monitoring-alerting/v2.0.x-v2.4.x/cluster-monitoring
|
||||
- /rancher/v2.x/en/monitoring-alerting/v2.0.x-v2.4.x/
|
||||
- /rancher/v2.x/en/monitoring-alerting/v2.0.x-v2.4.x/cluster-monitoring/
|
||||
---
|
||||
|
||||
_Available as of v2.2.0_
|
||||
|
||||
Using Rancher, you can monitor the state and processes of your cluster nodes, Kubernetes components, and software deployments through integration with [Prometheus](https://prometheus.io/), a leading open-source monitoring solution.
|
||||
|
||||
This section covers the following topics:
|
||||
|
||||
- [About Prometheus](#about-prometheus)
|
||||
- [Monitoring scope](#monitoring-scope)
|
||||
- [Enabling cluster monitoring](#enabling-cluster-monitoring)
|
||||
- [Resource consumption](#resource-consumption)
|
||||
- [Resource consumption of Prometheus pods](#resource-consumption-of-prometheus-pods)
|
||||
- [Resource consumption of other pods](#resource-consumption-of-other-pods)
|
||||
|
||||
# About Prometheus
|
||||
|
||||
Prometheus provides a _time series_ of your data, which is, according to [Prometheus documentation](https://prometheus.io/docs/concepts/data_model/):
|
||||
|
||||
You can configure these services to collect logs at either the cluster level or the project level. This page describes how to enable monitoring for a cluster. For details on enabling monitoring for a project, refer to the [project administration section]({{<baseurl>}}/rancher/v2.0-v2.4/en/project-admin/tools/monitoring/).
|
||||
|
||||
>A stream of timestamped values belonging to the same metric and the same set of labeled dimensions, along with comprehensive statistics and metrics of the monitored cluster.
|
||||
|
||||
In other words, Prometheus lets you view metrics from your different Rancher and Kubernetes objects. Using timestamps, Prometheus lets you query and view these metrics in easy-to-read graphs and visuals, either through the Rancher UI or [Grafana](https://grafana.com/), which is an analytics viewing platform deployed along with Prometheus.
|
||||
|
||||
By viewing data that Prometheus scrapes from your cluster control plane, nodes, and deployments, you can stay on top of everything happening in your cluster. You can then use these analytics to better run your organization: stop system emergencies before they start, develop maintenance strategies, restore crashed servers, etc.
|
||||
|
||||
Multi-tenancy support in terms of cluster-only and project-only Prometheus instances are also supported.
|
||||
|
||||
# Monitoring Scope
|
||||
|
||||
Using Prometheus, you can monitor Rancher at both the cluster level and [project level]({{<baseurl>}}/rancher/v2.0-v2.4/en/project-admin/tools/monitoring/). For each cluster and project that is enabled for monitoring, Rancher deploys a Prometheus server.
|
||||
|
||||
- Cluster monitoring allows you to view the health of your Kubernetes cluster. Prometheus collects metrics from the cluster components below, which you can view in graphs and charts.
|
||||
|
||||
- Kubernetes control plane
|
||||
- etcd database
|
||||
- All nodes (including workers)
|
||||
|
||||
- [Project monitoring]({{<baseurl>}}/rancher/v2.0-v2.4/en/project-admin/tools/monitoring/) allows you to view the state of pods running in a given project. Prometheus collects metrics from the project's deployed HTTP and TCP/UDP workloads.
|
||||
|
||||
# Enabling Cluster Monitoring
|
||||
|
||||
As an [administrator]({{<baseurl>}}/rancher/v2.0-v2.4/en/admin-settings/rbac/global-permissions/) or [cluster owner]({{<baseurl>}}/rancher/v2.0-v2.4/en/admin-settings/rbac/cluster-project-roles/#cluster-roles), you can configure Rancher to deploy Prometheus to monitor your Kubernetes cluster.
|
||||
|
||||
> **Prerequisites:** The following TCP ports need to be opened for metrics scraping:
|
||||
>
|
||||
> | Port | Node type | Component |
|
||||
> | --- | --- | --- |
|
||||
> | 9796 | Worker | Node exporter |
|
||||
> | 10254 | Worker | Nginx Ingress Controller |
|
||||
> | 10250 | Worker/Controlplane | Kubelet |
|
||||
> | 10251 | Controlplane | Kube scheduler |
|
||||
> | 10252 | Controlplane | Kube controller manager |
|
||||
> | 2379 | Etcd | Etcd server |
|
||||
|
||||
> Monitoring V1 requires a Kubernetes verison less than or equal to v1.20.x. To install monitoring on Kubernetes v1.21+, you will need to [migrate to Monitoring V2.]({{<baseurl>}}/rancher/v2.5/en/monitoring-alerting/migrating/)
|
||||
|
||||
1. From the **Global** view, navigate to the cluster that you want to configure cluster monitoring.
|
||||
|
||||
1. Select **Tools > Monitoring** in the navigation bar.
|
||||
|
||||
1. Select **Enable** to show the [Prometheus configuration options]({{<baseurl>}}/rancher/v2.0-v2.4/en/monitoring-alerting/legacy/monitoring/cluster-monitoring/prometheus/). Review the [resource consumption recommendations](#resource-consumption) to ensure you have enough resources for Prometheus and on your worker nodes to enable monitoring. Enter in your desired configuration options.
|
||||
|
||||
1. Click **Save**.
|
||||
|
||||
**Result:** The Prometheus server will be deployed as well as two monitoring applications. The two monitoring applications, `cluster-monitoring` and `monitoring-operator`, are added as an [application]({{<baseurl>}}/rancher/v2.0-v2.4/en/catalog/apps/) to the cluster's `system` project. After the applications are `active`, you can start viewing [cluster metrics]({{<baseurl>}}/rancher/v2.0-v2.4/en/monitoring-alerting/legacy/monitoring/cluster-monitoring/cluster-metrics/) through the Rancher dashboard or directly from Grafana.
|
||||
|
||||
> The default username and password for the Grafana instance will be `admin/admin`. However, Grafana dashboards are served via the Rancher authentication proxy, so only users who are currently authenticated into the Rancher server have access to the Grafana dashboard.
|
||||
|
||||
# Resource Consumption
|
||||
|
||||
When enabling cluster monitoring, you need to ensure your worker nodes and Prometheus pod have enough resources. The tables below provides a guide of how much resource consumption will be used. In larger deployments, it is strongly advised that the monitoring infrastructure be placed on dedicated nodes in the cluster.
|
||||
|
||||
### Resource Consumption of Prometheus Pods
|
||||
|
||||
This table is the resource consumption of the Prometheus pod, which is based on the number of all the nodes in the cluster. The count of nodes includes the worker, control plane and etcd nodes. Total disk space allocation should be approximated by the `rate * retention` period set at the cluster level. When enabling cluster level monitoring, you should adjust the CPU and Memory limits and reservation.
|
||||
|
||||
Number of Cluster Nodes | CPU (milli CPU) | Memory | Disk
|
||||
------------------------|-----|--------|------
|
||||
5 | 500 | 650 MB | ~1 GB/Day
|
||||
50| 2000 | 2 GB | ~5 GB/Day
|
||||
256| 4000 | 6 GB | ~18 GB/Day
|
||||
|
||||
Additional pod resource requirements for cluster level monitoring.
|
||||
|
||||
| Workload | Container | CPU - Request | Mem - Request | CPU - Limit | Mem - Limit | Configurable |
|
||||
|---------------------|---------------------------------|---------------|---------------|-------------|-------------|--------------|
|
||||
| Prometheus | prometheus | 750m | 750Mi | 1000m | 1000Mi | Y |
|
||||
| | prometheus-proxy | 50m | 50Mi | 100m | 100Mi | Y |
|
||||
| | prometheus-auth | 100m | 100Mi | 500m | 200Mi | Y |
|
||||
| | prometheus-config-reloader | - | - | 50m | 50Mi | N |
|
||||
| | rules-configmap-reloader | - | - | 100m | 25Mi | N |
|
||||
| Grafana | grafana-init-plugin-json-copy | 50m | 50Mi | 50m | 50Mi | Y |
|
||||
| | grafana-init-plugin-json-modify | 50m | 50Mi | 50m | 50Mi | Y |
|
||||
| | grafana | 100m | 100Mi | 200m | 200Mi | Y |
|
||||
| | grafana-proxy | 50m | 50Mi | 100m | 100Mi | Y |
|
||||
| Kube-State Exporter | kube-state | 100m | 130Mi | 100m | 200Mi | Y |
|
||||
| Node Exporter | exporter-node | 200m | 200Mi | 200m | 200Mi | Y |
|
||||
| Operator | prometheus-operator | 100m | 50Mi | 200m | 100Mi | Y |
|
||||
|
||||
|
||||
### Resource Consumption of Other Pods
|
||||
|
||||
Besides the Prometheus pod, there are components that are deployed that require additional resources on the worker nodes.
|
||||
|
||||
Pod | CPU (milli CPU) | Memory (MB)
|
||||
----|-----------------|------------
|
||||
Node Exporter (Per Node) | 100 | 30
|
||||
Kube State Cluster Monitor | 100 | 130
|
||||
Grafana | 100 | 150
|
||||
Prometheus Cluster Monitoring Nginx | 50 | 50
|
||||
+493
@@ -0,0 +1,493 @@
|
||||
---
|
||||
title: Prometheus Custom Metrics Adapter
|
||||
weight: 5
|
||||
aliases:
|
||||
- /rancher/v2.0-v2.4/en/project-admin/tools/monitoring/custom-metrics
|
||||
- /rancher/v2.0-v2.4/en/monitoring-alerting/legacy/monitoring/cluster-monitoring/custom-metrics
|
||||
- /rancher/v2.0-v2.4/en/cluster-admin/tools/monitoring/custom-metrics/
|
||||
- /rancher/v2.0-v2.4/en/monitoring-alerting/v2.0.x-v2.4.x/cluster-monitoring/custom-metrics
|
||||
- /rancher/v2.x/en/monitoring-alerting/v2.0.x-v2.4.x/cluster-monitoring/custom-metrics/
|
||||
---
|
||||
|
||||
After you've enabled [cluster level monitoring]({{< baseurl >}}/rancher/v2.0-v2.4/en/monitoring-alerting/legacy/monitoring/cluster-monitoring/), You can view the metrics data from Rancher. You can also deploy the Prometheus custom metrics adapter then you can use the HPA with metrics stored in cluster monitoring.
|
||||
|
||||
## Deploy Prometheus Custom Metrics Adapter
|
||||
|
||||
We are going to use the [Prometheus custom metrics adapter](https://github.com/DirectXMan12/k8s-prometheus-adapter/releases/tag/v0.5.0), version v0.5.0. This is a great example for the [custom metrics server](https://github.com/kubernetes-incubator/custom-metrics-apiserver). And you must be the *cluster owner* to execute following steps.
|
||||
|
||||
- Get the service account of the cluster monitoring is using. It should be configured in the workload ID: `statefulset:cattle-prometheus:prometheus-cluster-monitoring`. And if you didn't customize anything, the service account name should be `cluster-monitoring`.
|
||||
|
||||
- Grant permission to that service account. You will need two kinds of permission.
|
||||
One role is `extension-apiserver-authentication-reader` in `kube-system`, so you will need to create a `Rolebinding` to in `kube-system`. This permission is to get api aggregation configuration from config map in `kube-system`.
|
||||
|
||||
```yaml
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: RoleBinding
|
||||
metadata:
|
||||
name: custom-metrics-auth-reader
|
||||
namespace: kube-system
|
||||
roleRef:
|
||||
apiGroup: rbac.authorization.k8s.io
|
||||
kind: Role
|
||||
name: extension-apiserver-authentication-reader
|
||||
subjects:
|
||||
- kind: ServiceAccount
|
||||
name: cluster-monitoring
|
||||
namespace: cattle-prometheus
|
||||
```
|
||||
|
||||
The other one is cluster role `system:auth-delegator`, so you will need to create a `ClusterRoleBinding`. This permission is to have subject access review permission.
|
||||
|
||||
```yaml
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: ClusterRoleBinding
|
||||
metadata:
|
||||
name: custom-metrics:system:auth-delegator
|
||||
roleRef:
|
||||
apiGroup: rbac.authorization.k8s.io
|
||||
kind: ClusterRole
|
||||
name: system:auth-delegator
|
||||
subjects:
|
||||
- kind: ServiceAccount
|
||||
name: cluster-monitoring
|
||||
namespace: cattle-prometheus
|
||||
```
|
||||
|
||||
- Create configuration for custom metrics adapter. Following is an example configuration. There will be a configuration details in next session.
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: adapter-config
|
||||
namespace: cattle-prometheus
|
||||
data:
|
||||
config.yaml: |
|
||||
rules:
|
||||
- seriesQuery: '{__name__=~"^container_.*",container_name!="POD",namespace!="",pod_name!=""}'
|
||||
seriesFilters: []
|
||||
resources:
|
||||
overrides:
|
||||
namespace:
|
||||
resource: namespace
|
||||
pod_name:
|
||||
resource: pod
|
||||
name:
|
||||
matches: ^container_(.*)_seconds_total$
|
||||
as: ""
|
||||
metricsQuery: sum(rate(<<.Series>>{<<.LabelMatchers>>,container_name!="POD"}[1m])) by (<<.GroupBy>>)
|
||||
- seriesQuery: '{__name__=~"^container_.*",container_name!="POD",namespace!="",pod_name!=""}'
|
||||
seriesFilters:
|
||||
- isNot: ^container_.*_seconds_total$
|
||||
resources:
|
||||
overrides:
|
||||
namespace:
|
||||
resource: namespace
|
||||
pod_name:
|
||||
resource: pod
|
||||
name:
|
||||
matches: ^container_(.*)_total$
|
||||
as: ""
|
||||
metricsQuery: sum(rate(<<.Series>>{<<.LabelMatchers>>,container_name!="POD"}[1m])) by (<<.GroupBy>>)
|
||||
- seriesQuery: '{__name__=~"^container_.*",container_name!="POD",namespace!="",pod_name!=""}'
|
||||
seriesFilters:
|
||||
- isNot: ^container_.*_total$
|
||||
resources:
|
||||
overrides:
|
||||
namespace:
|
||||
resource: namespace
|
||||
pod_name:
|
||||
resource: pod
|
||||
name:
|
||||
matches: ^container_(.*)$
|
||||
as: ""
|
||||
metricsQuery: sum(<<.Series>>{<<.LabelMatchers>>,container_name!="POD"}) by (<<.GroupBy>>)
|
||||
- seriesQuery: '{namespace!="",__name__!~"^container_.*"}'
|
||||
seriesFilters:
|
||||
- isNot: .*_total$
|
||||
resources:
|
||||
template: <<.Resource>>
|
||||
name:
|
||||
matches: ""
|
||||
as: ""
|
||||
metricsQuery: sum(<<.Series>>{<<.LabelMatchers>>}) by (<<.GroupBy>>)
|
||||
- seriesQuery: '{namespace!="",__name__!~"^container_.*"}'
|
||||
seriesFilters:
|
||||
- isNot: .*_seconds_total
|
||||
resources:
|
||||
template: <<.Resource>>
|
||||
name:
|
||||
matches: ^(.*)_total$
|
||||
as: ""
|
||||
metricsQuery: sum(rate(<<.Series>>{<<.LabelMatchers>>}[1m])) by (<<.GroupBy>>)
|
||||
- seriesQuery: '{namespace!="",__name__!~"^container_.*"}'
|
||||
seriesFilters: []
|
||||
resources:
|
||||
template: <<.Resource>>
|
||||
name:
|
||||
matches: ^(.*)_seconds_total$
|
||||
as: ""
|
||||
metricsQuery: sum(rate(<<.Series>>{<<.LabelMatchers>>}[1m])) by (<<.GroupBy>>)
|
||||
resourceRules:
|
||||
cpu:
|
||||
containerQuery: sum(rate(container_cpu_usage_seconds_total{<<.LabelMatchers>>}[1m])) by (<<.GroupBy>>)
|
||||
nodeQuery: sum(rate(container_cpu_usage_seconds_total{<<.LabelMatchers>>, id='/'}[1m])) by (<<.GroupBy>>)
|
||||
resources:
|
||||
overrides:
|
||||
instance:
|
||||
resource: node
|
||||
namespace:
|
||||
resource: namespace
|
||||
pod_name:
|
||||
resource: pod
|
||||
containerLabel: container_name
|
||||
memory:
|
||||
containerQuery: sum(container_memory_working_set_bytes{<<.LabelMatchers>>}) by (<<.GroupBy>>)
|
||||
nodeQuery: sum(container_memory_working_set_bytes{<<.LabelMatchers>>,id='/'}) by (<<.GroupBy>>)
|
||||
resources:
|
||||
overrides:
|
||||
instance:
|
||||
resource: node
|
||||
namespace:
|
||||
resource: namespace
|
||||
pod_name:
|
||||
resource: pod
|
||||
containerLabel: container_name
|
||||
window: 1m
|
||||
```
|
||||
|
||||
- Create HTTPS TLS certs for your api server. You can use following command to create a self-signed cert.
|
||||
|
||||
```bash
|
||||
openssl req -new -newkey rsa:4096 -x509 -sha256 -days 365 -nodes -out serving.crt -keyout serving.key -subj "/C=CN/CN=custom-metrics-apiserver.cattle-prometheus.svc.cluster.local"
|
||||
# And you will find serving.crt and serving.key in your path. And then you are going to create a secret in cattle-prometheus namespace.
|
||||
kubectl create secret generic -n cattle-prometheus cm-adapter-serving-certs --from-file=serving.key=./serving.key --from-file=serving.crt=./serving.crt
|
||||
```
|
||||
|
||||
- Then you can create the prometheus custom metrics adapter. And you will need a service for this deployment too. Creating it via Import YAML or Rancher would do. Please create those resources in `cattle-prometheus` namespaces.
|
||||
|
||||
Here is the prometheus custom metrics adapter deployment.
|
||||
```yaml
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
labels:
|
||||
app: custom-metrics-apiserver
|
||||
name: custom-metrics-apiserver
|
||||
namespace: cattle-prometheus
|
||||
spec:
|
||||
replicas: 1
|
||||
selector:
|
||||
matchLabels:
|
||||
app: custom-metrics-apiserver
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: custom-metrics-apiserver
|
||||
name: custom-metrics-apiserver
|
||||
spec:
|
||||
serviceAccountName: cluster-monitoring
|
||||
containers:
|
||||
- name: custom-metrics-apiserver
|
||||
image: directxman12/k8s-prometheus-adapter-amd64:v0.5.0
|
||||
args:
|
||||
- --secure-port=6443
|
||||
- --tls-cert-file=/var/run/serving-cert/serving.crt
|
||||
- --tls-private-key-file=/var/run/serving-cert/serving.key
|
||||
- --logtostderr=true
|
||||
- --prometheus-url=http://prometheus-operated/
|
||||
- --metrics-relist-interval=1m
|
||||
- --v=10
|
||||
- --config=/etc/adapter/config.yaml
|
||||
ports:
|
||||
- containerPort: 6443
|
||||
volumeMounts:
|
||||
- mountPath: /var/run/serving-cert
|
||||
name: volume-serving-cert
|
||||
readOnly: true
|
||||
- mountPath: /etc/adapter/
|
||||
name: config
|
||||
readOnly: true
|
||||
- mountPath: /tmp
|
||||
name: tmp-vol
|
||||
volumes:
|
||||
- name: volume-serving-cert
|
||||
secret:
|
||||
secretName: cm-adapter-serving-certs
|
||||
- name: config
|
||||
configMap:
|
||||
name: adapter-config
|
||||
- name: tmp-vol
|
||||
emptyDir: {}
|
||||
|
||||
```
|
||||
|
||||
Here is the service of the deployment.
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: custom-metrics-apiserver
|
||||
namespace: cattle-prometheus
|
||||
spec:
|
||||
ports:
|
||||
- port: 443
|
||||
targetPort: 6443
|
||||
selector:
|
||||
app: custom-metrics-apiserver
|
||||
```
|
||||
|
||||
- Create API service for your custom metric server.
|
||||
|
||||
```yaml
|
||||
apiVersion: apiregistration.k8s.io/v1beta1
|
||||
kind: APIService
|
||||
metadata:
|
||||
name: v1beta1.custom.metrics.k8s.io
|
||||
spec:
|
||||
service:
|
||||
name: custom-metrics-apiserver
|
||||
namespace: cattle-prometheus
|
||||
group: custom.metrics.k8s.io
|
||||
version: v1beta1
|
||||
insecureSkipTLSVerify: true
|
||||
groupPriorityMinimum: 100
|
||||
versionPriority: 100
|
||||
|
||||
```
|
||||
|
||||
- Then you can verify your custom metrics server by `kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1`. If you see the return datas from the api, it means that the metrics server has been successfully set up.
|
||||
|
||||
- You create HPA with custom metrics now. Here is an example of HPA. You will need to create a nginx deployment in your namespace first.
|
||||
|
||||
```yaml
|
||||
kind: HorizontalPodAutoscaler
|
||||
apiVersion: autoscaling/v2beta1
|
||||
metadata:
|
||||
name: nginx
|
||||
spec:
|
||||
scaleTargetRef:
|
||||
# point the HPA at the nginx deployment you just created
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
name: nginx
|
||||
# autoscale between 1 and 10 replicas
|
||||
minReplicas: 1
|
||||
maxReplicas: 10
|
||||
metrics:
|
||||
# use a "Pods" metric, which takes the average of the
|
||||
# given metric across all pods controlled by the autoscaling target
|
||||
- type: Pods
|
||||
pods:
|
||||
metricName: memory_usage_bytes
|
||||
targetAverageValue: 5000000
|
||||
```
|
||||
|
||||
And then, you should see your nginx is scaling up. HPA with custom metrics works.
|
||||
|
||||
## Configuration of prometheus custom metrics adapter
|
||||
|
||||
> Refer to https://github.com/DirectXMan12/k8s-prometheus-adapter/blob/master/docs/config.md
|
||||
|
||||
The adapter determines which metrics to expose, and how to expose them,
|
||||
through a set of "discovery" rules. Each rule is executed independently
|
||||
(so make sure that your rules are mutually exclusive), and specifies each
|
||||
of the steps the adapter needs to take to expose a metric in the API.
|
||||
|
||||
Each rule can be broken down into roughly four parts:
|
||||
|
||||
- *Discovery*, which specifies how the adapter should find all Prometheus
|
||||
metrics for this rule.
|
||||
|
||||
- *Association*, which specifies how the adapter should determine which
|
||||
Kubernetes resources a particular metric is associated with.
|
||||
|
||||
- *Naming*, which specifies how the adapter should expose the metric in
|
||||
the custom metrics API.
|
||||
|
||||
- *Querying*, which specifies how a request for a particular metric on one
|
||||
or more Kubernetes objects should be turned into a query to Prometheus.
|
||||
|
||||
A basic config with one rule might look like:
|
||||
|
||||
```yaml
|
||||
rules:
|
||||
# this rule matches cumulative cAdvisor metrics measured in seconds
|
||||
- seriesQuery: '{__name__=~"^container_.*",container_name!="POD",namespace!="",pod_name!=""}'
|
||||
resources:
|
||||
# skip specifying generic resource<->label mappings, and just
|
||||
# attach only pod and namespace resources by mapping label names to group-resources
|
||||
overrides:
|
||||
namespace: {resource: "namespace"},
|
||||
pod_name: {resource: "pod"},
|
||||
# specify that the `container_` and `_seconds_total` suffixes should be removed.
|
||||
# this also introduces an implicit filter on metric family names
|
||||
name:
|
||||
# we use the value of the capture group implicitly as the API name
|
||||
# we could also explicitly write `as: "$1"`
|
||||
matches: "^container_(.*)_seconds_total$"
|
||||
# specify how to construct a query to fetch samples for a given series
|
||||
# This is a Go template where the `.Series` and `.LabelMatchers` string values
|
||||
# are available, and the delimiters are `<<` and `>>` to avoid conflicts with
|
||||
# the prometheus query language
|
||||
metricsQuery: "sum(rate(<<.Series>>{<<.LabelMatchers>>,container_name!="POD"}[2m])) by (<<.GroupBy>>)"
|
||||
```
|
||||
|
||||
### Discovery
|
||||
|
||||
Discovery governs the process of finding the metrics that you want to
|
||||
expose in the custom metrics API. There are two fields that factor into
|
||||
discovery: `seriesQuery` and `seriesFilters`.
|
||||
|
||||
`seriesQuery` specifies Prometheus series query (as passed to the
|
||||
`/api/v1/series` endpoint in Prometheus) to use to find some set of
|
||||
Prometheus series. The adapter will strip the label values from this
|
||||
series, and then use the resulting metric-name-label-names combinations
|
||||
later on.
|
||||
|
||||
In many cases, `seriesQuery` will be sufficient to narrow down the list of
|
||||
Prometheus series. However, sometimes (especially if two rules might
|
||||
otherwise overlap), it's useful to do additional filtering on metric
|
||||
names. In this case, `seriesFilters` can be used. After the list of
|
||||
series is returned from `seriesQuery`, each series has its metric name
|
||||
filtered through any specified filters.
|
||||
|
||||
Filters may be either:
|
||||
|
||||
- `is: <regex>`, which matches any series whose name matches the specified
|
||||
regex.
|
||||
|
||||
- `isNot: <regex>`, which matches any series whose name does not match the
|
||||
specified regex.
|
||||
|
||||
For example:
|
||||
|
||||
```yaml
|
||||
# match all cAdvisor metrics that aren't measured in seconds
|
||||
seriesQuery: '{__name__=~"^container_.*_total",container_name!="POD",namespace!="",pod_name!=""}'
|
||||
seriesFilters:
|
||||
isNot: "^container_.*_seconds_total"
|
||||
```
|
||||
|
||||
### Association
|
||||
|
||||
Association governs the process of figuring out which Kubernetes resources
|
||||
a particular metric could be attached to. The `resources` field controls
|
||||
this process.
|
||||
|
||||
There are two ways to associate resources with a particular metric. In
|
||||
both cases, the value of the label becomes the name of the particular
|
||||
object.
|
||||
|
||||
One way is to specify that any label name that matches some particular
|
||||
pattern refers to some group-resource based on the label name. This can
|
||||
be done using the `template` field. The pattern is specified as a Go
|
||||
template, with the `Group` and `Resource` fields representing group and
|
||||
resource. You don't necessarily have to use the `Group` field (in which
|
||||
case the group is guessed by the system). For instance:
|
||||
|
||||
```yaml
|
||||
# any label `kube_<group>_<resource>` becomes <group>.<resource> in Kubernetes
|
||||
resources:
|
||||
template: "kube_<<.Group>>_<<.Resource>>"
|
||||
```
|
||||
|
||||
The other way is to specify that some particular label represents some
|
||||
particular Kubernetes resource. This can be done using the `overrides`
|
||||
field. Each override maps a Prometheus label to a Kubernetes
|
||||
group-resource. For instance:
|
||||
|
||||
```yaml
|
||||
# the microservice label corresponds to the apps.deployment resource
|
||||
resource:
|
||||
overrides:
|
||||
microservice: {group: "apps", resource: "deployment"}
|
||||
```
|
||||
|
||||
These two can be combined, so you can specify both a template and some
|
||||
individual overrides.
|
||||
|
||||
The resources mentioned can be any resource available in your kubernetes
|
||||
cluster, as long as you've got a corresponding label.
|
||||
|
||||
### Naming
|
||||
|
||||
Naming governs the process of converting a Prometheus metric name into
|
||||
a metric in the custom metrics API, and vice versa. It's controlled by
|
||||
the `name` field.
|
||||
|
||||
Naming is controlled by specifying a pattern to extract an API name from
|
||||
a Prometheus name, and potentially a transformation on that extracted
|
||||
value.
|
||||
|
||||
The pattern is specified in the `matches` field, and is just a regular
|
||||
expression. If not specified, it defaults to `.*`.
|
||||
|
||||
The transformation is specified by the `as` field. You can use any
|
||||
capture groups defined in the `matches` field. If the `matches` field
|
||||
doesn't contain capture groups, the `as` field defaults to `$0`. If it
|
||||
contains a single capture group, the `as` field defautls to `$1`.
|
||||
Otherwise, it's an error not to specify the as field.
|
||||
|
||||
For example:
|
||||
|
||||
```yaml
|
||||
# match turn any name <name>_total to <name>_per_second
|
||||
# e.g. http_requests_total becomes http_requests_per_second
|
||||
name:
|
||||
matches: "^(.*)_total$"
|
||||
as: "${1}_per_second"
|
||||
```
|
||||
|
||||
### Querying
|
||||
|
||||
Querying governs the process of actually fetching values for a particular
|
||||
metric. It's controlled by the `metricsQuery` field.
|
||||
|
||||
The `metricsQuery` field is a Go template that gets turned into
|
||||
a Prometheus query, using input from a particular call to the custom
|
||||
metrics API. A given call to the custom metrics API is distilled down to
|
||||
a metric name, a group-resource, and one or more objects of that
|
||||
group-resource. These get turned into the following fields in the
|
||||
template:
|
||||
|
||||
- `Series`: the metric name
|
||||
- `LabelMatchers`: a comma-separated list of label matchers matching the
|
||||
given objects. Currently, this is the label for the particular
|
||||
group-resource, plus the label for namespace, if the group-resource is
|
||||
namespaced.
|
||||
- `GroupBy`: a comma-separated list of labels to group by. Currently,
|
||||
this contains the group-resource label used in `LabelMatchers`.
|
||||
|
||||
For instance, suppose we had a series `http_requests_total` (exposed as
|
||||
`http_requests_per_second` in the API) with labels `service`, `pod`,
|
||||
`ingress`, `namespace`, and `verb`. The first four correspond to
|
||||
Kubernetes resources. Then, if someone requested the metric
|
||||
`pods/http_request_per_second` for the pods `pod1` and `pod2` in the
|
||||
`somens` namespace, we'd have:
|
||||
|
||||
- `Series: "http_requests_total"`
|
||||
- `LabelMatchers: "pod=~\"pod1|pod2",namespace="somens"`
|
||||
- `GroupBy`: `pod`
|
||||
|
||||
Additionally, there are two advanced fields that are "raw" forms of other
|
||||
fields:
|
||||
|
||||
- `LabelValuesByName`: a map mapping the labels and values from the
|
||||
`LabelMatchers` field. The values are pre-joined by `|`
|
||||
(for used with the `=~` matcher in Prometheus).
|
||||
- `GroupBySlice`: the slice form of `GroupBy`.
|
||||
|
||||
In general, you'll probably want to use the `Series`, `LabelMatchers`, and
|
||||
`GroupBy` fields. The other two are for advanced usage.
|
||||
|
||||
The query is expected to return one value for each object requested. The
|
||||
adapter will use the labels on the returned series to associate a given
|
||||
series back to its corresponding object.
|
||||
|
||||
For example:
|
||||
|
||||
```yaml
|
||||
# convert cumulative cAdvisor metrics into rates calculated over 2 minutes
|
||||
metricsQuery: "sum(rate(<<.Series>>{<<.LabelMatchers>>,container_name!="POD"}[2m])) by (<<.GroupBy>>)"
|
||||
```
|
||||
+436
@@ -0,0 +1,436 @@
|
||||
---
|
||||
title: Prometheus Expressions
|
||||
weight: 4
|
||||
aliases:
|
||||
- /rancher/v2.0-v2.4/en/project-admin/tools/monitoring/expression
|
||||
- /rancher/v2.0-v2.4/en/cluster-admin/tools/monitoring/expression
|
||||
- /rancher/v2.0-v2.4/en/monitoring-alerting/legacy/monitoring/cluster-monitoring/expression
|
||||
- /rancher/v2.0-v2.4/en/monitoring-alerting/v2.0.x-v2.4.x/cluster-monitoring/expression
|
||||
- /rancher/v2.x/en/monitoring-alerting/v2.0.x-v2.4.x/cluster-monitoring/expression/
|
||||
---
|
||||
|
||||
The PromQL expressions in this doc can be used to configure [alerts.]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/tools/alerts/)
|
||||
|
||||
> Before expressions can be used in alerts, monitoring must be enabled. For more information, refer to the documentation on enabling monitoring [at the cluster level]({{<baseurl>}}/rancher/v2.0-v2.4/en/monitoring-alerting/legacy/monitoring/cluster-monitoring/) or [at the project level.]({{<baseurl>}}/rancher/v2.0-v2.4/en/project-admin/tools/monitoring/)
|
||||
|
||||
For more information about querying Prometheus, refer to the official [Prometheus documentation.](https://prometheus.io/docs/prometheus/latest/querying/basics/)
|
||||
|
||||
<!-- TOC -->
|
||||
|
||||
- [Cluster Metrics](#cluster-metrics)
|
||||
- [Cluster CPU Utilization](#cluster-cpu-utilization)
|
||||
- [Cluster Load Average](#cluster-load-average)
|
||||
- [Cluster Memory Utilization](#cluster-memory-utilization)
|
||||
- [Cluster Disk Utilization](#cluster-disk-utilization)
|
||||
- [Cluster Disk I/O](#cluster-disk-i-o)
|
||||
- [Cluster Network Packets](#cluster-network-packets)
|
||||
- [Cluster Network I/O](#cluster-network-i-o)
|
||||
- [Node Metrics](#node-metrics)
|
||||
- [Node CPU Utilization](#node-cpu-utilization)
|
||||
- [Node Load Average](#node-load-average)
|
||||
- [Node Memory Utilization](#node-memory-utilization)
|
||||
- [Node Disk Utilization](#node-disk-utilization)
|
||||
- [Node Disk I/O](#node-disk-i-o)
|
||||
- [Node Network Packets](#node-network-packets)
|
||||
- [Node Network I/O](#node-network-i-o)
|
||||
- [Etcd Metrics](#etcd-metrics)
|
||||
- [Etcd Has a Leader](#etcd-has-a-leader)
|
||||
- [Number of Times the Leader Changes](#number-of-times-the-leader-changes)
|
||||
- [Number of Failed Proposals](#number-of-failed-proposals)
|
||||
- [GRPC Client Traffic](#grpc-client-traffic)
|
||||
- [Peer Traffic](#peer-traffic)
|
||||
- [DB Size](#db-size)
|
||||
- [Active Streams](#active-streams)
|
||||
- [Raft Proposals](#raft-proposals)
|
||||
- [RPC Rate](#rpc-rate)
|
||||
- [Disk Operations](#disk-operations)
|
||||
- [Disk Sync Duration](#disk-sync-duration)
|
||||
- [Kubernetes Components Metrics](#kubernetes-components-metrics)
|
||||
- [API Server Request Latency](#api-server-request-latency)
|
||||
- [API Server Request Rate](#api-server-request-rate)
|
||||
- [Scheduling Failed Pods](#scheduling-failed-pods)
|
||||
- [Controller Manager Queue Depth](#controller-manager-queue-depth)
|
||||
- [Scheduler E2E Scheduling Latency](#scheduler-e2e-scheduling-latency)
|
||||
- [Scheduler Preemption Attempts](#scheduler-preemption-attempts)
|
||||
- [Ingress Controller Connections](#ingress-controller-connections)
|
||||
- [Ingress Controller Request Process Time](#ingress-controller-request-process-time)
|
||||
- [Rancher Logging Metrics](#rancher-logging-metrics)
|
||||
- [Fluentd Buffer Queue Rate](#fluentd-buffer-queue-rate)
|
||||
- [Fluentd Input Rate](#fluentd-input-rate)
|
||||
- [Fluentd Output Errors Rate](#fluentd-output-errors-rate)
|
||||
- [Fluentd Output Rate](#fluentd-output-rate)
|
||||
- [Workload Metrics](#workload-metrics)
|
||||
- [Workload CPU Utilization](#workload-cpu-utilization)
|
||||
- [Workload Memory Utilization](#workload-memory-utilization)
|
||||
- [Workload Network Packets](#workload-network-packets)
|
||||
- [Workload Network I/O](#workload-network-i-o)
|
||||
- [Workload Disk I/O](#workload-disk-i-o)
|
||||
- [Pod Metrics](#pod-metrics)
|
||||
- [Pod CPU Utilization](#pod-cpu-utilization)
|
||||
- [Pod Memory Utilization](#pod-memory-utilization)
|
||||
- [Pod Network Packets](#pod-network-packets)
|
||||
- [Pod Network I/O](#pod-network-i-o)
|
||||
- [Pod Disk I/O](#pod-disk-i-o)
|
||||
- [Container Metrics](#container-metrics)
|
||||
- [Container CPU Utilization](#container-cpu-utilization)
|
||||
- [Container Memory Utilization](#container-memory-utilization)
|
||||
- [Container Disk I/O](#container-disk-i-o)
|
||||
|
||||
<!-- /TOC -->
|
||||
|
||||
# Cluster Metrics
|
||||
|
||||
### Cluster CPU Utilization
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | `1 - (avg(irate(node_cpu_seconds_total{mode="idle"}[5m])) by (instance))` |
|
||||
| Summary | `1 - (avg(irate(node_cpu_seconds_total{mode="idle"}[5m])))` |
|
||||
|
||||
### Cluster Load Average
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | <table><tr><td>load1</td><td>`sum(node_load1) by (instance) / count(node_cpu_seconds_total{mode="system"}) by (instance)`</td></tr><tr><td>load5</td><td>`sum(node_load5) by (instance) / count(node_cpu_seconds_total{mode="system"}) by (instance)`</td></tr><tr><td>load15</td><td>`sum(node_load15) by (instance) / count(node_cpu_seconds_total{mode="system"}) by (instance)`</td></tr></table> |
|
||||
| Summary | <table><tr><td>load1</td><td>`sum(node_load1) by (instance) / count(node_cpu_seconds_total{mode="system"})`</td></tr><tr><td>load5</td><td>`sum(node_load5) by (instance) / count(node_cpu_seconds_total{mode="system"})`</td></tr><tr><td>load15</td><td>`sum(node_load15) by (instance) / count(node_cpu_seconds_total{mode="system"})`</td></tr></table> |
|
||||
|
||||
### Cluster Memory Utilization
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | `1 - sum(node_memory_MemAvailable_bytes) by (instance) / sum(node_memory_MemTotal_bytes) by (instance)` |
|
||||
| Summary | `1 - sum(node_memory_MemAvailable_bytes) / sum(node_memory_MemTotal_bytes)` |
|
||||
|
||||
### Cluster Disk Utilization
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | `(sum(node_filesystem_size_bytes{device!="rootfs"}) by (instance) - sum(node_filesystem_free_bytes{device!="rootfs"}) by (instance)) / sum(node_filesystem_size_bytes{device!="rootfs"}) by (instance)` |
|
||||
| Summary | `(sum(node_filesystem_size_bytes{device!="rootfs"}) - sum(node_filesystem_free_bytes{device!="rootfs"})) / sum(node_filesystem_size_bytes{device!="rootfs"})` |
|
||||
|
||||
### Cluster Disk I/O
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | <table><tr><td>read</td><td>`sum(rate(node_disk_read_bytes_total[5m])) by (instance)`</td></tr><tr><td>written</td><td>`sum(rate(node_disk_written_bytes_total[5m])) by (instance)`</td></tr></table> |
|
||||
| Summary | <table><tr><td>read</td><td>`sum(rate(node_disk_read_bytes_total[5m]))`</td></tr><tr><td>written</td><td>`sum(rate(node_disk_written_bytes_total[5m]))`</td></tr></table> |
|
||||
|
||||
### Cluster Network Packets
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | <table><tr><td>receive-dropped</td><td><code>sum(rate(node_network_receive_drop_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*"}[5m])) by (instance)</code></td></tr><tr><td>receive-errs</td><td><code>sum(rate(node_network_receive_errs_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*"}[5m])) by (instance)</code></td></tr><tr><td>receive-packets</td><td><code>sum(rate(node_network_receive_packets_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*"}[5m])) by (instance)</code></td></tr><tr><td>transmit-dropped</td><td><code>sum(rate(node_network_transmit_drop_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*"}[5m])) by (instance)</code></td></tr><tr><td>transmit-errs</td><td><code>sum(rate(node_network_transmit_errs_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*"}[5m])) by (instance)</code></td></tr><tr><td>transmit-packets</td><td><code>sum(rate(node_network_transmit_packets_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*"}[5m])) by (instance)</code></td></tr></table> |
|
||||
| Summary | <table><tr><td>receive-dropped</td><td><code>sum(rate(node_network_receive_drop_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*"}[5m]))</code></td></tr><tr><td>receive-errs</td><td><code>sum(rate(node_network_receive_errs_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*"}[5m]))</code></td></tr><tr><td>receive-packets</td><td><code>sum(rate(node_network_receive_packets_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*"}[5m]))</code></td></tr><tr><td>transmit-dropped</td><td><code>sum(rate(node_network_transmit_drop_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*"}[5m]))</code></td></tr><tr><td>transmit-errs</td><td><code>sum(rate(node_network_transmit_errs_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*"}[5m]))</code></td></tr><tr><td>transmit-packets</td><td><code>sum(rate(node_network_transmit_packets_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*"}[5m]))</code></td></tr></table> |
|
||||
|
||||
### Cluster Network I/O
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | <table><tr><td>receive</td><td><code>sum(rate(node_network_receive_bytes_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*"}[5m])) by (instance)</code></td></tr><tr><td>transmit</td><td><code>sum(rate(node_network_transmit_bytes_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*"}[5m])) by (instance)</code></td></tr></table> |
|
||||
| Summary | <table><tr><td>receive</td><td><code>sum(rate(node_network_receive_bytes_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*"}[5m]))</code></td></tr><tr><td>transmit</td><td><code>sum(rate(node_network_transmit_bytes_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*"}[5m]))</code></td></tr></table> |
|
||||
|
||||
# Node Metrics
|
||||
|
||||
### Node CPU Utilization
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | `avg(irate(node_cpu_seconds_total{mode!="idle", instance=~"$instance"}[5m])) by (mode)` |
|
||||
| Summary | `1 - (avg(irate(node_cpu_seconds_total{mode="idle", instance=~"$instance"}[5m])))` |
|
||||
|
||||
### Node Load Average
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | <table><tr><td>load1</td><td>`sum(node_load1{instance=~"$instance"}) / count(node_cpu_seconds_total{mode="system",instance=~"$instance"})`</td></tr><tr><td>load5</td><td>`sum(node_load5{instance=~"$instance"}) / count(node_cpu_seconds_total{mode="system",instance=~"$instance"})`</td></tr><tr><td>load15</td><td>`sum(node_load15{instance=~"$instance"}) / count(node_cpu_seconds_total{mode="system",instance=~"$instance"})`</td></tr></table> |
|
||||
| Summary | <table><tr><td>load1</td><td>`sum(node_load1{instance=~"$instance"}) / count(node_cpu_seconds_total{mode="system",instance=~"$instance"})`</td></tr><tr><td>load5</td><td>`sum(node_load5{instance=~"$instance"}) / count(node_cpu_seconds_total{mode="system",instance=~"$instance"})`</td></tr><tr><td>load15</td><td>`sum(node_load15{instance=~"$instance"}) / count(node_cpu_seconds_total{mode="system",instance=~"$instance"})`</td></tr></table> |
|
||||
|
||||
### Node Memory Utilization
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | `1 - sum(node_memory_MemAvailable_bytes{instance=~"$instance"}) / sum(node_memory_MemTotal_bytes{instance=~"$instance"})` |
|
||||
| Summary | `1 - sum(node_memory_MemAvailable_bytes{instance=~"$instance"}) / sum(node_memory_MemTotal_bytes{instance=~"$instance"}) ` |
|
||||
|
||||
### Node Disk Utilization
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | `(sum(node_filesystem_size_bytes{device!="rootfs",instance=~"$instance"}) by (device) - sum(node_filesystem_free_bytes{device!="rootfs",instance=~"$instance"}) by (device)) / sum(node_filesystem_size_bytes{device!="rootfs",instance=~"$instance"}) by (device)` |
|
||||
| Summary | `(sum(node_filesystem_size_bytes{device!="rootfs",instance=~"$instance"}) - sum(node_filesystem_free_bytes{device!="rootfs",instance=~"$instance"})) / sum(node_filesystem_size_bytes{device!="rootfs",instance=~"$instance"})` |
|
||||
|
||||
### Node Disk I/O
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | <table><tr><td>read</td><td>`sum(rate(node_disk_read_bytes_total{instance=~"$instance"}[5m]))`</td></tr><tr><td>written</td><td>`sum(rate(node_disk_written_bytes_total{instance=~"$instance"}[5m]))`</td></tr></table> |
|
||||
| Summary | <table><tr><td>read</td><td>`sum(rate(node_disk_read_bytes_total{instance=~"$instance"}[5m]))`</td></tr><tr><td>written</td><td>`sum(rate(node_disk_written_bytes_total{instance=~"$instance"}[5m]))`</td></tr></table> |
|
||||
|
||||
### Node Network Packets
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | <table><tr><td>receive-dropped</td><td><code>sum(rate(node_network_receive_drop_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*",instance=~"$instance"}[5m])) by (device)</code></td></tr><tr><td>receive-errs</td><td><code>sum(rate(node_network_receive_errs_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*",instance=~"$instance"}[5m])) by (device)</code></td></tr><tr><td>receive-packets</td><td><code>sum(rate(node_network_receive_packets_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*",instance=~"$instance"}[5m])) by (device)</code></td></tr><tr><td>transmit-dropped</td><td><code>sum(rate(node_network_transmit_drop_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*",instance=~"$instance"}[5m])) by (device)</code></td></tr><tr><td>transmit-errs</td><td><code>sum(rate(node_network_transmit_errs_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*",instance=~"$instance"}[5m])) by (device)</code></td></tr><tr><td>transmit-packets</td><td><code>sum(rate(node_network_transmit_packets_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*",instance=~"$instance"}[5m])) by (device)</code></td></tr></table> |
|
||||
| Summary | <table><tr><td>receive-dropped</td><td><code>sum(rate(node_network_receive_drop_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*",instance=~"$instance"}[5m]))</code></td></tr><tr><td>receive-errs</td><td><code>sum(rate(node_network_receive_errs_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*",instance=~"$instance"}[5m]))</code></td></tr><tr><td>receive-packets</td><td><code>sum(rate(node_network_receive_packets_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*",instance=~"$instance"}[5m]))</code></td></tr><tr><td>transmit-dropped</td><td><code>sum(rate(node_network_transmit_drop_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*",instance=~"$instance"}[5m]))</code></td></tr><tr><td>transmit-errs</td><td><code>sum(rate(node_network_transmit_errs_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*",instance=~"$instance"}[5m]))</code></td></tr><tr><td>transmit-packets</td><td><code>sum(rate(node_network_transmit_packets_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*",instance=~"$instance"}[5m]))</code></td></tr></table> |
|
||||
|
||||
### Node Network I/O
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | <table><tr><td>receive</td><td><code>sum(rate(node_network_receive_bytes_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*",instance=~"$instance"}[5m])) by (device)</code></td></tr><tr><td>transmit</td><td><code>sum(rate(node_network_transmit_bytes_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*",instance=~"$instance"}[5m])) by (device)</code></td></tr></table> |
|
||||
| Summary | <table><tr><td>receive</td><td><code>sum(rate(node_network_receive_bytes_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*",instance=~"$instance"}[5m]))</code></td></tr><tr><td>transmit</td><td><code>sum(rate(node_network_transmit_bytes_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*",instance=~"$instance"}[5m]))</code></td></tr></table> |
|
||||
|
||||
# Etcd Metrics
|
||||
|
||||
### Etcd Has a Leader
|
||||
|
||||
`max(etcd_server_has_leader)`
|
||||
|
||||
### Number of Times the Leader Changes
|
||||
|
||||
`max(etcd_server_leader_changes_seen_total)`
|
||||
|
||||
### Number of Failed Proposals
|
||||
|
||||
`sum(etcd_server_proposals_failed_total)`
|
||||
|
||||
### GRPC Client Traffic
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | <table><tr><td>in</td><td>`sum(rate(etcd_network_client_grpc_received_bytes_total[5m])) by (instance)`</td></tr><tr><td>out</td><td>`sum(rate(etcd_network_client_grpc_sent_bytes_total[5m])) by (instance)`</td></tr></table> |
|
||||
| Summary | <table><tr><td>in</td><td>`sum(rate(etcd_network_client_grpc_received_bytes_total[5m]))`</td></tr><tr><td>out</td><td>`sum(rate(etcd_network_client_grpc_sent_bytes_total[5m]))`</td></tr></table> |
|
||||
|
||||
### Peer Traffic
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | <table><tr><td>in</td><td>`sum(rate(etcd_network_peer_received_bytes_total[5m])) by (instance)`</td></tr><tr><td>out</td><td>`sum(rate(etcd_network_peer_sent_bytes_total[5m])) by (instance)`</td></tr></table> |
|
||||
| Summary | <table><tr><td>in</td><td>`sum(rate(etcd_network_peer_received_bytes_total[5m]))`</td></tr><tr><td>out</td><td>`sum(rate(etcd_network_peer_sent_bytes_total[5m]))`</td></tr></table> |
|
||||
|
||||
### DB Size
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | `sum(etcd_debugging_mvcc_db_total_size_in_bytes) by (instance)` |
|
||||
| Summary | `sum(etcd_debugging_mvcc_db_total_size_in_bytes)` |
|
||||
|
||||
### Active Streams
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | <table><tr><td>lease-watch</td><td>`sum(grpc_server_started_total{grpc_service="etcdserverpb.Lease",grpc_type="bidi_stream"}) by (instance) - sum(grpc_server_handled_total{grpc_service="etcdserverpb.Lease",grpc_type="bidi_stream"}) by (instance)`</td></tr><tr><td>watch</td><td>`sum(grpc_server_started_total{grpc_service="etcdserverpb.Watch",grpc_type="bidi_stream"}) by (instance) - sum(grpc_server_handled_total{grpc_service="etcdserverpb.Watch",grpc_type="bidi_stream"}) by (instance)`</td></tr></table> |
|
||||
| Summary | <table><tr><td>lease-watch</td><td>`sum(grpc_server_started_total{grpc_service="etcdserverpb.Lease",grpc_type="bidi_stream"}) - sum(grpc_server_handled_total{grpc_service="etcdserverpb.Lease",grpc_type="bidi_stream"})`</td></tr><tr><td>watch</td><td>`sum(grpc_server_started_total{grpc_service="etcdserverpb.Watch",grpc_type="bidi_stream"}) - sum(grpc_server_handled_total{grpc_service="etcdserverpb.Watch",grpc_type="bidi_stream"})`</td></tr></table> |
|
||||
|
||||
### Raft Proposals
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | <table><tr><td>applied</td><td>`sum(increase(etcd_server_proposals_applied_total[5m])) by (instance)`</td></tr><tr><td>committed</td><td>`sum(increase(etcd_server_proposals_committed_total[5m])) by (instance)`</td></tr><tr><td>pending</td><td>`sum(increase(etcd_server_proposals_pending[5m])) by (instance)`</td></tr><tr><td>failed</td><td>`sum(increase(etcd_server_proposals_failed_total[5m])) by (instance)`</td></tr></table> |
|
||||
| Summary | <table><tr><td>applied</td><td>`sum(increase(etcd_server_proposals_applied_total[5m]))`</td></tr><tr><td>committed</td><td>`sum(increase(etcd_server_proposals_committed_total[5m]))`</td></tr><tr><td>pending</td><td>`sum(increase(etcd_server_proposals_pending[5m]))`</td></tr><tr><td>failed</td><td>`sum(increase(etcd_server_proposals_failed_total[5m]))`</td></tr></table> |
|
||||
|
||||
### RPC Rate
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | <table><tr><td>total</td><td>`sum(rate(grpc_server_started_total{grpc_type="unary"}[5m])) by (instance)`</td></tr><tr><td>fail</td><td>`sum(rate(grpc_server_handled_total{grpc_type="unary",grpc_code!="OK"}[5m])) by (instance)`</td></tr></table> |
|
||||
| Summary | <table><tr><td>total</td><td>`sum(rate(grpc_server_started_total{grpc_type="unary"}[5m]))`</td></tr><tr><td>fail</td><td>`sum(rate(grpc_server_handled_total{grpc_type="unary",grpc_code!="OK"}[5m]))`</td></tr></table> |
|
||||
|
||||
### Disk Operations
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | <table><tr><td>commit-called-by-backend</td><td>`sum(rate(etcd_disk_backend_commit_duration_seconds_sum[1m])) by (instance)`</td></tr><tr><td>fsync-called-by-wal</td><td>`sum(rate(etcd_disk_wal_fsync_duration_seconds_sum[1m])) by (instance)`</td></tr></table> |
|
||||
| Summary | <table><tr><td>commit-called-by-backend</td><td>`sum(rate(etcd_disk_backend_commit_duration_seconds_sum[1m]))`</td></tr><tr><td>fsync-called-by-wal</td><td>`sum(rate(etcd_disk_wal_fsync_duration_seconds_sum[1m]))`</td></tr></table> |
|
||||
|
||||
### Disk Sync Duration
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | <table><tr><td>wal</td><td>`histogram_quantile(0.99, sum(rate(etcd_disk_wal_fsync_duration_seconds_bucket[5m])) by (instance, le))`</td></tr><tr><td>db</td><td>`histogram_quantile(0.99, sum(rate(etcd_disk_backend_commit_duration_seconds_bucket[5m])) by (instance, le))`</td></tr></table> |
|
||||
| Summary | <table><tr><td>wal</td><td>`sum(histogram_quantile(0.99, sum(rate(etcd_disk_wal_fsync_duration_seconds_bucket[5m])) by (instance, le)))`</td></tr><tr><td>db</td><td>`sum(histogram_quantile(0.99, sum(rate(etcd_disk_backend_commit_duration_seconds_bucket[5m])) by (instance, le)))`</td></tr></table> |
|
||||
|
||||
# Kubernetes Components Metrics
|
||||
|
||||
### API Server Request Latency
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | `avg(apiserver_request_latencies_sum / apiserver_request_latencies_count) by (instance, verb) /1e+06` |
|
||||
| Summary | `avg(apiserver_request_latencies_sum / apiserver_request_latencies_count) by (instance) /1e+06` |
|
||||
|
||||
### API Server Request Rate
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | `sum(rate(apiserver_request_count[5m])) by (instance, code)` |
|
||||
| Summary | `sum(rate(apiserver_request_count[5m])) by (instance)` |
|
||||
|
||||
### Scheduling Failed Pods
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | `sum(kube_pod_status_scheduled{condition="false"})` |
|
||||
| Summary | `sum(kube_pod_status_scheduled{condition="false"})` |
|
||||
|
||||
### Controller Manager Queue Depth
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | <table><tr><td>volumes</td><td>`sum(volumes_depth) by instance`</td></tr><tr><td>deployment</td><td>`sum(deployment_depth) by instance`</td></tr><tr><td>replicaset</td><td>`sum(replicaset_depth) by instance`</td></tr><tr><td>service</td><td>`sum(service_depth) by instance`</td></tr><tr><td>serviceaccount</td><td>`sum(serviceaccount_depth) by instance`</td></tr><tr><td>endpoint</td><td>`sum(endpoint_depth) by instance`</td></tr><tr><td>daemonset</td><td>`sum(daemonset_depth) by instance`</td></tr><tr><td>statefulset</td><td>`sum(statefulset_depth) by instance`</td></tr><tr><td>replicationmanager</td><td>`sum(replicationmanager_depth) by instance`</td></tr></table> |
|
||||
| Summary | <table><tr><td>volumes</td><td>`sum(volumes_depth)`</td></tr><tr><td>deployment</td><td>`sum(deployment_depth)`</td></tr><tr><td>replicaset</td><td>`sum(replicaset_depth)`</td></tr><tr><td>service</td><td>`sum(service_depth)`</td></tr><tr><td>serviceaccount</td><td>`sum(serviceaccount_depth)`</td></tr><tr><td>endpoint</td><td>`sum(endpoint_depth)`</td></tr><tr><td>daemonset</td><td>`sum(daemonset_depth)`</td></tr><tr><td>statefulset</td><td>`sum(statefulset_depth)`</td></tr><tr><td>replicationmanager</td><td>`sum(replicationmanager_depth)`</td></tr></table> |
|
||||
|
||||
### Scheduler E2E Scheduling Latency
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | `histogram_quantile(0.99, sum(scheduler_e2e_scheduling_latency_microseconds_bucket) by (le, instance)) / 1e+06` |
|
||||
| Summary | `sum(histogram_quantile(0.99, sum(scheduler_e2e_scheduling_latency_microseconds_bucket) by (le, instance)) / 1e+06)` |
|
||||
|
||||
### Scheduler Preemption Attempts
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | `sum(rate(scheduler_total_preemption_attempts[5m])) by (instance)` |
|
||||
| Summary | `sum(rate(scheduler_total_preemption_attempts[5m]))` |
|
||||
|
||||
### Ingress Controller Connections
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | <table><tr><td>reading</td><td>`sum(nginx_ingress_controller_nginx_process_connections{state="reading"}) by (instance)`</td></tr><tr><td>waiting</td><td>`sum(nginx_ingress_controller_nginx_process_connections{state="waiting"}) by (instance)`</td></tr><tr><td>writing</td><td>`sum(nginx_ingress_controller_nginx_process_connections{state="writing"}) by (instance)`</td></tr><tr><td>accepted</td><td>`sum(ceil(increase(nginx_ingress_controller_nginx_process_connections_total{state="accepted"}[5m]))) by (instance)`</td></tr><tr><td>active</td><td>`sum(ceil(increase(nginx_ingress_controller_nginx_process_connections_total{state="active"}[5m]))) by (instance)`</td></tr><tr><td>handled</td><td>`sum(ceil(increase(nginx_ingress_controller_nginx_process_connections_total{state="handled"}[5m]))) by (instance)`</td></tr></table> |
|
||||
| Summary | <table><tr><td>reading</td><td>`sum(nginx_ingress_controller_nginx_process_connections{state="reading"})`</td></tr><tr><td>waiting</td><td>`sum(nginx_ingress_controller_nginx_process_connections{state="waiting"})`</td></tr><tr><td>writing</td><td>`sum(nginx_ingress_controller_nginx_process_connections{state="writing"})`</td></tr><tr><td>accepted</td><td>`sum(ceil(increase(nginx_ingress_controller_nginx_process_connections_total{state="accepted"}[5m])))`</td></tr><tr><td>active</td><td>`sum(ceil(increase(nginx_ingress_controller_nginx_process_connections_total{state="active"}[5m])))`</td></tr><tr><td>handled</td><td>`sum(ceil(increase(nginx_ingress_controller_nginx_process_connections_total{state="handled"}[5m])))`</td></tr></table> |
|
||||
|
||||
### Ingress Controller Request Process Time
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | `topk(10, histogram_quantile(0.95,sum by (le, host, path)(rate(nginx_ingress_controller_request_duration_seconds_bucket{host!="_"}[5m]))))` |
|
||||
| Summary | `topk(10, histogram_quantile(0.95,sum by (le, host)(rate(nginx_ingress_controller_request_duration_seconds_bucket{host!="_"}[5m]))))` |
|
||||
|
||||
# Rancher Logging Metrics
|
||||
|
||||
|
||||
### Fluentd Buffer Queue Rate
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | `sum(rate(fluentd_output_status_buffer_queue_length[5m])) by (instance)` |
|
||||
| Summary | `sum(rate(fluentd_output_status_buffer_queue_length[5m]))` |
|
||||
|
||||
### Fluentd Input Rate
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | `sum(rate(fluentd_input_status_num_records_total[5m])) by (instance)` |
|
||||
| Summary | `sum(rate(fluentd_input_status_num_records_total[5m]))` |
|
||||
|
||||
### Fluentd Output Errors Rate
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | `sum(rate(fluentd_output_status_num_errors[5m])) by (type)` |
|
||||
| Summary | `sum(rate(fluentd_output_status_num_errors[5m]))` |
|
||||
|
||||
### Fluentd Output Rate
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | `sum(rate(fluentd_output_status_num_records_total[5m])) by (instance)` |
|
||||
| Summary | `sum(rate(fluentd_output_status_num_records_total[5m]))` |
|
||||
|
||||
# Workload Metrics
|
||||
|
||||
### Workload CPU Utilization
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | <table><tr><td>cfs throttled seconds</td><td>`sum(rate(container_cpu_cfs_throttled_seconds_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m])) by (pod_name)`</td></tr><tr><td>user seconds</td><td>`sum(rate(container_cpu_user_seconds_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m])) by (pod_name)`</td></tr><tr><td>system seconds</td><td>`sum(rate(container_cpu_system_seconds_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m])) by (pod_name)`</td></tr><tr><td>usage seconds</td><td>`sum(rate(container_cpu_usage_seconds_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m])) by (pod_name)`</td></tr></table> |
|
||||
| Summary | <table><tr><td>cfs throttled seconds</td><td>`sum(rate(container_cpu_cfs_throttled_seconds_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m]))`</td></tr><tr><td>user seconds</td><td>`sum(rate(container_cpu_user_seconds_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m]))`</td></tr><tr><td>system seconds</td><td>`sum(rate(container_cpu_system_seconds_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m]))`</td></tr><tr><td>usage seconds</td><td>`sum(rate(container_cpu_usage_seconds_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m]))`</td></tr></table> |
|
||||
|
||||
### Workload Memory Utilization
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | `sum(container_memory_working_set_bytes{namespace="$namespace",pod_name=~"$podName", container_name!=""}) by (pod_name)` |
|
||||
| Summary | `sum(container_memory_working_set_bytes{namespace="$namespace",pod_name=~"$podName", container_name!=""})` |
|
||||
|
||||
### Workload Network Packets
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | <table><tr><td>receive-packets</td><td>`sum(rate(container_network_receive_packets_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m])) by (pod_name)`</td></tr><tr><td>receive-dropped</td><td>`sum(rate(container_network_receive_packets_dropped_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m])) by (pod_name)`</td></tr><tr><td>receive-errors</td><td>`sum(rate(container_network_receive_errors_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m])) by (pod_name)`</td></tr><tr><td>transmit-packets</td><td>`sum(rate(container_network_transmit_packets_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m])) by (pod_name)`</td></tr><tr><td>transmit-dropped</td><td>`sum(rate(container_network_transmit_packets_dropped_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m])) by (pod_name)`</td></tr><tr><td>transmit-errors</td><td>`sum(rate(container_network_transmit_errors_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m])) by (pod_name)`</td></tr></table> |
|
||||
| Summary | <table><tr><td>receive-packets</td><td>`sum(rate(container_network_receive_packets_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m]))`</td></tr><tr><td>receive-dropped</td><td>`sum(rate(container_network_receive_packets_dropped_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m]))`</td></tr><tr><td>receive-errors</td><td>`sum(rate(container_network_receive_errors_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m]))`</td></tr><tr><td>transmit-packets</td><td>`sum(rate(container_network_transmit_packets_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m]))`</td></tr><tr><td>transmit-dropped</td><td>`sum(rate(container_network_transmit_packets_dropped_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m]))`</td></tr><tr><td>transmit-errors</td><td>`sum(rate(container_network_transmit_errors_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m]))`</td></tr></table> |
|
||||
|
||||
### Workload Network I/O
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | <table><tr><td>receive</td><td>`sum(rate(container_network_receive_bytes_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m])) by (pod_name)`</td></tr><tr><td>transmit</td><td>`sum(rate(container_network_transmit_bytes_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m])) by (pod_name)`</td></tr></table> |
|
||||
| Summary | <table><tr><td>receive</td><td>`sum(rate(container_network_receive_bytes_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m]))`</td></tr><tr><td>transmit</td><td>`sum(rate(container_network_transmit_bytes_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m]))`</td></tr></table> |
|
||||
|
||||
### Workload Disk I/O
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | <table><tr><td>read</td><td>`sum(rate(container_fs_reads_bytes_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m])) by (pod_name)`</td></tr><tr><td>write</td><td>`sum(rate(container_fs_writes_bytes_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m])) by (pod_name)`</td></tr></table> |
|
||||
| Summary | <table><tr><td>read</td><td>`sum(rate(container_fs_reads_bytes_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m]))`</td></tr><tr><td>write</td><td>`sum(rate(container_fs_writes_bytes_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m]))`</td></tr></table> |
|
||||
|
||||
# Pod Metrics
|
||||
|
||||
### Pod CPU Utilization
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | <table><tr><td>cfs throttled seconds</td><td>`sum(rate(container_cpu_cfs_throttled_seconds_total{container_name!="POD",namespace="$namespace",pod_name="$podName", container_name!=""}[5m])) by (container_name)`</td></tr><tr><td>usage seconds</td><td>`sum(rate(container_cpu_usage_seconds_total{container_name!="POD",namespace="$namespace",pod_name="$podName", container_name!=""}[5m])) by (container_name)`</td></tr><tr><td>system seconds</td><td>`sum(rate(container_cpu_system_seconds_total{container_name!="POD",namespace="$namespace",pod_name="$podName", container_name!=""}[5m])) by (container_name)`</td></tr><tr><td>user seconds</td><td>`sum(rate(container_cpu_user_seconds_total{container_name!="POD",namespace="$namespace",pod_name="$podName", container_name!=""}[5m])) by (container_name)`</td></tr></table> |
|
||||
| Summary | <table><tr><td>cfs throttled seconds</td><td>`sum(rate(container_cpu_cfs_throttled_seconds_total{container_name!="POD",namespace="$namespace",pod_name="$podName", container_name!=""}[5m]))`</td></tr><tr><td>usage seconds</td><td>`sum(rate(container_cpu_usage_seconds_total{container_name!="POD",namespace="$namespace",pod_name="$podName", container_name!=""}[5m]))`</td></tr><tr><td>system seconds</td><td>`sum(rate(container_cpu_system_seconds_total{container_name!="POD",namespace="$namespace",pod_name="$podName", container_name!=""}[5m]))`</td></tr><tr><td>user seconds</td><td>`sum(rate(container_cpu_user_seconds_total{container_name!="POD",namespace="$namespace",pod_name="$podName", container_name!=""}[5m]))`</td></tr></table> |
|
||||
|
||||
### Pod Memory Utilization
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | `sum(container_memory_working_set_bytes{container_name!="POD",namespace="$namespace",pod_name="$podName",container_name!=""}) by (container_name)` |
|
||||
| Summary | `sum(container_memory_working_set_bytes{container_name!="POD",namespace="$namespace",pod_name="$podName",container_name!=""})` |
|
||||
|
||||
### Pod Network Packets
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | <table><tr><td>receive-packets</td><td>`sum(rate(container_network_receive_packets_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`</td></tr><tr><td>receive-dropped</td><td>`sum(rate(container_network_receive_packets_dropped_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`</td></tr><tr><td>receive-errors</td><td>`sum(rate(container_network_receive_errors_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`</td></tr><tr><td>transmit-packets</td><td>`sum(rate(container_network_transmit_packets_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`</td></tr><tr><td>transmit-dropped</td><td>`sum(rate(container_network_transmit_packets_dropped_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`</td></tr><tr><td>transmit-errors</td><td>`sum(rate(container_network_transmit_errors_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`</td></tr></table> |
|
||||
| Summary | <table><tr><td>receive-packets</td><td>`sum(rate(container_network_receive_packets_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`</td></tr><tr><td>receive-dropped</td><td>`sum(rate(container_network_receive_packets_dropped_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`</td></tr><tr><td>receive-errors</td><td>`sum(rate(container_network_receive_errors_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`</td></tr><tr><td>transmit-packets</td><td>`sum(rate(container_network_transmit_packets_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`</td></tr><tr><td>transmit-dropped</td><td>`sum(rate(container_network_transmit_packets_dropped_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`</td></tr><tr><td>transmit-errors</td><td>`sum(rate(container_network_transmit_errors_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`</td></tr></table> |
|
||||
|
||||
### Pod Network I/O
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | <table><tr><td>receive</td><td>`sum(rate(container_network_receive_bytes_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`</td></tr><tr><td>transmit</td><td>`sum(rate(container_network_transmit_bytes_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`</td></tr></table> |
|
||||
| Summary | <table><tr><td>receive</td><td>`sum(rate(container_network_receive_bytes_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`</td></tr><tr><td>transmit</td><td>`sum(rate(container_network_transmit_bytes_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`</td></tr></table> |
|
||||
|
||||
### Pod Disk I/O
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| Detail | <table><tr><td>read</td><td>`sum(rate(container_fs_reads_bytes_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m])) by (container_name)`</td></tr><tr><td>write</td><td>`sum(rate(container_fs_writes_bytes_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m])) by (container_name)`</td></tr></table> |
|
||||
| Summary | <table><tr><td>read</td><td>`sum(rate(container_fs_reads_bytes_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`</td></tr><tr><td>write</td><td>`sum(rate(container_fs_writes_bytes_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`</td></tr></table> |
|
||||
|
||||
# Container Metrics
|
||||
|
||||
### Container CPU Utilization
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| cfs throttled seconds | `sum(rate(container_cpu_cfs_throttled_seconds_total{namespace="$namespace",pod_name="$podName",container_name="$containerName"}[5m]))` |
|
||||
| usage seconds | `sum(rate(container_cpu_usage_seconds_total{namespace="$namespace",pod_name="$podName",container_name="$containerName"}[5m]))` |
|
||||
| system seconds | `sum(rate(container_cpu_system_seconds_total{namespace="$namespace",pod_name="$podName",container_name="$containerName"}[5m]))` |
|
||||
| user seconds | `sum(rate(container_cpu_user_seconds_total{namespace="$namespace",pod_name="$podName",container_name="$containerName"}[5m]))` |
|
||||
|
||||
### Container Memory Utilization
|
||||
|
||||
`sum(container_memory_working_set_bytes{namespace="$namespace",pod_name="$podName",container_name="$containerName"})`
|
||||
|
||||
### Container Disk I/O
|
||||
|
||||
| Catalog | Expression |
|
||||
| --- | --- |
|
||||
| read | `sum(rate(container_fs_reads_bytes_total{namespace="$namespace",pod_name="$podName",container_name="$containerName"}[5m]))` |
|
||||
| write | `sum(rate(container_fs_writes_bytes_total{namespace="$namespace",pod_name="$podName",container_name="$containerName"}[5m]))` |
|
||||
+84
@@ -0,0 +1,84 @@
|
||||
---
|
||||
title: Project Monitoring
|
||||
weight: 2
|
||||
aliases:
|
||||
- /rancher/v2.0-v2.4/en/project-admin/tools/monitoring
|
||||
- /rancher/v2.0-v2.4/en/monitoring-alerting/v2.0.x-v2.4.x/monitoring/project-monitoring
|
||||
- /rancher/v2.0-v2.4/en/monitoring-alerting/v2.0.x-v2.4.x/cluster-monitoring/project-monitoring
|
||||
---
|
||||
|
||||
_Available as of v2.2.4_
|
||||
|
||||
Using Rancher, you can monitor the state and processes of your cluster nodes, Kubernetes components, and software deployments through integration with [Prometheus](https://prometheus.io/), a leading open-source monitoring solution.
|
||||
|
||||
This section covers the following topics:
|
||||
|
||||
- [Monitoring scope](#monitoring-scope)
|
||||
- [Permissions to configure project monitoring](#permissions-to-configure-project-monitoring)
|
||||
- [Enabling project monitoring](#enabling-project-monitoring)
|
||||
- [Project-level monitoring resource requirements](#project-level-monitoring-resource-requirements)
|
||||
- [Project metrics](#project-metrics)
|
||||
|
||||
### Monitoring Scope
|
||||
|
||||
Using Prometheus, you can monitor Rancher at both the [cluster level]({{<baseurl>}}/rancher/v2.0-v2.4/en/monitoring-alerting/legacy/monitoring/cluster-monitoring/) and project level. For each cluster and project that is enabled for monitoring, Rancher deploys a Prometheus server.
|
||||
|
||||
- [Cluster monitoring]({{<baseurl>}}/rancher/v2.0-v2.4/en/monitoring-alerting/legacy/monitoring/cluster-monitoring/) allows you to view the health of your Kubernetes cluster. Prometheus collects metrics from the cluster components below, which you can view in graphs and charts.
|
||||
|
||||
- Kubernetes control plane
|
||||
- etcd database
|
||||
- All nodes (including workers)
|
||||
|
||||
- Project monitoring allows you to view the state of pods running in a given project. Prometheus collects metrics from the project's deployed HTTP and TCP/UDP workloads.
|
||||
|
||||
### Permissions to Configure Project Monitoring
|
||||
|
||||
Only [administrators]({{<baseurl>}}/rancher/v2.0-v2.4/en/admin-settings/rbac/global-permissions/), [cluster owners or members]({{<baseurl>}}/rancher/v2.0-v2.4/en/admin-settings/rbac/cluster-project-roles/#cluster-roles), or [project owners]({{<baseurl>}}/rancher/v2.0-v2.4/en/admin-settings/rbac/cluster-project-roles/#project-roles) can configure project level monitoring. Project members can only view monitoring metrics.
|
||||
|
||||
### Enabling Project Monitoring
|
||||
|
||||
> **Prerequisite:** Cluster monitoring must be [enabled.]({{<baseurl>}}/rancher/v2.0-v2.4/en/monitoring-alerting/legacy/monitoring/cluster-monitoring/)
|
||||
|
||||
1. Go to the project where monitoring should be enabled. Note: When cluster monitoring is enabled, monitoring is also enabled by default in the **System** project.
|
||||
|
||||
1. Select **Tools > Monitoring** in the navigation bar.
|
||||
|
||||
1. Select **Enable** to show the [Prometheus configuration options]({{<baseurl>}}/rancher/v2.0-v2.4/en/monitoring-alerting/legacy/monitoring/cluster-monitoring/prometheus/). Enter in your desired configuration options.
|
||||
|
||||
1. Click **Save**.
|
||||
|
||||
### Project-Level Monitoring Resource Requirements
|
||||
|
||||
Container| CPU - Request | Mem - Request | CPU - Limit | Mem - Limit | Configurable
|
||||
---------|---------------|---------------|-------------|-------------|-------------
|
||||
Prometheus|750m| 750Mi | 1000m | 1000Mi | Yes
|
||||
Grafana | 100m | 100Mi | 200m | 200Mi | No
|
||||
|
||||
|
||||
**Result:** A single application,`project-monitoring`, is added as an [application]({{<baseurl>}}/rancher/v2.0-v2.4/en/catalog/apps/) to the project. After the application is `active`, you can start viewing project metrics through the [Rancher dashboard]({{<baseurl>}}/rancher/v2.0-v2.4/en/monitoring-alerting/legacy/monitoring/cluster-monitoring/) or directly from Grafana.
|
||||
|
||||
> The default username and password for the Grafana instance will be `admin/admin`. However, Grafana dashboards are served via the Rancher authentication proxy, so only users who are currently authenticated into the Rancher server have access to the Grafana dashboard.
|
||||
|
||||
### Project Metrics
|
||||
[Workload metrics]({{<baseurl>}}/rancher/v2.0-v2.4/en/monitoring-alerting/v2.0.x-v2.4.x/cluster-monitoring/expression/#workload-metrics) are available for the project if monitoring is enabled at the [cluster level]({{<baseurl>}}/rancher/v2.0-v2.4/en/monitoring-alerting/legacy/monitoring/cluster-monitoring/) and at the [project level.](#enabling-project-monitoring)
|
||||
|
||||
You can monitor custom metrics from any [exporters.](https://prometheus.io/docs/instrumenting/exporters/) You can also expose some custom endpoints on deployments without needing to configure Prometheus for your project.
|
||||
|
||||
> **Example:**
|
||||
> A [Redis](https://redis.io/) application is deployed in the namespace `redis-app` in the project `Datacenter`. It is monitored via [Redis exporter](https://github.com/oliver006/redis_exporter). After enabling project monitoring, you can edit the application to configure the <b>Advanced Options -> Custom Metrics</b> section. Enter the `Container Port` and `Path` and select the `Protocol`.
|
||||
|
||||
To access a project-level Grafana instance,
|
||||
|
||||
1. From the **Global** view, navigate to a cluster that has monitoring enabled.
|
||||
|
||||
1. Go to a project that has monitoring enabled.
|
||||
|
||||
1. From the project view, click **Apps.** In versions before v2.2.0, choose **Catalog Apps** on the main navigation bar.
|
||||
|
||||
1. Go to the `project-monitoring` application.
|
||||
|
||||
1. In the `project-monitoring` application, there are two `/index.html` links: one that leads to a Grafana instance and one that leads to a Prometheus instance. When you click the Grafana link, it will redirect you to a new webpage for Grafana, which shows metrics for the cluster.
|
||||
|
||||
1. You will be signed in to the Grafana instance automatically. The default username is `admin` and the default password is `admin`. For security, we recommend that you log out of Grafana, log back in with the `admin` password, and change your password.
|
||||
|
||||
**Results:** You will be logged into Grafana from the Grafana instance. After logging in, you can view the preset Grafana dashboards, which are imported via the [Grafana provisioning mechanism](http://docs.grafana.org/administration/provisioning/#dashboards), so you cannot modify them directly. For now, if you want to configure your own dashboards, clone the original and modify the new copy.
|
||||
+112
@@ -0,0 +1,112 @@
|
||||
---
|
||||
title: Prometheus Configuration
|
||||
weight: 1
|
||||
aliases:
|
||||
- /rancher/v2.0-v2.4/en/project-admin/tools/monitoring/prometheus
|
||||
- /rancher/v2.0-v2.4/en/cluster-admin/tools/monitoring/prometheus/
|
||||
- /rancher/v2.0-v2.4/en/monitoring-alerting/legacy/monitoring/cluster-monitoring/prometheus
|
||||
- /rancher/v2.0-v2.4/en/monitoring-alerting/v2.0.x-v2.4.x/cluster-monitoring/prometheus
|
||||
- /rancher/v2.x/en/monitoring-alerting/v2.0.x-v2.4.x/cluster-monitoring/prometheus/
|
||||
---
|
||||
|
||||
_Available as of v2.2.0_
|
||||
|
||||
While configuring monitoring at either the [cluster level]({{<baseurl>}}/rancher/v2.0-v2.4/en/monitoring-alerting/legacy/monitoring/cluster-monitoring/) or [project level]({{<baseurl>}}/rancher/v2.0-v2.4/en/project-admin/tools/monitoring/), there are multiple options that can be configured.
|
||||
|
||||
- [Basic Configuration](#basic-configuration)
|
||||
- [Advanced Options](#advanced-options)
|
||||
- [Node Exporter](#node-exporter)
|
||||
- [Persistent Storage](#persistent-storage)
|
||||
- [Remote Storage](#remote-storage)
|
||||
|
||||
# Basic Configuration
|
||||
|
||||
Option | Description
|
||||
-------|-------------
|
||||
Data Retention | How long your Prometheus instance retains monitoring data scraped from Rancher objects before it's purged.
|
||||
[Enable Node Exporter](#node-exporter) | Whether or not to deploy the node exporter.
|
||||
Node Exporter Host Port | The host port on which data is exposed, i.e. data that Prometheus collects from your node hardware. Required if you have enabled the node exporter.
|
||||
[Enable Persistent Storage](#persistent-storage) for Prometheus | Whether or not to configure storage for Prometheus so that metrics can be retained even if the Prometheus pod fails.
|
||||
[Enable Persistent Storage](#persistent-storage) for Grafana | Whether or not to configure storage for Grafana so that the Grafana dashboards and configuration can be retained even if the Grafana pod fails.
|
||||
Prometheus [CPU Limit](https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/#meaning-of-cpu) | CPU resource limit for the Prometheus pod.
|
||||
Prometheus [CPU Reservation](https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/#meaning-of-cpu) | CPU reservation for the Prometheus pod.
|
||||
Prometheus [Memory Limit](https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/#meaning-of-memory) | Memory resource limit for the Prometheus pod.
|
||||
Prometheus [Memory Reservation](https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/#meaning-of-memory) | Memory resource requests for the Prometheus pod.
|
||||
Selector | Ability to select the nodes in which Prometheus and Grafana pods are deployed to. To use this option, the nodes must have labels.
|
||||
|
||||
# Advanced Options
|
||||
|
||||
Since monitoring is an [application](https://github.com/rancher/system-charts/tree/dev/charts/rancher-monitoring) from the [Rancher catalog]({{<baseurl>}}/rancher/v2.0-v2.4/en/catalog/), it can be configured like any other catalog application, by passing in values to Helm.
|
||||
|
||||
> **Warning:** Any modification to the application without understanding the entire application can lead to catastrophic errors.
|
||||
|
||||
### Prometheus RemoteRead and RemoteWrite
|
||||
|
||||
_Available as of v2.4.0_
|
||||
|
||||
Prometheus RemoteRead and RemoteWrite can be configured as custom answers in the **Advanced Options** section.
|
||||
|
||||
For more information on remote endpoints and storage, refer to the [Prometheus documentation.](https://prometheus.io/docs/operating/integrations/#remote-endpoints-and-storage)
|
||||
|
||||
The Prometheus operator documentation contains the full [RemoteReadSpec](https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/api.md#remotereadspec) and [RemoteWriteSpec.](https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/api.md#remotewritespec)
|
||||
|
||||
An example configuration would be:
|
||||
|
||||
| Variable | Value |
|
||||
|--------------|------------|
|
||||
| `prometheus.remoteWrite[0].url` | `http://mytarget.com` |
|
||||
|
||||
### LivenessProbe and ReadinessProbe
|
||||
|
||||
_Available as of v2.4.0_
|
||||
|
||||
Prometheus LivenessProbe and ReadinessProbe can be configured as custom answers in the **Advanced Options** section.
|
||||
|
||||
The Kubernetes probe spec is [here.](https://v1-17.docs.kubernetes.io/docs/reference/generated/kubernetes-api/v1.17/#probe-v1-core)
|
||||
|
||||
Some example key-value pairs are:
|
||||
|
||||
| Variable | Value |
|
||||
|--------------|------------|
|
||||
| `prometheus.livenessProbe.timeoutSeconds` | 60 |
|
||||
| `prometheus.readinessProbe.timeoutSeconds` | 60 |
|
||||
|
||||
# Node Exporter
|
||||
|
||||
The [node exporter](https://github.com/prometheus/node_exporter/blob/master/README.md) is a popular open source exporter, which exposes the metrics for hardware and \*NIX kernels OS. It is designed to monitor the host system. However, there are still issues with namespaces when running it in a container, mostly around filesystem mount spaces. In order to monitor actual network metrics for the container network, the node exporter must be deployed with the `hostNetwork` mode.
|
||||
|
||||
When configuring Prometheus and enabling the node exporter, enter a host port in the **Node Exporter Host Port** that will not produce port conflicts with existing applications. The host port chosen must be open to allow internal traffic between Prometheus and the Node Exporter.
|
||||
|
||||
>**Warning:** In order for Prometheus to collect the metrics of the node exporter, after enabling cluster monitoring, you must open the <b>Node Exporter Host Port</b> in the host firewall rules to allow intranet access. By default, `9796` is used as that host port.
|
||||
|
||||
# Persistent Storage
|
||||
|
||||
>**Prerequisite:** Configure one or more StorageClasses to use as [persistent storage]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/volumes-and-storage/) for your Prometheus or Grafana pod.
|
||||
|
||||
By default, when you enable Prometheus for either a cluster or project, all monitoring data that Prometheus collects is stored on its own pod. With local storage, if the Prometheus or Grafana pods fail, all the data is lost. Rancher recommends configuring an external persistent storage to the cluster. With the external persistent storage, if the Prometheus or Grafana pods fail, the new pods can recover using data from the persistent storage.
|
||||
|
||||
When enabling persistent storage for Prometheus or Grafana, specify the size of the persistent volume and select the StorageClass.
|
||||
|
||||
# Remote Storage
|
||||
|
||||
>**Prerequisite:** Need a remote storage endpoint to be available. The possible list of integrations is available [here](https://prometheus.io/docs/operating/integrations/)
|
||||
|
||||
Using advanced options, remote storage integration for the Prometheus installation can be configured as follows:
|
||||
|
||||
```
|
||||
prometheus.remoteWrite[0].url = http://remote1/push
|
||||
prometheus.remoteWrite[0].remoteTimeout = 33s
|
||||
|
||||
prometheus.remoteWrite[1].url = http://remote2/push
|
||||
|
||||
|
||||
prometheus.remoteRead[0].url = http://remote1/read
|
||||
prometheus.remoteRead[0].proxyUrl = http://proxy.url
|
||||
prometheus.remoteRead[0].bearerToken = token-value
|
||||
|
||||
prometheus.remoteRead[1].url = http://remote2/read
|
||||
prometheus.remoteRead[1].remoteTimeout = 33s
|
||||
prometheus.remoteRead[1].readRecent = true
|
||||
```
|
||||
|
||||
Additional fields can be set up based on the [ReadSpec](https://github.com/coreos/prometheus-operator/blob/master/Documentation/api.md#remotereadspec) and [RemoteWriteSpec](https://github.com/coreos/prometheus-operator/blob/master/Documentation/api.md#remotewritespec)
|
||||
+66
@@ -0,0 +1,66 @@
|
||||
---
|
||||
title: Viewing Metrics
|
||||
weight: 2
|
||||
aliases:
|
||||
- /rancher/v2.0-v2.4/en/project-admin/tools/monitoring/viewing-metrics
|
||||
- /rancher/v2.0-v2.4/en/cluster-admin/tools/monitoring/viewing-metrics
|
||||
- /rancher/v2.0-v2.4/en/monitoring-alerting/legacy/monitoring/cluster-monitoring/viewing-metrics
|
||||
- /rancher/v2.0-v2.4/en/monitoring-alerting/v2.0.x-v2.4.x/cluster-monitoring/viewing-metrics
|
||||
- /rancher/v2.x/en/monitoring-alerting/v2.0.x-v2.4.x/cluster-monitoring/viewing-metrics/
|
||||
---
|
||||
|
||||
_Available as of v2.2.0_
|
||||
|
||||
After you've enabled monitoring at either the [cluster level]({{<baseurl>}}/rancher/v2.0-v2.4/en/monitoring-alerting/legacy/monitoring/cluster-monitoring/) or [project level]({{<baseurl>}}/rancher/v2.0-v2.4/en/project-admin/tools/monitoring/), you will want to be start viewing the data being collected. There are multiple ways to view this data.
|
||||
|
||||
## Rancher Dashboard
|
||||
|
||||
>**Note:** This is only available if you've enabled monitoring at the [cluster level]({{<baseurl>}}/rancher/v2.0-v2.4/en/monitoring-alerting/legacy/monitoring/cluster-monitoring/). Project specific analytics must be viewed using the project's Grafana instance.
|
||||
|
||||
Rancher's dashboards are available at multiple locations:
|
||||
|
||||
- **Cluster Dashboard**: From the **Global** view, navigate to the cluster.
|
||||
- **Node Metrics**: From the **Global** view, navigate to the cluster. Select **Nodes**. Find the individual node and click on its name. Click **Node Metrics.**
|
||||
- **Workload Metrics**: From the **Global** view, navigate to the project. From the main navigation bar, choose **Resources > Workloads.** (In versions before v2.3.0, choose **Workloads** on the main navigation bar.) Find the individual workload and click on its name. Click **Workload Metrics.**
|
||||
- **Pod Metrics**: From the **Global** view, navigate to the project. Select **Workloads > Workloads**. Find the individual workload and click on its name. Find the individual pod and click on its name. Click **Pod Metrics.**
|
||||
- **Container Metrics**: From the **Global** view, navigate to the project. From the main navigation bar, choose **Resources > Workloads.** (In versions before v2.3.0, choose **Workloads** on the main navigation bar.) Find the individual workload and click on its name. Find the individual pod and click on its name. Find the individual container and click on its name. Click **Container Metrics.**
|
||||
|
||||
Prometheus metrics are displayed and are denoted with the Grafana icon. If you click on the icon, the metrics will open a new tab in Grafana.
|
||||
|
||||
Within each Prometheus metrics widget, there are several ways to customize your view.
|
||||
|
||||
- Toggle between two views:
|
||||
- **Detail**: Displays graphs and charts that let you view each event in a Prometheus time series
|
||||
- **Summary** Displays events in a Prometheus time series that are outside the norm.
|
||||
- Change the range of the time series that you're viewing to see a more refined or expansive data sample.
|
||||
- Customize the data sample to display data between specific dates and times.
|
||||
|
||||
When analyzing these metrics, don't be concerned about any single standalone metric in the charts and graphs. Rather, you should establish a baseline for your metrics over the course of time, e.g. the range of values that your components usually operate within and are considered normal. After you establish the baseline, be on the lookout for any large deltas in the charts and graphs, as these big changes usually indicate a problem that you need to investigate.
|
||||
|
||||
## Grafana
|
||||
|
||||
If you've enabled monitoring at either the [cluster level]({{<baseurl>}}/rancher/v2.0-v2.4/en/monitoring-alerting/legacy/monitoring/cluster-monitoring/) or [project level]({{<baseurl>}}/rancher/v2.0-v2.4/en/project-admin/tools/monitoring/), Rancher automatically creates a link to Grafana instance. Use this link to view monitoring data.
|
||||
|
||||
Grafana allows you to query, visualize, alert, and ultimately, understand your cluster and workload data. For more information on Grafana and its capabilities, visit the [Grafana website](https://grafana.com/grafana).
|
||||
|
||||
### Authentication
|
||||
|
||||
Rancher determines which users can access the new Grafana instance, as well as the objects they can view within it, by validating them against the user's [cluster or project roles]({{<baseurl>}}/rancher/v2.0-v2.4/en/admin-settings/rbac/cluster-project-roles/). In other words, a user's access in Grafana mirrors their access in Rancher.
|
||||
|
||||
When you go to the Grafana instance, you will be logged in with the username `admin` and the password `admin`. If you log out and log in again, you will be prompted to change your password. You will only have access to the URL of the Grafana instance if you have access to view the corresponding metrics in Rancher. So for example, if your Rancher permissions are scoped to the project level, you won't be able to see the Grafana instance for cluster-level metrics.
|
||||
|
||||
### Accessing the Cluster-level Grafana Instance
|
||||
|
||||
1. From the **Global** view, navigate to a cluster that has monitoring enabled.
|
||||
|
||||
1. Go to the **System** project view. This project is where the cluster-level Grafana instance runs.
|
||||
|
||||
1. Click **Apps.** In versions before v2.2.0, choose **Catalog Apps** on the main navigation bar.
|
||||
|
||||
1. Go to the `cluster-monitoring` application.
|
||||
|
||||
1. In the `cluster-monitoring` application, there are two `/index.html` links: one that leads to a Grafana instance and one that leads to a Prometheus instance. When you click the Grafana link, it will redirect you to a new webpage for Grafana, which shows metrics for the cluster.
|
||||
|
||||
1. You will be signed in to the Grafana instance automatically. The default username is `admin` and the default password is `admin`. For security, we recommend that you log out of Grafana, log back in with the `admin` password, and change your password.
|
||||
|
||||
**Results:** You are logged into Grafana from the Grafana instance. After logging in, you can view the preset Grafana dashboards, which are imported via the [Grafana provisioning mechanism](http://docs.grafana.org/administration/provisioning/#dashboards), so you cannot modify them directly. For now, if you want to configure your own dashboards, clone the original and modify the new copy.
|
||||
+32
@@ -0,0 +1,32 @@
|
||||
---
|
||||
title: Disabling Istio
|
||||
weight: 4
|
||||
aliases:
|
||||
- /rancher/v2.0-v2.4/en/cluster-admin/tools/istio/disabling-istio
|
||||
- /rancher/v2.0-v2.4/en/istio/legacy/disabling-istio
|
||||
- /rancher/v2.0-v2.4/en/istio/v2.3.x-v2.4.x/disabling-istio
|
||||
- /rancher/v2.x/en/istio/v2.3.x-v2.4.x/disabling-istio/
|
||||
---
|
||||
|
||||
This section describes how to disable Istio in a cluster, namespace, or workload.
|
||||
|
||||
# Disable Istio in a Cluster
|
||||
|
||||
To disable Istio,
|
||||
|
||||
1. From the **Global** view, navigate to the cluster that you want to disable Istio for.
|
||||
1. Click **Tools > Istio.**
|
||||
1. Click **Disable,** then click the red button again to confirm the disable action.
|
||||
|
||||
**Result:** The `cluster-istio` application in the cluster's `system` project gets removed. The Istio sidecar cannot be deployed on any workloads in the cluster.
|
||||
|
||||
# Disable Istio in a Namespace
|
||||
|
||||
1. In the Rancher UI, go to the project that has the namespace where you want to disable Istio.
|
||||
1. On the **Workloads** tab, you will see a list of namespaces and the workloads deployed in them. Go to the namespace where you want to disable and click the **⋮ > Disable Istio Auto Injection.**
|
||||
|
||||
**Result:** When workloads are deployed in this namespace, they will not have the Istio sidecar.
|
||||
|
||||
# Remove the Istio Sidecar from a Workload
|
||||
|
||||
Disable Istio in the namespace, then redeploy the workloads with in it. They will be deployed without the Istio sidecar.
|
||||
@@ -0,0 +1,93 @@
|
||||
---
|
||||
title: Istio
|
||||
weight: 15
|
||||
aliases:
|
||||
- /rancher/v2.0-v2.4/en/dashboard/istio
|
||||
- /rancher/v2.0-v2.4/en/project-admin/istio/configuring-resource-allocations/
|
||||
- /rancher/v2.0-v2.4/en/cluster-admin/tools/istio/
|
||||
- /rancher/v2.0-v2.4/en/project-admin/istio
|
||||
- /rancher/v2.0-v2.4/en/istio/legacy/cluster-istio
|
||||
- /rancher/v2.x/en/istio/v2.3.x-v2.4.x/
|
||||
---
|
||||
_Available as of v2.3.0_
|
||||
|
||||
[Istio](https://istio.io/) is an open-source tool that makes it easier for DevOps teams to observe, control, troubleshoot, and secure the traffic within a complex network of microservices.
|
||||
|
||||
As a network of microservices changes and grows, the interactions between them can become more difficult to manage and understand. In such a situation, it is useful to have a service mesh as a separate infrastructure layer. Istio's service mesh lets you manipulate traffic between microservices without changing the microservices directly.
|
||||
|
||||
Our integration of Istio is designed so that a Rancher operator, such as an administrator or cluster owner, can deliver Istio to developers. Then developers can use Istio to enforce security policies, troubleshoot problems, or manage traffic for green/blue deployments, canary deployments, or A/B testing.
|
||||
|
||||
This service mesh provides features that include but are not limited to the following:
|
||||
|
||||
- Traffic management features
|
||||
- Enhanced monitoring and tracing
|
||||
- Service discovery and routing
|
||||
- Secure connections and service-to-service authentication with mutual TLS
|
||||
- Load balancing
|
||||
- Automatic retries, backoff, and circuit breaking
|
||||
|
||||
After Istio is enabled in a cluster, you can leverage Istio's control plane functionality with `kubectl`.
|
||||
|
||||
Rancher's Istio integration comes with comprehensive visualization aids:
|
||||
|
||||
- **Trace the root cause of errors with Jaeger.** [Jaeger](https://www.jaegertracing.io/) is an open-source tool that provides a UI for a distributed tracing system, which is useful for root cause analysis and for determining what causes poor performance. Distributed tracing allows you to view an entire chain of calls, which might originate with a user request and traverse dozens of microservices.
|
||||
- **Get the full picture of your microservice architecture with Kiali.** [Kiali](https://www.kiali.io/) provides a diagram that shows the services within a service mesh and how they are connected, including the traffic rates and latencies between them. You can check the health of the service mesh, or drill down to see the incoming and outgoing requests to a single component.
|
||||
- **Gain insights from time series analytics with Grafana dashboards.** [Grafana](https://grafana.com/) is an analytics platform that allows you to query, visualize, alert on and understand the data gathered by Prometheus.
|
||||
- **Write custom queries for time series data with the Prometheus UI.** [Prometheus](https://prometheus.io/) is a systems monitoring and alerting toolkit. Prometheus scrapes data from your cluster, which is then used by Grafana. A Prometheus UI is also integrated into Rancher, and lets you write custom queries for time series data and see the results in the UI.
|
||||
|
||||
|
||||
Istio needs to be set up by a Rancher administrator or cluster administrator before it can be used in a project.
|
||||
|
||||
# Prerequisites
|
||||
|
||||
Before enabling Istio, we recommend that you confirm that your Rancher worker nodes have enough [CPU and memory]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/tools/istio/resources) to run all of the components of Istio.
|
||||
|
||||
# Setup Guide
|
||||
|
||||
Refer to the [setup guide]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/tools/istio/setup) for instructions on how to set up Istio and use it in a project.
|
||||
|
||||
# Disabling Istio
|
||||
|
||||
To remove Istio components from a cluster, namespace, or workload, refer to the section on [disabling Istio.]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/tools/istio/disabling-istio)
|
||||
|
||||
# Accessing Visualizations
|
||||
|
||||
> By default, only cluster owners have access to Jaeger and Kiali. For instructions on how to allow project members to access them, see [this section.]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/tools/istio/rbac/)
|
||||
|
||||
After Istio is set up in a cluster, Grafana, Prometheus, Jaeger, and Kiali are available in the Rancher UI.
|
||||
|
||||
Your access to the visualizations depend on your role. Grafana and Prometheus are only available for cluster owners. The Kiali and Jaeger UIs are available only to cluster owners by default, but cluster owners can allow project members to access them by editing the Istio settings. When you go to your project and click **Resources > Istio,** you can go to each UI for Kiali, Jaeger, Grafana, and Prometheus by clicking their icons in the top right corner of the page.
|
||||
|
||||
To see the visualizations, go to the cluster where Istio is set up and click **Tools > Istio.** You should see links to each UI at the top of the page.
|
||||
|
||||
You can also get to the visualization tools from the project view.
|
||||
|
||||
# Viewing the Kiali Traffic Graph
|
||||
|
||||
1. From the project view in Rancher, click **Resources > Istio.**
|
||||
1. If you are a cluster owner, you can go to the **Traffic Graph** tab. This tab has the Kiali network visualization integrated into the UI.
|
||||
|
||||
# Viewing Traffic Metrics
|
||||
|
||||
Istio’s monitoring features provide visibility into the performance of all your services.
|
||||
|
||||
1. From the project view in Rancher, click **Resources > Istio.**
|
||||
1. Go to the **Traffic Metrics** tab. After traffic is generated in your cluster, you should be able to see metrics for **Success Rate, Request Volume, 4xx Response Count, Project 5xx Response Count** and **Request Duration.** Cluster owners can see all of the metrics, while project members can see a subset of the metrics.
|
||||
|
||||
# Architecture
|
||||
|
||||
Istio installs a service mesh that uses [Envoy](https://www.envoyproxy.io/learn/service-mesh) sidecar proxies to intercept traffic to each workload. These sidecars intercept and manage service-to-service communication, allowing fine-grained observation and control over traffic within the cluster.
|
||||
|
||||
Only workloads that have the Istio sidecar injected can be tracked and controlled by Istio.
|
||||
|
||||
Enabling Istio in Rancher enables monitoring in the cluster, and enables Istio in all new namespaces that are created in a cluster. You need to manually enable Istio in preexisting namespaces.
|
||||
|
||||
When a namespace has Istio enabled, new workloads deployed in the namespace will automatically have the Istio sidecar. You need to manually enable Istio in preexisting workloads.
|
||||
|
||||
For more information on the Istio sidecar, refer to the [Istio docs](https://istio.io/docs/setup/kubernetes/additional-setup/sidecar-injection/).
|
||||
|
||||
### Two Ingresses
|
||||
|
||||
By default, each Rancher-provisioned cluster has one NGINX ingress controller allowing traffic into the cluster. To allow Istio to receive external traffic, you need to enable the Istio ingress gateway for the cluster. The result is that your cluster will have two ingresses.
|
||||
|
||||
|
||||
@@ -0,0 +1,63 @@
|
||||
---
|
||||
title: Role-based Access Control
|
||||
weight: 3
|
||||
aliases:
|
||||
- /rancher/v2.0-v2.4/en/cluster-admin/tools/istio/rbac
|
||||
- /rancher/v2.0-v2.4/en/istio/legacy/rbac
|
||||
- /rancher/v2.0-v2.4/en/istio/v2.3.x-v2.4.x/rbac
|
||||
- /rancher/v2.x/en/istio/v2.3.x-v2.4.x/rbac/
|
||||
---
|
||||
|
||||
This section describes the permissions required to access Istio features and how to configure access to the Kiali and Jaeger visualizations.
|
||||
|
||||
# Cluster-level Access
|
||||
|
||||
By default, only cluster administrators can:
|
||||
|
||||
- Enable Istio for the cluster
|
||||
- Configure resource allocations for Istio
|
||||
- View each UI for Prometheus, Grafana, Kiali, and Jaeger
|
||||
|
||||
# Project-level Access
|
||||
|
||||
After Istio is enabled in a cluster, project owners and members have permission to:
|
||||
|
||||
- Enable and disable Istio sidecar auto-injection for namespaces
|
||||
- Add the Istio sidecar to workloads
|
||||
- View the traffic metrics and traffic graph for the cluster
|
||||
- View the Kiali and Jaeger visualizations if cluster administrators give access to project members
|
||||
- Configure Istio's resources (such as the gateway, destination rules, or virtual services) with `kubectl` (This does not apply to read-only project members)
|
||||
|
||||
# Access to Visualizations
|
||||
|
||||
By default, the Kiali and Jaeger visualizations are restricted to the cluster owner because the information in them could be sensitive.
|
||||
|
||||
**Jaeger** provides a UI for a distributed tracing system, which is useful for root cause analysis and for determining what causes poor performance.
|
||||
|
||||
**Kiali** provides a diagram that shows the services within a service mesh and how they are connected.
|
||||
|
||||
Rancher supports giving groups permission to access Kiali and Jaeger, but not individuals.
|
||||
|
||||
To configure who has permission to access the Kiali and Jaeger UI,
|
||||
|
||||
1. Go to the cluster view and click **Tools > Istio.**
|
||||
1. Then go to the **Member Access** section. If you want to restrict access to certain groups, choose **Allow cluster owner and specified members to access Kiali and Jaeger UI.** Search for the groups that you want to have access to Kiali and Jaeger. If you want all members to have access to the tools, click **Allow all members to access Kiali and Jaeger UI.**
|
||||
1. Click **Save.**
|
||||
|
||||
**Result:** The access levels for Kiali and Jaeger have been updated.
|
||||
|
||||
# Summary of Default Permissions for Istio Users
|
||||
|
||||
| Permission | Cluster Administrators | Project Owners | Project Members | Read-only Project Members |
|
||||
|------------------------------------------|----------------|----------------|-----------------|---------------------------|
|
||||
| Enable and disable Istio for the cluster | ✓ | | | |
|
||||
| Configure Istio resource limits | ✓ | | | |
|
||||
| Control who has access to Kiali and the Jaeger UI | ✓ | | | |
|
||||
| Enable and disable Istio for a namespace | ✓ | ✓ | ✓ | |
|
||||
| Enable and disable Istio on workloads | ✓ | ✓ | ✓ | |
|
||||
| Configure Istio with `kubectl` | ✓ | ✓ | ✓ | |
|
||||
| View Prometheus UI and Grafana UI | ✓ | | | |
|
||||
| View Kiali UI and Jaeger UI ([Configurable](#access-to-visualizations)) | ✓ | | | |
|
||||
| View Istio project dashboard, including traffic metrics* | ✓ | ✓ | ✓ | ✓ |
|
||||
|
||||
* By default, only the cluster owner will see the traffic graph. Project members will see only a subset of traffic metrics. Project members cannot see the traffic graph because it comes from Kiali, and access to Kiali is restricted to cluster owners by default.
|
||||
+23
@@ -0,0 +1,23 @@
|
||||
---
|
||||
title: Release Notes
|
||||
aliases:
|
||||
- /rancher/v2.0-v2.4/en/cluster-admin/tools/istio/release-notes
|
||||
- /rancher/v2.0-v2.4/en/istio/legacy/release-notes
|
||||
- /rancher/v2.0-v2.4/en/istio/v2.3.x-v2.4.x/release-notes
|
||||
- /rancher/v2.x/en/istio/v2.3.x-v2.4.x/release-notes/
|
||||
---
|
||||
|
||||
|
||||
# Istio 1.5.8
|
||||
|
||||
### Important note on 1.5.x versions
|
||||
|
||||
When upgrading from any 1.4 version of Istio to any 1.5 version, the Rancher installer will delete several resources in order to complete the upgrade, at which point they will be immediately re-installed. This includes the `istio-reader-service-account`. If your Istio installation is using this service account be aware that any secrets tied to the service account will be deleted. Most notably this will **break specific [multi-cluster deployments](https://archive.istio.io/v1.4/docs/setup/install/multicluster/)**. Downgrades back to 1.4 are not possible.
|
||||
|
||||
See the official upgrade notes for additional information on the 1.5 release and upgrading from 1.4: https://istio.io/latest/news/releases/1.5.x/announcing-1.5/upgrade-notes/
|
||||
|
||||
> **Note:** Rancher continues to use the Helm installation method, which produces a different architecture from an istioctl installation.
|
||||
|
||||
### Known Issues
|
||||
|
||||
* The Kiali traffic graph is currently not working [#24924](https://github.com/istio/istio/issues/24924)
|
||||
@@ -0,0 +1,156 @@
|
||||
---
|
||||
title: CPU and Memory Allocations
|
||||
weight: 1
|
||||
aliases:
|
||||
- /rancher/v2.0-v2.4/en/project-admin/istio/configuring-resource-allocations/
|
||||
- /rancher/v2.0-v2.4/en/project-admin/istio/config/
|
||||
- /rancher/v2.0-v2.4/en/cluster-admin/tools/istio/resources
|
||||
- /rancher/v2.0-v2.4/en/istio/legacy/resources
|
||||
- /rancher/v2.0-v2.4/en/istio/v2.3.x-v2.4.x/resources
|
||||
- /rancher/v2.x/en/istio/v2.3.x-v2.4.x/resources/
|
||||
---
|
||||
_Available as of v2.3.0_
|
||||
|
||||
This section describes the minimum recommended computing resources for the Istio components in a cluster.
|
||||
|
||||
The CPU and memory allocations for each component are [configurable.](#configuring-resource-allocations)
|
||||
|
||||
Before enabling Istio, we recommend that you confirm that your Rancher worker nodes have enough CPU and memory to run all of the components of Istio.
|
||||
|
||||
> **Tip:** In larger deployments, it is strongly advised that the infrastructure be placed on dedicated nodes in the cluster by adding a node selector for each Istio component.
|
||||
|
||||
The table below shows a summary of the minimum recommended resource requests and limits for the CPU and memory of each central Istio component.
|
||||
|
||||
In Kubernetes, the resource request indicates that the workload will not deployed on a node unless the node has at least the specified amount of memory and CPU available. If the workload surpasses the limit for CPU or memory, it can be terminated or evicted from the node. For more information on managing resource limits for containers, refer to the [Kubernetes documentation.](https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/)
|
||||
|
||||
Workload | Container | CPU - Request | Mem - Request | CPU - Limit | Mem - Limit | Configurable
|
||||
---------|-----------|---------------|---------------|-------------|-------------|-------------
|
||||
istio-pilot |discovery| 500m | 2048Mi | 1000m | 4096Mi | Y
|
||||
istio-telemetry |mixer| 1000m | 1024Mi | 4800m | 4096Mi | Y
|
||||
istio-policy | mixer | 1000m | 1024Mi | 4800m | 4096Mi | Y
|
||||
istio-tracing | jaeger | 100m | 100Mi | 500m | 1024Mi | Y
|
||||
prometheus | prometheus | 750m | 750Mi | 1000m | 1024Mi | Y
|
||||
grafana | grafana | 100m | 100Mi | 200m | 512Mi | Y
|
||||
Others | - | 500m | 500Mi | - | - | N
|
||||
**Total** | **-** | **3950m** | **5546Mi** | **>12300m** | **>14848Mi** | **-**
|
||||
|
||||
|
||||
# Configuring Resource Allocations
|
||||
|
||||
You can individually configure the resource allocation for each type of Istio component. This section includes the default resource allocations for each component.
|
||||
|
||||
To make it easier to schedule the workloads to a node, a cluster administrator can reduce the CPU and memory resource requests for the component. However, the default CPU and memory allocations are the minimum that we recommend.
|
||||
|
||||
You can find more information about Istio configuration in the [official Istio documentation](https://istio.io/docs/concepts/what-is-istio).
|
||||
|
||||
To configure the resources allocated to an Istio component,
|
||||
|
||||
1. In Rancher, go to the cluster where you have Istio installed.
|
||||
1. Click **Tools > Istio.** This opens the Istio configuration page.
|
||||
1. Change the CPU or memory allocations, the nodes where each component will be scheduled to, or the node tolerations.
|
||||
1. Click **Save.**
|
||||
|
||||
**Result:** The resource allocations for the Istio components are updated.
|
||||
|
||||
## Pilot
|
||||
|
||||
[Pilot](https://istio.io/docs/ops/deployment/architecture/#pilot) provides the following:
|
||||
|
||||
- Authentication configuration
|
||||
- Service discovery for the Envoy sidecars
|
||||
- Traffic management capabilities for intelligent routing (A/B tests and canary rollouts)
|
||||
- Configuration for resiliency (timeouts, retries, circuit breakers, etc)
|
||||
|
||||
For more information on Pilot, refer to the [documentation](https://istio.io/docs/concepts/traffic-management/#pilot-and-envoy).
|
||||
|
||||
Option | Description| Required | Default
|
||||
-------|------------|-------|-------
|
||||
Pilot CPU Limit | CPU resource limit for the istio-pilot pod.| Yes | 1000
|
||||
Pilot CPU Reservation | CPU reservation for the istio-pilot pod. | Yes | 500
|
||||
Pilot Memory Limit | Memory resource limit for the istio-pilot pod. | Yes | 4096
|
||||
Pilot Memory Reservation | Memory resource requests for the istio-pilot pod. | Yes | 2048
|
||||
Trace sampling Percentage | [Trace sampling percentage](https://istio.io/docs/tasks/telemetry/distributed-tracing/overview/#trace-sampling) | Yes | 1
|
||||
Pilot Selector | Ability to select the nodes in which istio-pilot pod is deployed to. To use this option, the nodes must have labels. | No | n/a
|
||||
|
||||
## Mixer
|
||||
|
||||
[Mixer](https://istio.io/docs/ops/deployment/architecture/#mixer) enforces access control and usage policies across the service mesh. It also integrates with plugins for monitoring tools such as Prometheus. The Envoy sidecar proxy passes telemetry data and monitoring data to Mixer, and Mixer passes the monitoring data to Prometheus.
|
||||
|
||||
For more information on Mixer, policies and telemetry, refer to the [documentation](https://istio.io/docs/concepts/policies-and-telemetry/).
|
||||
|
||||
Option | Description| Required | Default
|
||||
-------|------------|-------|-------
|
||||
Mixer Telemetry CPU Limit | CPU resource limit for the istio-telemetry pod.| Yes | 4800
|
||||
Mixer Telemetry CPU Reservation | CPU reservation for the istio-telemetry pod.| Yes | 1000
|
||||
Mixer Telemetry Memory Limit | Memory resource limit for the istio-telemetry pod.| Yes | 4096
|
||||
Mixer Telemetry Memory Reservation | Memory resource requests for the istio-telemetry pod.| Yes | 1024
|
||||
Enable Mixer Policy | Whether or not to deploy the istio-policy. | Yes | False
|
||||
Mixer Policy CPU Limit | CPU resource limit for the istio-policy pod. | Yes, when policy enabled | 4800
|
||||
Mixer Policy CPU Reservation | CPU reservation for the istio-policy pod. | Yes, when policy enabled | 1000
|
||||
Mixer Policy Memory Limit | Memory resource limit for the istio-policy pod. | Yes, when policy enabled | 4096
|
||||
Mixer Policy Memory Reservation | Memory resource requests for the istio-policy pod. | Yes, when policy enabled | 1024
|
||||
Mixer Selector | Ability to select the nodes in which istio-policy and istio-telemetry pods are deployed to. To use this option, the nodes must have labels. | No | n/a
|
||||
|
||||
## Tracing
|
||||
|
||||
[Distributed tracing](https://istio.io/docs/tasks/telemetry/distributed-tracing/overview/) enables users to track a request through a service mesh. This makes it easier to troubleshoot problems with latency, parallelism and serialization.
|
||||
|
||||
Option | Description| Required | Default
|
||||
-------|------------|-------|-------
|
||||
Enable Tracing | Whether or not to deploy the istio-tracing. | Yes | True
|
||||
Tracing CPU Limit | CPU resource limit for the istio-tracing pod. | Yes | 500
|
||||
Tracing CPU Reservation | CPU reservation for the istio-tracing pod. | Yes | 100
|
||||
Tracing Memory Limit | Memory resource limit for the istio-tracing pod. | Yes | 1024
|
||||
Tracing Memory Reservation | Memory resource requests for the istio-tracing pod. | Yes | 100
|
||||
Tracing Selector | Ability to select the nodes in which tracing pod is deployed to. To use this option, the nodes must have labels. | No | n/a
|
||||
|
||||
## Ingress Gateway
|
||||
|
||||
The Istio gateway allows Istio features such as monitoring and route rules to be applied to traffic entering the cluster. This gateway is a prerequisite for outside traffic to make requests to Istio.
|
||||
|
||||
For more information, refer to the [documentation](https://istio.io/docs/tasks/traffic-management/ingress/).
|
||||
|
||||
Option | Description| Required | Default
|
||||
-------|------------|-------|-------
|
||||
Enable Ingress Gateway | Whether or not to deploy the istio-ingressgateway. | Yes | False
|
||||
Service Type of Istio Ingress Gateway | How to expose the gateway. You can choose NodePort or Loadbalancer | Yes | NodePort
|
||||
Http2 Port | The NodePort for http2 requests | Yes | 31380
|
||||
Https Port | The NodePort for https requests | Yes | 31390
|
||||
Load Balancer IP | Ingress Gateway Load Balancer IP | No | n/a
|
||||
Load Balancer Source Ranges | Ingress Gateway Load Balancer Source Ranges | No | n/a
|
||||
Ingress Gateway CPU Limit | CPU resource limit for the istio-ingressgateway pod. | Yes | 2000
|
||||
Ingress Gateway CPU Reservation | CPU reservation for the istio-ingressgateway pod. | Yes | 100
|
||||
Ingress Gateway Memory Limit | Memory resource limit for the istio-ingressgateway pod. | Yes | 1024
|
||||
Ingress Gateway Memory Reservation | Memory resource requests for the istio-ingressgateway pod. | Yes | 128
|
||||
Ingress Gateway Selector | Ability to select the nodes in which istio-ingressgateway pod is deployed to. To use this option, the nodes must have labels. | No | n/a
|
||||
|
||||
## Prometheus
|
||||
|
||||
You can query for Istio metrics using Prometheus. Prometheus is an open-source systems monitoring and alerting toolkit.
|
||||
|
||||
Option | Description| Required | Default
|
||||
-------|------------|-------|-------
|
||||
Prometheus CPU Limit | CPU resource limit for the Prometheus pod.| Yes | 1000
|
||||
Prometheus CPU Reservation | CPU reservation for the Prometheus pod.| Yes | 750
|
||||
Prometheus Memory Limit | Memory resource limit for the Prometheus pod.| Yes | 1024
|
||||
Prometheus Memory Reservation | Memory resource requests for the Prometheus pod.| Yes | 750
|
||||
Retention for Prometheus | How long your Prometheus instance retains data | Yes | 6
|
||||
Prometheus Selector | Ability to select the nodes in which Prometheus pod is deployed to. To use this option, the nodes must have labels.| No | n/a
|
||||
|
||||
## Grafana
|
||||
|
||||
You can visualize metrics with Grafana. Grafana lets you visualize Istio traffic data scraped by Prometheus.
|
||||
|
||||
Option | Description| Required | Default
|
||||
-------|------------|-------|-------
|
||||
Enable Grafana | Whether or not to deploy the Grafana.| Yes | True
|
||||
Grafana CPU Limit | CPU resource limit for the Grafana pod.| Yes, when Grafana enabled | 200
|
||||
Grafana CPU Reservation | CPU reservation for the Grafana pod.| Yes, when Grafana enabled | 100
|
||||
Grafana Memory Limit | Memory resource limit for the Grafana pod.| Yes, when Grafana enabled | 512
|
||||
Grafana Memory Reservation | Memory resource requests for the Grafana pod.| Yes, when Grafana enabled | 100
|
||||
Grafana Selector | Ability to select the nodes in which Grafana pod is deployed to. To use this option, the nodes must have labels. | No | n/a
|
||||
Enable Persistent Storage for Grafana | Enable Persistent Storage for Grafana | Yes, when Grafana enabled | False
|
||||
Source | Use a Storage Class to provision a new persistent volume or Use an existing persistent volume claim | Yes, when Grafana enabled and enabled PV | Use SC
|
||||
Storage Class | Storage Class for provisioning PV for Grafana | Yes, when Grafana enabled, enabled PV and use storage class | Use the default class
|
||||
Persistent Volume Size | The size for the PV you would like to provision for Grafana | Yes, when Grafana enabled, enabled PV and use storage class | 5Gi
|
||||
Existing Claim | Use existing PVC for Grafana | Yes, when Grafana enabled, enabled PV and use existing PVC | n/a
|
||||
+327
@@ -0,0 +1,327 @@
|
||||
---
|
||||
title: 4. Add Deployments and Services with the Istio Sidecar
|
||||
weight: 4
|
||||
aliases:
|
||||
- /rancher/v2.0-v2.4/en/cluster-admin/tools/istio/setup/deploy-workloads
|
||||
- /rancher/v2.0-v2.4/en/istio/legacy/setup/deploy-workloads
|
||||
- /rancher/v2.0-v2.4/en/istio/v2.3.x-v2.4.x/setup/deploy-workloads
|
||||
- /rancher/v2.x/en/istio/v2.3.x-v2.4.x/setup/deploy-workloads/
|
||||
---
|
||||
|
||||
> **Prerequisite:** To enable Istio for a workload, the cluster and namespace must have Istio enabled.
|
||||
|
||||
Enabling Istio in a namespace only enables automatic sidecar injection for new workloads. To enable the Envoy sidecar for existing workloads, you need to enable it manually for each workload.
|
||||
|
||||
To inject the Istio sidecar on an existing workload in the namespace, go to the workload, click the **⋮,** and click **Redeploy.** When the workload is redeployed, it will have the Envoy sidecar automatically injected.
|
||||
|
||||
Wait a few minutes for the workload to upgrade to have the istio sidecar. Click it and go to the Containers section. You should be able to see istio-init and istio-proxy alongside your original workload. This means the Istio sidecar is enabled for the workload. Istio is doing all the wiring for the sidecar envoy. Now Istio can do all the features automatically if you enable them in the yaml.
|
||||
|
||||
### 3. Add Deployments and Services
|
||||
|
||||
Next we add the Kubernetes resources for the sample deployments and services for the BookInfo app in Istio's documentation.
|
||||
|
||||
1. Go to the project inside the cluster you want to deploy the workload on.
|
||||
1. In Workloads, click **Import YAML.**
|
||||
1. Copy the below resources into the form.
|
||||
1. Click **Import.**
|
||||
|
||||
This will set up the following sample resources from Istio's example BookInfo app:
|
||||
|
||||
Details service and deployment:
|
||||
|
||||
- A `details` Service
|
||||
- A ServiceAccount for `bookinfo-details`
|
||||
- A `details-v1` Deployment
|
||||
|
||||
Ratings service and deployment:
|
||||
|
||||
- A `ratings` Service
|
||||
- A ServiceAccount for `bookinfo-ratings`
|
||||
- A `ratings-v1` Deployment
|
||||
|
||||
Reviews service and deployments (three versions):
|
||||
|
||||
- A `reviews` Service
|
||||
- A ServiceAccount for `bookinfo-reviews`
|
||||
- A `reviews-v1` Deployment
|
||||
- A `reviews-v2` Deployment
|
||||
- A `reviews-v3` Deployment
|
||||
|
||||
Productpage service and deployment:
|
||||
|
||||
This is the main page of the app, which will be visible from a web browser. The other services will be called from this page.
|
||||
|
||||
- A `productpage` service
|
||||
- A ServiceAccount for `bookinfo-productpage`
|
||||
- A `productpage-v1` Deployment
|
||||
|
||||
### Resource YAML
|
||||
|
||||
```yaml
|
||||
# Copyright 2017 Istio Authors
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
##################################################################################################
|
||||
# Details service
|
||||
##################################################################################################
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: details
|
||||
labels:
|
||||
app: details
|
||||
service: details
|
||||
spec:
|
||||
ports:
|
||||
- port: 9080
|
||||
name: http
|
||||
selector:
|
||||
app: details
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: ServiceAccount
|
||||
metadata:
|
||||
name: bookinfo-details
|
||||
---
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: details-v1
|
||||
labels:
|
||||
app: details
|
||||
version: v1
|
||||
spec:
|
||||
replicas: 1
|
||||
selector:
|
||||
matchLabels:
|
||||
app: details
|
||||
version: v1
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: details
|
||||
version: v1
|
||||
spec:
|
||||
serviceAccountName: bookinfo-details
|
||||
containers:
|
||||
- name: details
|
||||
image: docker.io/istio/examples-bookinfo-details-v1:1.15.0
|
||||
imagePullPolicy: IfNotPresent
|
||||
ports:
|
||||
- containerPort: 9080
|
||||
---
|
||||
##################################################################################################
|
||||
# Ratings service
|
||||
##################################################################################################
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: ratings
|
||||
labels:
|
||||
app: ratings
|
||||
service: ratings
|
||||
spec:
|
||||
ports:
|
||||
- port: 9080
|
||||
name: http
|
||||
selector:
|
||||
app: ratings
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: ServiceAccount
|
||||
metadata:
|
||||
name: bookinfo-ratings
|
||||
---
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: ratings-v1
|
||||
labels:
|
||||
app: ratings
|
||||
version: v1
|
||||
spec:
|
||||
replicas: 1
|
||||
selector:
|
||||
matchLabels:
|
||||
app: ratings
|
||||
version: v1
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: ratings
|
||||
version: v1
|
||||
spec:
|
||||
serviceAccountName: bookinfo-ratings
|
||||
containers:
|
||||
- name: ratings
|
||||
image: docker.io/istio/examples-bookinfo-ratings-v1:1.15.0
|
||||
imagePullPolicy: IfNotPresent
|
||||
ports:
|
||||
- containerPort: 9080
|
||||
---
|
||||
##################################################################################################
|
||||
# Reviews service
|
||||
##################################################################################################
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: reviews
|
||||
labels:
|
||||
app: reviews
|
||||
service: reviews
|
||||
spec:
|
||||
ports:
|
||||
- port: 9080
|
||||
name: http
|
||||
selector:
|
||||
app: reviews
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: ServiceAccount
|
||||
metadata:
|
||||
name: bookinfo-reviews
|
||||
---
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: reviews-v1
|
||||
labels:
|
||||
app: reviews
|
||||
version: v1
|
||||
spec:
|
||||
replicas: 1
|
||||
selector:
|
||||
matchLabels:
|
||||
app: reviews
|
||||
version: v1
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: reviews
|
||||
version: v1
|
||||
spec:
|
||||
serviceAccountName: bookinfo-reviews
|
||||
containers:
|
||||
- name: reviews
|
||||
image: docker.io/istio/examples-bookinfo-reviews-v1:1.15.0
|
||||
imagePullPolicy: IfNotPresent
|
||||
ports:
|
||||
- containerPort: 9080
|
||||
---
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: reviews-v2
|
||||
labels:
|
||||
app: reviews
|
||||
version: v2
|
||||
spec:
|
||||
replicas: 1
|
||||
selector:
|
||||
matchLabels:
|
||||
app: reviews
|
||||
version: v2
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: reviews
|
||||
version: v2
|
||||
spec:
|
||||
serviceAccountName: bookinfo-reviews
|
||||
containers:
|
||||
- name: reviews
|
||||
image: docker.io/istio/examples-bookinfo-reviews-v2:1.15.0
|
||||
imagePullPolicy: IfNotPresent
|
||||
ports:
|
||||
- containerPort: 9080
|
||||
---
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: reviews-v3
|
||||
labels:
|
||||
app: reviews
|
||||
version: v3
|
||||
spec:
|
||||
replicas: 1
|
||||
selector:
|
||||
matchLabels:
|
||||
app: reviews
|
||||
version: v3
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: reviews
|
||||
version: v3
|
||||
spec:
|
||||
serviceAccountName: bookinfo-reviews
|
||||
containers:
|
||||
- name: reviews
|
||||
image: docker.io/istio/examples-bookinfo-reviews-v3:1.15.0
|
||||
imagePullPolicy: IfNotPresent
|
||||
ports:
|
||||
- containerPort: 9080
|
||||
---
|
||||
##################################################################################################
|
||||
# Productpage services
|
||||
##################################################################################################
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: productpage
|
||||
labels:
|
||||
app: productpage
|
||||
service: productpage
|
||||
spec:
|
||||
ports:
|
||||
- port: 9080
|
||||
name: http
|
||||
selector:
|
||||
app: productpage
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: ServiceAccount
|
||||
metadata:
|
||||
name: bookinfo-productpage
|
||||
---
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: productpage-v1
|
||||
labels:
|
||||
app: productpage
|
||||
version: v1
|
||||
spec:
|
||||
replicas: 1
|
||||
selector:
|
||||
matchLabels:
|
||||
app: productpage
|
||||
version: v1
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: productpage
|
||||
version: v1
|
||||
spec:
|
||||
serviceAccountName: bookinfo-productpage
|
||||
containers:
|
||||
- name: productpage
|
||||
image: docker.io/istio/examples-bookinfo-productpage-v1:1.15.0
|
||||
imagePullPolicy: IfNotPresent
|
||||
ports:
|
||||
- containerPort: 9080
|
||||
---
|
||||
```
|
||||
|
||||
### [Next: Set up the Istio Gateway]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/tools/istio/setup/gateway)
|
||||
+39
@@ -0,0 +1,39 @@
|
||||
---
|
||||
title: 1. Enable Istio in the Cluster
|
||||
weight: 1
|
||||
aliases:
|
||||
- /rancher/v2.0-v2.4/en/cluster-admin/tools/istio/setup/enable-istio-in-cluster
|
||||
- /rancher/v2.0-v2.4/en/istio/legacy/setup/enable-istio-in-cluster
|
||||
- /rancher/v2.0-v2.4/en/istio/v2.3.x-v2.4.x/setup/enable-istio-in-cluster
|
||||
- /rancher/v2.x/en/istio/v2.3.x-v2.4.x/setup/enable-istio-in-cluster/
|
||||
---
|
||||
|
||||
This cluster uses the default Nginx controller to allow traffic into the cluster.
|
||||
|
||||
A Rancher [administrator]({{<baseurl>}}/rancher/v2.0-v2.4/en/admin-settings/rbac/global-permissions/) or [cluster owner]({{<baseurl>}}/rancher/v2.0-v2.4/en/admin-settings/rbac/cluster-project-roles/#cluster-roles) can configure Rancher to deploy Istio in a Kubernetes cluster.
|
||||
|
||||
# Prerequisites
|
||||
|
||||
This guide assumes you have already [installed Rancher,]({{<baseurl>}}/rancher/v2.0-v2.4/en/installation) and you have already [provisioned a separate Kubernetes cluster]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-provisioning) on which you will install Istio.
|
||||
|
||||
The nodes in your cluster must meet the [CPU and memory requirements.]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/tools/istio/resources/)
|
||||
|
||||
The workloads and services that you want to be controlled by Istio must meet [Istio's requirements.](https://istio.io/docs/setup/additional-setup/requirements/)
|
||||
|
||||
> If the cluster has a Pod Security Policy enabled there are [additional prerequisites steps]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/tools/istio/setup/enable-istio-in-cluster/enable-istio-with-psp/)
|
||||
|
||||
# Enable Istio in the Cluster
|
||||
|
||||
1. From the **Global** view, navigate to the **cluster** where you want to enable Istio.
|
||||
1. Click **Tools > Istio.**
|
||||
1. Optional: Configure member access and [resource limits]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/tools/istio/resources/) for the Istio components. Ensure you have enough resources on your worker nodes to enable Istio.
|
||||
1. Click **Enable**.
|
||||
1. Click **Save**.
|
||||
|
||||
**Result:** Istio is enabled at the cluster level.
|
||||
|
||||
The Istio application, `cluster-istio`, is added as an application to the cluster's `system` project.
|
||||
|
||||
When Istio is enabled in the cluster, the label for Istio sidecar auto injection,`istio-injection=enabled`, will be automatically added to each new namespace in this cluster. This automatically enables Istio sidecar injection in all new workloads that are deployed in those namespaces. You will need to manually enable Istio in preexisting namespaces and workloads.
|
||||
|
||||
### [Next: Enable Istio in a Namespace]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/tools/istio/setup/enable-istio-in-namespace)
|
||||
+53
@@ -0,0 +1,53 @@
|
||||
---
|
||||
title: Enable Istio with Pod Security Policies
|
||||
aliases:
|
||||
- /rancher/v2.0-v2.4/en/cluster-admin/tools/istio/setup/enable-istio-in-cluster/enable-istio-with-psp
|
||||
- /rancher/v2.0-v2.4/en/istio/legacy/setup/enable-istio-in-cluster/enable-istio-with-psp
|
||||
- /rancher/v2.0-v2.4/en/istio/v2.3.x-v2.4.x/setup/enable-istio-in-cluster/enable-istio-with-psp
|
||||
- /rancher/v2.x/en/istio/v2.3.x-v2.4.x/setup/enable-istio-in-cluster/enable-istio-with-psp/
|
||||
---
|
||||
|
||||
>**Note:** The following guide is only for RKE provisioned clusters.
|
||||
|
||||
If you have restrictive Pod Security Policies enabled, then Istio may not be able to function correctly, because it needs certain permissions in order to install itself and manage pod infrastructure. In this section, we will configure a cluster with PSPs enabled for an Istio install, and also set up the Istio CNI plugin.
|
||||
|
||||
The Istio CNI plugin removes the need for each application pod to have a privileged `NET_ADMIN` container. For further information, see the [Istio CNI Plugin docs](https://istio.io/docs/setup/additional-setup/cni). Please note that the [Istio CNI Plugin is in alpha](https://istio.io/about/feature-stages/).
|
||||
|
||||
- 1. [Configure the System Project Policy to allow Istio install.](#1-configure-the-system-project-policy-to-allow-istio-install)
|
||||
- 2. [Install the CNI plugin in the System project.](#2-install-the-cni-plugin-in-the-system-project)
|
||||
- 3. [Install Istio.](#3-install-istio)
|
||||
|
||||
### 1. Configure the System Project Policy to allow Istio install
|
||||
|
||||
1. From the main menu of the **Dashboard**, select **Projects/Namespaces**.
|
||||
1. Find the **Project: System** project and select the **⋮ > Edit**.
|
||||
1. Change the Pod Security Policy option to be unrestricted, then click Save.
|
||||
|
||||
|
||||
### 2. Install the CNI Plugin in the System Project
|
||||
|
||||
1. From the main menu of the **Dashboard**, select **Projects/Namespaces**.
|
||||
1. Select the **Project: System** project.
|
||||
1. Choose **Tools > Catalogs** in the navigation bar.
|
||||
1. Add a catalog with the following:
|
||||
1. Name: istio-cni
|
||||
1. Catalog URL: https://github.com/istio/cni
|
||||
1. Branch: The branch that matches your current release, for example: `release-1.4`.
|
||||
1. From the main menu select **Apps**
|
||||
1. Click Launch and select istio-cni
|
||||
1. Update the namespace to be "kube-system"
|
||||
1. In the answers section, click "Edit as YAML" and paste in the following, then click launch:
|
||||
|
||||
```
|
||||
---
|
||||
logLevel: "info"
|
||||
excludeNamespaces:
|
||||
- "istio-system"
|
||||
- "kube-system"
|
||||
```
|
||||
|
||||
### 3. Install Istio
|
||||
|
||||
Follow the [primary instructions]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/tools/istio/setup/enable-istio-in-cluster/), adding a custom answer: `istio_cni.enabled: true`.
|
||||
|
||||
After Istio has finished installing, the Apps page in System Projects should show both istio and `istio-cni` applications deployed successfully. Sidecar injection will now be functional.
|
||||
+53
@@ -0,0 +1,53 @@
|
||||
---
|
||||
title: 2. Enable Istio in a Namespace
|
||||
weight: 2
|
||||
aliases:
|
||||
- /rancher/v2.0-v2.4/en/cluster-admin/tools/istio/setup/enable-istio-in-namespace
|
||||
- /rancher/v2.0-v2.4/en/istio/legacy/setup/enable-istio-in-namespace
|
||||
- /rancher/v2.0-v2.4/en/istio/v2.3.x-v2.4.x/setup/enable-istio-in-namespace
|
||||
- /rancher/v2.x/en/istio/v2.3.x-v2.4.x/setup/enable-istio-in-namespace/
|
||||
---
|
||||
|
||||
You will need to manually enable Istio in each namespace that you want to be tracked or controlled by Istio. When Istio is enabled in a namespace, the Envoy sidecar proxy will be automatically injected into all new workloads that are deployed in the namespace.
|
||||
|
||||
This namespace setting will only affect new workloads in the namespace. Any preexisting workloads will need to be re-deployed to leverage the sidecar auto injection.
|
||||
|
||||
> **Prerequisite:** To enable Istio in a namespace, the cluster must have Istio enabled.
|
||||
|
||||
1. In the Rancher UI, go to the cluster view. Click the **Projects/Namespaces** tab.
|
||||
1. Go to the namespace where you want to enable the Istio sidecar auto injection and click the **⋮.**
|
||||
1. Click **Edit.**
|
||||
1. In the **Istio sidecar auto injection** section, click **Enable.**
|
||||
1. Click **Save.**
|
||||
|
||||
**Result:** The namespace now has the label `istio-injection=enabled`. All new workloads deployed in this namespace will have the Istio sidecar injected by default.
|
||||
|
||||
### Verifying that Automatic Istio Sidecar Injection is Enabled
|
||||
|
||||
To verify that Istio is enabled, deploy a hello-world workload in the namespace. Go to the workload and click the pod name. In the **Containers** section, you should see the `istio-proxy` container.
|
||||
|
||||
### Excluding Workloads from Being Injected with the Istio Sidecar
|
||||
|
||||
If you need to exclude a workload from getting injected with the Istio sidecar, use the following annotation on the workload:
|
||||
|
||||
```
|
||||
sidecar.istio.io/inject: “false”
|
||||
```
|
||||
|
||||
To add the annotation to a workload,
|
||||
|
||||
1. From the **Global** view, open the project that has the workload that should not have the sidecar.
|
||||
1. Click **Resources > Workloads.**
|
||||
1. Go to the workload that should not have the sidecar and click **⋮ > Edit.**
|
||||
1. Click **Show Advanced Options.** Then expand the **Labels & Annotations** section.
|
||||
1. Click **Add Annotation.**
|
||||
1. In the **Key** field, enter `sidecar.istio.io/inject`.
|
||||
1. In the **Value** field, enter `false`.
|
||||
1. Click **Save.**
|
||||
|
||||
**Result:** The Istio sidecar will not be injected into the workload.
|
||||
|
||||
> **NOTE:** If you are having issues with a Job you deployed not completing, you will need to add this annotation to your pod using the provided steps. Since Istio Sidecars run indefinitely, a Job cannot be considered complete even after its task has completed.
|
||||
|
||||
|
||||
### [Next: Select the Nodes ]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/tools/istio/setup/node-selectors)
|
||||
@@ -0,0 +1,135 @@
|
||||
---
|
||||
title: 5. Set up the Istio Gateway
|
||||
weight: 5
|
||||
aliases:
|
||||
- /rancher/v2.0-v2.4/en/cluster-admin/tools/istio/setup/gateway
|
||||
- /rancher/v2.0-v2.4/en/istio/legacy/setup/gateway
|
||||
- /rancher/v2.0-v2.4/en/istio/v2.3.x-v2.4.x/setup/gateway
|
||||
- /rancher/v2.x/en/istio/v2.3.x-v2.4.x/setup/gateway/
|
||||
---
|
||||
|
||||
The gateway to each cluster can have its own port or load balancer, which is unrelated to a service mesh. By default, each Rancher-provisioned cluster has one NGINX ingress controller allowing traffic into the cluster.
|
||||
|
||||
You can use the NGINX ingress controller with or without Istio installed. If this is the only gateway to your cluster, Istio will be able to route traffic from service to service, but Istio will not be able to receive traffic from outside the cluster.
|
||||
|
||||
To allow Istio to receive external traffic, you need to enable Istio's gateway, which works as a north-south proxy for external traffic. When you enable the Istio gateway, the result is that your cluster will have two ingresses.
|
||||
|
||||
You will also need to set up a Kubernetes gateway for your services. This Kubernetes resource points to Istio's implementation of the ingress gateway to the cluster.
|
||||
|
||||
You can route traffic into the service mesh with a load balancer or just Istio's NodePort gateway. This section describes how to set up the NodePort gateway.
|
||||
|
||||
For more information on the Istio gateway, refer to the [Istio documentation.](https://istio.io/docs/reference/config/networking/v1alpha3/gateway/)
|
||||
|
||||
|
||||
|
||||
# Enable the Istio Gateway
|
||||
|
||||
The ingress gateway is a Kubernetes service that will be deployed in your cluster. There is only one Istio gateway per cluster.
|
||||
|
||||
1. Go to the cluster where you want to allow outside traffic into Istio.
|
||||
1. Click **Tools > Istio.**
|
||||
1. Expand the **Ingress Gateway** section.
|
||||
1. Under **Enable Ingress Gateway,** click **True.** The default type of service for the Istio gateway is NodePort. You can also configure it as a [load balancer.]({{<baseurl>}}/rancher/v2.0-v2.4/en/k8s-in-rancher/load-balancers-and-ingress/load-balancers/)
|
||||
1. Optionally, configure the ports, service types, node selectors and tolerations, and resource requests and limits for this service. The default resource requests for CPU and memory are the minimum recommended resources.
|
||||
1. Click **Save.**
|
||||
|
||||
**Result:** The gateway is deployed, which allows Istio to receive traffic from outside the cluster.
|
||||
|
||||
# Add a Kubernetes Gateway that Points to the Istio Gateway
|
||||
|
||||
To allow traffic to reach Ingress, you will also need to provide a Kubernetes gateway resource in your YAML that points to Istio's implementation of the ingress gateway to the cluster.
|
||||
|
||||
1. Go to the namespace where you want to deploy the Kubernetes gateway and click **Import YAML.**
|
||||
1. Upload the gateway YAML as a file or paste it into the form. An example gateway YAML is provided below.
|
||||
1. Click **Import.**
|
||||
|
||||
```yaml
|
||||
apiVersion: networking.istio.io/v1alpha3
|
||||
kind: Gateway
|
||||
metadata:
|
||||
name: bookinfo-gateway
|
||||
spec:
|
||||
selector:
|
||||
istio: ingressgateway # use istio default controller
|
||||
servers:
|
||||
- port:
|
||||
number: 80
|
||||
name: http
|
||||
protocol: HTTP
|
||||
hosts:
|
||||
- "*"
|
||||
---
|
||||
apiVersion: networking.istio.io/v1alpha3
|
||||
kind: VirtualService
|
||||
metadata:
|
||||
name: bookinfo
|
||||
spec:
|
||||
hosts:
|
||||
- "*"
|
||||
gateways:
|
||||
- bookinfo-gateway
|
||||
http:
|
||||
- match:
|
||||
- uri:
|
||||
exact: /productpage
|
||||
- uri:
|
||||
prefix: /static
|
||||
- uri:
|
||||
exact: /login
|
||||
- uri:
|
||||
exact: /logout
|
||||
- uri:
|
||||
prefix: /api/v1/products
|
||||
route:
|
||||
- destination:
|
||||
host: productpage
|
||||
port:
|
||||
number: 9080
|
||||
```
|
||||
|
||||
**Result:** You have configured your gateway resource so that Istio can receive traffic from outside the cluster.
|
||||
|
||||
Confirm that the resource exists by running:
|
||||
```
|
||||
kubectl get gateway -A
|
||||
```
|
||||
|
||||
The result should be something like this:
|
||||
```
|
||||
NAME AGE
|
||||
bookinfo-gateway 64m
|
||||
```
|
||||
|
||||
### Access the ProductPage Service from a Web Browser
|
||||
|
||||
To test and see if the BookInfo app deployed correctly, the app can be viewed a web browser using the Istio controller IP and port, combined with the request name specified in your Kubernetes gateway resource:
|
||||
|
||||
`http://<IP of Istio controller>:<Port of istio controller>/productpage`
|
||||
|
||||
To get the ingress gateway URL and port,
|
||||
|
||||
1. Go to the `System` project in your cluster.
|
||||
1. Within the `System` project, go to `Resources` > `Workloads` then scroll down to the `istio-system` namespace.
|
||||
1. Within `istio-system`, there is a workload named `istio-ingressgateway`. Under the name of this workload, you should see links, such as `80/tcp`.
|
||||
1. Click one of those links. This should show you the URL of the ingress gateway in your web browser. Append `/productpage` to the URL.
|
||||
|
||||
**Result:** You should see the BookInfo app in the web browser.
|
||||
|
||||
For help inspecting the Istio controller URL and ports, try the commands the [Istio documentation.](https://istio.io/docs/tasks/traffic-management/ingress/ingress-control/#determining-the-ingress-ip-and-ports)
|
||||
|
||||
# Troubleshooting
|
||||
|
||||
The [official Istio documentation](https://istio.io/docs/tasks/traffic-management/ingress/ingress-control/#troubleshooting) suggests `kubectl` commands to inspect the correct ingress host and ingress port for external requests.
|
||||
|
||||
### Confirming that the Kubernetes Gateway Matches Istio's Ingress Controller
|
||||
|
||||
You can try the steps in this section to make sure the Kubernetes gateway is configured properly.
|
||||
|
||||
In the gateway resource, the selector refers to Istio's default ingress controller by its label, in which the key of the label is `istio` and the value is `ingressgateway`. To make sure the label is appropriate for the gateway, do the following:
|
||||
|
||||
1. Go to the `System` project in your cluster.
|
||||
1. Within the `System` project, go to the namespace `istio-system`.
|
||||
1. Within `istio-system`, there is a workload named `istio-ingressgateway`.
|
||||
1. Click the name of this workload and go to the **Labels and Annotations** section. You should see that it has the key `istio` and the value `ingressgateway`. This confirms that the selector in the Gateway resource matches Istio's default ingress controller.
|
||||
|
||||
### [Next: Set up Istio's Components for Traffic Management]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/tools/istio/setup/set-up-traffic-management)
|
||||
+43
@@ -0,0 +1,43 @@
|
||||
---
|
||||
title: 3. Select the Nodes Where Istio Components Will be Deployed
|
||||
weight: 3
|
||||
aliases:
|
||||
- /rancher/v2.0-v2.4/en/cluster-admin/tools/istio/setup/node-selectors
|
||||
- /rancher/v2.0-v2.4/en/istio/legacy/setup/node-selectors
|
||||
- /rancher/v2.0-v2.4/en/istio/v2.3.x-v2.4.x/setup/node-selectors
|
||||
- /rancher/v2.x/en/istio/v2.3.x-v2.4.x/setup/node-selectors/
|
||||
---
|
||||
|
||||
> **Prerequisite:** Your cluster needs a worker node that can designated for Istio. The worker node should meet the [resource requirements.]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/tools/istio/resources)
|
||||
|
||||
This section describes how use node selectors to configure Istio components to be deployed on a designated node.
|
||||
|
||||
In larger deployments, it is strongly advised that Istio's infrastructure be placed on dedicated nodes in the cluster by adding a node selector for each Istio component.
|
||||
|
||||
# Adding a Label to the Istio Node
|
||||
|
||||
First, add a label to the node where Istio components should be deployed. This label can have any key-value pair. For this example, we will use the key `istio` and the value `enabled`.
|
||||
|
||||
1. From the cluster view, go to the **Nodes** tab.
|
||||
1. Go to a worker node that will host the Istio components and click **⋮ > Edit.**
|
||||
1. Expand the **Labels & Annotations** section.
|
||||
1. Click **Add Label.**
|
||||
1. In the fields that appear, enter `istio` for the key and `enabled` for the value.
|
||||
1. Click **Save.**
|
||||
|
||||
**Result:** A worker node has the label that will allow you to designate it for Istio components.
|
||||
|
||||
# Configuring Istio Components to Use the Labeled Node
|
||||
|
||||
Configure each Istio component to be deployed to the node with the Istio label. Each Istio component can be configured individually, but in this tutorial, we will configure all of the components to be scheduled on the same node for the sake of simplicity.
|
||||
|
||||
For larger deployments, it is recommended to schedule each component of Istio onto separate nodes.
|
||||
|
||||
1. From the cluster view, click **Tools > Istio.**
|
||||
1. Expand the **Pilot** section and click **Add Selector** in the form that appears. Enter the node selector label that you added to the Istio node. In our case, we are using the key `istio` and the value `enabled.`
|
||||
1. Repeat the previous step for the **Mixer** and **Tracing** sections.
|
||||
1. Click **Save.**
|
||||
|
||||
**Result:** The Istio components will be deployed on the Istio node.
|
||||
|
||||
### [Next: Add Deployments and Services]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/tools/istio/setup/deploy-workloads)
|
||||
+66
@@ -0,0 +1,66 @@
|
||||
---
|
||||
title: 6. Set up Istio's Components for Traffic Management
|
||||
weight: 6
|
||||
aliases:
|
||||
- /rancher/v2.0-v2.4/en/cluster-admin/tools/istio/setup/set-up-traffic-management
|
||||
- /rancher/v2.0-v2.4/en/istio/legacy/setup/set-up-traffic-management
|
||||
- /rancher/v2.0-v2.4/en/istio/v2.3.x-v2.4.x/setup/set-up-traffic-management
|
||||
- /rancher/v2.x/en/istio/v2.3.x-v2.4.x/setup/set-up-traffic-management/
|
||||
---
|
||||
|
||||
A central advantage of traffic management in Istio is that it allows dynamic request routing. Some common applications for dynamic request routing include canary deployments and blue/green deployments. The two key resources in Istio traffic management are *virtual services* and *destination rules*.
|
||||
|
||||
- [Virtual services](https://istio.io/docs/reference/config/networking/v1alpha3/virtual-service/) intercept and direct traffic to your Kubernetes services, allowing you to divide percentages of traffic from a request to different services. You can use them to define a set of routing rules to apply when a host is addressed.
|
||||
- [Destination rules](https://istio.io/docs/reference/config/networking/v1alpha3/destination-rule/) serve as the single source of truth about which service versions are available to receive traffic from virtual services. You can use these resources to define policies that apply to traffic that is intended for a service after routing has occurred.
|
||||
|
||||
This section describes how to add an example virtual service that corresponds to the `reviews` microservice in the sample BookInfo app. The purpose of this service is to divide traffic between two versions of the `reviews` service.
|
||||
|
||||
In this example, we take the traffic to the `reviews` service and intercept it so that 50 percent of it goes to `v1` of the service and 50 percent goes to `v2`.
|
||||
|
||||
After this virtual service is deployed, we will generate traffic and see from the Kiali visualization that traffic is being routed evenly between the two versions of the service.
|
||||
|
||||
To deploy the virtual service and destination rules for the `reviews` service,
|
||||
|
||||
1. Go to the project view and click **Import YAML.**
|
||||
1. Copy resources below into the form.
|
||||
1. Click **Import.**
|
||||
|
||||
```
|
||||
apiVersion: networking.istio.io/v1alpha3
|
||||
kind: VirtualService
|
||||
metadata:
|
||||
name: reviews
|
||||
spec:
|
||||
hosts:
|
||||
- reviews
|
||||
http:
|
||||
- route:
|
||||
- destination:
|
||||
host: reviews
|
||||
subset: v1
|
||||
weight: 50
|
||||
- destination:
|
||||
host: reviews
|
||||
subset: v3
|
||||
weight: 50
|
||||
---
|
||||
apiVersion: networking.istio.io/v1alpha3
|
||||
kind: DestinationRule
|
||||
metadata:
|
||||
name: reviews
|
||||
spec:
|
||||
host: reviews
|
||||
subsets:
|
||||
- name: v1
|
||||
labels:
|
||||
version: v1
|
||||
- name: v2
|
||||
labels:
|
||||
version: v2
|
||||
- name: v3
|
||||
labels:
|
||||
version: v3
|
||||
```
|
||||
**Result:** When you generate traffic to this service (for example, by refreshing the ingress gateway URL), the Kiali traffic graph will reflect that traffic to the `reviews` service is divided evenly between `v1` and `v3`.
|
||||
|
||||
### [Next: Generate and View Traffic]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/tools/istio/setup/view-traffic)
|
||||
@@ -0,0 +1,26 @@
|
||||
---
|
||||
title: Setup Guide
|
||||
weight: 2
|
||||
aliases:
|
||||
- /rancher/v2.0-v2.4/en/cluster-admin/tools/istio/setup
|
||||
- /rancher/v2.0-v2.4/en/istio/legacy/setup
|
||||
- /rancher/v2.0-v2.4/en/istio/v2.3.x-v2.4.x/setup
|
||||
- /rancher/v2.x/en/istio/v2.3.x-v2.4.x/setup/
|
||||
---
|
||||
|
||||
This section describes how to enable Istio and start using it in your projects.
|
||||
|
||||
This section assumes that you have Rancher installed, and you have a Rancher-provisioned Kubernetes cluster where you would like to set up Istio.
|
||||
|
||||
If you use Istio for traffic management, you will need to allow external traffic to the cluster. In that case, you will need to follow all of the steps below.
|
||||
|
||||
> **Quick Setup** If you don't need external traffic to reach Istio, and you just want to set up Istio for monitoring and tracing traffic within the cluster, skip the steps for [setting up the Istio gateway]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/tools/istio/setup/gateway) and [setting up Istio's components for traffic management.]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/tools/istio/setup/set-up-traffic-management)
|
||||
|
||||
1. [Enable Istio in the cluster.]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/tools/istio/setup/enable-istio-in-cluster)
|
||||
1. [Enable Istio in all the namespaces where you want to use it.]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/tools/istio/setup/enable-istio-in-namespace)
|
||||
1. [Select the nodes where the main Istio components will be deployed.]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/tools/istio/setup/node-selectors)
|
||||
1. [Add deployments and services that have the Istio sidecar injected.]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/tools/istio/setup/deploy-workloads)
|
||||
1. [Set up the Istio gateway. ]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/tools/istio/setup/gateway)
|
||||
1. [Set up Istio's components for traffic management.]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/tools/istio/setup/set-up-traffic-management)
|
||||
1. [Generate traffic and see Istio in action.]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/tools/istio/setup/#view-traffic)
|
||||
|
||||
+31
@@ -0,0 +1,31 @@
|
||||
---
|
||||
title: 7. Generate and View Traffic
|
||||
weight: 7
|
||||
aliases:
|
||||
- /rancher/v2.0-v2.4/en/cluster-admin/tools/istio/setup/view-traffic
|
||||
- /rancher/v2.0-v2.4/en/istio/legacy/setup/view-traffic
|
||||
- /rancher/v2.0-v2.4/en/istio/v2.3.x-v2.4.x/setup/view-traffic
|
||||
- /rancher/v2.x/en/istio/v2.3.x-v2.4.x/setup/view-traffic/
|
||||
---
|
||||
|
||||
This section describes how to view the traffic that is being managed by Istio.
|
||||
|
||||
# The Kiali Traffic Graph
|
||||
|
||||
Rancher integrates a Kiali graph into the Rancher UI. The Kiali graph provides a powerful way to visualize the topology of your Istio service mesh. It shows you which services communicate with each other.
|
||||
|
||||
To see the traffic graph,
|
||||
|
||||
1. From the project view in Rancher, click **Resources > Istio.**
|
||||
1. Go to the **Traffic Graph** tab. This tab has the Kiali network visualization integrated into the UI.
|
||||
|
||||
If you refresh the URL to the BookInfo app several times, you should be able to see green arrows on the Kiali graph showing traffic to `v1` and `v3` of the `reviews` service. The control panel on the right side of the graph lets you configure details including how many minutes of the most recent traffic should be shown on the graph.
|
||||
|
||||
For additional tools and visualizations, you can go to each UI for Kiali, Jaeger, Grafana, and Prometheus by clicking their icons in the top right corner of the page.
|
||||
|
||||
# Viewing Traffic Metrics
|
||||
|
||||
Istio’s monitoring features provide visibility into the performance of all your services.
|
||||
|
||||
1. From the project view in Rancher, click **Resources > Istio.**
|
||||
1. Go to the **Traffic Metrics** tab. After traffic is generated in your cluster, you should be able to see metrics for **Success Rate, Request Volume, 4xx Response Count, Project 5xx Response Count** and **Request Duration.**
|
||||
@@ -0,0 +1,207 @@
|
||||
---
|
||||
title: Notifiers
|
||||
weight: 4
|
||||
aliases:
|
||||
- /rancher/v2.0-v2.4/en/project-admin/tools/notifiers
|
||||
- /rancher/v2.0-v2.4/en/cluster-admin/tools/notifiers
|
||||
- /rancher/v2.0-v2.4/en/monitoring-alerting/legacy/notifiers
|
||||
- /rancher/v2.0-v2.4/en/monitoring-alerting/v2.0.x-v2.4.x/notifiers
|
||||
- /rancher/v2.x/en/monitoring-alerting/v2.0.x-v2.4.x/notifiers/
|
||||
- /rancher/v2.x/en/monitoring-alerting/v2.0.x-v2.4.x/cluster-alerts/default-alerts/
|
||||
---
|
||||
|
||||
Notifiers are services that inform you of alert events. You can configure notifiers to send alert notifications to staff best suited to take corrective action.
|
||||
|
||||
Rancher integrates with a variety of popular IT services, including:
|
||||
|
||||
- **Slack**: Send alert notifications to your Slack channels.
|
||||
- **Email**: Choose email recipients for alert notifications.
|
||||
- **PagerDuty**: Route notifications to staff by phone, SMS, or personal email.
|
||||
- **WebHooks**: Update a webpage with alert notifications.
|
||||
- **WeChat**: (Available as of v2.2.0) Send alert notifications to your Enterprise WeChat contacts.
|
||||
- **DingTalk**: (Available as of v2.4.6) Send alert notifications to DingTalk using a webhook.
|
||||
- **Microsoft Teams**: (Available as of v2.4.6) Send alert notifications to Teams using a webhook.
|
||||
|
||||
This section covers the following topics:
|
||||
|
||||
- [Roles-based access control for notifiers](#roles-based-access-control-for-notifiers)
|
||||
- [Adding notifiers](#adding-notifiers)
|
||||
- [Configuration](#configuration)
|
||||
- [Managing notifiers](#managing-notifiers)
|
||||
- [Example payload for a webhook alert notifier](#example-payload-for-a-webhook-alert-notifier)
|
||||
|
||||
# Roles-based Access Control for Notifiers
|
||||
|
||||
Notifiers are configured at the cluster level. This model ensures that only cluster owners need to configure notifiers, leaving project owners to simply configure alerts in the scope of their projects. You don't need to dispense privileges like SMTP server access or cloud account access.
|
||||
|
||||
# Adding Notifiers
|
||||
|
||||
Set up a notifier so that you can begin configuring and sending alerts.
|
||||
|
||||
1. From the **Global View**, open the cluster that you want to add a notifier.
|
||||
1. From the main menu, select **Tools > Notifiers**. Then click **Add Notifier**.
|
||||
1. Select the service you want to use as your notifier, and then fill out the form. For help filling out the form, refer to the configuration section below.
|
||||
1. Click **Test.** You should receive a notification confirming that the notifier is configured correctly.
|
||||
1. Click **Add** to complete adding the notifier.
|
||||
|
||||
**Result:** Your notifier is added to Rancher.
|
||||
|
||||
# Configuration
|
||||
|
||||
- [Slack](#slack)
|
||||
- [Email](#email)
|
||||
- [PagerDuty](#pagerduty)
|
||||
- [Webhook](#webhook)
|
||||
- [WeChat](#wechat)
|
||||
- [DingTalk](#dingtalk)
|
||||
- [Microsoft Teams](#microsoft-teams)
|
||||
|
||||
### Slack
|
||||
|
||||
| Field | Explanation |
|
||||
|----------|----------------------|
|
||||
| Name | Enter a **Name** for the notifier. |
|
||||
| URL | From Slack, create a webhook. For instructions, see the [Slack Documentation](https://get.slack.help/hc/en-us/articles/115005265063-Incoming-WebHooks-for-Slack). Then enter the Slack webhook URL. |
|
||||
| Default Channel | Enter the name of the channel that you want to send alert notifications in the following format: `#<channelname>`. Both public and private channels are supported. |
|
||||
| Proxy URL | Proxy for the Slack webhook. |
|
||||
| Send Resolved Alerts | _Available as of v2.3.0_ Whether to send a follow-up notification if an alert has been resolved (e.g. [Resolved] High CPU Usage) |
|
||||
|
||||
**Validation:** Click **Test**. If the test is successful, the Slack channel you're configuring for the notifier outputs **Slack setting validated.**
|
||||
|
||||
### Email
|
||||
|
||||
| Field | Explanation |
|
||||
|----------|----------------------|
|
||||
| Name | Enter a **Name** for the notifier. |
|
||||
| Default Recipient Address | Enter the email address that you want to receive the notification. |
|
||||
| Send Resolved Alerts | _Available as of v2.3.0_ Whether to send a follow-up notification if an alert has been resolved (e.g. [Resolved] High CPU Usage) |
|
||||
|
||||
SMTP Server Configuration:
|
||||
|
||||
| Field | Explanation |
|
||||
|----------|----------------------|
|
||||
| Sender | Enter an email address available on your mail server that you want to send the notification. |
|
||||
| Host | Enter the IP address or hostname for your SMTP server. Example: `smtp.email.com` |
|
||||
| Port | In the **Port** field, enter the port used for email. Typically, TLS uses `587` and SSL uses `465`. |
|
||||
| Use TLS | If you're using TLS, make sure **Use TLS** is selected. |
|
||||
| Username | Username to authenticate with the SMTP server. |
|
||||
| Password | Password to authenticate with the SMTP server. |
|
||||
|
||||
**Validation:** Click **Test**. If the test is successful, Rancher prints **settings validated** and you receive a test notification email.
|
||||
|
||||
### PagerDuty
|
||||
|
||||
| Field | Explanation |
|
||||
|----------|----------------------|
|
||||
| Name | Enter a **Name** for the notifier. |
|
||||
| Default Integration Key | From PagerDuty, create a Prometheus integration. For instructions, see the [PagerDuty Documentation](https://www.pagerduty.com/docs/guides/prometheus-integration-guide/). Then enter the integration key.
|
||||
| Service Key | The same as the integration key. For instructions on creating a Prometheus integration, see the [PagerDuty Documentation](https://www.pagerduty.com/docs/guides/prometheus-integration-guide/). Then enter the integration key. |
|
||||
| Send Resolved Alerts | _Available as of v2.3.0_ Whether to send a follow-up notification if an alert has been resolved (e.g. [Resolved] High CPU Usage) |
|
||||
|
||||
**Validation:** Click **Test**. If the test is successful, your PagerDuty endpoint outputs **PagerDuty setting validated.**
|
||||
|
||||
### Webhook
|
||||
|
||||
| Field | Explanation |
|
||||
|----------|----------------------|
|
||||
| Name | Enter a **Name** for the notifier. |
|
||||
| URL | Using the app of your choice, create a webhook URL. |
|
||||
| Proxy URL | Proxy for the webhook. |
|
||||
| Send Resolved Alerts | _Available as of v2.3.0_ Whether to send a follow-up notification if an alert has been resolved (e.g. [Resolved] High CPU Usage) |
|
||||
|
||||
**Validation:** Click **Test**. If the test is successful, the URL you're configuring as a notifier outputs **Webhook setting validated.**
|
||||
|
||||
### WeChat
|
||||
|
||||
_Available as of v2.2.0_
|
||||
|
||||
| Field | Explanation |
|
||||
|----------|----------------------|
|
||||
| Name | Enter a **Name** for the notifier. |
|
||||
| Corporation ID | Enter the "EnterpriseID" of your corporation. You can get it fro the [Profile page](https://work.weixin.qq.com/wework_admin/frame#profile). |
|
||||
| Application Agent ID | From Enterprise WeChat, create an application in the [Application page](https://work.weixin.qq.com/wework_admin/frame#apps), and then enter the "AgentId" of this application. You will also need to enter the application secret. |
|
||||
| Application Secret | The secret that corresponds to the Application Agent ID. |
|
||||
| Recipient Type | Party, tag, or user. |
|
||||
| Default Recipient | The default recipient ID should correspond to the recipient type. It should be the party ID, tag ID or user account that you want to receive the notification. You could get contact information from [Contacts page](https://work.weixin.qq.com/wework_admin/frame#contacts). |
|
||||
| Proxy URL | If you are using a proxy, enter the proxy URL. |
|
||||
| Send Resolved Alerts | _Available as of v2.3.0_ Whether to send a follow-up notification if an alert has been resolved (e.g. [Resolved] High CPU Usage) |
|
||||
|
||||
**Validation:** Click **Test.** If the test is successful, you should receive an alert message.
|
||||
|
||||
### DingTalk
|
||||
|
||||
_Available as of v2.4.6_
|
||||
|
||||
| Field | Explanation |
|
||||
|----------|----------------------|
|
||||
| Name | Enter a **Name** for the notifier. |
|
||||
| Webhook URL | Enter the DingTalk webhook URL. For help setting up the webhook, refer to the [DingTalk documentation.](https://www.alibabacloud.com/help/doc-detail/52872.htm) |
|
||||
| Secret | Optional: Enter a secret for the DingTalk webhook. |
|
||||
| Proxy URL | Optional: Enter a proxy for the DingTalk webhook. |
|
||||
| Send Resolved Alerts | Whether to send a follow-up notification if an alert has been resolved (e.g. [Resolved] High CPU Usage) |
|
||||
|
||||
**Validation:** Click **Test.** If the test is successful, the DingTalk notifier output is **DingTalk setting validated.**
|
||||
|
||||
### Microsoft Teams
|
||||
|
||||
_Available as of v2.4.6_
|
||||
|
||||
| Field | Explanation |
|
||||
|----------|----------------------|
|
||||
| Name | Enter a **Name** for the notifier. |
|
||||
| Webhook URL | Enter the Microsoft Teams webhook URL. For help setting up the webhook, refer to the [Teams Documentation.](https://docs.microsoft.com/en-us/microsoftteams/platform/webhooks-and-connectors/how-to/add-incoming-webhook) |
|
||||
| Proxy URL | Optional: Enter a proxy for the Teams webhook. |
|
||||
| Send Resolved Alerts | Whether to send a follow-up notification if an alert has been resolved (e.g. [Resolved] High CPU Usage) |
|
||||
|
||||
**Validation:** Click **Test.** If the test is successful, the Teams notifier output is **MicrosoftTeams setting validated.**
|
||||
|
||||
# Managing Notifiers
|
||||
|
||||
After you set up notifiers, you can manage them. From the **Global** view, open the cluster that you want to manage your notifiers. Select **Tools > Notifiers**. You can:
|
||||
|
||||
- **Edit** their settings that you configured during their initial setup.
|
||||
- **Clone** them, to quickly setup slightly different notifiers.
|
||||
- **Delete** them when they're no longer necessary.
|
||||
|
||||
# Example Payload for a Webhook Alert Notifier
|
||||
|
||||
```json
|
||||
{
|
||||
"receiver": "c-2a3bc:kube-components-alert",
|
||||
"status": "firing",
|
||||
"alerts": [
|
||||
{
|
||||
"status": "firing",
|
||||
"labels": {
|
||||
"alert_name": "Scheduler is unavailable",
|
||||
"alert_type": "systemService",
|
||||
"cluster_name": "mycluster (ID: c-2a3bc)",
|
||||
"component_name": "scheduler",
|
||||
"group_id": "c-2a3bc:kube-components-alert",
|
||||
"logs": "Get http://127.0.0.1:10251/healthz: dial tcp 127.0.0.1:10251: connect: connection refused",
|
||||
"rule_id": "c-2a3bc:kube-components-alert_scheduler-system-service",
|
||||
"severity": "critical"
|
||||
},
|
||||
"annotations": {},
|
||||
"startsAt": "2020-01-30T19:18:13.321684733Z",
|
||||
"endsAt": "0001-01-01T00:00:00Z",
|
||||
"generatorURL": ""
|
||||
}
|
||||
],
|
||||
"groupLabels": {
|
||||
"component_name": "scheduler",
|
||||
"rule_id": "c-2a3bc:kube-components-alert_scheduler-system-service"
|
||||
},
|
||||
"commonLabels": {
|
||||
"alert_name": "Scheduler is unavailable",
|
||||
"alert_type": "systemService",
|
||||
"cluster_name": "mycluster (ID: c-2a3bc)"
|
||||
}
|
||||
}
|
||||
```
|
||||
# What's Next?
|
||||
|
||||
After creating a notifier, set up alerts to receive notifications of Rancher system events.
|
||||
|
||||
- [Cluster owners]({{<baseurl>}}/rancher/v2.0-v2.4/en/admin-settings/rbac/cluster-project-roles/#cluster-roles) can set up alerts at the [cluster level]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/tools/alerts/).
|
||||
- [Project owners]({{<baseurl>}}/rancher/v2.0-v2.4/en/admin-settings/rbac/cluster-project-roles/#project-roles) can set up alerts at the [project level]({{<baseurl>}}/rancher/v2.0-v2.4/en/project-admin/tools/alerts/).
|
||||
@@ -0,0 +1,99 @@
|
||||
---
|
||||
title: OPA Gatekeeper
|
||||
weight: 17
|
||||
aliases:
|
||||
- /rancher/v2.0-v2.4/en/cluster-admin/tools/opa-gatekeeper
|
||||
- /rancher/v2.0-v2.4/en/opa-gatekeper/Open%20Policy%20Agent
|
||||
- /rancher/v2.0-v2.4/en/opa-gatekeper
|
||||
---
|
||||
_Available as of v2.4.0_
|
||||
|
||||
To ensure consistency and compliance, every organization needs the ability to define and enforce policies in its environment in an automated way. [OPA (Open Policy Agent)](https://www.openpolicyagent.org/) is a policy engine that facilitates policy-based control for cloud native environments. Rancher provides the ability to enable OPA Gatekeeper in Kubernetes clusters, and also installs a couple of built-in policy definitions, which are also called constraint templates.
|
||||
|
||||
OPA provides a high-level declarative language that lets you specify policy as code and ability to extend simple APIs to offload policy decision-making.
|
||||
|
||||
[OPA Gatekeeper](https://github.com/open-policy-agent/gatekeeper) is a project that provides integration between OPA and Kubernetes. OPA Gatekeeper provides:
|
||||
|
||||
- An extensible, parameterized policy library.
|
||||
- Native Kubernetes CRDs for instantiating the policy library, also called “constraints."
|
||||
- Native Kubernetes CRDs for extending the policy library, also called "constraint templates."
|
||||
- Audit functionality.
|
||||
|
||||
To read more about OPA, please refer to the [official documentation.](https://www.openpolicyagent.org/docs/latest/)
|
||||
|
||||
# How the OPA Gatekeeper Integration Works
|
||||
|
||||
Kubernetes provides the ability to extend API server functionality via admission controller webhooks, which are invoked whenever a resource is created, updated or deleted. Gatekeeper is installed as a validating webhook and enforces policies defined by Kubernetes custom resource definitions. In addition to the admission control usage, Gatekeeper provides the capability to audit existing resources in Kubernetes clusters and mark current violations of enabled policies.
|
||||
|
||||
OPA Gatekeeper is made available via Rancher's Helm system chart, and it is installed in a namespace named `gatekeeper-system.`
|
||||
|
||||
# Enabling OPA Gatekeeper in a Cluster
|
||||
|
||||
> **Prerequisites:**
|
||||
>
|
||||
> - Only administrators and cluster owners can enable OPA Gatekeeper.
|
||||
> - The dashboard needs to be enabled using the `dashboard` feature flag. For more information, refer to the [section on enabling experimental features.]({{<baseurl>}}/rancher/v2.0-v2.4/en/installation/options/feature-flags/)
|
||||
|
||||
1. Navigate to the cluster's **Dashboard** view.
|
||||
1. On the left side menu, expand the cluster menu and click on **OPA Gatekeeper.**
|
||||
1. To install Gatekeeper with the default configuration, click on **Enable Gatekeeper (v0.1.0) with defaults.**
|
||||
1. To change any default configuration, click on **Customize Gatekeeper yaml configuration.**
|
||||
|
||||
# Constraint Templates
|
||||
|
||||
[Constraint templates](https://github.com/open-policy-agent/gatekeeper#constraint-templates) are Kubernetes custom resources that define the schema and Rego logic of the OPA policy to be applied by Gatekeeper. For more information on the Rego policy language, refer to the [official documentation.](https://www.openpolicyagent.org/docs/latest/policy-language/)
|
||||
|
||||
When OPA Gatekeeper is enabled, Rancher installs some templates by default.
|
||||
|
||||
To list the constraint templates installed in the cluster, go to the left side menu under OPA Gatekeeper and click on **Templates.**
|
||||
|
||||
Rancher also provides the ability to create your own constraint templates by importing YAML definitions.
|
||||
|
||||
# Creating and Configuring Constraints
|
||||
|
||||
[Constraints](https://github.com/open-policy-agent/gatekeeper#constraints) are Kubernetes custom resources that define the scope of objects to which a specific constraint template applies to. The complete policy is defined by constraint templates and constraints together.
|
||||
|
||||
> **Prerequisites:** OPA Gatekeeper must be enabled in the cluster.
|
||||
|
||||
To list the constraints installed, go to the left side menu under OPA Gatekeeper, and click on **Constraints.**
|
||||
|
||||
New constraints can be created from a constraint template.
|
||||
|
||||
Rancher provides the ability to create a constraint by using a convenient form that lets you input the various constraint fields.
|
||||
|
||||
The **Edit as yaml** option is also available to configure the the constraint's yaml definition.
|
||||
|
||||
### Exempting Rancher's System Namespaces from Constraints
|
||||
|
||||
When a constraint is created, ensure that it does not apply to any Rancher or Kubernetes system namespaces. If the system namespaces are not excluded, then it is possible to see many resources under them marked as violations of the constraint.
|
||||
|
||||
To limit the scope of the constraint only to user namespaces, always specify these namespaces under the **Match** field of the constraint.
|
||||
|
||||
Also, the constraint may interfere with other Rancher functionality and deny system workloads from being deployed. To avoid this, exclude all Rancher-specific namespaces from your constraints.
|
||||
|
||||
# Enforcing Constraints in your Cluster
|
||||
|
||||
When the **Enforcement Action** is **Deny,** the constraint is immediately enabled and will deny any requests that violate the policy defined. By default, the enforcement value is **Deny.**
|
||||
|
||||
When the **Enforcement Action** is **Dryrun,** then any resources that violate the policy are only recorded under the constraint's status field.
|
||||
|
||||
To enforce constraints, create a constraint using the form. In the **Enforcement Action** field, choose **Deny.**
|
||||
|
||||
# Audit and Violations in your Cluster
|
||||
|
||||
OPA Gatekeeper runs a periodic audit to check if any existing resource violates any enforced constraint. The audit-interval (default 300s) can be configured while installing Gatekeeper.
|
||||
|
||||
On the Gatekeeper page, any violations of the defined constraints are listed.
|
||||
|
||||
Also under **Constraints,** the number of violations of the constraint can be found.
|
||||
|
||||
The detail view of each constraint lists information about the resource that violated the constraint.
|
||||
|
||||
# Disabling Gatekeeper
|
||||
|
||||
1. Navigate to the cluster's Dashboard view
|
||||
1. On the left side menu, expand the cluster menu and click on **OPA Gatekeeper.**
|
||||
1. Click the **⋮ > Disable**.
|
||||
|
||||
**Result:** Upon disabling OPA Gatekeeper, all constraint templates and constraints will also be deleted.
|
||||
|
||||
@@ -0,0 +1,71 @@
|
||||
---
|
||||
title: Tools for Logging, Monitoring, and More
|
||||
weight: 2033
|
||||
aliases:
|
||||
- /rancher/v2.0-v2.4/en/tools/notifiers-and-alerts/
|
||||
---
|
||||
|
||||
Rancher contains a variety of tools that aren't included in Kubernetes to assist in your DevOps operations. Rancher can integrate with external services to help your clusters run more efficiently. Tools are divided into following categories:
|
||||
|
||||
<!-- TOC -->
|
||||
|
||||
- [Logging](#logging)
|
||||
- [Monitoring](#monitoring)
|
||||
- [Alerts](#alerts)
|
||||
- [Notifiers](#notifiers)
|
||||
- [Istio](#istio)
|
||||
- [OPA Gatekeeper](#opa-gatekeeper)
|
||||
- [CIS Scans](#cis-scans)
|
||||
|
||||
<!-- /TOC -->
|
||||
|
||||
|
||||
# Logging
|
||||
|
||||
Logging is helpful because it allows you to:
|
||||
|
||||
- Capture and analyze the state of your cluster
|
||||
- Look for trends in your environment
|
||||
- Save your logs to a safe location outside of your cluster
|
||||
- Stay informed of events like a container crashing, a pod eviction, or a node dying
|
||||
- More easily debugg and troubleshoot problems
|
||||
|
||||
Rancher can integrate with Elasticsearch, splunk, kafka, syslog, and fluentd.
|
||||
|
||||
Refer to the logging documentation [here.](./cluster-logging)
|
||||
|
||||
# Monitoring
|
||||
|
||||
Using Rancher, you can monitor the state and processes of your cluster nodes, Kubernetes components, and software deployments through integration with [Prometheus](https://prometheus.io/), a leading open-source monitoring solution.
|
||||
|
||||
For details, refer to [Monitoring.](./cluster-monitoring)
|
||||
# Alerts
|
||||
|
||||
After monitoring is enabled, you can set up alerts and notifiers that provide the mechanism to receive them.
|
||||
|
||||
Alerts are rules that trigger notifications. Before you can receive alerts, you must configure one or more notifier in Rancher. The scope for alerts can be set at either the cluster or project level.
|
||||
|
||||
For details, refer to [Alerts.](./cluster-alerts)
|
||||
# Notifiers
|
||||
|
||||
Notifiers are services that inform you of alert events. You can configure notifiers to send alert notifications to staff best suited to take corrective action. Notifications can be sent with Slack, email, PagerDuty, WeChat, and webhooks.
|
||||
|
||||
For details, refer to [Notifiers.](./notifiers)
|
||||
# Istio
|
||||
|
||||
_Available as of v2.3_
|
||||
|
||||
[Istio](https://istio.io/) is an open-source tool that makes it easier for DevOps teams to observe, control, troubleshoot, and secure the traffic within a complex network of microservices.
|
||||
|
||||
Refer to the Istio documentation [here.](./istio)
|
||||
|
||||
# OPA Gatekeeper
|
||||
|
||||
[OPA Gatekeeper](https://github.com/open-policy-agent/gatekeeper) is an open-source project that provides integration between OPA and Kubernetes to provide policy control via admission controller webhooks. For details on how to enable Gatekeeper in Rancher, refer to the [OPA Gatekeeper section.](./opa-gatekeeper)
|
||||
|
||||
|
||||
# CIS Scans
|
||||
|
||||
Rancher can run a security scan to check whether Kubernetes is deployed according to security best practices as defined in the CIS Kubernetes Benchmark.
|
||||
|
||||
Refer to the CIS scan documentation [here.](./cis-scans)
|
||||
+167
@@ -0,0 +1,167 @@
|
||||
---
|
||||
title: Upgrading and Rolling Back Kubernetes
|
||||
weight: 70
|
||||
---
|
||||
|
||||
import Tabs from '@theme/Tabs';
|
||||
import TabItem from '@theme/TabItem';
|
||||
|
||||
Following an upgrade to the latest version of Rancher, downstream Kubernetes clusters can be upgraded to use the latest supported version of Kubernetes.
|
||||
|
||||
Rancher calls RKE (Rancher Kubernetes Engine) as a library when provisioning and editing RKE clusters. For more information on configuring the upgrade strategy for RKE clusters, refer to the [RKE documentation]({{<baseurl>}}/rke/latest/en/).
|
||||
|
||||
This section covers the following topics:
|
||||
|
||||
- [New Features](#new-features)
|
||||
- [Tested Kubernetes Versions](#tested-kubernetes-versions)
|
||||
- [How Upgrades Work](#how-upgrades-work)
|
||||
- [Recommended Best Practice for Upgrades](#recommended-best-practice-for-upgrades)
|
||||
- [Upgrading the Kubernetes Version](#upgrading-the-kubernetes-version)
|
||||
- [Rolling Back](#rolling-back)
|
||||
- [Configuring the Upgrade Strategy](#configuring-the-upgrade-strategy)
|
||||
- [Configuring the Maximum Unavailable Worker Nodes in the Rancher UI](#configuring-the-maximum-unavailable-worker-nodes-in-the-rancher-ui)
|
||||
- [Enabling Draining Nodes During Upgrades from the Rancher UI](#enabling-draining-nodes-during-upgrades-from-the-rancher-ui)
|
||||
- [Maintaining Availability for Applications During Upgrades](#maintaining-availability-for-applications-during-upgrades)
|
||||
- [Configuring the Upgrade Strategy in the cluster.yml](#configuring-the-upgrade-strategy-in-the-cluster-yml)
|
||||
- [Troubleshooting](#troubleshooting)
|
||||
|
||||
# New Features
|
||||
|
||||
As of Rancher v2.3.0, the Kubernetes metadata feature was added, which allows Rancher to ship Kubernetes patch versions without upgrading Rancher. For details, refer to the [section on Kubernetes metadata.]({{<baseurl>}}/rancher/v2.0-v2.4/en/admin-settings/k8s-metadata)
|
||||
|
||||
As of Rancher v2.4.0,
|
||||
|
||||
- The ability to import K3s Kubernetes clusters into Rancher was added, along with the ability to upgrade Kubernetes when editing those clusters. For details, refer to the [section on imported clusters.]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-provisioning/imported-clusters)
|
||||
- New advanced options are exposed in the Rancher UI for configuring the upgrade strategy of an RKE cluster: **Maximum Worker Nodes Unavailable** and **Drain nodes.** These options leverage the new cluster upgrade process of RKE v1.1.0, in which worker nodes are upgraded in batches, so that applications can remain available during cluster upgrades, under [certain conditions.](#maintaining-availability-for-applications-during-upgrades)
|
||||
|
||||
# Tested Kubernetes Versions
|
||||
|
||||
Before a new version of Rancher is released, it's tested with the latest minor versions of Kubernetes to ensure compatibility. For details on which versions of Kubernetes were tested on each Rancher version, refer to the [support maintenance terms.](https://rancher.com/support-maintenance-terms/all-supported-versions/rancher-v2.4.17/)
|
||||
|
||||
# How Upgrades Work
|
||||
|
||||
RKE v1.1.0 changed the way that clusters are upgraded.
|
||||
|
||||
In this section of the [RKE documentation,]({{<baseurl>}}/rke/latest/en/upgrades/how-upgrades-work) you'll learn what happens when you edit or upgrade your RKE Kubernetes cluster.
|
||||
|
||||
|
||||
# Recommended Best Practice for Upgrades
|
||||
|
||||
<Tabs>
|
||||
<TabItem value="Rancher v2.4+">
|
||||
|
||||
When upgrading the Kubernetes version of a cluster, we recommend that you:
|
||||
|
||||
1. Take a snapshot.
|
||||
1. Initiate a Kubernetes upgrade.
|
||||
1. If the upgrade fails, revert the cluster to the pre-upgrade Kubernetes version. This is achieved by selecting the **Restore etcd and Kubernetes version** option. This will return your cluster to the pre-upgrade kubernetes version before restoring the etcd snapshot.
|
||||
|
||||
The restore operation will work on a cluster that is not in a healthy or active state.
|
||||
|
||||
</TabItem>
|
||||
<TabItem value="Rancher before v2.4">
|
||||
|
||||
When upgrading the Kubernetes version of a cluster, we recommend that you:
|
||||
|
||||
1. Take a snapshot.
|
||||
1. Initiate a Kubernetes upgrade.
|
||||
1. If the upgrade fails, restore the cluster from the etcd snapshot.
|
||||
|
||||
The cluster cannot be downgraded to a previous Kubernetes version.
|
||||
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
|
||||
# Upgrading the Kubernetes Version
|
||||
|
||||
> **Prerequisites:**
|
||||
>
|
||||
> - The options below are available only for [Rancher-launched RKE Kubernetes clusters]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-provisioning/rke-clusters/) and imported/registered K3s Kubernetes clusters.
|
||||
> - Before upgrading Kubernetes, [back up your cluster.]({{<baseurl>}}/rancher/v2.0-v2.4/en/backups)
|
||||
|
||||
1. From the **Global** view, find the cluster for which you want to upgrade Kubernetes. Select **⋮ > Edit**.
|
||||
|
||||
1. Expand **Cluster Options**.
|
||||
|
||||
1. From the **Kubernetes Version** drop-down, choose the version of Kubernetes that you want to use for the cluster.
|
||||
|
||||
1. Click **Save**.
|
||||
|
||||
**Result:** Kubernetes begins upgrading for the cluster.
|
||||
|
||||
# Rolling Back
|
||||
|
||||
_Available as of v2.4_
|
||||
|
||||
A cluster can be restored to a backup in which the previous Kubernetes version was used. For more information, refer to the following sections:
|
||||
|
||||
- [Backing up a cluster]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/backing-up-etcd/#how-snapshots-work)
|
||||
- [Restoring a cluster from backup]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/restoring-etcd/#restoring-a-cluster-from-a-snapshot)
|
||||
|
||||
# Configuring the Upgrade Strategy
|
||||
|
||||
As of RKE v1.1.0, additional upgrade options became available to give you more granular control over the upgrade process. These options can be used to maintain availability of your applications during a cluster upgrade if certain [conditions and requirements]({{<baseurl>}}/rke/latest/en/upgrades/maintaining-availability) are met.
|
||||
|
||||
The upgrade strategy can be configured in the Rancher UI, or by editing the `cluster.yml`. More advanced options are available by editing the `cluster.yml`.
|
||||
|
||||
### Configuring the Maximum Unavailable Worker Nodes in the Rancher UI
|
||||
|
||||
From the Rancher UI, the maximum number of unavailable worker nodes can be configured. During a cluster upgrade, worker nodes will be upgraded in batches of this size.
|
||||
|
||||
By default, the maximum number of unavailable worker is defined as 10 percent of all worker nodes. This number can be configured as a percentage or as an integer. When defined as a percentage, the batch size is rounded down to the nearest node, with a minimum of one node.
|
||||
|
||||
To change the default number or percentage of worker nodes,
|
||||
|
||||
1. Go to the cluster view in the Rancher UI.
|
||||
1. Click **⋮ > Edit.**
|
||||
1. In the **Advanced Options** section, go to the **Maxiumum Worker Nodes Unavailable** field. Enter the percentage of worker nodes that can be upgraded in a batch. Optionally, select **Count** from the drop-down menu and enter the maximum unavailable worker nodes as an integer.
|
||||
1. Click **Save.**
|
||||
|
||||
**Result:** The cluster is updated to use the new upgrade strategy.
|
||||
|
||||
### Enabling Draining Nodes During Upgrades from the Rancher UI
|
||||
|
||||
By default, RKE [cordons](https://kubernetes.io/docs/concepts/architecture/nodes/#manual-node-administration) each node before upgrading it. [Draining](https://kubernetes.io/docs/tasks/administer-cluster/safely-drain-node/) is disabled during upgrades by default. If draining is enabled in the cluster configuration, RKE will both cordon and drain the node before it is upgraded.
|
||||
|
||||
To enable draining each node during a cluster upgrade,
|
||||
|
||||
1. Go to the cluster view in the Rancher UI.
|
||||
1. Click **⋮ > Edit.**
|
||||
1. In the **Advanced Options** section, go to the **Drain nodes** field and click **Yes.**
|
||||
1. Choose a safe or aggressive drain option. For more information about each option, refer to [this section.]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/nodes/#aggressive-and-safe-draining-options)
|
||||
1. Optionally, configure a grace period. The grace period is the timeout given to each pod for cleaning things up, so they will have chance to exit gracefully. Pods might need to finish any outstanding requests, roll back transactions or save state to some external storage. If this value is negative, the default value specified in the pod will be used.
|
||||
1. Optionally, configure a timeout, which is the amount of time the drain should continue to wait before giving up.
|
||||
1. Click **Save.**
|
||||
|
||||
**Result:** The cluster is updated to use the new upgrade strategy.
|
||||
|
||||
> **Note:** As of Rancher v2.4.0, there is a [known issue](https://github.com/rancher/rancher/issues/25478) in which the Rancher UI doesn't show state of etcd and controlplane as drained, even though they are being drained.
|
||||
|
||||
### Maintaining Availability for Applications During Upgrades
|
||||
|
||||
_Available as of RKE v1.1.0_
|
||||
|
||||
In [this section of the RKE documentation,]({{<baseurl>}}/rke/latest/en/upgrades/maintaining-availability/) you'll learn the requirements to prevent downtime for your applications when upgrading the cluster.
|
||||
|
||||
### Configuring the Upgrade Strategy in the cluster.yml
|
||||
|
||||
More advanced upgrade strategy configuration options are available by editing the `cluster.yml`.
|
||||
|
||||
For details, refer to [Configuring the Upgrade Strategy]({{<baseurl>}}/rke/latest/en/upgrades/configuring-strategy) in the RKE documentation. The section also includes an example `cluster.yml` for configuring the upgrade strategy.
|
||||
|
||||
# Troubleshooting
|
||||
|
||||
If a node doesn't come up after an upgrade, the `rke up` command errors out.
|
||||
|
||||
No upgrade will proceed if the number of unavailable nodes exceeds the configured maximum.
|
||||
|
||||
If an upgrade stops, you may need to fix an unavailable node or remove it from the cluster before the upgrade can continue.
|
||||
|
||||
A failed node could be in many different states:
|
||||
|
||||
- Powered off
|
||||
- Unavailable
|
||||
- User drains a node while upgrade is in process, so there are no kubelets on the node
|
||||
- The upgrade itself failed
|
||||
|
||||
If the max unavailable number of nodes is reached during an upgrade, Rancher user clusters will be stuck in updating state and not move forward with upgrading any other control plane nodes. It will continue to evaluate the set of unavailable nodes in case one of the nodes becomes available. If the node cannot be fixed, you must remove the node in order to continue the upgrade.
|
||||
+104
@@ -0,0 +1,104 @@
|
||||
---
|
||||
title: Setting up Existing Storage
|
||||
weight: 1
|
||||
---
|
||||
|
||||
This section describes how to set up existing persistent storage for workloads in Rancher.
|
||||
|
||||
> This section assumes that you understand the Kubernetes concepts of persistent volumes and persistent volume claims. For more information, refer to the section on [how storage works.](../how-storage-works)
|
||||
|
||||
To set up storage, follow these steps:
|
||||
|
||||
1. [Set up persistent storage.](#1-set-up-persistent-storage)
|
||||
2. [Add a persistent volume that refers to the persistent storage.](#2-add-a-persistent-volume-that-refers-to-the-persistent-storage)
|
||||
3. [Add a persistent volume claim that refers to the persistent volume.](#3-add-a-persistent-volume-claim-that-refers-to-the-persistent-volume)
|
||||
4. [Mount the persistent volume claim as a volume in your workload.](#4-mount-the-persistent-volume-claim-as-a-volume-in-your-workload)
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- To create a persistent volume as a Kubernetes resource, you must have the `Manage Volumes` [role.]({{<baseurl>}}/rancher/v2.0-v2.4/en/admin-settings/rbac/cluster-project-roles/#project-role-reference)
|
||||
- If you are provisioning storage for a cluster hosted in the cloud, the storage and cluster hosts must have the same cloud provider.
|
||||
|
||||
### 1. Set up persistent storage
|
||||
|
||||
Creating a persistent volume in Rancher will not create a storage volume. It only creates a Kubernetes resource that maps to an existing volume. Therefore, before you can create a persistent volume as a Kubernetes resource, you must have storage provisioned.
|
||||
|
||||
The steps to set up a persistent storage device will differ based on your infrastructure. We provide examples of how to set up storage using [vSphere,](../examples/vsphere) [NFS,](../examples/nfs) or Amazon's [EBS.](../examples/ebs)
|
||||
|
||||
If you have a pool of block storage, and you don't want to use a cloud provider, Longhorn could help you provide persistent storage to your Kubernetes cluster.
|
||||
|
||||
### 2. Add a persistent volume that refers to the persistent storage
|
||||
|
||||
These steps describe how to set up a persistent volume at the cluster level in Kubernetes.
|
||||
|
||||
1. From the cluster view, select **Storage > Persistent Volumes**.
|
||||
|
||||
1. Click **Add Volume**.
|
||||
|
||||
1. Enter a **Name** for the persistent volume.
|
||||
|
||||
1. Select the **Volume Plugin** for the disk type or service that you're using. When adding storage to a cluster that's hosted by a cloud provider, use the cloud provider's plug-in for cloud storage. For example, if you have a Amazon EC2 cluster and you want to use cloud storage for it, you must use the `Amazon EBS Disk` volume plugin.
|
||||
|
||||
1. Enter the **Capacity** of your volume in gigabytes.
|
||||
|
||||
1. Complete the **Plugin Configuration** form. Each plugin type requires information specific to the vendor of disk type. For help regarding each plugin's form and the information that's required, refer to the plug-in's vendor documentation.
|
||||
|
||||
1. Optional: In the **Customize** form, configure the [access modes.](https://kubernetes.io/docs/concepts/storage/persistent-volumes/#access-modes) This options sets how many nodes can access the volume, along with the node read/write permissions. The [Kubernetes Documentation](https://kubernetes.io/docs/concepts/storage/persistent-volumes/#access-modes) includes a table that lists which access modes are supported by the plugins available.
|
||||
|
||||
1. Optional: In the **Customize** form, configure the [mount options.](https://kubernetes.io/docs/concepts/storage/persistent-volumes/#mount-options) Each volume plugin allows you to specify additional command line options during the mounting process. Consult each plugin's vendor documentation for the mount options available.
|
||||
|
||||
1. Click **Save**.
|
||||
|
||||
**Result:** Your new persistent volume is created.
|
||||
|
||||
### 3. Add a persistent volume claim that refers to the persistent volume
|
||||
|
||||
These steps describe how to set up a PVC in the namespace where your stateful workload will be deployed.
|
||||
|
||||
1. Go to the project containing a workload that you want to add a persistent volume claim to.
|
||||
|
||||
1. Then click the **Volumes** tab and click **Add Volume**. (In versions before v2.3.0, click **Workloads** on the main navigation bar, then **Volumes.**)
|
||||
|
||||
1. Enter a **Name** for the volume claim.
|
||||
|
||||
1. Select the namespace of the workload that you want to add the persistent storage to.
|
||||
|
||||
1. In the section called **Use an existing persistent volume,** go to the **Persistent Volume** drop-down and choose the persistent volume that you created.
|
||||
|
||||
1. **Optional:** From **Customize**, select the [Access Modes](https://kubernetes.io/docs/concepts/storage/persistent-volumes/#access-modes) that you want to use.
|
||||
|
||||
1. Click **Create.**
|
||||
|
||||
**Result:** Your PVC is created. You can now attach it to any workload in the project.
|
||||
|
||||
### 4. Mount the persistent volume claim as a volume in your workload
|
||||
|
||||
Mount PVCs to stateful workloads so that your applications can store their data.
|
||||
|
||||
You can mount PVCs during the deployment of a workload, or following workload creation.
|
||||
|
||||
The following steps describe how to assign existing storage to a new workload that is a stateful set:
|
||||
|
||||
1. From the **Project** view, go to the **Workloads** tab.
|
||||
1. Click **Deploy.**
|
||||
1. Enter a name for the workload.
|
||||
1. Next to the **Workload Type** field, click **More Options.**
|
||||
1. Click **Stateful set of 1 pod.** Optionally, configure the number of pods.
|
||||
1. Choose the namespace where the workload will be deployed.
|
||||
1. Expand the **Volumes** section and click **Add Volume > Use an existing persistent volume (claim).**.
|
||||
1. In the **Persistent Volume Claim** field, select the PVC that you created.
|
||||
1. In the **Mount Point** field, enter the path that the workload will use to access the volume.
|
||||
1. Click **Launch.**
|
||||
|
||||
**Result:** When the workload is deployed, it will make a request for the specified amount of disk space to the Kubernetes master. If a PV with the specified resources is available when the workload is deployed, the Kubernetes master will bind the PV to the PVC.
|
||||
|
||||
The following steps describe how to assign persistent storage to an existing workload:
|
||||
|
||||
1. From the **Project** view, go to the **Workloads** tab.
|
||||
1. Go to the workload that you want to add the persistent storage to. The workload type should be a stateful set. Click **⋮ > Edit.**
|
||||
1. Expand the **Volumes** section and click **Add Volume > Use an existing persistent volume (claim).**.
|
||||
1. In the **Persistent Volume Claim** field, select the PVC that you created.
|
||||
1. In the **Mount Point** field, enter the path that the workload will use to access the volume.
|
||||
1. Click **Save.**
|
||||
|
||||
**Result:** The workload will make a request for the specified amount of disk space to the Kubernetes master. If a PV with the specified resources is available when the workload is deployed, the Kubernetes master will bind the PV to the PVC.
|
||||
@@ -0,0 +1,16 @@
|
||||
---
|
||||
title: Creating Persistent Storage in Amazon's EBS
|
||||
weight: 3053
|
||||
---
|
||||
|
||||
This section describes how to set up Amazon's Elastic Block Store in EC2.
|
||||
|
||||
1. From the EC2 console, go to the **ELASTIC BLOCK STORE** section in the left panel and click **Volumes.**
|
||||
1. Click **Create Volume.**
|
||||
1. Optional: Configure the size of the volume or other options. The volume should be created in the same availability zone as the instance it will be attached to.
|
||||
1. Click **Create Volume.**
|
||||
1. Click **Close.**
|
||||
|
||||
**Result:** Persistent storage has been created.
|
||||
|
||||
For details on how to set up the newly created storage in Rancher, refer to the section on [setting up existing storage.]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/volumes-and-storage/attaching-existing-storage/)
|
||||
@@ -0,0 +1,15 @@
|
||||
---
|
||||
title: Provisioning Storage Examples
|
||||
weight: 3053
|
||||
aliases:
|
||||
- /rancher/v2.0-v2.4/en/tasks/clusters/adding-storage/provisioning-storage/
|
||||
- /rancher/v2.0-v2.4/en/k8s-in-rancher/volumes-and-storage/examples/
|
||||
---
|
||||
|
||||
Rancher supports persistent storage with a variety of volume plugins. However, before you use any of these plugins to bind persistent storage to your workloads, you have to configure the storage itself, whether its a cloud-based solution from a service-provider or an on-prem solution that you manage yourself.
|
||||
|
||||
For your convenience, Rancher offers documentation on how to configure some of the popular storage methods:
|
||||
|
||||
- [NFS](./nfs)
|
||||
- [vSphere](./vsphere)
|
||||
- [EBS](./ebs)
|
||||
@@ -0,0 +1,68 @@
|
||||
---
|
||||
title: NFS Storage
|
||||
weight: 3054
|
||||
aliases:
|
||||
- /rancher/v2.0-v2.4/en/tasks/clusters/adding-storage/provisioning-storage/nfs/
|
||||
---
|
||||
|
||||
Before you can use the NFS storage volume plug-in with Rancher deployments, you need to provision an NFS server.
|
||||
|
||||
>**Note:**
|
||||
>
|
||||
>- If you already have an NFS share, you don't need to provision a new NFS server to use the NFS volume plugin within Rancher. Instead, skip the rest of this procedure and complete [adding storage]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/volumes-and-storage/).
|
||||
>
|
||||
>- This procedure demonstrates how to set up an NFS server using Ubuntu, although you should be able to use these instructions for other Linux distros (e.g. Debian, RHEL, Arch Linux, etc.). For official instruction on how to create an NFS server using another Linux distro, consult the distro's documentation.
|
||||
|
||||
>**Recommended:** To simplify the process of managing firewall rules, use NFSv4.
|
||||
|
||||
1. Using a remote Terminal connection, log into the Ubuntu server that you intend to use for NFS storage.
|
||||
|
||||
1. Enter the following command:
|
||||
|
||||
```
|
||||
sudo apt-get install nfs-kernel-server
|
||||
```
|
||||
|
||||
1. Enter the command below, which sets the directory used for storage, along with user access rights. Modify the command if you'd like to keep storage at a different directory.
|
||||
|
||||
```
|
||||
mkdir -p /nfs && chown nobody:nogroup /nfs
|
||||
```
|
||||
- The `-p /nfs` parameter creates a directory named `nfs` at root.
|
||||
- The `chown nobody:nogroup /nfs` parameter allows all access to the storage directory.
|
||||
|
||||
1. Create an NFS exports table. This table sets the directory paths on your NFS server that are exposed to the nodes that will use the server for storage.
|
||||
|
||||
1. Open `/etc/exports` using your text editor of choice.
|
||||
1. Add the path of the `/nfs` folder that you created in step 3, along with the IP addresses of your cluster nodes. Add an entry for each IP address in your cluster. Follow each address and its accompanying parameters with a single space that is a delimiter.
|
||||
|
||||
```
|
||||
/nfs <IP_ADDRESS1>(rw,sync,no_subtree_check) <IP_ADDRESS2>(rw,sync,no_subtree_check) <IP_ADDRESS3>(rw,sync,no_subtree_check)
|
||||
```
|
||||
|
||||
**Tip:** You can replace the IP addresses with a subnet. For example: `10.212.50.12/24`
|
||||
|
||||
1. Update the NFS table by entering the following command:
|
||||
|
||||
```
|
||||
exportfs -ra
|
||||
```
|
||||
|
||||
1. Open the ports used by NFS.
|
||||
|
||||
1. To find out what ports NFS is using, enter the following command:
|
||||
|
||||
```
|
||||
rpcinfo -p | grep nfs
|
||||
```
|
||||
2. [Open the ports](https://help.ubuntu.com/lts/serverguide/firewall.html.en) that the previous command outputs. For example, the following command opens port 2049:
|
||||
|
||||
```
|
||||
sudo ufw allow 2049
|
||||
```
|
||||
|
||||
**Result:** Your NFS server is configured to be used for storage with your Rancher nodes.
|
||||
|
||||
## What's Next?
|
||||
|
||||
Within Rancher, add the NFS server as a storage volume and/or storage class. After adding the server, you can use it for storage for your deployments.
|
||||
+78
@@ -0,0 +1,78 @@
|
||||
---
|
||||
title: vSphere Storage
|
||||
weight: 3055
|
||||
aliases:
|
||||
- /rancher/v2.0-v2.4/en/tasks/clusters/adding-storage/provisioning-storage/vsphere/
|
||||
---
|
||||
|
||||
To provide stateful workloads with vSphere storage, we recommend creating a vSphereVolume StorageClass. This practice dynamically provisions vSphere storage when workloads request volumes through a [persistent volume claim]({{<baseurl>}}/rancher/v2.0-v2.4/en/k8s-in-rancher/volumes-and-storage/persistent-volume-claims/).
|
||||
|
||||
In order to dynamically provision storage in vSphere, the vSphere provider must be [enabled.]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-provisioning/rke-clusters/cloud-providers/vsphere)
|
||||
|
||||
- [Prerequisites](#prerequisites)
|
||||
- [Creating a StorageClass](#creating-a-storageclass)
|
||||
- [Creating a Workload with a vSphere Volume](#creating-a-workload-with-a-vsphere-volume)
|
||||
- [Verifying Persistence of the Volume](#verifying-persistence-of-the-volume)
|
||||
- [Why to Use StatefulSets Instead of Deployments](#why-to-use-statefulsets-instead-of-deployments)
|
||||
|
||||
### Prerequisites
|
||||
|
||||
In order to provision vSphere volumes in a cluster created with the [Rancher Kubernetes Engine (RKE)]({{< baseurl>}}/rancher/v2.0-v2.4/en/cluster-provisioning/rke-clusters/), the [vSphere cloud provider]({{<baseurl>}}/rke/latest/en/config-options/cloud-providers/vsphere) must be explicitly enabled in the [cluster options]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-provisioning/rke-clusters/options/).
|
||||
|
||||
### Creating a StorageClass
|
||||
|
||||
> **Note:**
|
||||
>
|
||||
> The following steps can also be performed using the `kubectl` command line tool. See [Kubernetes documentation on persistent volumes](https://kubernetes.io/docs/concepts/storage/persistent-volumes/) for details.
|
||||
|
||||
1. From the Global view, open the cluster where you want to provide vSphere storage.
|
||||
2. From the main menu, select **Storage > Storage Classes**. Then click **Add Class**.
|
||||
3. Enter a **Name** for the class.
|
||||
4. Under **Provisioner**, select **VMWare vSphere Volume**.
|
||||
|
||||
{{< img "/img/rancher/vsphere-storage-class.png" "vsphere-storage-class">}}
|
||||
|
||||
5. Optionally, specify additional properties for this storage class under **Parameters**. Refer to the [vSphere storage documentation](https://vmware.github.io/vsphere-storage-for-kubernetes/documentation/storageclass.html) for details.
|
||||
5. Click **Save**.
|
||||
|
||||
### Creating a Workload with a vSphere Volume
|
||||
|
||||
1. From the cluster where you configured vSphere storage, begin creating a workload as you would in [Deploying Workloads]({{<baseurl>}}/rancher/v2.0-v2.4/en/k8s-in-rancher/workloads/deploy-workloads/).
|
||||
2. For **Workload Type**, select **Stateful set of 1 pod**.
|
||||
3. Expand the **Volumes** section and click **Add Volume**.
|
||||
4. Choose **Add a new persistent volume (claim)**. This option will implicitly create the claim once you deploy the workload.
|
||||
5. Assign a **Name** for the claim, ie. `test-volume` and select the vSphere storage class created in the previous step.
|
||||
6. Enter the required **Capacity** for the volume. Then click **Define**.
|
||||
|
||||
{{< img "/img/rancher/workload-add-volume.png" "workload-add-volume">}}
|
||||
|
||||
7. Assign a path in the **Mount Point** field. This is the full path where the volume will be mounted in the container file system, e.g. `/persistent`.
|
||||
8. Click **Launch** to create the workload.
|
||||
|
||||
### Verifying Persistence of the Volume
|
||||
|
||||
1. From the context menu of the workload you just created, click **Execute Shell**.
|
||||
2. Note the directory at root where the volume has been mounted to (in this case `/persistent`).
|
||||
3. Create a file in the volume by executing the command `touch /<volumeMountPoint>/data.txt`.
|
||||
4. **Close** the shell window.
|
||||
5. Click on the name of the workload to reveal detail information.
|
||||
6. Open the context menu next to the Pod in the *Running* state.
|
||||
7. Delete the Pod by selecting **Delete**.
|
||||
8. Observe that the pod is deleted. Then a new pod is scheduled to replace it so that the workload maintains its configured scale of a single stateful pod.
|
||||
9. Once the replacement pod is running, click **Execute Shell**.
|
||||
10. Inspect the contents of the directory where the volume is mounted by entering `ls -l /<volumeMountPoint>`. Note that the file you created earlier is still present.
|
||||
|
||||
|
||||
|
||||
### Why to Use StatefulSets Instead of Deployments
|
||||
|
||||
You should always use [StatefulSets](https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/) for workloads consuming vSphere storage, as this resource type is designed to address a VMDK block storage caveat.
|
||||
|
||||
Since vSphere volumes are backed by VMDK block storage, they only support an [access mode](https://kubernetes.io/docs/concepts/storage/persistent-volumes/#persistentvolumeclaims) of `ReadWriteOnce`. This setting restricts the volume so that it can only be mounted to a single pod at a time, unless all pods consuming that volume are co-located on the same node. This behavior makes a deployment resource unusable for scaling beyond a single replica if it consumes vSphere volumes.
|
||||
|
||||
Even using a deployment resource with just a single replica may result in a deadlock situation while updating the deployment. If the updated pod is scheduled to a node different from where the existing pod lives, it will fail to start because the VMDK is still attached to the other node.
|
||||
|
||||
### Related Links
|
||||
|
||||
- [vSphere Storage for Kubernetes](https://vmware.github.io/vsphere-storage-for-kubernetes/documentation/)
|
||||
- [Kubernetes Persistent Volumes](https://kubernetes.io/docs/concepts/storage/persistent-volumes/)
|
||||
+32
@@ -0,0 +1,32 @@
|
||||
---
|
||||
title: GlusterFS Volumes
|
||||
weight: 5000
|
||||
---
|
||||
|
||||
> This section only applies to [RKE clusters.]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-provisioning/rke-clusters/)
|
||||
|
||||
In clusters that store data on GlusterFS volumes, you may experience an issue where pods fail to mount volumes after restarting the `kubelet`. The logging of the `kubelet` will show: `transport endpoint is not connected`. To prevent this from happening, you can configure your cluster to mount the `systemd-run` binary in the `kubelet` container. There are two requirements before you can change the cluster configuration:
|
||||
|
||||
- The node needs to have the `systemd-run` binary installed (this can be checked by using the command `which systemd-run` on each cluster node)
|
||||
- The `systemd-run` binary needs to be compatible with Debian OS on which the hyperkube image is based (this can be checked using the following command on each cluster node, replacing the image tag with the Kubernetes version you want to use)
|
||||
|
||||
```
|
||||
docker run -v /usr/bin/systemd-run:/usr/bin/systemd-run --entrypoint /usr/bin/systemd-run rancher/hyperkube:v1.16.2-rancher1 --version
|
||||
```
|
||||
|
||||
>**Note:**
|
||||
>
|
||||
>Before updating your Kubernetes YAML to mount the `systemd-run` binary, make sure the `systemd` package is installed on your cluster nodes. If this package isn't installed _before_ the bind mounts are created in your Kubernetes YAML, Docker will automatically create the directories and files on each node and will not allow the package install to succeed.
|
||||
|
||||
```
|
||||
services:
|
||||
kubelet:
|
||||
extra_binds:
|
||||
- "/usr/bin/systemd-run:/usr/bin/systemd-run"
|
||||
```
|
||||
|
||||
After the cluster has finished provisioning, you can check the `kubelet` container logging to see if the functionality is activated by looking for the following logline:
|
||||
|
||||
```
|
||||
Detected OS with systemd
|
||||
```
|
||||
+78
@@ -0,0 +1,78 @@
|
||||
---
|
||||
title: How Persistent Storage Works
|
||||
weight: 1
|
||||
aliases:
|
||||
- /rancher/v2.0-v2.4/en/tasks/workloads/add-persistent-volume-claim
|
||||
---
|
||||
|
||||
A persistent volume (PV) is a piece of storage in the Kubernetes cluster, while a persistent volume claim (PVC) is a request for storage.
|
||||
|
||||
There are two ways to use persistent storage in Kubernetes:
|
||||
|
||||
- Use an existing persistent volume
|
||||
- Dynamically provision new persistent volumes
|
||||
|
||||
To use an existing PV, your application will need to use a PVC that is bound to a PV, and the PV should include the minimum resources that the PVC requires.
|
||||
|
||||
For dynamic storage provisioning, your application will need to use a PVC that is bound to a storage class. The storage class contains the authorization to provision new persistent volumes.
|
||||
|
||||
|
||||
|
||||
For more information, refer to the [official Kubernetes documentation on storage](https://kubernetes.io/docs/concepts/storage/volumes/)
|
||||
|
||||
This section covers the following topics:
|
||||
|
||||
- [About persistent volume claims](#about-persistent-volume-claims)
|
||||
- [PVCs are required for both new and existing persistent storage](#pvcs-are-required-for-both-new-and-existing-persistent-storage)
|
||||
- [Setting up existing storage with a PVC and PV](#setting-up-existing-storage-with-a-pvc-and-pv)
|
||||
- [Binding PVs to PVCs](#binding-pvs-to-pvcs)
|
||||
- [Provisioning new storage with a PVC and storage class](#provisioning-new-storage-with-a-pvc-and-storage-class)
|
||||
|
||||
# About Persistent Volume Claims
|
||||
|
||||
Persistent volume claims (PVCs) are objects that request storage resources from your cluster. They're similar to a voucher that your deployment can redeem for storage access. A PVC is mounted into a workloads as a volume so that the workload can claim its specified share of the persistent storage.
|
||||
|
||||
To access persistent storage, a pod must have a PVC mounted as a volume. This PVC lets your deployment application store its data in an external location, so that if a pod fails, it can be replaced with a new pod and continue accessing its data stored externally, as though an outage never occurred.
|
||||
|
||||
Each Rancher project contains a list of PVCs that you've created, available from **Resources > Workloads > Volumes.** (In versions before v2.3.0, the PVCs are in the **Volumes** tab.) You can reuse these PVCs when creating deployments in the future.
|
||||
|
||||
### PVCs are Required for Both New and Existing Persistent Storage
|
||||
|
||||
A PVC is required for pods to use any persistent storage, regardless of whether the workload is intended to use storage that already exists, or the workload will need to dynamically provision new storage on demand.
|
||||
|
||||
If you are setting up existing storage for a workload, the workload mounts a PVC, which refers to a PV, which corresponds to existing storage infrastructure.
|
||||
|
||||
If a workload should request new storage, the workload mounts PVC, which refers to a storage class, which has the capability to create a new PV along with its underlying storage infrastructure.
|
||||
|
||||
Rancher lets you create as many PVCs within a project as you'd like.
|
||||
|
||||
You can mount PVCs to a deployment as you create it, or later, after the deployment is running.
|
||||
|
||||
# Setting up Existing Storage with a PVC and PV
|
||||
|
||||
Your pods can store data in [volumes,](https://kubernetes.io/docs/concepts/storage/volumes/) but if the pod fails, that data is lost. To solve this issue, Kubernetes offers persistent volumes (PVs), which are Kubernetes resources that correspond to external storage disks or file systems that your pods can access. If a pod crashes, its replacement pod can access the data in persistent storage without any data loss.
|
||||
|
||||
PVs can represent a physical disk or file system that you host on premise, or a vendor-hosted storage resource, such as Amazon EBS or Azure Disk.
|
||||
|
||||
Creating a persistent volume in Rancher will not create a storage volume. It only creates a Kubernetes resource that maps to an existing volume. Therefore, before you can create a persistent volume as a Kubernetes resource, you must have storage provisioned.
|
||||
|
||||
> **Important:** PVs are created at the cluster level, which means that in a multi-tenant cluster, teams with access to separate namespaces could have access to the same PV.
|
||||
|
||||
### Binding PVs to PVCs
|
||||
|
||||
When pods are set up to use persistent storage, they mount a persistent volume claim (PVC) that is mounted the same way as any other Kubernetes volume. When each PVC is created, the Kubernetes master considers it to be a request for storage and binds it to a PV that matches the minimum resource requirements of the PVC. Not every PVC is guaranteed to be bound to a PV. According to the Kubernetes [documentation,](https://kubernetes.io/docs/concepts/storage/persistent-volumes/)
|
||||
|
||||
> Claims will remain unbound indefinitely if a matching volume does not exist. Claims will be bound as matching volumes become available. For example, a cluster provisioned with many 50Gi PVs would not match a PVC requesting 100Gi. The PVC can be bound when a 100Gi PV is added to the cluster.
|
||||
|
||||
In other words, you can create unlimited PVCs, but they will only be bound to PVs if the Kubernetes master can find a sufficient PVs that has at least the amount of disk space required by the PVC.
|
||||
|
||||
To dynamically provision new storage, the PVC mounted in the pod would have to correspond to a storage class instead of a persistent volume.
|
||||
|
||||
# Provisioning New Storage with a PVC and Storage Class
|
||||
|
||||
Storage Classes allow you to create PVs dynamically without having to create persistent storage in an infrastructure provider first.
|
||||
|
||||
For example, if a workload is bound to a PVC and the PVC refers to an Amazon EBS Storage Class, the storage class can dynamically create an EBS volume and a corresponding PV.
|
||||
|
||||
The Kubernetes master will then bind the newly created PV to your workload's PVC, allowing your workload to use the persistent storage.
|
||||
|
||||
+30
@@ -0,0 +1,30 @@
|
||||
---
|
||||
title: iSCSI Volumes
|
||||
weight: 6000
|
||||
---
|
||||
|
||||
In [Rancher Launched Kubernetes clusters]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-provisioning/rke-clusters/) that store data on iSCSI volumes, you may experience an issue where kubelets fail to automatically connect with iSCSI volumes. This failure is likely due to an incompatibility issue involving the iSCSI initiator tool. You can resolve this issue by installing the iSCSI initiator tool on each of your cluster nodes.
|
||||
|
||||
Rancher Launched Kubernetes clusters storing data on iSCSI volumes leverage the [iSCSI initiator tool](http://www.open-iscsi.com/), which is embedded in the kubelet's `rancher/hyperkube` Docker image. From each kubelet (i.e., the _initiator_), the tool discovers and launches sessions with an iSCSI volume (i.e., the _target_). However, in some instances, the versions of the iSCSI initiator tool installed on the initiator and the target may not match, resulting in a connection failure.
|
||||
|
||||
If you encounter this issue, you can work around it by installing the initiator tool on each node in your cluster. You can install the iSCSI initiator tool by logging into your cluster nodes and entering one of the following commands:
|
||||
|
||||
| Platform | Package Name | Install Command |
|
||||
| ------------- | ----------------------- | -------------------------------------- |
|
||||
| Ubuntu/Debian | `open-iscsi` | `sudo apt install open-iscsi` |
|
||||
| RHEL | `iscsi-initiator-utils` | `yum install iscsi-initiator-utils -y` |
|
||||
|
||||
|
||||
After installing the initiator tool on your nodes, edit the YAML for your cluster, editing the kubelet configuration to mount the iSCSI binary and configuration, as shown in the sample below.
|
||||
|
||||
>**Note:**
|
||||
>
|
||||
>Before updating your Kubernetes YAML to mount the iSCSI binary and configuration, make sure either the `open-iscsi` (deb) or `iscsi-initiator-utils` (yum) package is installed on your cluster nodes. If this package isn't installed _before_ the bind mounts are created in your Kubernetes YAML, Docker will automatically create the directories and files on each node and will not allow the package install to succeed.
|
||||
|
||||
```
|
||||
services:
|
||||
kubelet:
|
||||
extra_binds:
|
||||
- "/etc/iscsi:/etc/iscsi"
|
||||
- "/sbin/iscsiadm:/sbin/iscsiadm"
|
||||
```
|
||||
+113
@@ -0,0 +1,113 @@
|
||||
---
|
||||
title: Dynamically Provisioning New Storage in Rancher
|
||||
weight: 2
|
||||
---
|
||||
|
||||
This section describes how to provision new persistent storage for workloads in Rancher.
|
||||
|
||||
This section assumes that you understand the Kubernetes concepts of storage classes and persistent volume claims. For more information, refer to the section on [how storage works.](../how-storage-works)
|
||||
|
||||
New storage is often provisioned by a cloud provider such as Amazon EBS. However, new storage doesn't have to be in the cloud.
|
||||
|
||||
If you have a pool of block storage, and you don't want to use a cloud provider, Longhorn could help you provide persistent storage to your Kubernetes cluster.
|
||||
|
||||
To provision new storage for your workloads, follow these steps:
|
||||
|
||||
1. [Add a storage class and configure it to use your storage.](#1-add-a-storage-class-and-configure-it-to-use-your-storage)
|
||||
2. [Add a persistent volume claim that refers to the storage class.](#2-add-a-persistent-volume-claim-that-refers-to-the-storage-class)
|
||||
3. [Mount the persistent volume claim as a volume for your workload.](#3-mount-the-persistent-volume-claim-as-a-volume-for-your-workload)
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- To set up persistent storage, the `Manage Volumes` [role]({{<baseurl>}}/rancher/v2.0-v2.4/en/admin-settings/rbac/cluster-project-roles/#project-role-reference) is required.
|
||||
- If you are provisioning storage for a cluster hosted in the cloud, the storage and cluster hosts must have the same cloud provider.
|
||||
- The cloud provider must be enabled. For details on enabling cloud providers, refer to [this page.]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-provisioning/rke-clusters/options/cloud-providers/)
|
||||
- Make sure your storage provisioner is available to be enabled.
|
||||
|
||||
The following storage provisioners are enabled by default:
|
||||
|
||||
Name | Plugin
|
||||
--------|----------
|
||||
Amazon EBS Disk | `aws-ebs`
|
||||
AzureFile | `azure-file`
|
||||
AzureDisk | `azure-disk`
|
||||
Google Persistent Disk | `gce-pd`
|
||||
Longhorn | `flex-volume-longhorn`
|
||||
VMware vSphere Volume | `vsphere-volume`
|
||||
Local | `local`
|
||||
Network File System | `nfs`
|
||||
hostPath | `host-path`
|
||||
|
||||
To use a storage provisioner that is not on the above list, you will need to use a [feature flag to enable unsupported storage drivers.]({{<baseurl>}}/rancher/v2.0-v2.4/en/installation/options/feature-flags/enable-not-default-storage-drivers/)
|
||||
|
||||
### 1. Add a storage class and configure it to use your storage
|
||||
|
||||
These steps describe how to set up a storage class at the cluster level.
|
||||
|
||||
1. Go to the cluster for which you want to dynamically provision persistent storage volumes.
|
||||
|
||||
1. From the cluster view, select `Storage > Storage Classes`. Click `Add Class`.
|
||||
|
||||
1. Enter a `Name` for your storage class.
|
||||
|
||||
1. From the `Provisioner` drop-down, select the service that you want to use to dynamically provision storage volumes. For example, if you have a Amazon EC2 cluster and you want to use cloud storage for it, use the `Amazon EBS Disk` provisioner.
|
||||
|
||||
1. From the `Parameters` section, fill out the information required for the service to dynamically provision storage volumes. Each provisioner requires different information to dynamically provision storage volumes. Consult the service's documentation for help on how to obtain this information.
|
||||
|
||||
1. Click `Save`.
|
||||
|
||||
**Result:** The storage class is available to be consumed by a PVC.
|
||||
|
||||
For full information about the storage class parameters, refer to the official [Kubernetes documentation.](https://kubernetes.io/docs/concepts/storage/storage-classes/#parameters).
|
||||
|
||||
### 2. Add a persistent volume claim that refers to the storage class
|
||||
|
||||
These steps describe how to set up a PVC in the namespace where your stateful workload will be deployed.
|
||||
|
||||
1. Go to the project containing a workload that you want to add a PVC to.
|
||||
|
||||
1. From the main navigation bar, choose **Resources > Workloads.** (In versions before v2.3.0, choose **Workloads** on the main navigation bar.) Then select the **Volumes** tab. Click **Add Volume**.
|
||||
|
||||
1. Enter a **Name** for the volume claim.
|
||||
|
||||
1. Select the namespace of the volume claim.
|
||||
|
||||
1. In the **Source** field, click **Use a Storage Class to provision a new persistent volume.**
|
||||
|
||||
1. Go to the **Storage Class** drop-down and select the storage class that you created.
|
||||
|
||||
1. Enter a volume **Capacity**.
|
||||
|
||||
1. Optional: Expand the **Customize** section and select the [Access Modes](https://kubernetes.io/docs/concepts/storage/persistent-volumes/#access-modes) that you want to use.
|
||||
|
||||
1. Click **Create.**
|
||||
|
||||
**Result:** Your PVC is created. You can now attach it to any workload in the project.
|
||||
|
||||
### 3. Mount the persistent volume claim as a volume for your workload
|
||||
|
||||
Mount PVCs to workloads so that your applications can store their data.
|
||||
|
||||
You can mount PVCs during the deployment of a workload, or following workload creation.
|
||||
|
||||
To attach the PVC to a new workload,
|
||||
|
||||
1. Create a workload as you would in [Deploying Workloads]({{<baseurl>}}/rancher/v2.0-v2.4/en/k8s-in-rancher/workloads/deploy-workloads/).
|
||||
1. For **Workload Type**, select **Stateful set of 1 pod**.
|
||||
1. Expand the **Volumes** section and click **Add Volume > Add a New Persistent Volume (Claim).**
|
||||
1. In the **Persistent Volume Claim** section, select the newly created persistent volume claim that is attached to the storage class.
|
||||
1. In the **Mount Point** field, enter the path that the workload will use to access the volume.
|
||||
1. Click **Launch.**
|
||||
|
||||
**Result:** When the workload is deployed, it will make a request for the specified amount of disk space to the Kubernetes master. If a PV with the specified resources is available when the workload is deployed, the Kubernetes master will bind the PV to the PVC.
|
||||
|
||||
To attach the PVC to an existing workload,
|
||||
|
||||
1. Go to the project that has the workload that will have the PVC attached.
|
||||
1. Go to the workload that will have persistent storage and click **⋮ > Edit.**
|
||||
1. Expand the **Volumes** section and click **Add Volume > Add a New Persistent Volume (Claim).**
|
||||
1. In the **Persistent Volume Claim** section, select the newly created persistent volume claim that is attached to the storage class.
|
||||
1. In the **Mount Point** field, enter the path that the workload will use to access the volume.
|
||||
1. Click **Save.**
|
||||
|
||||
**Result:** The workload will make a request for the specified amount of disk space to the Kubernetes master. If a PV with the specified resources is available when the workload is deployed, the Kubernetes master will bind the PV to the PVC. If not, Rancher will provision new persistent storage.
|
||||
+70
@@ -0,0 +1,70 @@
|
||||
---
|
||||
title: "Kubernetes Persistent Storage: Volumes and Storage Classes"
|
||||
description: "Learn about the two ways with which you can create persistent storage in Kubernetes: persistent volumes and storage classes"
|
||||
weight: 2031
|
||||
aliases:
|
||||
- /rancher/v2.0-v2.4/en/tasks/clusters/adding-storage/
|
||||
- /rancher/v2.0-v2.4/en/cluster-admin/volumes-and-storage/persistent-volume-claims/
|
||||
- /rancher/v2.0-v2.4/en/k8s-in-rancher/volumes-and-storage/persistent-volume-claims/
|
||||
---
|
||||
When deploying an application that needs to retain data, you'll need to create persistent storage. Persistent storage allows you to store application data external from the pod running your application. This storage practice allows you to maintain application data, even if the application's pod fails.
|
||||
|
||||
The documents in this section assume that you understand the Kubernetes concepts of persistent volumes, persistent volume claims, and storage classes. For more information, refer to the section on [how storage works.](./how-storage-works)
|
||||
|
||||
### Prerequisites
|
||||
|
||||
To set up persistent storage, the `Manage Volumes` [role]({{<baseurl>}}/rancher/v2.0-v2.4/en/admin-settings/rbac/cluster-project-roles/#project-role-reference) is required.
|
||||
|
||||
If you are provisioning storage for a cluster hosted in the cloud, the storage and cluster hosts must have the same cloud provider.
|
||||
|
||||
For provisioning new storage with Rancher, the cloud provider must be enabled. For details on enabling cloud providers, refer to [this page.]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-provisioning/rke-clusters/options/cloud-providers/)
|
||||
|
||||
For attaching existing persistent storage to a cluster, the cloud provider does not need to be enabled.
|
||||
|
||||
### Setting up Existing Storage
|
||||
|
||||
The overall workflow for setting up existing storage is as follows:
|
||||
|
||||
1. Set up your persistent storage. This may be storage in an infrastructure provider, or it could be your own storage.
|
||||
2. Add a persistent volume (PV) that refers to the persistent storage.
|
||||
3. Add a persistent volume claim (PVC) that refers to the PV.
|
||||
4. Mount the PVC as a volume in your workload.
|
||||
|
||||
For details and prerequisites, refer to [this page.](./attaching-existing-storage)
|
||||
|
||||
### Dynamically Provisioning New Storage in Rancher
|
||||
|
||||
The overall workflow for provisioning new storage is as follows:
|
||||
|
||||
1. Add a StorageClass and configure it to use your storage provider. The StorageClass could refer to storage in an infrastructure provider, or it could refer to your own storage.
|
||||
2. Add a persistent volume claim (PVC) that refers to the storage class.
|
||||
3. Mount the PVC as a volume for your workload.
|
||||
|
||||
For details and prerequisites, refer to [this page.](./provisioning-new-storage)
|
||||
|
||||
### Longhorn Storage
|
||||
|
||||
[Longhorn](https://longhorn.io/) is a lightweight, reliable and easy-to-use distributed block storage system for Kubernetes.
|
||||
|
||||
Longhorn is free, open source software. Originally developed by Rancher Labs, it is now being developed as a sandbox project of the Cloud Native Computing Foundation. It can be installed on any Kubernetes cluster with Helm, with kubectl, or with the Rancher UI.
|
||||
|
||||
If you have a pool of block storage, Longhorn can help you provide persistent storage to your Kubernetes cluster without relying on cloud providers. For more information about Longhorn features, refer to the [documentation.](https://longhorn.io/docs/1.0.2/what-is-longhorn/)
|
||||
|
||||
### Provisioning Storage Examples
|
||||
|
||||
We provide examples of how to provision storage with [NFS,](./examples/nfs) [vSphere,](./examples/vsphere) and [Amazon's EBS.](./examples/ebs)
|
||||
|
||||
### GlusterFS Volumes
|
||||
|
||||
In clusters that store data on GlusterFS volumes, you may experience an issue where pods fail to mount volumes after restarting the `kubelet`. For details on preventing this from happening, refer to [this page.](./glusterfs-volumes)
|
||||
|
||||
### iSCSI Volumes
|
||||
|
||||
In [Rancher Launched Kubernetes clusters]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-provisioning/rke-clusters/) that store data on iSCSI volumes, you may experience an issue where kubelets fail to automatically connect with iSCSI volumes. For details on resolving this issue, refer to [this page.](./iscsi-volumes)
|
||||
|
||||
### hostPath Volumes
|
||||
Before you create a hostPath volume, you need to set up an [extra_bind]({{<baseurl>}}/rke/latest/en/config-options/services/services-extras/#extra-binds/) in your cluster configuration. This will mount the path as a volume in your kubelets, which can then be used for hostPath volumes in your workloads.
|
||||
|
||||
### Related Links
|
||||
|
||||
- [Kubernetes Documentation: Storage](https://kubernetes.io/docs/concepts/storage/)
|
||||
Reference in New Issue
Block a user