mirror of
https://github.com/rancher/rancher-docs.git
synced 2026-09-25 20:48:11 +00:00
Remove unneeded intermediate folders
This commit is contained in:
Billy Tat
@@ -0,0 +1,166 @@
|
||||
---
|
||||
title: Backing up a Cluster
|
||||
weight: 2045
|
||||
---
|
||||
|
||||
In the Rancher UI, etcd backup and recovery for [Rancher launched Kubernetes clusters]({{<baseurl>}}/rancher/v2.6/en/cluster-provisioning/rke-clusters/) can be easily performed.
|
||||
|
||||
Rancher recommends configuring recurrent `etcd` snapshots for all production clusters. Additionally, one-time snapshots can easily be taken as well.
|
||||
|
||||
Snapshots of the etcd database are taken and saved either [locally onto the etcd nodes](#local-backup-target) or to a [S3 compatible target](#s3-backup-target). The advantages of configuring S3 is that if all etcd nodes are lost, your snapshot is saved remotely and can be used to restore the cluster.
|
||||
|
||||
This section covers the following topics:
|
||||
|
||||
- [How snapshots work](#how-snapshots-work)
|
||||
- [Configuring recurring snapshots](#configuring-recurring-snapshots)
|
||||
- [One-time snapshots](#one-time-snapshots)
|
||||
- [Snapshot backup targets](#snapshot-backup-targets)
|
||||
- [Local backup target](#local-backup-target)
|
||||
- [S3 backup target](#s3-backup-target)
|
||||
- [Using a custom CA certificate for S3](#using-a-custom-ca-certificate-for-s3)
|
||||
- [IAM Support for storing snapshots in S3](#iam-support-for-storing-snapshots-in-s3)
|
||||
- [Viewing available snapshots](#viewing-available-snapshots)
|
||||
- [Safe timestamps](#safe-timestamps)
|
||||
- [Enabling snapshot features for clusters created before Rancher v2.2.0](#enabling-snapshot-features-for-clusters-created-before-rancher-v2-2-0)
|
||||
|
||||
# How Snapshots Work
|
||||
|
||||
### Snapshot Components
|
||||
|
||||
When Rancher creates a snapshot, it includes three components:
|
||||
|
||||
- The cluster data in etcd
|
||||
- The Kubernetes version
|
||||
- The cluster configuration in the form of the `cluster.yml`
|
||||
|
||||
Because the Kubernetes version is now included in the snapshot, it is possible to restore a cluster to a prior Kubernetes version.
|
||||
|
||||
The multiple components of the snapshot allow you to select from the following options if you need to restore a cluster from a snapshot:
|
||||
|
||||
- **Restore just the etcd contents:** This restore is similar to restoring to snapshots in Rancher before v2.4.0.
|
||||
- **Restore etcd and Kubernetes version:** This option should be used if a Kubernetes upgrade is the reason that your cluster is failing, and you haven't made any cluster configuration changes.
|
||||
- **Restore etcd, Kubernetes versions and cluster configuration:** This option should be used if you changed both the Kubernetes version and cluster configuration when upgrading.
|
||||
|
||||
It's always recommended to take a new snapshot before any upgrades.
|
||||
|
||||
### Generating the Snapshot from etcd Nodes
|
||||
|
||||
For each etcd node in the cluster, the etcd cluster health is checked. If the node reports that the etcd cluster is healthy, a snapshot is created from it and optionally uploaded to S3.
|
||||
|
||||
The snapshot is stored in `/opt/rke/etcd-snapshots`. If the directory is configured on the nodes as a shared mount, it will be overwritten. On S3, the snapshot will always be from the last node that uploads it, as all etcd nodes upload it and the last will remain.
|
||||
|
||||
In the case when multiple etcd nodes exist, any created snapshot is created after the cluster has been health checked, so it can be considered a valid snapshot of the data in the etcd cluster.
|
||||
|
||||
### Snapshot Naming Conventions
|
||||
|
||||
The name of the snapshot is auto-generated. The `--name` option can be used to override the name of the snapshot when creating one-time snapshots with the RKE CLI.
|
||||
|
||||
When Rancher creates a snapshot of an RKE cluster, the snapshot name is based on the type (whether the snapshot is manual or recurring) and the target (whether the snapshot is saved locally or uploaded to S3). The naming convention is as follows:
|
||||
|
||||
- `m` stands for manual
|
||||
- `r` stands for recurring
|
||||
- `l` stands for local
|
||||
- `s` stands for S3
|
||||
|
||||
Some example snapshot names are:
|
||||
|
||||
- c-9dmxz-rl-8b2cx
|
||||
- c-9dmxz-ml-kr56m
|
||||
- c-9dmxz-ms-t6bjb
|
||||
- c-9dmxz-rs-8gxc8
|
||||
|
||||
### How Restoring from a Snapshot Works
|
||||
|
||||
On restore, the following process is used:
|
||||
|
||||
1. The snapshot is retrieved from S3, if S3 is configured.
|
||||
2. The snapshot is unzipped (if zipped).
|
||||
3. One of the etcd nodes in the cluster serves that snapshot file to the other nodes.
|
||||
4. The other etcd nodes download the snapshot and validate the checksum so that they all use the same snapshot for the restore.
|
||||
5. The cluster is restored and post-restore actions will be done in the cluster.
|
||||
|
||||
# Configuring Recurring Snapshots
|
||||
|
||||
Select how often you want recurring snapshots to be taken as well as how many snapshots to keep. The amount of time is measured in hours. With timestamped snapshots, the user has the ability to do a point-in-time recovery.
|
||||
|
||||
By default, [Rancher launched Kubernetes clusters]({{<baseurl>}}/rancher/v2.6/en/cluster-provisioning/rke-clusters/) are configured to take recurring snapshots (saved to local disk). To protect against local disk failure, using the [S3 Target](#s3-backup-target) or replicating the path on disk is advised.
|
||||
|
||||
During cluster provisioning or editing the cluster, the configuration for snapshots can be found in the advanced section for **Cluster Options**. Click on **Show advanced options**.
|
||||
|
||||
In the **Advanced Cluster Options** section, there are several options available to configure:
|
||||
|
||||
| Option | Description | Default Value|
|
||||
| --- | ---| --- |
|
||||
| etcd Snapshot Backup Target | Select where you want the snapshots to be saved. Options are either local or in S3 | local|
|
||||
|Recurring etcd Snapshot Enabled| Enable/Disable recurring snapshots | Yes|
|
||||
| Recurring etcd Snapshot Creation Period | Time in hours between recurring snapshots| 12 hours |
|
||||
| Recurring etcd Snapshot Retention Count | Number of snapshots to retain| 6 |
|
||||
|
||||
# One-Time Snapshots
|
||||
|
||||
In addition to recurring snapshots, you may want to take a "one-time" snapshot. For example, before upgrading the Kubernetes version of a cluster it's best to backup the state of the cluster to protect against upgrade failure.
|
||||
|
||||
1. In the upper left corner, click **☰ > Cluster Management**.
|
||||
1. On the **Clusters** page, navigate to the cluster where you want to take a one-time snapshot.
|
||||
1. Click **⋮ > Take Snapshot**.
|
||||
|
||||
**Result:** Based on your [snapshot backup target](#snapshot-backup-targets), a one-time snapshot will be taken and saved in the selected backup target.
|
||||
|
||||
# Snapshot Backup Targets
|
||||
|
||||
Rancher supports two different backup targets:
|
||||
|
||||
* [Local Target](#local-backup-target)
|
||||
* [S3 Target](#s3-backup-target)
|
||||
|
||||
### Local Backup Target
|
||||
|
||||
By default, the `local` backup target is selected. The benefits of this option is that there is no external configuration. Snapshots are automatically saved locally to the etcd nodes in the [Rancher launched Kubernetes clusters]({{<baseurl>}}/rancher/v2.6/en/cluster-provisioning/rke-clusters/) in `/opt/rke/etcd-snapshots`. All recurring snapshots are taken at configured intervals. The downside of using the `local` backup target is that if there is a total disaster and _all_ etcd nodes are lost, there is no ability to restore the cluster.
|
||||
|
||||
### S3 Backup Target
|
||||
|
||||
The `S3` backup target allows users to configure a S3 compatible backend to store the snapshots. The primary benefit of this option is that if the cluster loses all the etcd nodes, the cluster can still be restored as the snapshots are stored externally. Rancher recommends external targets like `S3` backup, however its configuration requirements do require additional effort that should be considered.
|
||||
|
||||
| Option | Description | Required|
|
||||
|---|---|---|
|
||||
|S3 Bucket Name| S3 bucket name where backups will be stored| *|
|
||||
|S3 Region|S3 region for the backup bucket| |
|
||||
|S3 Region Endpoint|S3 regions endpoint for the backup bucket|* |
|
||||
|S3 Access Key|S3 access key with permission to access the backup bucket|*|
|
||||
|S3 Secret Key|S3 secret key with permission to access the backup bucket|*|
|
||||
| Custom CA Certificate | A custom certificate used to access private S3 backends ||
|
||||
|
||||
### Using a custom CA certificate for S3
|
||||
|
||||
The backup snapshot can be stored on a custom `S3` backup like [minio](https://min.io/). If the S3 back end uses a self-signed or custom certificate, provide a custom certificate using the `Custom CA Certificate` option to connect to the S3 backend.
|
||||
|
||||
### IAM Support for Storing Snapshots in S3
|
||||
|
||||
The `S3` backup target supports using IAM authentication to AWS API in addition to using API credentials. An IAM role gives temporary permissions that an application can use when making API calls to S3 storage. To use IAM authentication, the following requirements must be met:
|
||||
|
||||
- The cluster etcd nodes must have an instance role that has read/write access to the designated backup bucket.
|
||||
- The cluster etcd nodes must have network access to the specified S3 endpoint.
|
||||
- The Rancher Server worker node(s) must have an instance role that has read/write to the designated backup bucket.
|
||||
- The Rancher Server worker node(s) must have network access to the specified S3 endpoint.
|
||||
|
||||
To give an application access to S3, refer to the AWS documentation on [Using an IAM Role to Grant Permissions to Applications Running on Amazon EC2 Instances.](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html)
|
||||
|
||||
# Viewing Available Snapshots
|
||||
|
||||
The list of all available snapshots for the cluster is available in the Rancher UI.
|
||||
|
||||
1. In the upper left corner, click **☰ > Cluster Management**.
|
||||
1. In the **Clusters** page, go to the cluster where you want to view the snapshots and click its name.
|
||||
1. Click the **Snapshots** tab to view the list of saved snapshots. These snapshots include a timestamp of when they were created.
|
||||
|
||||
# Safe Timestamps
|
||||
|
||||
Snapshot files are timestamped to simplify processing the files using external tools and scripts, but in some S3 compatible backends, these timestamps were unusable.
|
||||
|
||||
The option `safe_timestamp` is added to support compatible file names. When this flag is set to `true`, all special characters in the snapshot filename timestamp are replaced.
|
||||
|
||||
This option is not available directly in the UI, and is only available through the `Edit as Yaml` interface.
|
||||
|
||||
# Enabling Snapshot Features for Clusters Created Before Rancher v2.2.0
|
||||
|
||||
If you have any Rancher launched Kubernetes clusters that were created before v2.2.0, after upgrading Rancher, you must [edit the cluster]({{<baseurl>}}/rancher/v2.6/en/cluster-admin/editing-clusters/) and _save_ it, in order to enable the updated snapshot features. Even if you were already creating snapshots before v2.2.0, you must do this step as the older snapshots will not be available to use to [back up and restore etcd through the UI]({{<baseurl>}}/rancher/v2.6/en/cluster-admin/restoring-etcd/).
|
||||
@@ -0,0 +1,75 @@
|
||||
---
|
||||
title: Certificate Rotation
|
||||
weight: 2040
|
||||
---
|
||||
|
||||
import Tabs from '@theme/Tabs';
|
||||
import TabItem from '@theme/TabItem';
|
||||
|
||||
> **Warning:** Rotating Kubernetes certificates may result in your cluster being temporarily unavailable as components are restarted. For production environments, it's recommended to perform this action during a maintenance window.
|
||||
|
||||
By default, Kubernetes clusters require certificates and Rancher launched Kubernetes clusters automatically generate certificates for the Kubernetes components. Rotating these certificates is important before the certificates expire as well as if a certificate is compromised. After the certificates are rotated, the Kubernetes components are automatically restarted.
|
||||
|
||||
Certificates can be rotated for the following services:
|
||||
|
||||
<Tabs>
|
||||
<TabItem value="RKE">
|
||||
|
||||
- etcd
|
||||
- kubelet (node certificate)
|
||||
- kubelet (serving certificate, if [enabled]({{<baseurl>}}/rke/latest/en/config-options/services/#kubelet-options))
|
||||
- kube-apiserver
|
||||
- kube-proxy
|
||||
- kube-scheduler
|
||||
- kube-controller-manager
|
||||
|
||||
</TabItem>
|
||||
<TabItem value="RKE2">
|
||||
|
||||
- admin
|
||||
- api-server
|
||||
- controller-manager
|
||||
- scheduler
|
||||
- rke2-controller
|
||||
- rke2-server
|
||||
- cloud-controller
|
||||
- etcd
|
||||
- auth-proxy
|
||||
- kubelet
|
||||
- kube-proxy
|
||||
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
|
||||
> **Note:** For users who didn't rotate their webhook certificates, and they have expired after one year, please see this [page]({{<baseurl>}}/rancher/v2.6/en/troubleshooting/expired-webhook-certificates/) for help.
|
||||
|
||||
|
||||
### Certificate Rotation
|
||||
|
||||
Rancher launched Kubernetes clusters have the ability to rotate the auto-generated certificates through the UI.
|
||||
|
||||
1. In the upper left corner, click **☰ > Cluster Management**.
|
||||
1. On the **Clusters** page, go to the cluster you want to rotate certificates for amd click **⋮ > Rotate Certificates**.
|
||||
1. Select which certificates that you want to rotate.
|
||||
|
||||
* Rotate all Service certificates (keep the same CA)
|
||||
* Rotate an individual service and choose one of the services from the drop-down menu
|
||||
|
||||
1. Click **Save**.
|
||||
|
||||
**Results:** The selected certificates will be rotated and the related services will be restarted to start using the new certificate.
|
||||
|
||||
### Additional Notes
|
||||
|
||||
<Tabs>
|
||||
<TabItem value="RKE">
|
||||
|
||||
Even though the RKE CLI can use custom certificates for the Kubernetes cluster components, Rancher currently doesn't allow the ability to upload these in Rancher launched Kubernetes clusters.
|
||||
|
||||
</TabItem>
|
||||
<TabItem value="RKE2">
|
||||
|
||||
In RKE2, both etcd and control plane nodes are treated as the same `server` concept. As such, when rotating certificates of services specific to either of these components will result in certificates being rotated on both. The certificates will only change for the specified service, but you will see nodes for both components go into an updating state. You may also see worker only nodes go into an updating state. This is to restart the workers after a certificate change to ensure they get the latest client certs.
|
||||
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
+285
@@ -0,0 +1,285 @@
|
||||
---
|
||||
title: Removing Kubernetes Components from Nodes
|
||||
description: Learn about cluster cleanup when removing nodes from your Rancher-launched Kubernetes cluster. What is removed, how to do it manually
|
||||
weight: 2055
|
||||
---
|
||||
|
||||
import Tabs from '@theme/Tabs';
|
||||
import TabItem from '@theme/TabItem';
|
||||
|
||||
This section describes how to disconnect a node from a Rancher-launched Kubernetes cluster and remove all of the Kubernetes components from the node. This process allows you to use the node for other purposes.
|
||||
|
||||
When you use Rancher to install Kubernetes on new nodes in an infrastructure provider, resources (containers/virtual network interfaces) and configuration items (certificates/configuration files) are created.
|
||||
|
||||
When removing nodes from your Rancher launched Kubernetes cluster (provided that they are in `Active` state), those resources are automatically cleaned, and the only action needed is to restart the node. When a node has become unreachable and the automatic cleanup process cannot be used, we describe the steps that need to be executed before the node can be added to a cluster again.
|
||||
|
||||
## What Gets Removed?
|
||||
|
||||
When cleaning nodes provisioned using Rancher, the following components are deleted based on the type of cluster node you're removing.
|
||||
|
||||
| Removed Component | [Nodes Hosted by Infrastructure Provider][1] | [Custom Nodes][2] | [Hosted Cluster][3] | [Registered Nodes][4] |
|
||||
| ------------------------------------------------------------------------------ | --------------- | ----------------- | ------------------- | ------------------- |
|
||||
| The Rancher deployment namespace (`cattle-system` by default) | ✓ | ✓ | ✓ | ✓ |
|
||||
| `serviceAccount`, `clusterRoles`, and `clusterRoleBindings` labeled by Rancher | ✓ | ✓ | ✓ | ✓ |
|
||||
| Labels, Annotations, and Finalizers | ✓ | ✓ | ✓ | ✓ |
|
||||
| Rancher Deployment | ✓ | ✓ | ✓ | |
|
||||
| Machines, clusters, projects, and user custom resource definitions (CRDs) | ✓ | ✓ | ✓ | |
|
||||
| All resources create under the `management.cattle.io` API Group | ✓ | ✓ | ✓ | |
|
||||
| All CRDs created by Rancher v2.x | ✓ | ✓ | ✓ | |
|
||||
|
||||
[1]: {{<baseurl>}}/rancher/v2.6/en/cluster-provisioning/rke-clusters/node-pools/
|
||||
[2]: {{<baseurl>}}/rancher/v2.6/en/cluster-provisioning/rke-clusters/custom-nodes/
|
||||
[3]: {{<baseurl>}}/rancher/v2.6/en/cluster-provisioning/hosted-kubernetes-clusters/
|
||||
[4]: {{<baseurl>}}/rancher/v2.6/en/cluster-provisioning/registered-clusters/
|
||||
|
||||
## Removing a Node from a Cluster by Rancher UI
|
||||
|
||||
When the node is in `Active` state, removing the node from a cluster will trigger a process to clean up the node. Please restart the node after the automatic cleanup process is done to make sure any non-persistent data is properly removed.
|
||||
|
||||
**To restart a node:**
|
||||
|
||||
```
|
||||
# using reboot
|
||||
$ sudo reboot
|
||||
|
||||
# using shutdown
|
||||
$ sudo shutdown -r now
|
||||
```
|
||||
|
||||
## Removing Rancher Components from a Cluster Manually
|
||||
|
||||
When a node is unreachable and removed from the cluster, the automatic cleaning process can't be triggered because the node is unreachable. Please follow the steps below to manually remove the Rancher components.
|
||||
|
||||
>**Warning:** The commands listed below will remove data from the node. Make sure you have created a backup of files you want to keep before executing any of the commands as data will be lost.
|
||||
|
||||
### Removing Rancher Components from Registered Clusters
|
||||
|
||||
For registered clusters, the process for removing Rancher is a little different. You have the option of simply deleting the cluster in the Rancher UI, or your can run a script that removes Rancher components from the nodes. Both options make the same deletions.
|
||||
|
||||
After the registered cluster is detached from Rancher, the cluster's workloads will be unaffected and you can access the cluster using the same methods that you did before the cluster was registered into Rancher.
|
||||
|
||||
<Tabs>
|
||||
<TabItem value="By UI / API">
|
||||
|
||||
>**Warning:** This process will remove data from your cluster. Make sure you have created a backup of files you want to keep before executing the command, as data will be lost.
|
||||
|
||||
After you initiate the removal of a registered cluster using the Rancher UI (or API), the following events occur.
|
||||
|
||||
1. Rancher creates a `serviceAccount` that it uses to remove the Rancher components from the cluster. This account is assigned the [clusterRole](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#role-and-clusterrole) and [clusterRoleBinding](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#rolebinding-and-clusterrolebinding) permissions, which are required to remove the Rancher components.
|
||||
|
||||
1. Using the `serviceAccount`, Rancher schedules and runs a [job](https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/) that cleans the Rancher components off of the cluster. This job also references the `serviceAccount` and its roles as dependencies, so the job deletes them before its completion.
|
||||
|
||||
1. Rancher is removed from the cluster. However, the cluster persists, running the native version of Kubernetes.
|
||||
|
||||
**Result:** All components listed for registered clusters in [What Gets Removed?](#what-gets-removed) are deleted.
|
||||
|
||||
|
||||
</TabItem>
|
||||
<TabItem value="By Script">
|
||||
|
||||
Rather than cleaning registered cluster nodes using the Rancher UI, you can run a script instead.
|
||||
|
||||
>**Prerequisite:**
|
||||
>
|
||||
>Install [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/).
|
||||
|
||||
1. Open a web browser, navigate to [GitHub](https://github.com/rancher/rancher/blob/master/cleanup/user-cluster.sh), and download `user-cluster.sh`.
|
||||
|
||||
1. Make the script executable by running the following command from the same directory as `user-cluster.sh`:
|
||||
|
||||
```
|
||||
chmod +x user-cluster.sh
|
||||
```
|
||||
|
||||
1. **Air Gap Environments Only:** Open `user-cluster.sh` and replace `yaml_url` with the URL in `user-cluster.yml`.
|
||||
|
||||
If you don't have an air gap environment, skip this step.
|
||||
|
||||
1. From the same directory, run the script and provide the `rancher/rancher-agent` image version which should be equal to the version of Rancher used to manage the cluster. (`<RANCHER_VERSION>`):
|
||||
|
||||
>**Tip:**
|
||||
>
|
||||
>Add the `-dry-run` flag to preview the script's outcome without making changes.
|
||||
```
|
||||
./user-cluster.sh rancher/rancher-agent:<RANCHER_VERSION>
|
||||
```
|
||||
|
||||
**Result:** The script runs. All components listed for registered clusters in [What Gets Removed?](#what-gets-removed) are deleted.
|
||||
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
|
||||
### Windows Nodes
|
||||
|
||||
To clean up a Windows node, you can run a cleanup script located in `c:\etc\rancher`. The script deletes Kubernetes generated resources and the execution binary. It also drops the firewall rules and network settings.
|
||||
|
||||
To run the script, you can use this command in the PowerShell:
|
||||
|
||||
```
|
||||
pushd c:\etc\rancher
|
||||
.\cleanup.ps1
|
||||
popd
|
||||
```
|
||||
|
||||
**Result:** The node is reset and can be re-added to a Kubernetes cluster.
|
||||
|
||||
### Docker Containers, Images, and Volumes
|
||||
|
||||
Based on what role you assigned to the node, there are Kubernetes components in containers, containers belonging to overlay networking, DNS, ingress controller and Rancher agent. (and pods you created that have been scheduled to this node)
|
||||
|
||||
**To clean all Docker containers, images and volumes:**
|
||||
|
||||
```
|
||||
docker rm -f $(docker ps -qa)
|
||||
docker rmi -f $(docker images -q)
|
||||
docker volume rm $(docker volume ls -q)
|
||||
```
|
||||
|
||||
### Mounts
|
||||
|
||||
Kubernetes components and secrets leave behind mounts on the system that need to be unmounted.
|
||||
|
||||
Mounts |
|
||||
--------|
|
||||
`/var/lib/kubelet/pods/XXX` (miscellaneous mounts) |
|
||||
`/var/lib/kubelet` |
|
||||
`/var/lib/rancher` |
|
||||
|
||||
**To unmount all mounts:**
|
||||
|
||||
```
|
||||
for mount in $(mount | grep tmpfs | grep '/var/lib/kubelet' | awk '{ print $3 }') /var/lib/kubelet /var/lib/rancher; do umount $mount; done
|
||||
```
|
||||
|
||||
### Directories and Files
|
||||
|
||||
The following directories are used when adding a node to a cluster, and should be removed. You can remove a directory using `rm -rf /directory_name`.
|
||||
|
||||
>**Note:** Depending on the role you assigned to the node, some of the directories will or won't be present on the node.
|
||||
|
||||
Directories |
|
||||
--------|
|
||||
`/etc/ceph` |
|
||||
`/etc/cni` |
|
||||
`/etc/kubernetes` |
|
||||
`/opt/cni` |
|
||||
`/opt/rke` |
|
||||
`/run/secrets/kubernetes.io` |
|
||||
`/run/calico` |
|
||||
`/run/flannel` |
|
||||
`/var/lib/calico` |
|
||||
`/var/lib/etcd` |
|
||||
`/var/lib/cni` |
|
||||
`/var/lib/kubelet` |
|
||||
`/var/lib/rancher/rke/log` |
|
||||
`/var/log/containers` |
|
||||
`/var/log/kube-audit` |
|
||||
`/var/log/pods` |
|
||||
`/var/run/calico` |
|
||||
|
||||
**To clean the directories:**
|
||||
|
||||
```
|
||||
rm -rf /etc/ceph \
|
||||
/etc/cni \
|
||||
/etc/kubernetes \
|
||||
/opt/cni \
|
||||
/opt/rke \
|
||||
/run/secrets/kubernetes.io \
|
||||
/run/calico \
|
||||
/run/flannel \
|
||||
/var/lib/calico \
|
||||
/var/lib/etcd \
|
||||
/var/lib/cni \
|
||||
/var/lib/kubelet \
|
||||
/var/lib/rancher/rke/log \
|
||||
/var/log/containers \
|
||||
/var/log/kube-audit \
|
||||
/var/log/pods \
|
||||
/var/run/calico
|
||||
```
|
||||
|
||||
### Network Interfaces and Iptables
|
||||
|
||||
The remaining two components that are changed/configured are (virtual) network interfaces and iptables rules. Both are non-persistent to the node, meaning that they will be cleared after a restart of the node. To remove these components, a restart is recommended.
|
||||
|
||||
**To restart a node:**
|
||||
|
||||
```
|
||||
# using reboot
|
||||
$ sudo reboot
|
||||
|
||||
# using shutdown
|
||||
$ sudo shutdown -r now
|
||||
```
|
||||
|
||||
If you want to know more on (virtual) network interfaces or iptables rules, please see the specific subjects below.
|
||||
|
||||
### Network Interfaces
|
||||
|
||||
>**Note:** Depending on the network provider configured for the cluster the node was part of, some of the interfaces will or won't be present on the node.
|
||||
|
||||
Interfaces |
|
||||
--------|
|
||||
`flannel.1` |
|
||||
`cni0` |
|
||||
`tunl0` |
|
||||
`caliXXXXXXXXXXX` (random interface names) |
|
||||
`vethXXXXXXXX` (random interface names) |
|
||||
|
||||
**To list all interfaces:**
|
||||
|
||||
```
|
||||
# Using ip
|
||||
ip address show
|
||||
|
||||
# Using ifconfig
|
||||
ifconfig -a
|
||||
```
|
||||
|
||||
**To remove an interface:**
|
||||
|
||||
```
|
||||
ip link delete interface_name
|
||||
```
|
||||
|
||||
### Iptables
|
||||
|
||||
>**Note:** Depending on the network provider configured for the cluster the node was part of, some of the chains will or won't be present on the node.
|
||||
|
||||
Iptables rules are used to route traffic from and to containers. The created rules are not persistent, so restarting the node will restore iptables to its original state.
|
||||
|
||||
Chains |
|
||||
--------|
|
||||
`cali-failsafe-in` |
|
||||
`cali-failsafe-out` |
|
||||
`cali-fip-dnat` |
|
||||
`cali-fip-snat` |
|
||||
`cali-from-hep-forward` |
|
||||
`cali-from-host-endpoint` |
|
||||
`cali-from-wl-dispatch` |
|
||||
`cali-fw-caliXXXXXXXXXXX` (random chain names) |
|
||||
`cali-nat-outgoing` |
|
||||
`cali-pri-kns.NAMESPACE` (chain per namespace) |
|
||||
`cali-pro-kns.NAMESPACE` (chain per namespace) |
|
||||
`cali-to-hep-forward` |
|
||||
`cali-to-host-endpoint` |
|
||||
`cali-to-wl-dispatch` |
|
||||
`cali-tw-caliXXXXXXXXXXX` (random chain names) |
|
||||
`cali-wl-to-host` |
|
||||
`KUBE-EXTERNAL-SERVICES` |
|
||||
`KUBE-FIREWALL` |
|
||||
`KUBE-MARK-DROP` |
|
||||
`KUBE-MARK-MASQ` |
|
||||
`KUBE-NODEPORTS` |
|
||||
`KUBE-SEP-XXXXXXXXXXXXXXXX` (random chain names) |
|
||||
`KUBE-SERVICES` |
|
||||
`KUBE-SVC-XXXXXXXXXXXXXXXX` (random chain names) |
|
||||
|
||||
**To list all iptables rules:**
|
||||
|
||||
```
|
||||
iptables -L -t nat
|
||||
iptables -L -t mangle
|
||||
iptables -L
|
||||
```
|
||||
@@ -0,0 +1,99 @@
|
||||
---
|
||||
title: Cloning Clusters
|
||||
weight: 2035
|
||||
---
|
||||
|
||||
If you have a cluster in Rancher that you want to use as a template for creating similar clusters, you can use Rancher CLI to clone the cluster's configuration, edit it, and then use it to quickly launch the cloned cluster.
|
||||
|
||||
Duplication of registered clusters is not supported.
|
||||
|
||||
| Cluster Type | Cloneable? |
|
||||
|----------------------------------|---------------|
|
||||
| [Nodes Hosted by Infrastructure Provider]({{<baseurl>}}/rancher/v2.6/en/cluster-provisioning/rke-clusters/node-pools/) | ✓ |
|
||||
| [Hosted Kubernetes Providers]({{<baseurl>}}/rancher/v2.6/en/cluster-provisioning/hosted-kubernetes-clusters/) | ✓ |
|
||||
| [Custom Cluster]({{<baseurl>}}/rancher/v2.6/en/cluster-provisioning/rke-clusters/custom-nodes) | ✓ |
|
||||
| [Registered Cluster]({{<baseurl>}}/rancher/v2.6/en/cluster-provisioning/registered-clusters/) | |
|
||||
|
||||
> **Warning:** During the process of duplicating a cluster, you will edit a config file full of cluster settings. However, we recommend editing only values explicitly listed in this document, as cluster duplication is designed for simple cluster copying, _not_ wide scale configuration changes. Editing other values may invalidate the config file, which will lead to cluster deployment failure.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
Download and install [Rancher CLI]({{<baseurl>}}/rancher/v2.6/en/cli). Remember to [create an API bearer token]({{<baseurl>}}/rancher/v2.6/en/user-settings/api-keys) if necessary.
|
||||
|
||||
|
||||
## 1. Export Cluster Config
|
||||
|
||||
Begin by using Rancher CLI to export the configuration for the cluster that you want to clone.
|
||||
|
||||
1. Open Terminal and change your directory to the location of the Rancher CLI binary, `rancher`.
|
||||
|
||||
1. Enter the following command to list the clusters managed by Rancher.
|
||||
|
||||
|
||||
./rancher cluster ls
|
||||
|
||||
|
||||
1. Find the cluster that you want to clone, and copy either its resource `ID` or `NAME` to your clipboard. From this point on, we'll refer to the resource `ID` or `NAME` as `<RESOURCE_ID>`, which is used as a placeholder in the next step.
|
||||
|
||||
1. Enter the following command to export the configuration for your cluster.
|
||||
|
||||
|
||||
./rancher clusters export <RESOURCE_ID>
|
||||
|
||||
|
||||
**Step Result:** The YAML for a cloned cluster prints to Terminal.
|
||||
|
||||
1. Copy the YAML to your clipboard and paste it in a new file. Save the file as `cluster-template.yml` (or any other name, as long as it has a `.yml` extension).
|
||||
|
||||
## 2. Modify Cluster Config
|
||||
|
||||
Use your favorite text editor to modify the cluster configuration in `cluster-template.yml` for your cloned cluster.
|
||||
|
||||
> **Note:** Cluster configuration directives must be nested under the `rancher_kubernetes_engine_config` directive in `cluster.yml`. For more information, refer to the section on [the config file structure in Rancher v2.3.0+.]({{<baseurl>}}/rancher/v2.6/en/cluster-provisioning/rke-clusters/options/#config-file-structure-in-rancher-v2-3-0)
|
||||
|
||||
1. Open `cluster-template.yml` (or whatever you named your config) in your favorite text editor.
|
||||
|
||||
>**Warning:** Only edit the cluster config values explicitly called out below. Many of the values listed in this file are used to provision your cloned cluster, and editing their values may break the provisioning process.
|
||||
|
||||
|
||||
1. As depicted in the example below, at the `<CLUSTER_NAME>` placeholder, replace your original cluster's name with a unique name (`<CLUSTER_NAME>`). If your cloned cluster has a duplicate name, the cluster will not provision successfully.
|
||||
|
||||
```yml
|
||||
Version: v3
|
||||
clusters:
|
||||
<CLUSTER_NAME>: # ENTER UNIQUE NAME
|
||||
dockerRootDir: /var/lib/docker
|
||||
enableNetworkPolicy: false
|
||||
rancherKubernetesEngineConfig:
|
||||
addonJobTimeout: 30
|
||||
authentication:
|
||||
strategy: x509
|
||||
authorization: {}
|
||||
bastionHost: {}
|
||||
cloudProvider: {}
|
||||
ignoreDockerVersion: true
|
||||
```
|
||||
|
||||
1. For each `nodePools` section, replace the original nodepool name with a unique name at the `<NODEPOOL_NAME>` placeholder. If your cloned cluster has a duplicate nodepool name, the cluster will not provision successfully.
|
||||
|
||||
```yml
|
||||
nodePools:
|
||||
<NODEPOOL_NAME>:
|
||||
clusterId: do
|
||||
controlPlane: true
|
||||
etcd: true
|
||||
hostnamePrefix: mark-do
|
||||
nodeTemplateId: do
|
||||
quantity: 1
|
||||
worker: true
|
||||
```
|
||||
|
||||
1. When you're done, save and close the configuration.
|
||||
|
||||
## 3. Launch Cloned Cluster
|
||||
|
||||
Move `cluster-template.yml` into the same directory as the Rancher CLI binary. Then run this command:
|
||||
|
||||
./rancher up --file cluster-template.yml
|
||||
|
||||
**Result:** Your cloned cluster begins provisioning. Enter `./rancher cluster ls` to confirm.
|
||||
@@ -0,0 +1,44 @@
|
||||
---
|
||||
title: How the Authorized Cluster Endpoint Works
|
||||
weight: 2015
|
||||
---
|
||||
|
||||
This section describes how the kubectl CLI, the kubeconfig file, and the authorized cluster endpoint work together to allow you to access a downstream Kubernetes cluster directly, without authenticating through the Rancher server. It is intended to provide background information and context to the instructions for [how to set up kubectl to directly access a cluster.](../kubectl/#authenticating-directly-with-a-downstream-cluster)
|
||||
|
||||
### About the kubeconfig File
|
||||
|
||||
The _kubeconfig file_ is a file used to configure access to Kubernetes when used in conjunction with the kubectl command line tool (or other clients).
|
||||
|
||||
This kubeconfig file and its contents are specific to the cluster you are viewing. It can be downloaded from the cluster view in Rancher. You will need a separate kubeconfig file for each cluster that you have access to in Rancher.
|
||||
|
||||
After you download the kubeconfig file, you will be able to use the kubeconfig file and its Kubernetes [contexts](https://kubernetes.io/docs/reference/kubectl/cheatsheet/#kubectl-context-and-configuration) to access your downstream cluster.
|
||||
|
||||
If admins have [enforced TTL on kubeconfig tokens]({{<baseurl>}}/rancher/v2.6/en/api/api-tokens/#setting-ttl-on-kubeconfig-tokens), the kubeconfig file requires [rancher cli]({{<baseurl>}}/rancher/v2.6/en/cluster-admin/cluster-access/ace) to be present in your PATH.
|
||||
|
||||
|
||||
### Two Authentication Methods for RKE Clusters
|
||||
|
||||
If the cluster is not an [RKE cluster,]({{<baseurl>}}/rancher/v2.6/en/cluster-provisioning/rke-clusters/) the kubeconfig file allows you to access the cluster in only one way: it lets you be authenticated with the Rancher server, then Rancher allows you to run kubectl commands on the cluster.
|
||||
|
||||
For RKE clusters, the kubeconfig file allows you to be authenticated in two ways:
|
||||
|
||||
- **Through the Rancher server authentication proxy:** Rancher's authentication proxy validates your identity, then connects you to the downstream cluster that you want to access.
|
||||
- **Directly with the downstream cluster's API server:** RKE clusters have an authorized cluster endpoint enabled by default. This endpoint allows you to access your downstream Kubernetes cluster with the kubectl CLI and a kubeconfig file, and it is enabled by default for RKE clusters. In this scenario, the downstream cluster's Kubernetes API server authenticates you by calling a webhook (the `kube-api-auth` microservice) that Rancher set up.
|
||||
|
||||
This second method, the capability to connect directly to the cluster's Kubernetes API server, is important because it lets you access your downstream cluster if you can't connect to Rancher.
|
||||
|
||||
To use the authorized cluster endpoint, you will need to configure kubectl to use the extra kubectl context in the kubeconfig file that Rancher generates for you when the RKE cluster is created. This file can be downloaded from the cluster view in the Rancher UI, and the instructions for configuring kubectl are on [this page.](../kubectl/#authenticating-directly-with-a-downstream-cluster)
|
||||
|
||||
These methods of communicating with downstream Kubernetes clusters are also explained in the [architecture page]({{<baseurl>}}/rancher/v2.6/en/overview/architecture/#communicating-with-downstream-user-clusters) in the larger context of explaining how Rancher works and how Rancher communicates with downstream clusters.
|
||||
|
||||
### About the kube-api-auth Authentication Webhook
|
||||
|
||||
The `kube-api-auth` microservice is deployed to provide the user authentication functionality for the [authorized cluster endpoint,]({{<baseurl>}}/rancher/v2.6/en/overview/architecture/#4-authorized-cluster-endpoint) which is only available for [RKE clusters.]({{<baseurl>}}/rancher/v2.6/en/cluster-provisioning/rke-clusters/) When you access the user cluster using `kubectl`, the cluster's Kubernetes API server authenticates you by using the `kube-api-auth` service as a webhook.
|
||||
|
||||
During cluster provisioning, the file `/etc/kubernetes/kube-api-authn-webhook.yaml` is deployed and `kube-apiserver` is configured with `--authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml`. This configures the `kube-apiserver` to query `http://127.0.0.1:6440/v1/authenticate` to determine authentication for bearer tokens.
|
||||
|
||||
The scheduling rules for `kube-api-auth` are listed below:
|
||||
|
||||
| Component | nodeAffinity nodeSelectorTerms | nodeSelector | Tolerations |
|
||||
| -------------------- | ------------------------------------------ | ------------ | ------------------------------------------------------------------------------ |
|
||||
| kube-api-auth | `beta.kubernetes.io/os:NotIn:windows`<br/>`node-role.kubernetes.io/controlplane:In:"true"` | none | `operator:Exists` |
|
||||
@@ -0,0 +1,32 @@
|
||||
---
|
||||
title: Cluster Access
|
||||
weight: 1
|
||||
---
|
||||
|
||||
This section is about what tools can be used to access clusters managed by Rancher.
|
||||
|
||||
For information on how to give users permission to access a cluster, see the section on [adding users to clusters.]({{<baseurl>}}/rancher/v2.6/en/cluster-admin/cluster-access/cluster-members/)
|
||||
|
||||
For more information on roles-based access control, see [this section.]({{<baseurl>}}/rancher/v2.6/en/admin-settings/rbac/)
|
||||
|
||||
For information on how to set up an authentication system, see [this section.]({{<baseurl>}}/rancher/v2.6/en/admin-settings/authentication/)
|
||||
|
||||
|
||||
### Rancher UI
|
||||
|
||||
Rancher provides an intuitive user interface for interacting with your clusters. All options available in the UI use the Rancher API. Therefore any action possible in the UI is also possible in the Rancher CLI or Rancher API.
|
||||
|
||||
### kubectl
|
||||
|
||||
You can use the Kubernetes command-line tool, [kubectl](https://kubernetes.io/docs/reference/kubectl/overview/), to manage your clusters. You have two options for using kubectl:
|
||||
|
||||
- **Rancher kubectl shell:** Interact with your clusters by launching a kubectl shell available in the Rancher UI. This option requires no configuration actions on your part. For more information, see [Accessing Clusters with kubectl Shell]({{<baseurl>}}/rancher/v2.6/en/cluster-admin/cluster-access/kubectl/).
|
||||
- **Terminal remote connection:** You can also interact with your clusters by installing [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/) on your local desktop and then copying the cluster's kubeconfig file to your local `~/.kube/config` directory. For more information, see [Accessing Clusters with kubectl and a kubeconfig File](./kubectl/).
|
||||
|
||||
### Rancher CLI
|
||||
|
||||
You can control your clusters by downloading Rancher's own command-line interface, [Rancher CLI]({{<baseurl>}}/rancher/v2.6/en/cli/). This CLI tool can interact directly with different clusters and projects or pass them `kubectl` commands.
|
||||
|
||||
### Rancher API
|
||||
|
||||
Finally, you can interact with your clusters over the Rancher API. Before you use the API, you must obtain an [API key]({{<baseurl>}}/rancher/v2.6/en/user-settings/api-keys/). To view the different resource fields and actions for an API object, open the API UI, which can be accessed by clicking on **View in API** for any Rancher UI object.
|
||||
+52
@@ -0,0 +1,52 @@
|
||||
---
|
||||
title: Adding Users to Clusters
|
||||
weight: 2020
|
||||
---
|
||||
|
||||
If you want to provide a user with access and permissions to _all_ projects, nodes, and resources within a cluster, assign the user a cluster membership.
|
||||
|
||||
>**Tip:** Want to provide a user with access to a _specific_ project within a cluster? See [Adding Project Members]({{<baseurl>}}/rancher/v2.6/en/cluster-admin/projects-and-namespaces/project-members/) instead.
|
||||
|
||||
There are two contexts where you can add cluster members:
|
||||
|
||||
- Adding Members to a New Cluster
|
||||
|
||||
You can add members to a cluster as you create it (recommended if possible).
|
||||
|
||||
- [Adding Members to an Existing Cluster](#editing-cluster-membership)
|
||||
|
||||
You can always add members to a cluster after a cluster is provisioned.
|
||||
|
||||
## Editing Cluster Membership
|
||||
|
||||
Cluster administrators can edit the membership for a cluster, controlling which Rancher users can access the cluster and what features they can use.
|
||||
|
||||
1. Click **☰ > Cluster Management**.
|
||||
1. Go to the cluster you want to add members to and click **⋮ > Edit Config**.
|
||||
1. In the **Member Roles** tab, click **Add Member**.
|
||||
1. Search for the user or group that you want to add to the cluster.
|
||||
|
||||
If external authentication is configured:
|
||||
|
||||
- Rancher returns users from your [external authentication]({{<baseurl>}}/rancher/v2.6/en/admin-settings/authentication/) source as you type.
|
||||
|
||||
>**Using AD but can't find your users?**
|
||||
>There may be an issue with your search attribute configuration. See [Configuring Active Directory Authentication: Step 5]({{<baseurl>}}/rancher/v2.6/en/admin-settings/authentication/ad/).
|
||||
|
||||
- A drop-down allows you to add groups instead of individual users. The drop-down only lists groups that you, the logged in user, are part of.
|
||||
|
||||
>**Note:** If you are logged in as a local user, external users do not display in your search results. For more information, see [External Authentication Configuration and Principal Users]({{<baseurl>}}/rancher/v2.6/en/admin-settings/authentication/#external-authentication-configuration-and-principal-users).
|
||||
|
||||
1. Assign the user or group **Cluster** roles.
|
||||
|
||||
[What are Cluster Roles?]({{<baseurl>}}/rancher/v2.6/en/admin-settings/rbac/cluster-project-roles/)
|
||||
|
||||
>**Tip:** For Custom Roles, you can modify the list of individual roles available for assignment.
|
||||
>
|
||||
> - To add roles to the list, [Add a Custom Role]({{<baseurl>}}/rancher/v2.6/en/admin-settings/rbac/default-custom-roles/).
|
||||
> - To remove roles from the list, [Lock/Unlock Roles]({{<baseurl>}}/rancher/v2.6/en/admin-settings/rbac/locked-roles).
|
||||
|
||||
**Result:** The chosen users are added to the cluster.
|
||||
|
||||
- To revoke cluster membership, select the user and click **Delete**. This action deletes membership, not the user.
|
||||
- To modify a user's roles in the cluster, delete them from the cluster, and then re-add them with modified roles.
|
||||
@@ -0,0 +1,103 @@
|
||||
---
|
||||
title: "Access a Cluster with Kubectl and kubeconfig"
|
||||
description: "Learn how you can access and manage your Kubernetes clusters using kubectl with kubectl Shell or with kubectl CLI and kubeconfig file. A kubeconfig file is used to configure access to Kubernetes. When you create a cluster with Rancher, it automatically creates a kubeconfig for your cluster."
|
||||
weight: 2010
|
||||
---
|
||||
|
||||
This section describes how to manipulate your downstream Kubernetes cluster with kubectl from the Rancher UI or from your workstation.
|
||||
|
||||
For more information on using kubectl, see [Kubernetes Documentation: Overview of kubectl](https://kubernetes.io/docs/reference/kubectl/overview/).
|
||||
|
||||
- [Accessing clusters with kubectl shell in the Rancher UI](#accessing-clusters-with-kubectl-shell-in-the-rancher-ui)
|
||||
- [Accessing clusters with kubectl from your workstation](#accessing-clusters-with-kubectl-from-your-workstation)
|
||||
- [Note on Resources created using kubectl](#note-on-resources-created-using-kubectl)
|
||||
- [Authenticating Directly with a Downstream Cluster](#authenticating-directly-with-a-downstream-cluster)
|
||||
- [Connecting Directly to Clusters with FQDN Defined](#connecting-directly-to-clusters-with-fqdn-defined)
|
||||
- [Connecting Directly to Clusters without FQDN Defined](#connecting-directly-to-clusters-without-fqdn-defined)
|
||||
|
||||
|
||||
### Accessing Clusters with kubectl Shell in the Rancher UI
|
||||
|
||||
You can access and manage your clusters by logging into Rancher and opening the kubectl shell in the UI. No further configuration necessary.
|
||||
|
||||
1. Click **☰ > Cluster Management**.
|
||||
1. Go to the cluster you want to access with kubectl and click **Explore**.
|
||||
1. In the top navigation menu, click the **Kubectl Shell** button. Use the window that opens to interact with your Kubernetes cluster.
|
||||
|
||||
### Accessing Clusters with kubectl from Your Workstation
|
||||
|
||||
This section describes how to download your cluster's kubeconfig file, launch kubectl from your workstation, and access your downstream cluster.
|
||||
|
||||
This alternative method of accessing the cluster allows you to authenticate with Rancher and manage your cluster without using the Rancher UI.
|
||||
|
||||
> **Prerequisites:** These instructions assume that you have already created a Kubernetes cluster, and that kubectl is installed on your workstation. For help installing kubectl, refer to the official [Kubernetes documentation.](https://kubernetes.io/docs/tasks/tools/install-kubectl/)
|
||||
|
||||
1. Log into Rancher. Click **☰ > Cluster Management**.
|
||||
1. Go to the cluster that you want to access with kubectl and click **Explore**.
|
||||
1. In the top navigation bar, click **Download KubeConfig** button.
|
||||
1. Save the YAML file on your local computer. Move the file to `~/.kube/config`. Note: The default location that kubectl uses for the kubeconfig file is `~/.kube/config`, but you can use any directory and specify it using the `--kubeconfig` flag, as in this command:
|
||||
```
|
||||
kubectl --kubeconfig /custom/path/kube.config get pods
|
||||
```
|
||||
1. From your workstation, launch kubectl. Use it to interact with your kubernetes cluster.
|
||||
|
||||
|
||||
### Note on Resources Created Using kubectl
|
||||
|
||||
Rancher will discover and show resources created by `kubectl`. However, these resources might not have all the necessary annotations on discovery. If an operation (for instance, scaling the workload) is done to the resource using the Rancher UI/API, this may trigger recreation of the resources due to the missing annotations. This should only happen the first time an operation is done to the discovered resource.
|
||||
|
||||
# Authenticating Directly with a Downstream Cluster
|
||||
|
||||
This section intended to help you set up an alternative method to access an [RKE cluster.]({{<baseurl>}}/rancher/v2.6/en/cluster-provisioning/rke-clusters)
|
||||
|
||||
This method is only available for RKE clusters that have the [authorized cluster endpoint]({{<baseurl>}}/rancher/v2.6/en/overview/architecture/#4-authorized-cluster-endpoint) enabled. When Rancher creates this RKE cluster, it generates a kubeconfig file that includes additional kubectl context(s) for accessing your cluster. This additional context allows you to use kubectl to authenticate with the downstream cluster without authenticating through Rancher. For a longer explanation of how the authorized cluster endpoint works, refer to [this page.](../ace)
|
||||
|
||||
We recommend that as a best practice, you should set up this method to access your RKE cluster, so that just in case you can’t connect to Rancher, you can still access the cluster.
|
||||
|
||||
> **Prerequisites:** The following steps assume that you have created a Kubernetes cluster and followed the steps to [connect to your cluster with kubectl from your workstation.](#accessing-clusters-with-kubectl-from-your-workstation)
|
||||
|
||||
To find the name of the context(s) in your downloaded kubeconfig file, run:
|
||||
|
||||
```
|
||||
kubectl config get-contexts --kubeconfig /custom/path/kube.config
|
||||
CURRENT NAME CLUSTER AUTHINFO NAMESPACE
|
||||
* my-cluster my-cluster user-46tmn
|
||||
my-cluster-controlplane-1 my-cluster-controlplane-1 user-46tmn
|
||||
```
|
||||
|
||||
In this example, when you use `kubectl` with the first context, `my-cluster`, you will be authenticated through the Rancher server.
|
||||
|
||||
With the second context, `my-cluster-controlplane-1`, you would authenticate with the authorized cluster endpoint, communicating with an downstream RKE cluster directly.
|
||||
|
||||
We recommend using a load balancer with the authorized cluster endpoint. For details, refer to the [recommended architecture section.]({{<baseurl>}}/rancher/v2.6/en/overview/architecture-recommendations/#architecture-for-an-authorized-cluster-endpoint)
|
||||
|
||||
Now that you have the name of the context needed to authenticate directly with the cluster, you can pass the name of the context in as an option when running kubectl commands. The commands will differ depending on whether your cluster has an FQDN defined. Examples are provided in the sections below.
|
||||
|
||||
When `kubectl` works normally, it confirms that you can access your cluster while bypassing Rancher's authentication proxy.
|
||||
|
||||
### Connecting Directly to Clusters with FQDN Defined
|
||||
|
||||
If an FQDN is defined for the cluster, a single context referencing the FQDN will be created. The context will be named `<CLUSTER_NAME>-fqdn`. When you want to use `kubectl` to access this cluster without Rancher, you will need to use this context.
|
||||
|
||||
Assuming the kubeconfig file is located at `~/.kube/config`:
|
||||
|
||||
```
|
||||
kubectl --context <CLUSTER_NAME>-fqdn get nodes
|
||||
```
|
||||
Directly referencing the location of the kubeconfig file:
|
||||
```
|
||||
kubectl --kubeconfig /custom/path/kube.config --context <CLUSTER_NAME>-fqdn get pods
|
||||
```
|
||||
|
||||
### Connecting Directly to Clusters without FQDN Defined
|
||||
|
||||
If there is no FQDN defined for the cluster, extra contexts will be created referencing the IP address of each node in the control plane. Each context will be named `<CLUSTER_NAME>-<NODE_NAME>`. When you want to use `kubectl` to access this cluster without Rancher, you will need to use this context.
|
||||
|
||||
Assuming the kubeconfig file is located at `~/.kube/config`:
|
||||
```
|
||||
kubectl --context <CLUSTER_NAME>-<NODE_NAME> get nodes
|
||||
```
|
||||
Directly referencing the location of the kubeconfig file:
|
||||
```
|
||||
kubectl --kubeconfig /custom/path/kube.config --context <CLUSTER_NAME>-<NODE_NAME> get pods
|
||||
```
|
||||
@@ -0,0 +1,33 @@
|
||||
---
|
||||
title: Cluster Administration
|
||||
weight: 8
|
||||
---
|
||||
|
||||
After you provision a cluster in Rancher, you can begin using powerful Kubernetes features to deploy and scale your containerized applications in development, testing, or production environments.
|
||||
|
||||
This page covers the following topics:
|
||||
|
||||
- [Switching between clusters](#switching-between-clusters)
|
||||
- [Managing clusters in Rancher](#managing-clusters-in-rancher)
|
||||
- [Configuring tools](#configuring-tools)
|
||||
|
||||
> This section assumes a basic familiarity with Docker and Kubernetes. For a brief explanation of how Kubernetes components work together, refer to the [concepts]({{<baseurl>}}/rancher/v2.6/en/overview/concepts) page.
|
||||
|
||||
## Managing Clusters in Rancher
|
||||
|
||||
After clusters have been [provisioned into Rancher]({{<baseurl>}}/rancher/v2.6/en/cluster-provisioning/), [cluster owners]({{<baseurl>}}/rancher/v2.6/en/admin-settings/rbac/cluster-project-roles/#cluster-roles) will need to manage these clusters. There are many different options of how to manage your cluster.
|
||||
|
||||
{{% include file="/rancher/v2.6/en/cluster-provisioning/cluster-capabilities-table</summary>
|
||||
|
||||
## Configuring Tools
|
||||
|
||||
Rancher contains a variety of tools that aren't included in Kubernetes to assist in your DevOps operations. Rancher can integrate with external services to help your clusters run more efficiently. Tools are divided into following categories:
|
||||
|
||||
- Alerts
|
||||
- Notifiers
|
||||
- Logging
|
||||
- Monitoring
|
||||
- Istio Service Mesh
|
||||
- OPA Gatekeeper
|
||||
|
||||
Tools can be installed through **Apps & Marketplace.**
|
||||
@@ -0,0 +1,580 @@
|
||||
---
|
||||
title: Cluster Autoscaler with AWS EC2 Auto Scaling Groups
|
||||
weight: 1
|
||||
---
|
||||
|
||||
This guide will show you how to install and use [Kubernetes cluster-autoscaler](https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/) on Rancher custom clusters using AWS EC2 Auto Scaling Groups.
|
||||
|
||||
We are going to install a Rancher RKE custom cluster with a fixed number of nodes with the etcd and controlplane roles, and a variable nodes with the worker role, managed by `cluster-autoscaler`.
|
||||
|
||||
- [Prerequisites](#prerequisites)
|
||||
- [1. Create a Custom Cluster](#1-create-a-custom-cluster)
|
||||
- [2. Configure the Cloud Provider](#2-configure-the-cloud-provider)
|
||||
- [3. Deploy Nodes](#3-deploy-nodes)
|
||||
- [4. Install cluster-autoscaler](#4-install-cluster-autoscaler)
|
||||
- [Parameters](#parameters)
|
||||
- [Deployment](#deployment)
|
||||
- [Testing](#testing)
|
||||
- [Generating Load](#generating-load)
|
||||
- [Checking Scale](#checking-scale)
|
||||
|
||||
# Prerequisites
|
||||
|
||||
These elements are required to follow this guide:
|
||||
|
||||
* The Rancher server is up and running
|
||||
* You have an AWS EC2 user with proper permissions to create virtual machines, auto scaling groups, and IAM profiles and roles
|
||||
|
||||
### 1. Create a Custom Cluster
|
||||
|
||||
On Rancher server, we should create a custom k8s cluster v1.18.x. Be sure that cloud_provider name is set to `amazonec2`. Once cluster is created we need to get:
|
||||
|
||||
* clusterID: `c-xxxxx` will be used on EC2 `kubernetes.io/cluster/<clusterID>` instance tag
|
||||
* clusterName: will be used on EC2 `k8s.io/cluster-autoscaler/<clusterName>` instance tag
|
||||
* nodeCommand: will be added on EC2 instance user_data to include new nodes on cluster
|
||||
|
||||
```sh
|
||||
sudo docker run -d --privileged --restart=unless-stopped --net=host -v /etc/kubernetes:/etc/kubernetes -v /var/run:/var/run rancher/rancher-agent:<RANCHER_VERSION> --server https://<RANCHER_URL> --token <RANCHER_TOKEN> --ca-checksum <RANCHER_CHECKSUM> <roles>
|
||||
```
|
||||
|
||||
### 2. Configure the Cloud Provider
|
||||
|
||||
On AWS EC2, we should create a few objects to configure our system. We've defined three distinct groups and IAM profiles to configure on AWS.
|
||||
|
||||
1. Autoscaling group: Nodes that will be part of the EC2 Auto Scaling Group (ASG). The ASG will be used by `cluster-autoscaler` to scale up and down.
|
||||
* IAM profile: Required by k8s nodes where cluster-autoscaler will be running. It is recommended for Kubernetes master nodes. This profile is called `K8sAutoscalerProfile`.
|
||||
|
||||
```json
|
||||
{
|
||||
"Version": "2012-10-17",
|
||||
"Statement": [
|
||||
{
|
||||
"Effect": "Allow",
|
||||
"Action": [
|
||||
"autoscaling:DescribeAutoScalingGroups",
|
||||
"autoscaling:DescribeAutoScalingInstances",
|
||||
"autoscaling:DescribeLaunchConfigurations",
|
||||
"autoscaling:SetDesiredCapacity",
|
||||
"autoscaling:TerminateInstanceInAutoScalingGroup",
|
||||
"autoscaling:DescribeTags",
|
||||
"autoscaling:DescribeLaunchConfigurations",
|
||||
"ec2:DescribeLaunchTemplateVersions"
|
||||
],
|
||||
"Resource": [
|
||||
"*"
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
2. Master group: Nodes that will be part of the Kubernetes etcd and/or control planes. This will be out of the ASG.
|
||||
* IAM profile: Required by the Kubernetes cloud_provider integration. Optionally, `AWS_ACCESS_KEY` and `AWS_SECRET_KEY` can be used instead [using-aws-credentials.](https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/aws/README.md#using-aws-credentials) This profile is called `K8sMasterProfile`.
|
||||
|
||||
```json
|
||||
{
|
||||
"Version": "2012-10-17",
|
||||
"Statement": [
|
||||
{
|
||||
"Effect": "Allow",
|
||||
"Action": [
|
||||
"autoscaling:DescribeAutoScalingGroups",
|
||||
"autoscaling:DescribeLaunchConfigurations",
|
||||
"autoscaling:DescribeTags",
|
||||
"ec2:DescribeInstances",
|
||||
"ec2:DescribeRegions",
|
||||
"ec2:DescribeRouteTables",
|
||||
"ec2:DescribeSecurityGroups",
|
||||
"ec2:DescribeSubnets",
|
||||
"ec2:DescribeVolumes",
|
||||
"ec2:CreateSecurityGroup",
|
||||
"ec2:CreateTags",
|
||||
"ec2:CreateVolume",
|
||||
"ec2:ModifyInstanceAttribute",
|
||||
"ec2:ModifyVolume",
|
||||
"ec2:AttachVolume",
|
||||
"ec2:AuthorizeSecurityGroupIngress",
|
||||
"ec2:CreateRoute",
|
||||
"ec2:DeleteRoute",
|
||||
"ec2:DeleteSecurityGroup",
|
||||
"ec2:DeleteVolume",
|
||||
"ec2:DetachVolume",
|
||||
"ec2:RevokeSecurityGroupIngress",
|
||||
"ec2:DescribeVpcs",
|
||||
"elasticloadbalancing:AddTags",
|
||||
"elasticloadbalancing:AttachLoadBalancerToSubnets",
|
||||
"elasticloadbalancing:ApplySecurityGroupsToLoadBalancer",
|
||||
"elasticloadbalancing:CreateLoadBalancer",
|
||||
"elasticloadbalancing:CreateLoadBalancerPolicy",
|
||||
"elasticloadbalancing:CreateLoadBalancerListeners",
|
||||
"elasticloadbalancing:ConfigureHealthCheck",
|
||||
"elasticloadbalancing:DeleteLoadBalancer",
|
||||
"elasticloadbalancing:DeleteLoadBalancerListeners",
|
||||
"elasticloadbalancing:DescribeLoadBalancers",
|
||||
"elasticloadbalancing:DescribeLoadBalancerAttributes",
|
||||
"elasticloadbalancing:DetachLoadBalancerFromSubnets",
|
||||
"elasticloadbalancing:DeregisterInstancesFromLoadBalancer",
|
||||
"elasticloadbalancing:ModifyLoadBalancerAttributes",
|
||||
"elasticloadbalancing:RegisterInstancesWithLoadBalancer",
|
||||
"elasticloadbalancing:SetLoadBalancerPoliciesForBackendServer",
|
||||
"elasticloadbalancing:AddTags",
|
||||
"elasticloadbalancing:CreateListener",
|
||||
"elasticloadbalancing:CreateTargetGroup",
|
||||
"elasticloadbalancing:DeleteListener",
|
||||
"elasticloadbalancing:DeleteTargetGroup",
|
||||
"elasticloadbalancing:DescribeListeners",
|
||||
"elasticloadbalancing:DescribeLoadBalancerPolicies",
|
||||
"elasticloadbalancing:DescribeTargetGroups",
|
||||
"elasticloadbalancing:DescribeTargetHealth",
|
||||
"elasticloadbalancing:ModifyListener",
|
||||
"elasticloadbalancing:ModifyTargetGroup",
|
||||
"elasticloadbalancing:RegisterTargets",
|
||||
"elasticloadbalancing:SetLoadBalancerPoliciesOfListener",
|
||||
"iam:CreateServiceLinkedRole",
|
||||
"ecr:GetAuthorizationToken",
|
||||
"ecr:BatchCheckLayerAvailability",
|
||||
"ecr:GetDownloadUrlForLayer",
|
||||
"ecr:GetRepositoryPolicy",
|
||||
"ecr:DescribeRepositories",
|
||||
"ecr:ListImages",
|
||||
"ecr:BatchGetImage",
|
||||
"kms:DescribeKey"
|
||||
],
|
||||
"Resource": [
|
||||
"*"
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
* IAM role: `K8sMasterRole: [K8sMasterProfile,K8sAutoscalerProfile]`
|
||||
* Security group: `K8sMasterSg` More info at[RKE ports (custom nodes tab)]({{<baseurl>}}/rancher/v2.6/en/installation/requirements/ports/#downstream-kubernetes-cluster-nodes)
|
||||
* Tags:
|
||||
`kubernetes.io/cluster/<clusterID>: owned`
|
||||
* User data: `K8sMasterUserData` Ubuntu 18.04(ami-0e11cbb34015ff725), installs docker and add etcd+controlplane node to the k8s cluster
|
||||
|
||||
```sh
|
||||
#!/bin/bash -x
|
||||
|
||||
cat <<EOF > /etc/sysctl.d/90-kubelet.conf
|
||||
vm.overcommit_memory = 1
|
||||
vm.panic_on_oom = 0
|
||||
kernel.panic = 10
|
||||
kernel.panic_on_oops = 1
|
||||
kernel.keys.root_maxkeys = 1000000
|
||||
kernel.keys.root_maxbytes = 25000000
|
||||
EOF
|
||||
sysctl -p /etc/sysctl.d/90-kubelet.conf
|
||||
|
||||
curl -sL https://releases.rancher.com/install-docker/19.03.sh | sh
|
||||
sudo usermod -aG docker ubuntu
|
||||
|
||||
TOKEN=$(curl -s -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600")
|
||||
PRIVATE_IP=$(curl -H "X-aws-ec2-metadata-token: ${TOKEN}" -s http://169.254.169.254/latest/meta-data/local-ipv4)
|
||||
PUBLIC_IP=$(curl -H "X-aws-ec2-metadata-token: ${TOKEN}" -s http://169.254.169.254/latest/meta-data/public-ipv4)
|
||||
K8S_ROLES="--etcd --controlplane"
|
||||
|
||||
sudo docker run -d --privileged --restart=unless-stopped --net=host -v /etc/kubernetes:/etc/kubernetes -v /var/run:/var/run rancher/rancher-agent:<RANCHER_VERSION> --server https://<RANCHER_URL> --token <RANCHER_TOKEN> --ca-checksum <RANCHER_CA_CHECKSUM> --address ${PUBLIC_IP} --internal-address ${PRIVATE_IP} ${K8S_ROLES}
|
||||
```
|
||||
|
||||
3. Worker group: Nodes that will be part of the k8s worker plane. Worker nodes will be scaled by cluster-autoscaler using the ASG.
|
||||
* IAM profile: Provides cloud_provider worker integration.
|
||||
This profile is called `K8sWorkerProfile`.
|
||||
|
||||
```json
|
||||
{
|
||||
"Version": "2012-10-17",
|
||||
"Statement": [
|
||||
{
|
||||
"Effect": "Allow",
|
||||
"Action": [
|
||||
"ec2:DescribeInstances",
|
||||
"ec2:DescribeRegions",
|
||||
"ecr:GetAuthorizationToken",
|
||||
"ecr:BatchCheckLayerAvailability",
|
||||
"ecr:GetDownloadUrlForLayer",
|
||||
"ecr:GetRepositoryPolicy",
|
||||
"ecr:DescribeRepositories",
|
||||
"ecr:ListImages",
|
||||
"ecr:BatchGetImage"
|
||||
],
|
||||
"Resource": "*"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
* IAM role: `K8sWorkerRole: [K8sWorkerProfile]`
|
||||
* Security group: `K8sWorkerSg` More info at [RKE ports (custom nodes tab)]({{<baseurl>}}/rancher/v2.6/en/installation/requirements/ports/#downstream-kubernetes-cluster-nodes)
|
||||
* Tags:
|
||||
* `kubernetes.io/cluster/<clusterID>: owned`
|
||||
* `k8s.io/cluster-autoscaler/<clusterName>: true`
|
||||
* `k8s.io/cluster-autoscaler/enabled: true`
|
||||
* User data: `K8sWorkerUserData` Ubuntu 18.04(ami-0e11cbb34015ff725), installs docker and add worker node to the k8s cluster
|
||||
|
||||
```sh
|
||||
#!/bin/bash -x
|
||||
|
||||
cat <<EOF > /etc/sysctl.d/90-kubelet.conf
|
||||
vm.overcommit_memory = 1
|
||||
vm.panic_on_oom = 0
|
||||
kernel.panic = 10
|
||||
kernel.panic_on_oops = 1
|
||||
kernel.keys.root_maxkeys = 1000000
|
||||
kernel.keys.root_maxbytes = 25000000
|
||||
EOF
|
||||
sysctl -p /etc/sysctl.d/90-kubelet.conf
|
||||
|
||||
curl -sL https://releases.rancher.com/install-docker/19.03.sh | sh
|
||||
sudo usermod -aG docker ubuntu
|
||||
|
||||
TOKEN=$(curl -s -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600")
|
||||
PRIVATE_IP=$(curl -H "X-aws-ec2-metadata-token: ${TOKEN}" -s http://169.254.169.254/latest/meta-data/local-ipv4)
|
||||
PUBLIC_IP=$(curl -H "X-aws-ec2-metadata-token: ${TOKEN}" -s http://169.254.169.254/latest/meta-data/public-ipv4)
|
||||
K8S_ROLES="--worker"
|
||||
|
||||
sudo docker run -d --privileged --restart=unless-stopped --net=host -v /etc/kubernetes:/etc/kubernetes -v /var/run:/var/run rancher/rancher-agent:<RANCHER_VERSION> --server https://<RANCHER_URL> --token <RANCHER_TOKEN> --ca-checksum <RANCHER_CA_CHECKCSUM> --address ${PUBLIC_IP} --internal-address ${PRIVATE_IP} ${K8S_ROLES}
|
||||
```
|
||||
|
||||
More info is at [RKE clusters on AWS]({{<baseurl>}}/rancher/v2.6/en/cluster-provisioning/rke-clusters/cloud-providers/amazon/) and [Cluster Autoscaler on AWS.](https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/aws/README.md)
|
||||
|
||||
### 3. Deploy Nodes
|
||||
|
||||
Once we've configured AWS, let's create VMs to bootstrap our cluster:
|
||||
|
||||
* master (etcd+controlplane): Depending your needs, deploy three master instances with proper size. More info is at [the recommendations for production-ready clusters.]({{<baseurl>}}/rancher/v2.6/en/cluster-provisioning/production/)
|
||||
* IAM role: `K8sMasterRole`
|
||||
* Security group: `K8sMasterSg`
|
||||
* Tags:
|
||||
* `kubernetes.io/cluster/<clusterID>: owned`
|
||||
* User data: `K8sMasterUserData`
|
||||
|
||||
* worker: Define an ASG on EC2 with the following settings:
|
||||
* Name: `K8sWorkerAsg`
|
||||
* IAM role: `K8sWorkerRole`
|
||||
* Security group: `K8sWorkerSg`
|
||||
* Tags:
|
||||
* `kubernetes.io/cluster/<clusterID>: owned`
|
||||
* `k8s.io/cluster-autoscaler/<clusterName>: true`
|
||||
* `k8s.io/cluster-autoscaler/enabled: true`
|
||||
* User data: `K8sWorkerUserData`
|
||||
* Instances:
|
||||
* minimum: 2
|
||||
* desired: 2
|
||||
* maximum: 10
|
||||
|
||||
Once the VMs are deployed, you should have a Rancher custom cluster up and running with three master and two worker nodes.
|
||||
|
||||
### 4. Install Cluster-autoscaler
|
||||
|
||||
At this point, we should have rancher cluster up and running. We are going to install cluster-autoscaler on master nodes and `kube-system` namespace, following cluster-autoscaler recommendation.
|
||||
|
||||
#### Parameters
|
||||
|
||||
This table shows cluster-autoscaler parameters for fine tuning:
|
||||
|
||||
| Parameter | Default | Description |
|
||||
|---|---|---|
|
||||
|cluster-name|-|Autoscaled cluster name, if available|
|
||||
|address|:8085|The address to expose Prometheus metrics|
|
||||
|kubernetes|-|Kubernetes master location. Leave blank for default|
|
||||
|kubeconfig|-|Path to kubeconfig file with authorization and master location information|
|
||||
|cloud-config|-|The path to the cloud provider configuration file. Empty string for no configuration file|
|
||||
|namespace|"kube-system"|Namespace in which cluster-autoscaler run|
|
||||
|scale-down-enabled|true|Should CA scale down the cluster|
|
||||
|scale-down-delay-after-add|"10m"|How long after scale up that scale down evaluation resumes|
|
||||
|scale-down-delay-after-delete|0|How long after node deletion that scale down evaluation resumes, defaults to scanInterval|
|
||||
|scale-down-delay-after-failure|"3m"|How long after scale down failure that scale down evaluation resumes|
|
||||
|scale-down-unneeded-time|"10m"|How long a node should be unneeded before it is eligible for scale down|
|
||||
|scale-down-unready-time|"20m"|How long an unready node should be unneeded before it is eligible for scale down|
|
||||
|scale-down-utilization-threshold|0.5|Sum of cpu or memory of all pods running on the node divided by node's corresponding allocatable resource, below which a node can be considered for scale down|
|
||||
|scale-down-gpu-utilization-threshold|0.5|Sum of gpu requests of all pods running on the node divided by node's allocatable resource, below which a node can be considered for scale down|
|
||||
|scale-down-non-empty-candidates-count|30|Maximum number of non empty nodes considered in one iteration as candidates for scale down with drain|
|
||||
|scale-down-candidates-pool-ratio|0.1|A ratio of nodes that are considered as additional non empty candidates for scale down when some candidates from previous iteration are no longer valid|
|
||||
|scale-down-candidates-pool-min-count|50|Minimum number of nodes that are considered as additional non empty candidates for scale down when some candidates from previous iteration are no longer valid|
|
||||
|node-deletion-delay-timeout|"2m"|Maximum time CA waits for removing delay-deletion.cluster-autoscaler.kubernetes.io/ annotations before deleting the node|
|
||||
|scan-interval|"10s"|How often cluster is reevaluated for scale up or down|
|
||||
|max-nodes-total|0|Maximum number of nodes in all node groups. Cluster autoscaler will not grow the cluster beyond this number|
|
||||
|cores-total|"0:320000"|Minimum and maximum number of cores in cluster, in the format `<min>:<max>.` Cluster autoscaler will not scale the cluster beyond these numbers|
|
||||
|memory-total|"0:6400000"|Minimum and maximum number of gigabytes of memory in cluster, in the format `<min>:<max>.` Cluster autoscaler will not scale the cluster beyond these numbers|
|
||||
cloud-provider|-|Cloud provider type|
|
||||
|max-bulk-soft-taint-count|10|Maximum number of nodes that can be tainted/untainted PreferNoSchedule at the same time. Set to 0 to turn off such tainting|
|
||||
|max-bulk-soft-taint-time|"3s"|Maximum duration of tainting/untainting nodes as PreferNoSchedule at the same time|
|
||||
|max-empty-bulk-delete|10|Maximum number of empty nodes that can be deleted at the same time|
|
||||
|max-graceful-termination-sec|600|Maximum number of seconds CA waits for pod termination when trying to scale down a node|
|
||||
|max-total-unready-percentage|45|Maximum percentage of unready nodes in the cluster. After this is exceeded, CA halts operations|
|
||||
|ok-total-unready-count|3|Number of allowed unready nodes, irrespective of max-total-unready-percentage|
|
||||
|scale-up-from-zero|true|Should CA scale up when there 0 ready nodes|
|
||||
|max-node-provision-time|"15m"|Maximum time CA waits for node to be provisioned|
|
||||
|nodes|-|sets min,max size and other configuration data for a node group in a format accepted by cloud provider. Can be used multiple times. Format: `<min>:<max>:<other...>`|
|
||||
|node-group-auto-discovery|-|One or more definition(s) of node group auto-discovery. A definition is expressed `<name of discoverer>:[<key>[=<value>]]`|
|
||||
|estimator|-|"binpacking"|Type of resource estimator to be used in scale up. Available values: ["binpacking"]|
|
||||
|expander|"random"|Type of node group expander to be used in scale up. Available values: `["random","most-pods","least-waste","price","priority"]`|
|
||||
|ignore-daemonsets-utilization|false|Should CA ignore DaemonSet pods when calculating resource utilization for scaling down|
|
||||
|ignore-mirror-pods-utilization|false|Should CA ignore Mirror pods when calculating resource utilization for scaling down|
|
||||
|write-status-configmap|true|Should CA write status information to a configmap|
|
||||
|max-inactivity|"10m"|Maximum time from last recorded autoscaler activity before automatic restart|
|
||||
|max-failing-time|"15m"|Maximum time from last recorded successful autoscaler run before automatic restart|
|
||||
|balance-similar-node-groups|false|Detect similar node groups and balance the number of nodes between them|
|
||||
|node-autoprovisioning-enabled|false|Should CA autoprovision node groups when needed|
|
||||
|max-autoprovisioned-node-group-count|15|The maximum number of autoprovisioned groups in the cluster|
|
||||
|unremovable-node-recheck-timeout|"5m"|The timeout before we check again a node that couldn't be removed before|
|
||||
|expendable-pods-priority-cutoff|-10|Pods with priority below cutoff will be expendable. They can be killed without any consideration during scale down and they don't cause scale up. Pods with null priority (PodPriority disabled) are non expendable|
|
||||
|regional|false|Cluster is regional|
|
||||
|new-pod-scale-up-delay|"0s"|Pods less than this old will not be considered for scale-up|
|
||||
|ignore-taint|-|Specifies a taint to ignore in node templates when considering to scale a node group|
|
||||
|balancing-ignore-label|-|Specifies a label to ignore in addition to the basic and cloud-provider set of labels when comparing if two node groups are similar|
|
||||
|aws-use-static-instance-list|false|Should CA fetch instance types in runtime or use a static list. AWS only|
|
||||
|profiling|false|Is debug/pprof endpoint enabled|
|
||||
|
||||
#### Deployment
|
||||
|
||||
Based on [cluster-autoscaler-run-on-master.yaml](https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/aws/examples/cluster-autoscaler-run-on-master.yaml) example, we've created our own `cluster-autoscaler-deployment.yaml` to use preferred [auto-discovery setup](https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler/cloudprovider/aws#auto-discovery-setup), updating tolerations, nodeSelector, image version and command config:
|
||||
|
||||
|
||||
```yml
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: ServiceAccount
|
||||
metadata:
|
||||
labels:
|
||||
k8s-addon: cluster-autoscaler.addons.k8s.io
|
||||
k8s-app: cluster-autoscaler
|
||||
name: cluster-autoscaler
|
||||
namespace: kube-system
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: ClusterRole
|
||||
metadata:
|
||||
name: cluster-autoscaler
|
||||
labels:
|
||||
k8s-addon: cluster-autoscaler.addons.k8s.io
|
||||
k8s-app: cluster-autoscaler
|
||||
rules:
|
||||
- apiGroups: [""]
|
||||
resources: ["events", "endpoints"]
|
||||
verbs: ["create", "patch"]
|
||||
- apiGroups: [""]
|
||||
resources: ["pods/eviction"]
|
||||
verbs: ["create"]
|
||||
- apiGroups: [""]
|
||||
resources: ["pods/status"]
|
||||
verbs: ["update"]
|
||||
- apiGroups: [""]
|
||||
resources: ["endpoints"]
|
||||
resourceNames: ["cluster-autoscaler"]
|
||||
verbs: ["get", "update"]
|
||||
- apiGroups: [""]
|
||||
resources: ["nodes"]
|
||||
verbs: ["watch", "list", "get", "update"]
|
||||
- apiGroups: [""]
|
||||
resources:
|
||||
- "pods"
|
||||
- "services"
|
||||
- "replicationcontrollers"
|
||||
- "persistentvolumeclaims"
|
||||
- "persistentvolumes"
|
||||
verbs: ["watch", "list", "get"]
|
||||
- apiGroups: ["extensions"]
|
||||
resources: ["replicasets", "daemonsets"]
|
||||
verbs: ["watch", "list", "get"]
|
||||
- apiGroups: ["policy"]
|
||||
resources: ["poddisruptionbudgets"]
|
||||
verbs: ["watch", "list"]
|
||||
- apiGroups: ["apps"]
|
||||
resources: ["statefulsets", "replicasets", "daemonsets"]
|
||||
verbs: ["watch", "list", "get"]
|
||||
- apiGroups: ["storage.k8s.io"]
|
||||
resources: ["storageclasses", "csinodes"]
|
||||
verbs: ["watch", "list", "get"]
|
||||
- apiGroups: ["batch", "extensions"]
|
||||
resources: ["jobs"]
|
||||
verbs: ["get", "list", "watch", "patch"]
|
||||
- apiGroups: ["coordination.k8s.io"]
|
||||
resources: ["leases"]
|
||||
verbs: ["create"]
|
||||
- apiGroups: ["coordination.k8s.io"]
|
||||
resourceNames: ["cluster-autoscaler"]
|
||||
resources: ["leases"]
|
||||
verbs: ["get", "update"]
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: Role
|
||||
metadata:
|
||||
name: cluster-autoscaler
|
||||
namespace: kube-system
|
||||
labels:
|
||||
k8s-addon: cluster-autoscaler.addons.k8s.io
|
||||
k8s-app: cluster-autoscaler
|
||||
rules:
|
||||
- apiGroups: [""]
|
||||
resources: ["configmaps"]
|
||||
verbs: ["create","list","watch"]
|
||||
- apiGroups: [""]
|
||||
resources: ["configmaps"]
|
||||
resourceNames: ["cluster-autoscaler-status", "cluster-autoscaler-priority-expander"]
|
||||
verbs: ["delete", "get", "update", "watch"]
|
||||
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: ClusterRoleBinding
|
||||
metadata:
|
||||
name: cluster-autoscaler
|
||||
labels:
|
||||
k8s-addon: cluster-autoscaler.addons.k8s.io
|
||||
k8s-app: cluster-autoscaler
|
||||
roleRef:
|
||||
apiGroup: rbac.authorization.k8s.io
|
||||
kind: ClusterRole
|
||||
name: cluster-autoscaler
|
||||
subjects:
|
||||
- kind: ServiceAccount
|
||||
name: cluster-autoscaler
|
||||
namespace: kube-system
|
||||
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: RoleBinding
|
||||
metadata:
|
||||
name: cluster-autoscaler
|
||||
namespace: kube-system
|
||||
labels:
|
||||
k8s-addon: cluster-autoscaler.addons.k8s.io
|
||||
k8s-app: cluster-autoscaler
|
||||
roleRef:
|
||||
apiGroup: rbac.authorization.k8s.io
|
||||
kind: Role
|
||||
name: cluster-autoscaler
|
||||
subjects:
|
||||
- kind: ServiceAccount
|
||||
name: cluster-autoscaler
|
||||
namespace: kube-system
|
||||
|
||||
---
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: cluster-autoscaler
|
||||
namespace: kube-system
|
||||
labels:
|
||||
app: cluster-autoscaler
|
||||
spec:
|
||||
replicas: 1
|
||||
selector:
|
||||
matchLabels:
|
||||
app: cluster-autoscaler
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: cluster-autoscaler
|
||||
annotations:
|
||||
prometheus.io/scrape: 'true'
|
||||
prometheus.io/port: '8085'
|
||||
spec:
|
||||
serviceAccountName: cluster-autoscaler
|
||||
tolerations:
|
||||
- effect: NoSchedule
|
||||
operator: "Equal"
|
||||
value: "true"
|
||||
key: node-role.kubernetes.io/controlplane
|
||||
nodeSelector:
|
||||
node-role.kubernetes.io/controlplane: "true"
|
||||
containers:
|
||||
- image: eu.gcr.io/k8s-artifacts-prod/autoscaling/cluster-autoscaler:v1.18.1
|
||||
name: cluster-autoscaler
|
||||
resources:
|
||||
limits:
|
||||
cpu: 100m
|
||||
memory: 300Mi
|
||||
requests:
|
||||
cpu: 100m
|
||||
memory: 300Mi
|
||||
command:
|
||||
- ./cluster-autoscaler
|
||||
- --v=4
|
||||
- --stderrthreshold=info
|
||||
- --cloud-provider=aws
|
||||
- --skip-nodes-with-local-storage=false
|
||||
- --expander=least-waste
|
||||
- --node-group-auto-discovery=asg:tag=k8s.io/cluster-autoscaler/enabled,k8s.io/cluster-autoscaler/<clusterName>
|
||||
volumeMounts:
|
||||
- name: ssl-certs
|
||||
mountPath: /etc/ssl/certs/ca-certificates.crt
|
||||
readOnly: true
|
||||
imagePullPolicy: "Always"
|
||||
volumes:
|
||||
- name: ssl-certs
|
||||
hostPath:
|
||||
path: "/etc/ssl/certs/ca-certificates.crt"
|
||||
|
||||
```
|
||||
|
||||
Once the manifest file is prepared, deploy it in the Kubernetes cluster (Rancher UI can be used instead):
|
||||
|
||||
```sh
|
||||
kubectl -n kube-system apply -f cluster-autoscaler-deployment.yaml
|
||||
```
|
||||
|
||||
**Note:** Cluster-autoscaler deployment can also be set up using [manual configuration](https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler/cloudprovider/aws#manual-configuration)
|
||||
|
||||
# Testing
|
||||
|
||||
At this point, we should have a cluster-scaler up and running in our Rancher custom cluster. Cluster-scale should manage `K8sWorkerAsg` ASG to scale up and down between 2 and 10 nodes, when one of the following conditions is true:
|
||||
|
||||
* There are pods that failed to run in the cluster due to insufficient resources. In this case, the cluster is scaled up.
|
||||
* There are nodes in the cluster that have been underutilized for an extended period of time and their pods can be placed on other existing nodes. In this case, the cluster is scaled down.
|
||||
|
||||
### Generating Load
|
||||
|
||||
We've prepared a `test-deployment.yaml` just to generate load on the Kubernetes cluster and see if cluster-autoscaler is working properly. The test deployment is requesting 1000m CPU and 1024Mi memory by three replicas. Adjust the requested resources and/or replica to be sure you exhaust the Kubernetes cluster resources:
|
||||
|
||||
```yaml
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
labels:
|
||||
app: hello-world
|
||||
name: hello-world
|
||||
spec:
|
||||
replicas: 3
|
||||
selector:
|
||||
matchLabels:
|
||||
app: hello-world
|
||||
strategy:
|
||||
rollingUpdate:
|
||||
maxSurge: 1
|
||||
maxUnavailable: 0
|
||||
type: RollingUpdate
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: hello-world
|
||||
spec:
|
||||
containers:
|
||||
- image: rancher/hello-world
|
||||
imagePullPolicy: Always
|
||||
name: hello-world
|
||||
ports:
|
||||
- containerPort: 80
|
||||
protocol: TCP
|
||||
resources:
|
||||
limits:
|
||||
cpu: 1000m
|
||||
memory: 1024Mi
|
||||
requests:
|
||||
cpu: 1000m
|
||||
memory: 1024Mi
|
||||
```
|
||||
|
||||
Once the test deployment is prepared, deploy it in the Kubernetes cluster default namespace (Rancher UI can be used instead):
|
||||
|
||||
```
|
||||
kubectl -n default apply -f test-deployment.yaml
|
||||
```
|
||||
|
||||
### Checking Scale
|
||||
|
||||
Once the Kubernetes resources got exhausted, cluster-autoscaler should scale up worker nodes where pods failed to be scheduled. It should scale up until up until all pods became scheduled. You should see the new nodes on the ASG and on the Kubernetes cluster. Check the logs on the `kube-system` cluster-autoscaler pod.
|
||||
|
||||
Once scale up is checked, let check for scale down. To do it, reduce the replica number on the test deployment until you release enough Kubernetes cluster resources to scale down. You should see nodes disappear on the ASG and on the Kubernetes cluster. Check the logs on the `kube-system` cluster-autoscaler pod.
|
||||
@@ -0,0 +1,25 @@
|
||||
---
|
||||
title: Cluster Autoscaler
|
||||
weight: 1
|
||||
---
|
||||
|
||||
In this section, you'll learn how to install and use the [Kubernetes cluster-autoscaler](https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/) on Rancher custom clusters using AWS EC2 Auto Scaling Groups.
|
||||
|
||||
The cluster autoscaler is a tool that automatically adjusts the size of the Kubernetes cluster when one of the following conditions is true:
|
||||
|
||||
* There are pods that failed to run in the cluster due to insufficient resources.
|
||||
* There are nodes in the cluster that have been underutilized for an extended period of time and their pods can be placed on other existing nodes.
|
||||
|
||||
To prevent your pod from being evicted, set a `priorityClassName: system-cluster-critical` property on your pod spec.
|
||||
|
||||
Cluster Autoscaler is designed to run on Kubernetes master nodes. It can run in the `kube-system` namespace. Cluster Autoscaler doesn't scale down nodes with non-mirrored `kube-system` pods running on them.
|
||||
|
||||
It's possible to run a customized deployment of Cluster Autoscaler on worker nodes, but extra care needs to be taken to ensure that Cluster Autoscaler remains up and running.
|
||||
|
||||
# Cloud Providers
|
||||
|
||||
Cluster Autoscaler provides support to distinct cloud providers. For more information, go to [cluster-autoscaler supported cloud providers.](https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler#deployment)
|
||||
|
||||
### Setting up Cluster Autoscaler on Amazon Cloud Provider
|
||||
|
||||
For details on running the cluster autoscaler on Amazon cloud provider, refer to [this page.]({{<baseurl>}}/rancher/v2.6/en/cluster-admin/cluster-autoscaler/amazon)
|
||||
+218
@@ -0,0 +1,218 @@
|
||||
---
|
||||
shortTitle: AKS Cluster Configuration
|
||||
title: AKS Cluster Configuration Reference
|
||||
weight: 4
|
||||
---
|
||||
|
||||
# Changes in Rancher v2.6
|
||||
|
||||
- Support for adding more than one node pool
|
||||
- Support for private clusters
|
||||
- Enabled autoscaling node pools
|
||||
- The AKS permissions are now configured in cloud credentials
|
||||
|
||||
# Role-based Access Control
|
||||
|
||||
When provisioning an AKS cluster in the Rancher UI, RBAC cannot be disabled. If role-based access control is disabled for the cluster in AKS, the cluster cannot be registered or imported into Rancher.
|
||||
|
||||
Rancher can configure member roles for AKS clusters in the same way as any other cluster. For more information, see the section on [role-based access control.]({{<baseurl>}}/rancher/v2.6/en/admin-settings/rbac)
|
||||
|
||||
# Cloud Credentials
|
||||
|
||||
> The configuration information in this section assumes you have already set up a service principal for Rancher. For step-by-step instructions for how to set up the service principal, see [this section.]({{<baseurl>}}/rancher/v2.6/en/cluster-provisioning/hosted-kubernetes-clusters/aks/#prerequisites-in-microsoft-azure)
|
||||
|
||||
### Subscription ID
|
||||
|
||||
To get the subscription ID, click **All Services** in the left navigation bar. Then click **Subscriptions**. Go to the name of the subscription that you want to associate with your Kubernetes cluster and copy the **Subscription ID**.
|
||||
|
||||
### Client ID
|
||||
|
||||
To get the client ID, go to the Azure Portal, then click **Azure Active Directory**, then click **App registrations,** then click the name of the service principal. The client ID is listed on the app registration detail page as **Application (client) ID**.
|
||||
|
||||
### Client Secret
|
||||
|
||||
You can't retrieve the client secret value after it is created, so if you don't already have a client secret value, you will need to create a new client secret.
|
||||
|
||||
To get a new client secret, go to the Azure Portal, then click **Azure Active Directory**, then click **App registrations,** then click the name of the service principal.
|
||||
|
||||
Then click **Certificates & secrets** and click **New client secret**. Click **Add**. Then copy the **Value** of the new client secret.
|
||||
|
||||
### Environment
|
||||
|
||||
Microsoft provides multiple [clouds](https://docs.microsoft.com/en-us/cli/azure/cloud?view=azure-cli-latest) for compliance with regional laws, which are available for your use:
|
||||
|
||||
- AzurePublicCloud
|
||||
- AzureGermanCloud
|
||||
- AzureChinaCloud
|
||||
- AzureUSGovernmentCloud
|
||||
|
||||
# Account Access
|
||||
|
||||
In this section you will need to select an existing Azure cloud credential or create a new one.
|
||||
|
||||
For help configuring your Azure cloud credential, see [this section.](#cloud-credentials)
|
||||
|
||||
# Cluster Location
|
||||
|
||||
Configure the cluster and node location. For more information on availability zones for AKS, see the [AKS documentation.](https://docs.microsoft.com/en-us/azure/aks/availability-zones)
|
||||
|
||||
The high availability locations include multiple availability zones.
|
||||
|
||||
# Cluster Options
|
||||
|
||||
### Kubernetes Version
|
||||
|
||||
The available Kubernetes versions are dynamically fetched from the Azure API.
|
||||
|
||||
### Cluster Resource Group
|
||||
|
||||
A resource group is a container that holds related resources for an Azure solution. The resource group can include all the resources for the solution, or only those resources that you want to manage as a group. You decide how you want to allocate resources to resource groups based on what makes the most sense for your organization. Generally, add resources that share the same lifecycle to the same resource group so you can easily deploy, update, and delete them as a group.
|
||||
|
||||
Use an existing resource group or enter a resource group name and one will be created for you.
|
||||
|
||||
Using a resource group containing an existing AKS cluster will create a new resource group. Azure AKS only allows one AKS cluster per resource group.
|
||||
|
||||
For information on managing resource groups, see the [Azure documentation.](https://docs.microsoft.com/en-us/azure/azure-resource-manager/management/manage-resource-groups-portal)
|
||||
|
||||
### Linux Admin Username
|
||||
|
||||
The username used to create an SSH connection to the Linux nodes.
|
||||
|
||||
The default username for AKS nodes is `azureuser`.
|
||||
|
||||
### SSH Public Key
|
||||
|
||||
The key used to create an SSH connection to the Linux nodes.
|
||||
|
||||
### Tags
|
||||
|
||||
Cluster tags can be useful if your organization uses tags as a way to organize resources across multiple Azure services. These tags don't apply to resources within the cluster.
|
||||
|
||||
# Networking Options
|
||||
|
||||
### LoadBalancer SKU
|
||||
|
||||
Azure load balancers support both standard and basic SKUs (stock keeping units).
|
||||
|
||||
For a comparison of standard and basic load balancers, see the official [Azure documentation.](https://docs.microsoft.com/en-us/azure/load-balancer/skus#skus) Microsoft recommends the Standard load balancer.
|
||||
|
||||
The Standard load balancer is required if you have selected one or more availability zones, or if you have more than one node pool.
|
||||
|
||||
### Network Policy
|
||||
|
||||
All pods in an AKS cluster can send and receive traffic without limitations, by default. To improve security, you can define rules that control the flow of traffic. The Network Policy feature in Kubernetes lets you define rules for ingress and egress traffic between pods in a cluster.
|
||||
|
||||
Azure provides two ways to implement network policy. You choose a network policy option when you create an AKS cluster. The policy option can't be changed after the cluster is created:
|
||||
|
||||
- Azure's own implementation, called Azure Network Policies. The Azure network policy requires the Azure CNI.
|
||||
- Calico Network Policies, an open-source network and network security solution founded by [Tigera](https://www.tigera.io/).
|
||||
|
||||
You can also choose to have no network policy.
|
||||
|
||||
For more information about the differences between Azure and Calico network policies and their capabilities, see the [AKS documentation.](https://docs.microsoft.com/en-us/azure/aks/use-network-policies#differences-between-azure-and-calico-policies-and-their-capabilities)
|
||||
|
||||
### DNS Prefix
|
||||
Enter a unique DNS prefix for your cluster's Kubernetes API server FQDN.
|
||||
|
||||
### Network Plugin
|
||||
There are two network plugins: kubenet and Azure CNI.
|
||||
|
||||
The [kubenet](https://kubernetes.io/docs/concepts/cluster-administration/network-plugins/#kubenet) Kubernetes plugin is the default configuration for AKS cluster creation. When kubenet is used, each node in the cluster receives a routable IP address. The pods use NAT to communicate with other resources outside the AKS cluster. This approach reduces the number of IP addresses you need to reserve in your network space for pods to use.
|
||||
|
||||
With the Azure CNI (advanced) networking plugin, pods get full virtual network connectivity and can be directly reached via their private IP address from connected networks. This plugin requires more IP address space.
|
||||
|
||||
For more information on the differences between kubenet and Azure CNI, see the [AKS documentation.](https://docs.microsoft.com/en-us/azure/aks/concepts-network#compare-network-models)
|
||||
|
||||
### HTTP Application Routing
|
||||
|
||||
When enabled, the HTTP application routing add-on makes it easier to access applications deployed to the AKS cluster. It deploys two components: a [Kubernetes Ingress controller](https://kubernetes.io/docs/concepts/services-networking/ingress/) and an [External-DNS](https://github.com/kubernetes-incubator/external-dns) controller.
|
||||
|
||||
For more information, see the [AKS documentation.](https://docs.microsoft.com/en-us/azure/aks/http-application-routing)
|
||||
|
||||
### Set Authorized IP Ranges
|
||||
|
||||
You can secure access to the Kubernetes API server using [authorized IP address ranges.](https://docs.microsoft.com/en-us/azure/aks/api-server-authorized-ip-ranges#overview-of-api-server-authorized-ip-ranges)
|
||||
|
||||
The Kubernetes API server exposes the Kubernetes API. This component provides the interaction for management tools, such as kubectl. AKS provides a single-tenant cluster control plane with a dedicated API server. By default, the API server is assigned a public IP address, and you should control access to it using Kubernetes-based or Azure-based RBAC.
|
||||
|
||||
To secure access to the otherwise publicly accessible AKS control plane and API server, you can enable and use authorized IP ranges. These authorized IP ranges only allow defined IP address ranges to communicate with the API server.
|
||||
|
||||
However, even if you use authorized IP address ranges, you should still use Kubernetes RBAC or Azure RBAC to authorize users and the actions they request.
|
||||
|
||||
### Container Monitoring
|
||||
|
||||
Container monitoring gives you performance visibility by collecting memory and processor metrics from controllers, nodes, and containers that are available in Kubernetes through the Metrics API. Container logs are also collected. After you enable monitoring, metrics and logs are automatically collected for you through a containerized version of the Log Analytics agent for Linux. Metrics are written to the metrics store and log data is written to the logs store associated with your [Log Analytics](https://docs.microsoft.com/en-us/azure/azure-monitor/logs/log-query-overview) workspace.
|
||||
|
||||
### Log Analytics Workspace Resource Group
|
||||
|
||||
The [resource group](https://docs.microsoft.com/en-us/azure/azure-resource-manager/management/overview#resource-groups) containing the Log Analytics Workspace. You must create at least one workspace to use Azure Monitor Logs.
|
||||
|
||||
### Log Analytics Workspace Name
|
||||
|
||||
Data collected by Azure Monitor Logs is stored in one or more [Log Analytics workspaces.](https://docs.microsoft.com/en-us/azure/azure-monitor/logs/design-logs-deployment) The workspace defines the geographic location of the data, access rights defining which users can access data, and configuration settings such as the pricing tier and data retention.
|
||||
|
||||
You must create at least one workspace to use Azure Monitor Logs. A single workspace may be suffxicient for all of your monitoring data, or may choose to create multiple workspaces depending on your requirements. For example, you might have one workspace for your production data and another for testing.
|
||||
|
||||
For more information about Azure Monitor Logs, see the [Azure documentation.](https://docs.microsoft.com/en-us/azure/azure-monitor/logs/data-platform-logs)
|
||||
|
||||
### Support Private Kubernetes Service
|
||||
|
||||
Typically, AKS worker nodes do not get public IPs, regardless of whether the cluster is private. In a private cluster, the control plane does not have a public endpoint.
|
||||
|
||||
Rancher can connect to a private AKS cluster in one of two ways.
|
||||
|
||||
The first way to ensure that Rancher is running on the same [NAT](https://docs.microsoft.com/en-us/azure/virtual-network/nat-overview) as the AKS nodes.
|
||||
|
||||
The second way is to run a command to register the cluster with Rancher. Once the cluster is provisioned, you can run the displayed command anywhere you can connect to the cluster’s Kubernetes API. This command is displayed in a pop-up when you provision an AKS cluster with a private API endpoint enabled.
|
||||
|
||||
> **Note:** Please be aware that when registering an existing AKS cluster, the cluster might take some time, possibly hours, to appear in the `Cluster To register` dropdown list. This outcome will be based on region.
|
||||
|
||||
For more information about connecting to an AKS private cluster, see the [AKS documentation.](https://docs.microsoft.com/en-us/azure/aks/private-clusters#options-for-connecting-to-the-private-cluster)
|
||||
|
||||
# Node Pools
|
||||
|
||||
### Mode
|
||||
|
||||
The Azure interface allows users to specify whether a Primary Node Pool relies on either `system` (normally used for control planes) or `user` (what is most typically needed for Rancher).
|
||||
|
||||
For Primary Node Pools, you can specify Mode, OS, Count and Size.
|
||||
|
||||
System node pools always require running nodes, so they cannot be scaled below one node. At least one system node pool is required.
|
||||
|
||||
For subsequent node pools, the Rancher UI forces the default of User. User node pools allow you to scale to zero nodes. User node pools don't run any part of the Kubernetes controlplane.
|
||||
|
||||
AKS doesn't expose the nodes that run the Kubernetes controlplane components.
|
||||
|
||||
### Availability Zones
|
||||
|
||||
[Availability zones](https://docs.microsoft.com/en-us/azure/availability-zones/az-overview) are unique physical locations within a region. Each zone is made up of one or more data centers equipped with independent power, cooling, and networking.
|
||||
|
||||
Not all regions have support for availability zones. For a list of Azure regions with availability zones, see the [Azure documentation.](https://docs.microsoft.com/en-us/azure/availability-zones/az-region#azure-regions-with-availability-zones)
|
||||
|
||||
### VM Size
|
||||
|
||||
Choose a size for each VM in the node pool. For details about each VM size, see [this page.](https://azure.microsoft.com/en-us/pricing/details/virtual-machines/linux/)
|
||||
|
||||
### OS Disk Type
|
||||
|
||||
The nodes in the node pool can have either managed or ephemeral disks.
|
||||
|
||||
[Ephemeral OS disks](https://docs.microsoft.com/en-us/azure/virtual-machines/ephemeral-os-disks) are created on the local virtual machine storage and not saved to the remote Azure Storage. Ephemeral OS disks work well for stateless workloads, where applications are tolerant of individual VM failures, but are more affected by VM deployment time or reimaging the individual VM instances. With Ephemeral OS disk, you get lower read/write latency to the OS disk and faster VM reimage.
|
||||
|
||||
[Azure managed disks](https://docs.microsoft.com/en-us/azure/virtual-machines/managed-disks-overview) are block-level storage volumes that are managed by Azure and used with Azure Virtual Machines. Managed disks are designed for 99.999% availability. Managed disks achieve this by providing you with three replicas of your data, allowing for high durability.
|
||||
|
||||
### OS Disk Size
|
||||
|
||||
The size in GB for the disk for each node.
|
||||
|
||||
### Node Count
|
||||
The number of nodes in the node pool. The maximum number of nodes may be limited by your [Azure subscription.](https://docs.microsoft.com/en-us/azure/azure-resource-manager/management/azure-subscription-service-limits)
|
||||
|
||||
### Max Pods Per Node
|
||||
The maximum number of pods per node defaults to 110 with a maximum of 250.
|
||||
|
||||
### Enable Auto Scaling
|
||||
|
||||
When auto scaling is enabled, you will need to enter a minimum and maximum node count.
|
||||
|
||||
When Auto Scaling is enabled, you can't manually scale the node pool. The scale is controlled by the AKS autoscaler.
|
||||
@@ -0,0 +1,28 @@
|
||||
---
|
||||
title: Cluster Configuration
|
||||
weight: 2025
|
||||
---
|
||||
|
||||
After you provision a Kubernetes cluster using Rancher, you can still edit options and settings for the cluster.
|
||||
|
||||
For information on editing cluster membership, go to [this page.]({{<baseurl>}}/rancher/v2.6/en/cluster-admin/cluster-access/cluster-members)
|
||||
|
||||
### Cluster Configuration References
|
||||
|
||||
The cluster configuration options depend on the type of Kubernetes cluster:
|
||||
|
||||
- [RKE Cluster Configuration](./rke-config-reference)
|
||||
- [RKE2 Cluster Configuration](./rke2-config-reference)
|
||||
- [K3s Cluster Configuration](./k3s-config-reference)
|
||||
- [EKS Cluster Configuration](./eks-config-reference)
|
||||
- [GKE Cluster Configuration](./gke-config-reference)
|
||||
- [AKS Cluster Configuration](./aks-config-reference)
|
||||
|
||||
### Cluster Management Capabilities by Cluster Type
|
||||
|
||||
The options and settings available for an existing cluster change based on the method that you used to provision it.
|
||||
|
||||
The following table summarizes the options and settings available for each cluster type:
|
||||
|
||||
{{% include file="/rancher/v2.6/en/cluster-provisioning/cluster-capabilities-table</summary>
|
||||
|
||||
+145
@@ -0,0 +1,145 @@
|
||||
---
|
||||
title: EKS Cluster Configuration Reference
|
||||
shortTitle: EKS Cluster Configuration
|
||||
weight: 2
|
||||
---
|
||||
|
||||
### Account Access
|
||||
|
||||
Complete each drop-down and field using the information obtained for your IAM policy.
|
||||
|
||||
| Setting | Description |
|
||||
| ---------- | -------------------------------------------------------------------------------------------------------------------- |
|
||||
| Region | From the drop-down choose the geographical region in which to build your cluster. |
|
||||
| Cloud Credentials | Select the cloud credentials that you created for your IAM policy. For more information on creating cloud credentials in Rancher, refer to [this page.]({{<baseurl>}}/rancher/v2.6/en/user-settings/cloud-credentials/) |
|
||||
|
||||
### Service Role
|
||||
|
||||
Choose a [service role](https://docs.aws.amazon.com/IAM/latest/UserGuide/using-service-linked-roles.html).
|
||||
|
||||
Service Role | Description
|
||||
-------------|---------------------------
|
||||
Standard: Rancher generated service role | If you choose this role, Rancher automatically adds a service role for use with the cluster.
|
||||
Custom: Choose from your existing service roles | If you choose this role, Rancher lets you choose from service roles that you're already created within AWS. For more information on creating a custom service role in AWS, see the [Amazon documentation](https://docs.aws.amazon.com/IAM/latest/UserGuide/using-service-linked-roles.html#create-service-linked-role).
|
||||
|
||||
### Secrets Encryption
|
||||
|
||||
Optional: To encrypt secrets, select or enter a key created in [AWS Key Management Service (KMS)](https://docs.aws.amazon.com/kms/latest/developerguide/overview.html)
|
||||
|
||||
### API Server Endpoint Access
|
||||
|
||||
Configuring Public/Private API access is an advanced use case. For details, refer to the EKS cluster endpoint access control [documentation.](https://docs.aws.amazon.com/eks/latest/userguide/cluster-endpoint.html)
|
||||
|
||||
### Private-only API Endpoints
|
||||
|
||||
If you enable private and disable public API endpoint access when creating a cluster, then there is an extra step you must take in order for Rancher to connect to the cluster successfully. In this case, a pop-up will be displayed with a command that you will run on the cluster to register it with Rancher. Once the cluster is provisioned, you can run the displayed command anywhere you can connect to the cluster's Kubernetes API.
|
||||
|
||||
There are two ways to avoid this extra manual step:
|
||||
- You can create the cluster with both private and public API endpoint access on cluster creation. You can disable public access after the cluster is created and in an active state and Rancher will continue to communicate with the EKS cluster.
|
||||
- You can ensure that Rancher shares a subnet with the EKS cluster. Then security groups can be used to enable Rancher to communicate with the cluster's API endpoint. In this case, the command to register the cluster is not needed, and Rancher will be able to communicate with your cluster. For more information on configuring security groups, refer to the [security groups documentation](https://docs.aws.amazon.com/vpc/latest/userguide/VPC_SecurityGroups.html).
|
||||
|
||||
### Public Access Endpoints
|
||||
|
||||
Optionally limit access to the public endpoint via explicit CIDR blocks.
|
||||
|
||||
If you limit access to specific CIDR blocks, then it is recommended that you also enable the private access to avoid losing network communication to the cluster.
|
||||
|
||||
One of the following is required to enable private access:
|
||||
- Rancher's IP must be part of an allowed CIDR block
|
||||
- Private access should be enabled, and Rancher must share a subnet with the cluster and have network access to the cluster, which can be configured with a security group
|
||||
|
||||
For more information about public and private access to the cluster endpoint, refer to the [Amazon EKS documentation.](https://docs.aws.amazon.com/eks/latest/userguide/cluster-endpoint.html)
|
||||
|
||||
### Subnet
|
||||
|
||||
| Option | Description |
|
||||
| ------- | ------------ |
|
||||
| Standard: Rancher generated VPC and Subnet | While provisioning your cluster, Rancher generates a new VPC with 3 public subnets. |
|
||||
| Custom: Choose from your existing VPC and Subnets | While provisioning your cluster, Rancher configures your Control Plane and nodes to use a VPC and Subnet that you've already [created in AWS](https://docs.aws.amazon.com/vpc/latest/userguide/what-is-amazon-vpc.html). |
|
||||
|
||||
For more information, refer to the AWS documentation for [Cluster VPC Considerations](https://docs.aws.amazon.com/eks/latest/userguide/network_reqs.html). Follow one of the sets of instructions below based on your selection from the previous step.
|
||||
|
||||
- [What Is Amazon VPC?](https://docs.aws.amazon.com/vpc/latest/userguide/what-is-amazon-vpc.html)
|
||||
- [VPCs and Subnets](https://docs.aws.amazon.com/vpc/latest/userguide/VPC_Subnets.html)
|
||||
|
||||
### Security Group
|
||||
|
||||
Amazon Documentation:
|
||||
|
||||
- [Cluster Security Group Considerations](https://docs.aws.amazon.com/eks/latest/userguide/sec-group-reqs.html)
|
||||
- [Security Groups for Your VPC](https://docs.aws.amazon.com/vpc/latest/userguide/VPC_SecurityGroups.html)
|
||||
- [Create a Security Group](https://docs.aws.amazon.com/vpc/latest/userguide/getting-started-ipv4.html#getting-started-create-security-group)
|
||||
|
||||
### Logging
|
||||
|
||||
Configure control plane logs to send to Amazon CloudWatch. You are charged the standard CloudWatch Logs data ingestion and storage costs for any logs sent to CloudWatch Logs from your clusters.
|
||||
|
||||
Each log type corresponds to a component of the Kubernetes control plane. To learn more about these components, see [Kubernetes Components](https://kubernetes.io/docs/concepts/overview/components/) in the Kubernetes documentation.
|
||||
|
||||
For more information on EKS control plane logging, refer to the official [documentation.](https://docs.aws.amazon.com/eks/latest/userguide/control-plane-logs.html)
|
||||
|
||||
### Managed Node Groups
|
||||
|
||||
Amazon EKS managed node groups automate the provisioning and lifecycle management of nodes (Amazon EC2 instances) for Amazon EKS Kubernetes clusters.
|
||||
|
||||
For more information about how node groups work and how they are configured, refer to the [EKS documentation.](https://docs.aws.amazon.com/eks/latest/userguide/managed-node-groups.html)
|
||||
|
||||
#### Bring your own launch template
|
||||
|
||||
A launch template ID and version can be provided in order to easily configure the EC2 instances in a node group. If a launch template is provided, then none of the settings below will be configurable in Rancher. Therefore, using a launch template would require that all the necessary and desired settings from the list below would need to be specified in the launch template. Also note that if a launch template ID and version is provided, then only the template version can be updated. Using a new template ID would require creating a new managed node group.
|
||||
|
||||
| Option | Description | Required/Optional |
|
||||
| ------ | ----------- | ----------------- |
|
||||
| Instance Type | Choose the [hardware specs](https://aws.amazon.com/ec2/instance-types/) for the instance you're provisioning. | Required |
|
||||
| Image ID | Specify a custom AMI for the nodes. Custom AMIs used with EKS must be [configured properly](https://aws.amazon.com/premiumsupport/knowledge-center/eks-custom-linux-ami/) | Optional |
|
||||
| Node Volume Size | The launch template must specify an EBS volume with the desired size | Required |
|
||||
| SSH Key | A key to be added to the instances to provide SSH access to the nodes | Optional |
|
||||
| User Data | Cloud init script in [MIME multi-part format](https://docs.aws.amazon.com/eks/latest/userguide/launch-templates.html#launch-template-user-data) | Optional |
|
||||
| Instance Resource Tags | Tag each EC2 instance in the node group | Optional |
|
||||
|
||||
#### Rancher-managed launch templates
|
||||
|
||||
If you do not specify a launch template, then you will be able to configure the above options in the Rancher UI and all of them can be updated after creation. In order to take advantage of all of these options, Rancher will create and manage a launch template for you. Each cluster in Rancher will have one Rancher-managed launch template and each managed node group that does not have a specified launch template will have one version of the managed launch template. The name of this launch template will have the prefix "rancher-managed-lt-" followed by the display name of the cluster. In addition, the Rancher-managed launch template will be tagged with the key "rancher-managed-template" and value "do-not-modify-or-delete" to help identify it as Rancher-managed. It is important that this launch template and its versions not be modified, deleted, or used with any other clusters or managed node groups. Doing so could result in your node groups being "degraded" and needing to be destroyed and recreated.
|
||||
|
||||
#### Custom AMIs
|
||||
|
||||
If you specify a custom AMI, whether in a launch template or in Rancher, then the image must be [configured properly](https://aws.amazon.com/premiumsupport/knowledge-center/eks-custom-linux-ami/) and you must provide user data to [bootstrap the node](https://docs.aws.amazon.com/eks/latest/userguide/launch-templates.html#launch-template-custom-ami). This is considered an advanced use case and understanding the requirements is imperative.
|
||||
|
||||
If you specify a launch template that does not contain a custom AMI, then Amazon will use the [EKS-optimized AMI](https://docs.aws.amazon.com/eks/latest/userguide/eks-optimized-ami.html) for the Kubernetes version and selected region. You can also select a [GPU enabled instance](https://docs.aws.amazon.com/eks/latest/userguide/eks-optimized-ami.html#gpu-ami) for workloads that would benefit from it.
|
||||
|
||||
>**Note**
|
||||
>The GPU enabled instance setting in Rancher is ignored if a custom AMI is provided, either in the dropdown or in a launch template.
|
||||
|
||||
#### Spot instances
|
||||
|
||||
Spot instances are now [supported by EKS](https://docs.aws.amazon.com/eks/latest/userguide/managed-node-groups.html#managed-node-group-capacity-types-spot). If a launch template is specified, Amazon recommends that the template not provide an instance type. Instead, Amazon recommends providing multiple instance types. If the "Request Spot Instances" checkbox is enabled for a node group, then you will have the opportunity to provide multiple instance types.
|
||||
|
||||
>**Note**
|
||||
>Any selection you made in the instance type dropdown will be ignored in this situation and you must specify at least one instance type to the "Spot Instance Types" section. Furthermore, a launch template used with EKS cannot request spot instances. Requesting spot instances must be part of the EKS configuration.
|
||||
|
||||
#### Node Group Settings
|
||||
|
||||
The following settings are also configurable. All of these except for the "Node Group Name" are editable after the node group is created.
|
||||
|
||||
| Option | Description |
|
||||
| ------- | ------------ |
|
||||
| Node Group Name | The name of the node group. |
|
||||
| Desired ASG Size | The desired number of instances. |
|
||||
| Maximum ASG Size | The maximum number of instances. This setting won't take effect until the [Cluster Autoscaler](https://docs.aws.amazon.com/eks/latest/userguide/cluster-autoscaler.html) is installed. |
|
||||
| Minimum ASG Size | The minimum number of instances. This setting won't take effect until the [Cluster Autoscaler](https://docs.aws.amazon.com/eks/latest/userguide/cluster-autoscaler.html) is installed. |
|
||||
| Labels | Kubernetes labels applied to the nodes in the managed node group. |
|
||||
| Tags | These are tags for the managed node group and do not propagate to any of the associated resources. |
|
||||
|
||||
|
||||
### Configuring the Refresh Interval
|
||||
|
||||
The `eks-refresh-cron` setting is deprecated. It has been migrated to the `eks-refresh` setting, which is an integer representing seconds.
|
||||
|
||||
The default value is 300 seconds.
|
||||
|
||||
The syncing interval can be changed by running `kubectl edit setting eks-refresh`.
|
||||
|
||||
If the `eks-refresh-cron` setting was previously set, the migration will happen automatically.
|
||||
|
||||
The shorter the refresh window, the less likely any race conditions will occur, but it does increase the likelihood of encountering request limits that may be in place for AWS APIs.
|
||||
|
||||
+307
@@ -0,0 +1,307 @@
|
||||
---
|
||||
title: GKE Cluster Configuration Reference
|
||||
shortTitle: GKE Cluster Configuration
|
||||
weight: 3
|
||||
---
|
||||
|
||||
# Changes in Rancher v2.6
|
||||
|
||||
- Support for additional configuration options:
|
||||
- Project network isolation
|
||||
- Network tags
|
||||
|
||||
# Cluster Location
|
||||
|
||||
| Value | Description |
|
||||
|--------|--------------|
|
||||
| Location Type | Zonal or Regional. With GKE, you can create a cluster tailored to the availability requirements of your workload and your budget. By default, a cluster's nodes run in a single compute zone. When multiple zones are selected, the cluster's nodes will span multiple compute zones, while the controlplane is located in a single zone. Regional clusters increase the availability of the controlplane as well. For help choosing the type of cluster availability, refer to [these docs.](https://cloud.google.com/kubernetes-engine/docs/best-practices/scalability#choosing_a_regional_or_zonal_control_plane) |
|
||||
| Zone | Each region in Compute engine contains a number of zones. For more information about available regions and zones, refer to [these docs.](https://cloud.google.com/compute/docs/regions-zones#available) |
|
||||
| Additional Zones | For zonal clusters, you can select additional zones to create a [multi-zone cluster.](https://cloud.google.com/kubernetes-engine/docs/concepts/types-of-clusters#multi-zonal_clusters) |
|
||||
| Region | For [regional clusters,](https://cloud.google.com/kubernetes-engine/docs/concepts/types-of-clusters#regional_clusters) you can select a region. For more information about available regions and zones, refer to [this section](https://cloud.google.com/compute/docs/regions-zones#available). The first part of each zone name is the name of the region. |
|
||||
|
||||
# Cluster Options
|
||||
|
||||
### Kubernetes Version
|
||||
|
||||
_Mutable: yes_
|
||||
|
||||
For more information on GKE Kubernetes versions, refer to [these docs.](https://cloud.google.com/kubernetes-engine/versioning)
|
||||
|
||||
### Container Address Range
|
||||
|
||||
_Mutable: no_
|
||||
|
||||
The IP address range for pods in the cluster. Must be a valid CIDR range, e.g. 10.42.0.0/16. If not specified, a random range is automatically chosen from 10.0.0.0/8 and will exclude ranges already allocated to VMs, other clusters, or routes. Automatically chosen ranges may conflict with reserved IP addresses, dynamic routes, or routes within VPCs peering with the cluster.
|
||||
|
||||
### Network
|
||||
|
||||
_Mutable: no_
|
||||
|
||||
The Compute Engine Network that the cluster connects to. Routes and firewalls will be created using this network. If using [Shared VPCs](https://cloud.google.com/vpc/docs/shared-vpc), the VPC networks that are shared to your project will appear here. will be available to select in this field. For more information, refer to [this page](https://cloud.google.com/vpc/docs/vpc#vpc_networks_and_subnets).
|
||||
|
||||
### Node Subnet / Subnet
|
||||
|
||||
_Mutable: no_
|
||||
|
||||
The Compute Engine subnetwork that the cluster connects to. This subnetwork must belong to the network specified in the **Network** field. Select an existing subnetwork, or select "Auto Create Subnetwork" to have one automatically created. If not using an existing network, **Subnetwork Name** is required to generate one. If using [Shared VPCs](https://cloud.google.com/vpc/docs/shared-vpc), the VPC subnets that are shared to your project will appear here. If using a Shared VPC network, you cannot select "Auto Create Subnetwork". For more information, refer to [this page.](https://cloud.google.com/vpc/docs/vpc#vpc_networks_and_subnets)
|
||||
|
||||
### Subnetwork Name
|
||||
|
||||
_Mutable: no_
|
||||
|
||||
Automatically create a subnetwork with the provided name. Required if "Auto Create Subnetwork" is selected for **Node Subnet** or **Subnet**. For more information on subnetworks, refer to [this page.](https://cloud.google.com/vpc/docs/vpc#vpc_networks_and_subnets)
|
||||
|
||||
### Ip Aliases
|
||||
|
||||
_Mutable: no_
|
||||
|
||||
Enable [alias IPs](https://cloud.google.com/vpc/docs/alias-ip). This enables VPC-native traffic routing. Required if using [Shared VPCs](https://cloud.google.com/vpc/docs/shared-vpc).
|
||||
|
||||
### Network Policy
|
||||
|
||||
_Mutable: yes_
|
||||
|
||||
Enable network policy enforcement on the cluster. A network policy defines the level of communication that can occur between pods and services in the cluster. For more information, refer to [this page.](https://cloud.google.com/kubernetes-engine/docs/how-to/network-policy)
|
||||
|
||||
### Project Network Isolation
|
||||
|
||||
_Mutable: yes_
|
||||
|
||||
choose whether to enable or disable inter-project communication. Note that enabling Project Network Isolation will automatically enable Network Policy and Network Policy Config, but not vice versa.
|
||||
|
||||
### Node Ipv4 CIDR Block
|
||||
|
||||
_Mutable: no_
|
||||
|
||||
The IP address range of the instance IPs in this cluster. Can be set if "Auto Create Subnetwork" is selected for **Node Subnet** or **Subnet**. Must be a valid CIDR range, e.g. 10.96.0.0/14. For more information on how to determine the IP address range, refer to [this page.](https://cloud.google.com/kubernetes-engine/docs/concepts/alias-ips#cluster_sizing)
|
||||
|
||||
### Cluster Secondary Range Name
|
||||
|
||||
_Mutable: no_
|
||||
|
||||
The name of an existing secondary range for Pod IP addresses. If selected, **Cluster Pod Address Range** will automatically be populated. Required if using a Shared VPC network.
|
||||
|
||||
### Cluster Pod Address Range
|
||||
|
||||
_Mutable: no_
|
||||
|
||||
The IP address range assigned to pods in the cluster. Must be a valid CIDR range, e.g. 10.96.0.0/11. If not provided, will be created automatically. Must be provided if using a Shared VPC network. For more information on how to determine the IP address range for your pods, refer to [this section.](https://cloud.google.com/kubernetes-engine/docs/concepts/alias-ips#cluster_sizing_secondary_range_pods)
|
||||
|
||||
### Services Secondary Range Name
|
||||
|
||||
_Mutable: no_
|
||||
|
||||
The name of an existing secondary range for service IP addresses. If selected, **Service Address Range** will be automatically populated. Required if using a Shared VPC network.
|
||||
|
||||
### Service Address Range
|
||||
|
||||
_Mutable: no_
|
||||
|
||||
The address range assigned to the services in the cluster. Must be a valid CIDR range, e.g. 10.94.0.0/18. If not provided, will be created automatically. Must be provided if using a Shared VPC network. For more information on how to determine the IP address range for your services, refer to [this section.](https://cloud.google.com/kubernetes-engine/docs/concepts/alias-ips#cluster_sizing_secondary_range_svcs)
|
||||
|
||||
### Private Cluster
|
||||
|
||||
_Mutable: no_
|
||||
|
||||
> Warning: private clusters require additional planning and configuration outside of Rancher. Refer to the [private cluster guide]({{< baseurl >}}/rancher/v2.6/en/cluster-admin/editing-clusters/gke-config-reference/private-clusters/).
|
||||
|
||||
Assign nodes only internal IP addresses. Private cluster nodes cannot access the public internet unless additional networking steps are taken in GCP.
|
||||
|
||||
### Enable Private Endpoint
|
||||
|
||||
> Warning: private clusters require additional planning and configuration outside of Rancher. Refer to the [private cluster guide]({{< baseurl >}}/rancher/v2.6/en/cluster-admin/editing-clusters/gke-config-reference/private-clusters/).
|
||||
|
||||
_Mutable: no_
|
||||
|
||||
Locks down external access to the control plane endpoint. Only available if **Private Cluster** is also selected. If selected, and if Rancher does not have direct access to the Virtual Private Cloud network the cluster is running in, Rancher will provide a registration command to run on the cluster to enable Rancher to connect to it.
|
||||
|
||||
### Master IPV4 CIDR Block
|
||||
|
||||
_Mutable: no_
|
||||
|
||||
The IP range for the control plane VPC.
|
||||
|
||||
### Master Authorized Network
|
||||
|
||||
_Mutable: yes_
|
||||
|
||||
Enable control plane authorized networks to block untrusted non-GCP source IPs from accessing the Kubernetes master through HTTPS. If selected, additional authorized networks may be added. If the cluster is created with a public endpoint, this option is useful for locking down access to the public endpoint to only certain networks, such as the network where your Rancher service is running. If the cluster only has a private endpoint, this setting is required.
|
||||
|
||||
# Additional Options
|
||||
|
||||
### Cluster Addons
|
||||
|
||||
Additional Kubernetes cluster components. For more information, refer to [this page.](https://cloud.google.com/kubernetes-engine/docs/reference/rest/v1/projects.locations.clusters#Cluster.AddonsConfig)
|
||||
|
||||
#### Horizontal Pod Autoscaling
|
||||
|
||||
_Mutable: yes_
|
||||
|
||||
The Horizontal Pod Autoscaler changes the shape of your Kubernetes workload by automatically increasing or decreasing the number of Pods in response to the workload's CPU or memory consumption, or in response to custom metrics reported from within Kubernetes or external metrics from sources outside of your cluster. For more information, see [this page.](https://cloud.google.com/kubernetes-engine/docs/concepts/horizontalpodautoscaler)
|
||||
|
||||
#### HTTP (L7) Load Balancing
|
||||
|
||||
_Mutable: yes_
|
||||
|
||||
HTTP (L7) Load Balancing distributes HTTP and HTTPS traffic to backends hosted on GKE. For more information, refer to [this page.](https://cloud.google.com/kubernetes-engine/docs/tutorials/http-balancer)
|
||||
|
||||
#### Network Policy Config (master only)
|
||||
|
||||
_Mutable: yes_
|
||||
|
||||
Configuration for NetworkPolicy. This only tracks whether the addon is enabled or not on the master, it does not track whether network policy is enabled for the nodes.
|
||||
|
||||
### Cluster Features (Alpha Features)
|
||||
|
||||
_Mutable: no_
|
||||
|
||||
Turns on all Kubernetes alpha API groups and features for the cluster. When enabled, the cluster cannot be upgraded and will be deleted automatically after 30 days. Alpha clusters are not recommended for production use as they are not covered by the GKE SLA. For more information, refer to [this page.](https://cloud.google.com/kubernetes-engine/docs/concepts/alpha-clusters)
|
||||
|
||||
### Logging Service
|
||||
|
||||
_Mutable: yes_
|
||||
|
||||
The logging service the cluster uses to write logs. Use either [Cloud Logging](https://cloud.google.com/logging) or no logging service in which case no logs are exported from the cluster.
|
||||
|
||||
### Monitoring Service
|
||||
|
||||
_Mutable: yes_
|
||||
|
||||
The monitoring service the cluster uses to write metrics. Use either [Cloud Monitoring](https://cloud.google.com/monitoring) or monitoring service in which case no metrics are exported from the cluster.
|
||||
|
||||
|
||||
### Maintenance Window
|
||||
|
||||
_Mutable: yes_
|
||||
|
||||
Set the start time for a 4 hour maintenance window. The time is specified in the UTC time zone using the HH:MM format. For more information, refer to [this page.](https://cloud.google.com/kubernetes-engine/docs/concepts/maintenance-windows-and-exclusions)
|
||||
|
||||
# Node Pools
|
||||
|
||||
In this section, enter details describing the configuration of each node in the node pool.
|
||||
|
||||
### Kubernetes Version
|
||||
|
||||
_Mutable: yes_
|
||||
|
||||
The Kubernetes version for each node in the node pool. For more information on GKE Kubernetes versions, refer to [these docs.](https://cloud.google.com/kubernetes-engine/versioning)
|
||||
|
||||
### Image Type
|
||||
|
||||
_Mutable: yes_
|
||||
|
||||
The node operating system image. For more information for the node image options that GKE offers for each OS, refer to [this page.](https://cloud.google.com/kubernetes-engine/docs/concepts/node-images#available_node_images)
|
||||
|
||||
> Note: the default option is "Container-Optimized OS with Docker". The read-only filesystem on GCP's Container-Optimized OS is not compatible with the [legacy logging]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/tools/logging) implementation in Rancher. If you need to use the legacy logging feature, select "Ubuntu with Docker" or "Ubuntu with Containerd". The [current logging feature]({{<baseurl>}}/rancher/v2.6/en/logging) is compatible with the Container-Optimized OS image.
|
||||
|
||||
> Note: if selecting "Windows Long Term Service Channel" or "Windows Semi-Annual Channel" for the node pool image type, you must also add at least one Container-Optimized OS or Ubuntu node pool.
|
||||
|
||||
### Machine Type
|
||||
|
||||
_Mutable: no_
|
||||
|
||||
The virtualized hardware resources available to node instances. For more information on Google Cloud machine types, refer to [this page.](https://cloud.google.com/compute/docs/machine-types#machine_types)
|
||||
|
||||
### Root Disk Type
|
||||
|
||||
_Mutable: no_
|
||||
|
||||
Standard persistent disks are backed by standard hard disk drives (HDD), while SSD persistent disks are backed by solid state drives (SSD). For more information, refer to [this section.](https://cloud.google.com/compute/docs/disks)
|
||||
|
||||
### Local SSD Disks
|
||||
|
||||
_Mutable: no_
|
||||
|
||||
Configure each node's local SSD disk storage in GB. Local SSDs are physically attached to the server that hosts your VM instance. Local SSDs have higher throughput and lower latency than standard persistent disks or SSD persistent disks. The data that you store on a local SSD persists only until the instance is stopped or deleted. For more information, see [this section.](https://cloud.google.com/compute/docs/disks#localssds)
|
||||
|
||||
### Preemptible nodes (beta)
|
||||
|
||||
_Mutable: no_
|
||||
|
||||
Preemptible nodes, also called preemptible VMs, are Compute Engine VM instances that last a maximum of 24 hours in general, and provide no availability guarantees. For more information, see [this page.](https://cloud.google.com/kubernetes-engine/docs/how-to/preemptible-vms)
|
||||
|
||||
### Taints
|
||||
|
||||
_Mutable: no_
|
||||
|
||||
When you apply a taint to a node, only Pods that tolerate the taint are allowed to run on the node. In a GKE cluster, you can apply a taint to a node pool, which applies the taint to all nodes in the pool.
|
||||
|
||||
### Node Labels
|
||||
|
||||
_Mutable: no_
|
||||
|
||||
You can apply labels to the node pool, which applies the labels to all nodes in the pool.
|
||||
|
||||
Invalid labels can prevent upgrades or can prevent Rancher from starting. For details on label syntax requirements, see the [Kubernetes documentation.](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#syntax-and-character-set)
|
||||
|
||||
### Network Tags
|
||||
|
||||
_Mutable: no_
|
||||
|
||||
You can add network tags to the node pool to make firewall rules and routes between subnets. Tags will apply to all nodes in the pool.
|
||||
|
||||
For details on tag syntax and requirements, see the [Kubernetes documentation](https://cloud.google.com/vpc/docs/add-remove-network-tags).
|
||||
|
||||
# Group Details
|
||||
|
||||
In this section, enter details describing the node pool.
|
||||
|
||||
### Name
|
||||
|
||||
_Mutable: no_
|
||||
|
||||
Enter a name for the node pool.
|
||||
|
||||
### Initial Node Count
|
||||
|
||||
_Mutable: yes_
|
||||
|
||||
Integer for the starting number of nodes in the node pool.
|
||||
|
||||
### Max Pod Per Node
|
||||
|
||||
_Mutable: no_
|
||||
|
||||
GKE has a hard limit of 110 Pods per node. For more information on the Kubernetes limits, see [this section.](https://cloud.google.com/kubernetes-engine/docs/best-practices/scalability#dimension_limits)
|
||||
|
||||
### Autoscaling
|
||||
|
||||
_Mutable: yes_
|
||||
|
||||
Node pool autoscaling dynamically creates or deletes nodes based on the demands of your workload. For more information, see [this page.](https://cloud.google.com/kubernetes-engine/docs/concepts/cluster-autoscaler)
|
||||
|
||||
### Auto Repair
|
||||
|
||||
_Mutable: yes_
|
||||
|
||||
GKE's node auto-repair feature helps you keep the nodes in your cluster in a healthy, running state. When enabled, GKE makes periodic checks on the health state of each node in your cluster. If a node fails consecutive health checks over an extended time period, GKE initiates a repair process for that node. For more information, see the section on [auto-repairing nodes.](https://cloud.google.com/kubernetes-engine/docs/how-to/node-auto-repair)
|
||||
|
||||
### Auto Upgrade
|
||||
|
||||
_Mutable: yes_
|
||||
|
||||
When enabled, the auto-upgrade feature keeps the nodes in your cluster up-to-date with the cluster control plane (master) version when your control plane is [updated on your behalf.](https://cloud.google.com/kubernetes-engine/upgrades#automatic_cp_upgrades) For more information about auto-upgrading nodes, see [this page.](https://cloud.google.com/kubernetes-engine/docs/how-to/node-auto-upgrades)
|
||||
|
||||
### Access Scopes
|
||||
|
||||
_Mutable: no_
|
||||
|
||||
Access scopes are the legacy method of specifying permissions for your nodes.
|
||||
|
||||
- **Allow default access:** The default access for new clusters is the [Compute Engine default service account.](https://cloud.google.com/compute/docs/access/service-accounts?hl=en_US#default_service_account)
|
||||
- **Allow full access to all Cloud APIs:** Generally, you can just set the cloud-platform access scope to allow full access to all Cloud APIs, then grant the service account only relevant IAM roles. The combination of access scopes granted to the virtual machine instance and the IAM roles granted to the service account determines the amount of access the service account has for that instance.
|
||||
- **Set access for each API:** Alternatively, you can choose to set specific scopes that permit access to the particular API methods that the service will call.
|
||||
|
||||
For more information, see the [section about enabling service accounts for a VM.](https://cloud.google.com/compute/docs/access/create-enable-service-accounts-for-instances)
|
||||
|
||||
|
||||
### Configuring the Refresh Interval
|
||||
|
||||
The refresh interval can be configured through the setting "gke-refresh", which is an integer representing seconds.
|
||||
|
||||
The default value is 300 seconds.
|
||||
|
||||
The syncing interval can be changed by running `kubectl edit setting gke-refresh`.
|
||||
|
||||
The shorter the refresh window, the less likely any race conditions will occur, but it does increase the likelihood of encountering request limits that may be in place for GCP APIs.
|
||||
|
||||
+42
@@ -0,0 +1,42 @@
|
||||
---
|
||||
title: Private Clusters
|
||||
weight: 2
|
||||
---
|
||||
|
||||
In GKE, [private clusters](https://cloud.google.com/kubernetes-engine/docs/concepts/private-cluster-concept) are clusters whose nodes are isolated from inbound and outbound traffic by assigning them internal IP addresses only. Private clusters in GKE have the option of exposing the control plane endpoint as a publicly accessible address or as a private address. This is different from other Kubernetes providers, which may refer to clusters with private control plane endpoints as "private clusters" but still allow traffic to and from nodes. You may want to create a cluster with private nodes, with or without a public control plane endpoint, depending on your organization's networking and security requirements. A GKE cluster provisioned from Rancher can use isolated nodes by selecting "Private Cluster" in the Cluster Options (under "Show advanced options"). The control plane endpoint can optionally be made private by selecting "Enable Private Endpoint".
|
||||
|
||||
### Private Nodes
|
||||
|
||||
Because the nodes in a private cluster only have internal IP addresses, they will not be able to install the cluster agent and Rancher will not be able to fully manage the cluster. This can be overcome in a few ways.
|
||||
|
||||
#### Cloud NAT
|
||||
|
||||
>**Note**
|
||||
>Cloud NAT will [incur charges](https://cloud.google.com/nat/pricing).
|
||||
|
||||
If restricting outgoing internet access is not a concern for your organization, use Google's [Cloud NAT](https://cloud.google.com/nat/docs/using-nat) service to allow nodes in the private network to access the internet, enabling them to download the required images from Dockerhub and contact the Rancher management server. This is the simplest solution.
|
||||
|
||||
#### Private registry
|
||||
|
||||
>**Note**
|
||||
>This scenario is not officially supported, but is described for cases in which using the Cloud NAT service is not sufficient.
|
||||
|
||||
If restricting both incoming and outgoing traffic to nodes is a requirement, follow the air-gapped installation instructions to set up a private container image [registry](https://rancher.com/docs/rancher/v2.6/en/installation/other-installation-methods/air-gap/) on the VPC where the cluster is going to be, allowing the cluster nodes to access and download the images they need to run the cluster agent. If the control plane endpoint is also private, Rancher will need [direct access](#direct-access) to it.
|
||||
|
||||
### Private Control Plane Endpoint
|
||||
|
||||
If the cluster has a public endpoint exposed, Rancher will be able to reach the cluster, and no additional steps need to be taken. However, if the cluster has no public endpoint, then considerations must be made to ensure Rancher can access the cluster.
|
||||
|
||||
#### Cloud NAT
|
||||
|
||||
>**Note**
|
||||
>Cloud NAT will [incur charges](https://cloud.google.com/nat/pricing).
|
||||
|
||||
As above, if restricting outgoing internet access to the nodes is not a concern, then Google's [Cloud NAT](https://cloud.google.com/nat/docs/using-nat) service can be used to allow the nodes to access the internet. While the cluster is provisioning, Rancher will provide a registration command to run on the cluster. Download the [kubeconfig](https://cloud.google.com/kubernetes-engine/docs/how-to/cluster-access-for-kubectl) for the new cluster and run the provided kubectl command on the cluster. Gaining access
|
||||
to the cluster in order to run this command can be done by creating a temporary node or using an existing node in the VPC, or by logging on to or creating an SSH tunnel through one of the cluster nodes.
|
||||
|
||||
#### Direct access
|
||||
|
||||
If the Rancher server is run on the same VPC as the cluster's control plane, it will have direct access to the control plane's private endpoint. The cluster nodes will need to have access to a [private registry](#private-registry) to download images as described above.
|
||||
|
||||
You can also use services from Google such as [Cloud VPN](https://cloud.google.com/network-connectivity/docs/vpn/concepts/overview) or [Cloud Interconnect VLAN](https://cloud.google.com/network-connectivity/docs/interconnect) to facilitate connectivity between your organization's network and your Google VPC.
|
||||
+143
@@ -0,0 +1,143 @@
|
||||
---
|
||||
title: K3s Cluster Configuration Reference
|
||||
shortTitle: K3s Cluster Configuration
|
||||
weight: 6
|
||||
---
|
||||
|
||||
This section covers the configuration options that are available in Rancher for a new or existing K3s Kubernetes cluster.
|
||||
|
||||
# Overview
|
||||
|
||||
You can configure the Kubernetes options one of two ways:
|
||||
|
||||
- [Rancher UI](#configuration-options-in-the-rancher-ui): Use the Rancher UI to select options that are commonly customized when setting up a Kubernetes cluster.
|
||||
- [Cluster Config File](#cluster-config-file): Instead of using the Rancher UI to choose Kubernetes options for the cluster, advanced users can create a K3s config file. Using a config file allows you to set any of the [options](https://rancher.com/docs/k3s/latest/en/installation/install-options/) available in an K3s installation.
|
||||
|
||||
# Configuration Options in the Rancher UI
|
||||
|
||||
> Some advanced configuration options are not exposed in the Rancher UI forms, but they can be enabled by editing the K3s cluster configuration file in YAML. For the complete reference of configurable options for K3s clusters in YAML, see the [K3s documentation.](https://rancher.com/docs/k3s/latest/en/installation/install-options/)
|
||||
|
||||
### Basics
|
||||
#### Kubernetes Version
|
||||
|
||||
The version of Kubernetes installed on your cluster nodes. Rancher packages its own version of Kubernetes based on [hyperkube](https://github.com/rancher/hyperkube).
|
||||
|
||||
For more detail, see [Upgrading Kubernetes]({{<baseurl>}}/rancher/v2.6/en/cluster-admin/upgrading-kubernetes).
|
||||
|
||||
#### Encrypt Secrets
|
||||
|
||||
Option to enable or disable secrets encryption. When enabled, secrets will be encrypted using a AES-CBC key. If disabled, any previously secrets will not be readable until encryption is enabled again. Refer to the [K3s documentation](https://rancher.com/docs/k3s/latest/en/advanced/#secrets-encryption-config-experimental) for details.
|
||||
|
||||
#### Project Network Isolation
|
||||
|
||||
If your network provider allows project network isolation, you can choose whether to enable or disable inter-project communication.
|
||||
|
||||
#### SELinux
|
||||
|
||||
Option to enable or disable [SELinux](https://rancher.com/docs/k3s/latest/en/advanced/#selinux-support) support.
|
||||
|
||||
#### CoreDNS
|
||||
|
||||
By default, [CoreDNS](https://coredns.io/) is installed as the default DNS provider. If CoreDNS is not installed, an alternate DNS provider must be installed yourself. Refer to the [K3s documentation](https://rancher.com/docs/k3s/latest/en/networking/#coredns) for details..
|
||||
|
||||
#### Klipper Service LB
|
||||
|
||||
Option to enable or disable the [Klipper](https://github.com/rancher/klipper-lb) service load balancer. Refer to the [K3s documentation](https://rancher.com/docs/k3s/latest/en/networking/#service-load-balancer) for details.
|
||||
|
||||
#### Traefik Ingress
|
||||
|
||||
Option to enable or disable the [Traefik](https://traefik.io/) HTTP reverse proxy and load balancer. For more details and configuration options, see the [K3s documentation](https://rancher.com/docs/k3s/latest/en/networking/#traefik-ingress-controller).
|
||||
|
||||
#### Local Storage
|
||||
|
||||
Option to enable or disable [local storage](https://rancher.com/docs/k3s/latest/en/storage/) on the node(s).
|
||||
|
||||
#### Metrics Server
|
||||
|
||||
Option to enable or disable the [metrics server](https://github.com/kubernetes-incubator/metrics-server). If enabled, ensure port 10250 is opened for inbound TCP traffic.
|
||||
|
||||
### Add-On Config
|
||||
|
||||
Additional Kubernetes manifests, managed as a [Add-on](https://kubernetes.io/docs/concepts/cluster-administration/addons/), to apply to the cluster on startup. Refer to the [K3s documentation](https://rancher.com/docs/k3s/latest/en/helm/#automatically-deploying-manifests-and-helm-charts) for details.
|
||||
|
||||
### Agent Environment Vars
|
||||
|
||||
Option to set environment variables for [K3s agents](https://rancher.com/docs/k3s/latest/en/architecture/). The environment variables can be set using key value pairs. Refer to the [K3 documentation](https://rancher.com/docs/k3s/latest/en/installation/install-options/agent-config/) for more details.
|
||||
|
||||
### etcd
|
||||
|
||||
#### Automatic Snapshots
|
||||
|
||||
Option to enable or disable recurring etcd snapshots. If enabled, users have the option to configure the frequency of snapshots. For details, refer to the [K3s documentation](https://rancher.com/docs/k3s/latest/en/backup-restore/#creating-snapshots).
|
||||
|
||||
#### Metrics
|
||||
|
||||
Option to choose whether to expose etcd metrics to the public or only within the cluster.
|
||||
|
||||
### Networking
|
||||
|
||||
#### Cluster CIDR
|
||||
|
||||
IPv4/IPv6 network CIDRs to use for pod IPs (default: 10.42.0.0/16).
|
||||
|
||||
#### Service CIDR
|
||||
|
||||
IPv4/IPv6 network CIDRs to use for service IPs (default: 10.43.0.0/16).
|
||||
|
||||
#### Cluster DNS
|
||||
|
||||
IPv4 Cluster IP for coredns service. Should be in your service-cidr range (default: 10.43.0.10).
|
||||
|
||||
#### Cluster Domain
|
||||
|
||||
Select the domain for the cluster. The default is `cluster.local`.
|
||||
|
||||
#### NodePort Service Port Range
|
||||
|
||||
Option to change the range of ports that can be used for [NodePort services](https://kubernetes.io/docs/concepts/services-networking/service/#nodeport). The default is `30000-32767`.
|
||||
|
||||
#### TLS Alternate Names
|
||||
|
||||
Add additional hostnames or IPv4/IPv6 addresses as Subject Alternative Names on the server TLS cert.
|
||||
|
||||
#### Authorized Cluster Endpoint
|
||||
|
||||
Authorized Cluster Endpoint can be used to directly access the Kubernetes API server, without requiring communication through Rancher.
|
||||
|
||||
For more detail on how an authorized cluster endpoint works and why it is used, refer to the [architecture section.]({{<baseurl>}}/rancher/v2.6/en/overview/architecture/#4-authorized-cluster-endpoint)
|
||||
|
||||
We recommend using a load balancer with the authorized cluster endpoint. For details, refer to the [recommended architecture section.]({{<baseurl>}}/rancher/v2.6/en/overview/architecture-recommendations/#architecture-for-an-authorized-cluster-endpoint)
|
||||
|
||||
### Registries
|
||||
|
||||
Select the image repository to pull Rancher images from. For more details and configuration options, see the [K3s documentation](https://rancher.com/docs/k3s/latest/en/installation/private-registry/).
|
||||
|
||||
### Upgrade Strategy
|
||||
|
||||
#### Controle Plane Concurrency
|
||||
|
||||
Select how many nodes can be upgraded at the same time. Can be a fixed number or percentage.
|
||||
|
||||
#### Worker Concurrency
|
||||
|
||||
Select how many nodes can be upgraded at the same time. Can be a fixed number or percentage.
|
||||
|
||||
#### Drain Nodes (Control Plane)
|
||||
|
||||
Option to remove all pods from the node prior to upgrading.
|
||||
|
||||
#### Drain Nodes (Worker Nodes)
|
||||
|
||||
Option to remove all pods from the node prior to upgrading.
|
||||
|
||||
### Advanced
|
||||
|
||||
Option to set kubelet options for different nodes. For available options, refer to the [Kubernetes documentation](https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/).
|
||||
|
||||
# Cluster Config File
|
||||
|
||||
Instead of using the Rancher UI forms to choose Kubernetes options for the cluster, advanced users can create an K3s config file. Using a config file allows you to set any of the [options](https://rancher.com/docs/k3s/latest/en/installation/install-options/) available in an K3s installation.
|
||||
|
||||
To edit an K3s config file directly from the Rancher UI, click **Edit as YAML**.
|
||||
|
||||
|
||||
+360
@@ -0,0 +1,360 @@
|
||||
---
|
||||
title: RKE Cluster Configuration Reference
|
||||
shortTitle: RKE Cluster Configuration
|
||||
weight: 1
|
||||
---
|
||||
|
||||
When Rancher installs Kubernetes, it uses [RKE]({{<baseurl>}}/rancher/v2.6/en/cluster-provisioning/rke-clusters/) or [RKE2](https://docs.rke2.io/) as the Kubernetes distribution.
|
||||
|
||||
This section covers the configuration options that are available in Rancher for a new or existing RKE Kubernetes cluster.
|
||||
|
||||
- [Overview](#overview)
|
||||
- [Editing Clusters with a Form in the Rancher UI](#editing-clusters-with-a-form-in-the-rancher-ui)
|
||||
- [Editing Clusters with YAML](#editing-clusters-with-yaml)
|
||||
- [Configuration Options in the Rancher UI](#configuration-options-in-the-rancher-ui)
|
||||
- [Kubernetes Version](#kubernetes-version)
|
||||
- [Network Provider](#network-provider)
|
||||
- [Project Network Isolation](#project-network-isolation)
|
||||
- [Kubernetes Cloud Providers](#kubernetes-cloud-providers)
|
||||
- [Private Registries](#private-registries)
|
||||
- [Authorized Cluster Endpoint](#authorized-cluster-endpoint)
|
||||
- [Node Pools](#node-pools)
|
||||
- [NGINX Ingress](#nginx-ingress)
|
||||
- [Metrics Server Monitoring](#metrics-server-monitoring)
|
||||
- [Pod Security Policy Support](#pod-security-policy-support)
|
||||
- [Docker Version on Nodes](#docker-version-on-nodes)
|
||||
- [Docker Root Directory](#docker-root-directory)
|
||||
- [Default Pod Security Policy](#default-pod-security-policy)
|
||||
- [Node Port Range](#node-port-range)
|
||||
- [Recurring etcd Snapshots](#recurring-etcd-snapshots)
|
||||
- [Agent Environment Variables](#agent-environment-variables)
|
||||
- [Updating ingress-nginx](#updating-ingress-nginx)
|
||||
- [RKE Cluster Config File Reference](#rke-cluster-config-file-reference)
|
||||
- [Config File Structure in Rancher](#config-file-structure-in-rancher)
|
||||
- [Default DNS Provider](#default-dns-provider)
|
||||
- [Rancher Specific Parameters in YAML](#rancher-specific-parameters-in-yaml)
|
||||
- [docker_root_dir](#docker_root_dir)
|
||||
- [enable_cluster_monitoring](#enable_cluster_monitoring)
|
||||
- [enable_network_policy](#enable_network_policy)
|
||||
- [local_cluster_auth_endpoint](#local_cluster_auth_endpoint)
|
||||
- [Custom Network Plug-in](#custom-network-plug-in)
|
||||
|
||||
# Overview
|
||||
|
||||
You can configure the Kubernetes options one of two ways:
|
||||
|
||||
- [Rancher UI](#rancher-ui-options): Use the Rancher UI to select options that are commonly customized when setting up a Kubernetes cluster.
|
||||
- [Cluster Config File](#cluster-config-file): Instead of using the Rancher UI to choose Kubernetes options for the cluster, advanced users can create an RKE config file. Using a config file allows you to set any of the options available in an RKE installation, except for system_images configuration, by specifying them in YAML.
|
||||
|
||||
The RKE cluster config options are nested under the `rancher_kubernetes_engine_config` directive. For more information, see the section about the [cluster config file.](#cluster-config-file)
|
||||
|
||||
In [clusters launched by RKE]({{<baseurl>}}/rancher/v2.6/en/cluster-provisioning/rke-clusters/), you can edit any of the remaining options that follow.
|
||||
|
||||
For an example of RKE config file syntax, see the [RKE documentation]({{<baseurl>}}/rke/latest/en/example-yamls/).
|
||||
|
||||
The forms in the Rancher UI don't include all advanced options for configuring RKE. For the complete reference of configurable options for RKE Kubernetes clusters in YAML, see the [RKE documentation.]({{<baseurl>}}/rke/latest/en/config-options/)
|
||||
|
||||
# Editing Clusters with a Form in the Rancher UI
|
||||
|
||||
To edit your cluster,
|
||||
|
||||
1. In the upper left corner, click **☰ > Cluster Management**.
|
||||
1. Go to the cluster you want to configure and click **⋮ > Edit Config**.
|
||||
|
||||
|
||||
# Editing Clusters with YAML
|
||||
|
||||
Instead of using the Rancher UI to choose Kubernetes options for the cluster, advanced users can create an RKE config file. Using a config file allows you to set any of the options available in an RKE installation, except for system_images configuration, by specifying them in YAML.
|
||||
|
||||
RKE clusters (also called RKE1 clusters) are edited differently than RKE2 and K3s clusters.
|
||||
|
||||
To edit an RKE config file directly from the Rancher UI,
|
||||
|
||||
1. Click **☰ > Cluster Management**.
|
||||
1. Go to the RKE cluster you want to configure. Click and click **⋮ > Edit Config**. This take you to the RKE configuration form. Note: Because cluster provisioning changed in Rancher 2.6, the **⋮ > Edit as YAML** can be used for configuring RKE2 clusters, but it can't be used for editing RKE1 configuration.
|
||||
1. In the configuration form, scroll down and click **Edit as YAML**.
|
||||
1. Edit the RKE options under the `rancher_kubernetes_engine_config` directive.
|
||||
|
||||
# Configuration Options in the Rancher UI
|
||||
|
||||
> Some advanced configuration options are not exposed in the Rancher UI forms, but they can be enabled by editing the RKE cluster configuration file in YAML. For the complete reference of configurable options for RKE Kubernetes clusters in YAML, see the [RKE documentation.]({{<baseurl>}}/rke/latest/en/config-options/)
|
||||
|
||||
### Kubernetes Version
|
||||
|
||||
The version of Kubernetes installed on your cluster nodes. Rancher packages its own version of Kubernetes based on [hyperkube](https://github.com/rancher/hyperkube).
|
||||
|
||||
For more detail, see [Upgrading Kubernetes]({{<baseurl>}}/rancher/v2.6/en/cluster-admin/upgrading-kubernetes).
|
||||
|
||||
### Network Provider
|
||||
|
||||
The [Network Provider](https://kubernetes.io/docs/concepts/cluster-administration/networking/) that the cluster uses. For more details on the different networking providers, please view our [Networking FAQ]({{<baseurl>}}/rancher/v2.6/en/faq/networking/cni-providers/).
|
||||
|
||||
> After you launch the cluster, you cannot change your network provider. Therefore, choose which network provider you want to use carefully, as Kubernetes doesn't allow switching between network providers. Once a cluster is created with a network provider, changing network providers would require you tear down the entire cluster and all its applications.
|
||||
|
||||
Out of the box, Rancher is compatible with the following network providers:
|
||||
|
||||
- [Canal](https://github.com/projectcalico/canal)
|
||||
- [Flannel](https://github.com/coreos/flannel#flannel)
|
||||
- [Calico](https://docs.projectcalico.org/v3.11/introduction/)
|
||||
- [Weave](https://github.com/weaveworks/weave)
|
||||
|
||||
**Notes on Weave:**
|
||||
|
||||
When Weave is selected as network provider, Rancher will automatically enable encryption by generating a random password. If you want to specify the password manually, please see how to configure your cluster using a [Config File]({{<baseurl>}}/rancher/v2.6/en/cluster-provisioning/rke-clusters/options/#cluster-config-file) and the [Weave Network Plug-in Options]({{<baseurl>}}/rke/latest/en/config-options/add-ons/network-plugins/#weave-network-plug-in-options).
|
||||
|
||||
### Project Network Isolation
|
||||
|
||||
If your network provider allows project network isolation, you can choose whether to enable or disable inter-project communication.
|
||||
|
||||
Project network isolation is available if you are using any RKE network plugin that supports the enforcement of Kubernetes network policies, such as Canal or the Cisco ACI plugin.
|
||||
|
||||
### Kubernetes Cloud Providers
|
||||
|
||||
You can configure a [Kubernetes cloud provider]({{<baseurl>}}/rancher/v2.6/en/cluster-provisioning/rke-clusters/cloud-providers). If you want to use dynamically provisioned [volumes and storage]({{<baseurl>}}/rancher/v2.6/en/cluster-admin/volumes-and-storage/) in Kubernetes, typically you must select the specific cloud provider in order to use it. For example, if you want to use Amazon EBS, you would need to select the `aws` cloud provider.
|
||||
|
||||
>**Note:** If the cloud provider you want to use is not listed as an option, you will need to use the [config file option](#cluster-config-file) to configure the cloud provider. Please reference the [RKE cloud provider documentation]({{<baseurl>}}/rke/latest/en/config-options/cloud-providers/) on how to configure the cloud provider.
|
||||
|
||||
### Private Registries
|
||||
|
||||
The cluster-level private registry configuration is only used for provisioning clusters.
|
||||
|
||||
There are two main ways to set up private registries in Rancher: by setting up the [global default registry]({{<baseurl>}}/rancher/v2.6/en/admin-settings/config-private-registry) through the **Settings** tab in the global view, and by setting up a private registry in the advanced options in the cluster-level settings. The global default registry is intended to be used for air-gapped setups, for registries that do not require credentials. The cluster-level private registry is intended to be used in all setups in which the private registry requires credentials.
|
||||
|
||||
If your private registry requires credentials, you need to pass the credentials to Rancher by editing the cluster options for each cluster that needs to pull images from the registry.
|
||||
|
||||
The private registry configuration option tells Rancher where to pull the [system images]({{<baseurl>}}/rke/latest/en/config-options/system-images/) or [addon images]({{<baseurl>}}/rke/latest/en/config-options/add-ons/) that will be used in your cluster.
|
||||
|
||||
- **System images** are components needed to maintain the Kubernetes cluster.
|
||||
- **Add-ons** are used to deploy several cluster components, including network plug-ins, the ingress controller, the DNS provider, or the metrics server.
|
||||
|
||||
For more information on setting up a private registry for components applied during the provisioning of the cluster, see the [RKE documentation on private registries]({{<baseurl>}}/rke/latest/en/config-options/private-registries/).
|
||||
|
||||
Rancher v2.6 introduced the ability to configure [ECR registries for RKE clusters]({{<baseurl>}}/rke/latest/en/config-options/private-registries/#amazon-elastic-container-registry-ecr-private-registry-setup).
|
||||
|
||||
### Authorized Cluster Endpoint
|
||||
|
||||
Authorized Cluster Endpoint can be used to directly access the Kubernetes API server, without requiring communication through Rancher.
|
||||
|
||||
> The authorized cluster endpoint only works on Rancher-launched Kubernetes clusters. In other words, it only works in clusters where Rancher [used RKE]({{<baseurl>}}/rancher/v2.6/en/overview/architecture/#tools-for-provisioning-kubernetes-clusters) to provision the cluster. It is not available for clusters in a hosted Kubernetes provider, such as Amazon's EKS.
|
||||
|
||||
This is enabled by default in Rancher-launched Kubernetes clusters, using the IP of the node with the `controlplane` role and the default Kubernetes self signed certificates.
|
||||
|
||||
For more detail on how an authorized cluster endpoint works and why it is used, refer to the [architecture section.]({{<baseurl>}}/rancher/v2.6/en/overview/architecture/#4-authorized-cluster-endpoint)
|
||||
|
||||
We recommend using a load balancer with the authorized cluster endpoint. For details, refer to the [recommended architecture section.]({{<baseurl>}}/rancher/v2.6/en/overview/architecture-recommendations/#architecture-for-an-authorized-cluster-endpoint)
|
||||
|
||||
### Node Pools
|
||||
|
||||
For information on using the Rancher UI to set up node pools in an RKE cluster, refer to [this page.]({{<baseurl>}}/rancher/v2.6/en/cluster-provisioning/rke-clusters/node-pools)
|
||||
|
||||
### NGINX Ingress
|
||||
|
||||
If you want to publish your applications in a high-availability configuration, and you're hosting your nodes with a cloud-provider that doesn't have a native load-balancing feature, enable this option to use NGINX Ingress within the cluster.
|
||||
|
||||
### Metrics Server Monitoring
|
||||
|
||||
Option to enable or disable [Metrics Server]({{<baseurl>}}/rke/latest/en/config-options/add-ons/metrics-server/).
|
||||
|
||||
Each cloud provider capable of launching a cluster using RKE can collect metrics and monitor for your cluster nodes. Enable this option to view your node metrics from your cloud provider's portal.
|
||||
|
||||
### Pod Security Policy Support
|
||||
|
||||
Enables [pod security policies]({{<baseurl>}}/rancher/v2.6/en/admin-settings/pod-security-policies/) for the cluster. After enabling this option, choose a policy using the **Default Pod Security Policy** drop-down.
|
||||
|
||||
You must have an existing Pod Security Policy configured before you can use this option.
|
||||
|
||||
### Docker Version on Nodes
|
||||
|
||||
Configures whether nodes are allowed to run versions of Docker that Rancher doesn't officially support.
|
||||
|
||||
If you choose to require a supported Docker version, Rancher will stop pods from running on nodes that don't have a supported Docker version installed.
|
||||
|
||||
For details on which Docker versions were tested with each Rancher version, refer to the [support maintenance terms.](https://rancher.com/support-maintenance-terms/)
|
||||
|
||||
### Docker Root Directory
|
||||
|
||||
If the nodes you are adding to the cluster have Docker configured with a non-default Docker Root Directory (default is `/var/lib/docker`), specify the correct Docker Root Directory in this option.
|
||||
|
||||
### Default Pod Security Policy
|
||||
|
||||
If you enable **Pod Security Policy Support**, use this drop-down to choose the pod security policy that's applied to the cluster.
|
||||
|
||||
### Node Port Range
|
||||
|
||||
Option to change the range of ports that can be used for [NodePort services](https://kubernetes.io/docs/concepts/services-networking/service/#nodeport). Default is `30000-32767`.
|
||||
|
||||
### Recurring etcd Snapshots
|
||||
|
||||
Option to enable or disable [recurring etcd snapshots]({{<baseurl>}}/rke/latest/en/etcd-snapshots/#etcd-recurring-snapshots).
|
||||
|
||||
### Agent Environment Variables
|
||||
|
||||
Option to set environment variables for [rancher agents]({{<baseurl>}}/rancher/v2.6/en/cluster-provisioning/rke-clusters/rancher-agents/). The environment variables can be set using key value pairs. If rancher agent requires use of proxy to communicate with Rancher server, `HTTP_PROXY`, `HTTPS_PROXY` and `NO_PROXY` environment variables can be set using agent environment variables.
|
||||
|
||||
### Updating ingress-nginx
|
||||
|
||||
Clusters that were created before Kubernetes 1.16 will have an `ingress-nginx` `updateStrategy` of `OnDelete`. Clusters that were created with Kubernetes 1.16 or newer will have `RollingUpdate`.
|
||||
|
||||
If the `updateStrategy` of `ingress-nginx` is `OnDelete`, you will need to delete these pods to get the correct version for your deployment.
|
||||
|
||||
|
||||
|
||||
# RKE Cluster Config File Reference
|
||||
|
||||
Instead of using the Rancher UI to choose Kubernetes options for the cluster, advanced users can create an RKE config file. Using a config file allows you to set any of the [options available]({{<baseurl>}}/rke/latest/en/config-options/) in an RKE installation, except for `system_images` configuration. The `system_images` option is not supported when creating a cluster with the Rancher UI or API.
|
||||
|
||||
For the complete reference for configurable options for RKE Kubernetes clusters in YAML, see the [RKE documentation.]({{<baseurl>}}/rke/latest/en/config-options/)
|
||||
|
||||
### Config File Structure in Rancher
|
||||
|
||||
RKE (Rancher Kubernetes Engine) is the tool that Rancher uses to provision Kubernetes clusters. Rancher's cluster config files used to have the same structure as [RKE config files,]({{<baseurl>}}/rke/latest/en/example-yamls/) but the structure changed so that in Rancher, RKE cluster config items are separated from non-RKE config items. Therefore, configuration for your cluster needs to be nested under the `rancher_kubernetes_engine_config` directive in the cluster config file. Cluster config files created with earlier versions of Rancher will need to be updated for this format. An example cluster config file is included below.
|
||||
|
||||
<details id="v2.3.0-cluster-config-file">
|
||||
<summary>Example Cluster Config File</summary>
|
||||
|
||||
```yaml
|
||||
#
|
||||
# Cluster Config
|
||||
#
|
||||
docker_root_dir: /var/lib/docker
|
||||
enable_cluster_alerting: false
|
||||
enable_cluster_monitoring: false
|
||||
enable_network_policy: false
|
||||
local_cluster_auth_endpoint:
|
||||
enabled: true
|
||||
#
|
||||
# Rancher Config
|
||||
#
|
||||
rancher_kubernetes_engine_config: # Your RKE template config goes here.
|
||||
addon_job_timeout: 30
|
||||
authentication:
|
||||
strategy: x509
|
||||
ignore_docker_version: true
|
||||
#
|
||||
# # Currently only nginx ingress provider is supported.
|
||||
# # To disable ingress controller, set `provider: none`
|
||||
# # To enable ingress on specific nodes, use the node_selector, eg:
|
||||
# provider: nginx
|
||||
# node_selector:
|
||||
# app: ingress
|
||||
#
|
||||
ingress:
|
||||
provider: nginx
|
||||
kubernetes_version: v1.15.3-rancher3-1
|
||||
monitoring:
|
||||
provider: metrics-server
|
||||
#
|
||||
# If you are using calico on AWS
|
||||
#
|
||||
# network:
|
||||
# plugin: calico
|
||||
# calico_network_provider:
|
||||
# cloud_provider: aws
|
||||
#
|
||||
# # To specify flannel interface
|
||||
#
|
||||
# network:
|
||||
# plugin: flannel
|
||||
# flannel_network_provider:
|
||||
# iface: eth1
|
||||
#
|
||||
# # To specify flannel interface for canal plugin
|
||||
#
|
||||
# network:
|
||||
# plugin: canal
|
||||
# canal_network_provider:
|
||||
# iface: eth1
|
||||
#
|
||||
network:
|
||||
options:
|
||||
flannel_backend_type: vxlan
|
||||
plugin: canal
|
||||
#
|
||||
# services:
|
||||
# kube-api:
|
||||
# service_cluster_ip_range: 10.43.0.0/16
|
||||
# kube-controller:
|
||||
# cluster_cidr: 10.42.0.0/16
|
||||
# service_cluster_ip_range: 10.43.0.0/16
|
||||
# kubelet:
|
||||
# cluster_domain: cluster.local
|
||||
# cluster_dns_server: 10.43.0.10
|
||||
#
|
||||
services:
|
||||
etcd:
|
||||
backup_config:
|
||||
enabled: true
|
||||
interval_hours: 12
|
||||
retention: 6
|
||||
safe_timestamp: false
|
||||
creation: 12h
|
||||
extra_args:
|
||||
election-timeout: 5000
|
||||
heartbeat-interval: 500
|
||||
gid: 0
|
||||
retention: 72h
|
||||
snapshot: false
|
||||
uid: 0
|
||||
kube_api:
|
||||
always_pull_images: false
|
||||
pod_security_policy: false
|
||||
service_node_port_range: 30000-32767
|
||||
ssh_agent_auth: false
|
||||
windows_prefered_cluster: false
|
||||
```
|
||||
</details>
|
||||
|
||||
### Default DNS provider
|
||||
|
||||
The table below indicates what DNS provider is deployed by default. See [RKE documentation on DNS provider]({{<baseurl>}}/rke/latest/en/config-options/add-ons/dns/) for more information how to configure a different DNS provider. CoreDNS can only be used on Kubernetes v1.12.0 and higher.
|
||||
|
||||
| Rancher version | Kubernetes version | Default DNS provider |
|
||||
|-------------|--------------------|----------------------|
|
||||
| v2.2.5 and higher | v1.14.0 and higher | CoreDNS |
|
||||
| v2.2.5 and higher | v1.13.x and lower | kube-dns |
|
||||
| v2.2.4 and lower | any | kube-dns |
|
||||
|
||||
# Rancher Specific Parameters in YAML
|
||||
|
||||
Besides the RKE config file options, there are also Rancher specific settings that can be configured in the Config File (YAML):
|
||||
|
||||
### docker_root_dir
|
||||
|
||||
See [Docker Root Directory](#docker-root-directory).
|
||||
|
||||
### enable_cluster_monitoring
|
||||
|
||||
Option to enable or disable [Cluster Monitoring]({{<baseurl>}}/rancher/v2.6/en/monitoring-alerting/).
|
||||
|
||||
### enable_network_policy
|
||||
|
||||
Option to enable or disable Project Network Isolation.
|
||||
|
||||
Project network isolation is available if you are using any RKE network plugin that supports the enforcement of Kubernetes network policies, such as Canal or the Cisco ACI plugin.
|
||||
|
||||
### local_cluster_auth_endpoint
|
||||
|
||||
See [Authorized Cluster Endpoint](#authorized-cluster-endpoint).
|
||||
|
||||
Example:
|
||||
|
||||
```yaml
|
||||
local_cluster_auth_endpoint:
|
||||
enabled: true
|
||||
fqdn: "FQDN"
|
||||
ca_certs: |-
|
||||
-----BEGIN CERTIFICATE-----
|
||||
...
|
||||
-----END CERTIFICATE-----
|
||||
```
|
||||
|
||||
### Custom Network Plug-in
|
||||
|
||||
You can add a custom network plug-in by using the [user-defined add-on functionality]({{<baseurl>}}/rke/latest/en/config-options/add-ons/user-defined-add-ons/) of RKE. You define any add-on that you want deployed after the Kubernetes cluster is deployed.
|
||||
|
||||
There are two ways that you can specify an add-on:
|
||||
|
||||
- [In-line Add-ons]({{<baseurl>}}/rke/latest/en/config-options/add-ons/user-defined-add-ons/#in-line-add-ons)
|
||||
- [Referencing YAML Files for Add-ons]({{<baseurl>}}/rke/latest/en/config-options/add-ons/user-defined-add-ons/#referencing-yaml-files-for-add-ons)
|
||||
|
||||
For an example of how to configure a custom network plug-in by editing the `cluster.yml`, refer to the [RKE documentation.]({{<baseurl>}}/rke/latest/en/config-options/add-ons/network-plugins/custom-network-plugin-example)
|
||||
+188
@@ -0,0 +1,188 @@
|
||||
---
|
||||
title: RKE2 Cluster Configuration Reference
|
||||
shortTitle: RKE2 Cluster Configuration
|
||||
weight: 5
|
||||
---
|
||||
|
||||
This section covers the configuration options that are available in Rancher for a new or existing RKE2 Kubernetes cluster.
|
||||
|
||||
# Overview
|
||||
|
||||
You can configure the Kubernetes options in one of the two following ways:
|
||||
|
||||
- [Rancher UI](#configuration-options-in-the-rancher-ui): Use the Rancher UI to select options that are commonly customized when setting up a Kubernetes cluster.
|
||||
- [Cluster Config File](#cluster-config-file): Instead of using the Rancher UI to choose Kubernetes options for the cluster, advanced users can create an RKE2 config file. Using a config file allows you to set any of the [options](https://docs.rke2.io/install/install_options/install_options) available in an RKE2 installation.
|
||||
|
||||
# Configuration Options in the Rancher UI
|
||||
|
||||
> Some advanced configuration options are not exposed in the Rancher UI forms, but they can be enabled by editing the RKE2 cluster configuration file in YAML. For the complete reference of configurable options for RKE2 Kubernetes clusters in YAML, see the [RKE2 documentation.](https://docs.rke2.io/install/install_options/install_options/)
|
||||
|
||||
### Basics
|
||||
#### Kubernetes Version
|
||||
|
||||
The version of Kubernetes installed on your cluster nodes. Rancher packages its own version of Kubernetes based on [hyperkube](https://github.com/rancher/hyperkube).
|
||||
|
||||
For more detail, see [Upgrading Kubernetes]({{<baseurl>}}/rancher/v2.6/en/cluster-admin/upgrading-kubernetes).
|
||||
|
||||
#### Container Network Provider
|
||||
|
||||
The [Network Provider](https://kubernetes.io/docs/concepts/cluster-administration/networking/) that the cluster uses.
|
||||
|
||||
> After you launch the cluster, you cannot change your network provider. Therefore, choose which network provider you want to use carefully, as Kubernetes doesn't allow switching between network providers. Once a cluster is created with a network provider, changing network providers would require you to tear down the entire cluster and all its applications.
|
||||
|
||||
Out of the box, Rancher is compatible with the following network providers:
|
||||
|
||||
- [Canal](https://github.com/projectcalico/canal)
|
||||
- [Cilium](https://cilium.io/)*
|
||||
- [Calico](https://docs.projectcalico.org/v3.11/introduction/)
|
||||
- [Multus](https://github.com/k8snetworkplumbingwg/multus-cni)
|
||||
|
||||
\* When using [project network isolation](#project-network-isolation) in the [Cilium CNI]({{<baseurl>}}/rancher/v2.6/en/faq/networking/cni-providers/#cilium), it is possible to enable cross-node ingress routing. Click the [CNI provider docs]({{<baseurl>}}/rancher/v2.6/en/faq/networking/cni-providers/#ingress-routing-across-nodes-in-cilium) to learn more.
|
||||
|
||||
For more details on the different networking providers and how to configure them, please view our [RKE2 documentation](https://docs.rke2.io/install/network_options/).
|
||||
|
||||
##### Dual-stack Networking
|
||||
|
||||
[Dual-stack](https://docs.rke2.io/install/network_options/#dual-stack-configuration) networking is supported for all CNI providers. To configure RKE2 in dual-stack mode, set valid IPv4/IPv6 CIDRs for your [Cluster CIDR](#cluster-cidr) and/or [Service CIDR](#service-cidr).
|
||||
|
||||
###### Additional Configuration {#dual-stack-additional-config}
|
||||
|
||||
When using `cilium` or `multus,cilium` as your container network interface provider, ensure the **Enable IPv6 Support** option is also enabled.
|
||||
|
||||
#### Cloud Provider
|
||||
|
||||
You can configure a [Kubernetes cloud provider]({{<baseurl>}}/rancher/v2.6/en/cluster-provisioning/rke-clusters/cloud-providers). If you want to use dynamically provisioned [volumes and storage]({{<baseurl>}}/rancher/v2.6/en/cluster-admin/volumes-and-storage/) in Kubernetes, typically you must select the specific cloud provider in order to use it. For example, if you want to use Amazon EBS, you would need to select the `aws` cloud provider.
|
||||
|
||||
>**Note:** If the cloud provider you want to use is not listed as an option, you will need to use the [config file option](#cluster-config-file) to configure the cloud provider. Please reference [this documentation]({{<baseurl>}}/rke/latest/en/config-options/cloud-providers/) on how to configure the cloud provider.
|
||||
|
||||
#### Default Pod Security Policy
|
||||
|
||||
Choose the default [pod security policy]({{<baseurl>}}/rancher/v2.6/en/admin-settings/pod-security-policies/) for the cluster. Please refer to the [RKE2 documentation](https://docs.rke2.io/security/policies/) on the specifications of each available policy.
|
||||
|
||||
#### Worker CIS Profile
|
||||
|
||||
Select a [CIS benchmark]({{<baseurl>}}/rancher/v2.6/en/cis-scans/) to validate the system configuration against.
|
||||
|
||||
#### Project Network Isolation
|
||||
|
||||
If your network provider allows project network isolation, you can choose whether to enable or disable inter-project communication.
|
||||
|
||||
Project network isolation is available if you are using any RKE2 network plugin that supports the enforcement of Kubernetes network policies, such as Canal.
|
||||
|
||||
#### SELinux
|
||||
|
||||
Option to enable or disable [SELinux](https://docs.rke2.io/security/selinux) support.
|
||||
|
||||
#### CoreDNS
|
||||
|
||||
By default, [CoreDNS](https://coredns.io/) is installed as the default DNS provider. If CoreDNS is not installed, an alternate DNS provider must be installed yourself. Refer to the [RKE2 documentation](https://docs.rke2.io/networking/#coredns) for additional CoreDNS configurations.
|
||||
|
||||
#### NGINX Ingress
|
||||
|
||||
If you want to publish your applications in a high-availability configuration, and you're hosting your nodes with a cloud-provider that doesn't have a native load-balancing feature, enable this option to use NGINX Ingress within the cluster. Refer to the [RKE2 documentation](https://docs.rke2.io/networking/#nginx-ingress-controller) for additional configuration options.
|
||||
|
||||
Refer to the [RKE2 documentation](https://docs.rke2.io/networking/#nginx-ingress-controller) for additional configuration options.
|
||||
|
||||
#### Metrics Server
|
||||
|
||||
Option to enable or disable [Metrics Server]({{<baseurl>}}/rke/latest/en/config-options/add-ons/metrics-server/).
|
||||
|
||||
Each cloud provider capable of launching a cluster using RKE2 can collect metrics and monitor for your cluster nodes. Enable this option to view your node metrics from your cloud provider's portal.
|
||||
|
||||
### Add-On Config
|
||||
|
||||
Additional Kubernetes manifests, managed as an [Add-on](https://kubernetes.io/docs/concepts/cluster-administration/addons/), to apply to the cluster on startup. Refer to the [RKE2 documentation](https://docs.rke2.io/helm/#automatically-deploying-manifests-and-helm-charts) for details.
|
||||
|
||||
### Agent Environment Vars
|
||||
|
||||
Option to set environment variables for [Rancher agents](https://rancher.com/docs/rancher/v2.6/en/cluster-provisioning/rke-clusters/rancher-agents/). The environment variables can be set using key value pairs. Refer to the [RKE2 documentation](https://docs.rke2.io/install/install_options/linux_agent_config/) for more details.
|
||||
|
||||
### etcd
|
||||
|
||||
#### Automatic Snapshots
|
||||
|
||||
Option to enable or disable recurring etcd snapshots. If enabled, users have the option to configure the frequency of snapshots. For details, refer to the [RKE2 documentation](https://docs.rke2.io/backup_restore/#creating-snapshots). Note that with RKE2, snapshots are stored on each etcd node. This varies from RKE1 which only stores one snapshot per cluster.
|
||||
|
||||
#### Metrics
|
||||
|
||||
Option to choose whether to expose etcd metrics to the public or only within the cluster.
|
||||
|
||||
### Networking
|
||||
|
||||
#### Cluster CIDR
|
||||
|
||||
IPv4 and/or IPv6 network CIDRs to use for pod IPs (default: 10.42.0.0/16).
|
||||
|
||||
##### Dual-stack Networking
|
||||
|
||||
To configure [dual-stack](https://docs.rke2.io/install/network_options/#dual-stack-configuration) mode, enter a valid IPv4/IPv6 CIDR. For example `10.42.0.0/16,2001:cafe:42:0::/56`.
|
||||
|
||||
[Additional configuration](#dual-stack-additional-config) is required when using `cilium` or `multus,cilium` as your [container network](#container-network) interface provider.
|
||||
|
||||
#### Service CIDR
|
||||
|
||||
IPv4/IPv6 network CIDRs to use for service IPs (default: 10.43.0.0/16).
|
||||
|
||||
##### Dual-stack Networking
|
||||
|
||||
To configure [dual-stack](https://docs.rke2.io/install/network_options/#dual-stack-configuration) mode, enter a valid IPv4/IPv6 CIDR. For example `10.42.0.0/16,2001:cafe:42:0::/56`.
|
||||
|
||||
[Additional configuration](#dual-stack-additional-config) is required when using `cilium ` or `multus,cilium` as your [container network](#container-network) interface provider.
|
||||
|
||||
#### Cluster DNS
|
||||
|
||||
IPv4 Cluster IP for coredns service. Should be in your service-cidr range (default: 10.43.0.10).
|
||||
|
||||
#### Cluster Domain
|
||||
|
||||
Select the domain for the cluster. The default is `cluster.local`.
|
||||
|
||||
#### NodePort Service Port Range
|
||||
|
||||
Option to change the range of ports that can be used for [NodePort services](https://kubernetes.io/docs/concepts/services-networking/service/#nodeport). The default is `30000-32767`.
|
||||
|
||||
#### TLS Alternate Names
|
||||
|
||||
Add additional hostnames or IPv4/IPv6 addresses as Subject Alternative Names on the server TLS cert.
|
||||
|
||||
#### Authorized Cluster Endpoint
|
||||
|
||||
Authorized Cluster Endpoint can be used to directly access the Kubernetes API server, without requiring communication through Rancher.
|
||||
|
||||
This is enabled by default in Rancher-launched Kubernetes clusters, using the IP of the node with the `controlplane` role and the default Kubernetes self signed certificates.
|
||||
|
||||
For more detail on how an authorized cluster endpoint works and why it is used, refer to the [architecture section.]({{<baseurl>}}/rancher/v2.6/en/overview/architecture/#4-authorized-cluster-endpoint)
|
||||
|
||||
We recommend using a load balancer with the authorized cluster endpoint. For details, refer to the [recommended architecture section.]({{<baseurl>}}/rancher/v2.6/en/overview/architecture-recommendations/#architecture-for-an-authorized-cluster-endpoint)
|
||||
|
||||
### Registries
|
||||
|
||||
Select the image repository to pull Rancher images from. For more details and configuration options, see the [RKE2 documentation](https://docs.rke2.io/install/containerd_registry_configuration/).
|
||||
|
||||
### Upgrade Strategy
|
||||
|
||||
#### Control Plane Concurrency
|
||||
|
||||
Select how many nodes can be upgraded at the same time. Can be a fixed number or percentage.
|
||||
|
||||
#### Worker Concurrency
|
||||
|
||||
Select how many nodes can be upgraded at the same time. Can be a fixed number or percentage.
|
||||
|
||||
#### Drain Nodes (Control Plane)
|
||||
|
||||
Option to remove all pods from the node prior to upgrading.
|
||||
|
||||
#### Drain Nodes (Worker Nodes)
|
||||
|
||||
Option to remove all pods from the node prior to upgrading.
|
||||
|
||||
### Advanced
|
||||
|
||||
Option to set kubelet options for different nodes. For available options, refer to the [Kubernetes documentation](https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/).
|
||||
|
||||
# Cluster Config File
|
||||
|
||||
Instead of using the Rancher UI forms to choose Kubernetes options for the cluster, advanced users can create an RKE2 config file. Using a config file allows you to set any of the [options](https://docs.rke2.io/install/install_options/install_options) available in an RKE2 installation.
|
||||
|
||||
To edit an RKE2 config file directly from the Rancher UI, click **Edit as YAML**.
|
||||
@@ -0,0 +1,35 @@
|
||||
---
|
||||
title: Syncing
|
||||
weight: 10
|
||||
---
|
||||
|
||||
Syncing is the feature for EKS and GKE clusters that causes Rancher to update the clusters' values so they are up to date with their corresponding cluster object in the hosted Kubernetes provider. This enables Rancher to not be the sole owner of a hosted cluster’s state. Its largest limitation is that processing an update from Rancher and another source at the same time or within 5 minutes of one finishing may cause the state from one source to completely overwrite the other.
|
||||
|
||||
### How it works
|
||||
|
||||
There are two fields on the Rancher Cluster object that must be understood to understand how syncing works:
|
||||
|
||||
1. The config object for the cluster, located on the Spec of the Cluster:
|
||||
|
||||
* For EKS, the field is called EKSConfig
|
||||
* For GKE, the field is called GKEConfig
|
||||
|
||||
2. The UpstreamSpec object
|
||||
|
||||
* For EKS, this is located on the EKSStatus field on the Status of the Cluster.
|
||||
* For GKE, this is located on the GKEStatus field on the Status of the Cluster.
|
||||
|
||||
The struct types that define these objects can be found in their corresponding operator projects:
|
||||
|
||||
* [eks-operator](https://github.com/rancher/eks-operator/blob/master/pkg/apis/eks.cattle.io/v1/types.go)
|
||||
* [gke-operator](https://github.com/rancher/gke-operator/blob/master/pkg/apis/gke.cattle.io/v1/types.go)
|
||||
|
||||
All fields with the exception of the cluster name, the location (region or zone), Imported, and the cloud credential reference, are nillable on this Spec object.
|
||||
|
||||
The EKSConfig or GKEConfig represents desired state for its non-nil values. Fields that are non-nil in the config object can be thought of as “managed". When a cluster is created in Rancher, all fields are non-nil and therefore “managed”. When a pre-existing cluster is registered in rancher all nillable fields are nil and are not “managed”. Those fields become managed once their value has been changed by Rancher.
|
||||
|
||||
UpstreamSpec represents the cluster as it is in the hosted Kubernetes provider and is refreshed on an interval of 5 minutes. After the UpstreamSpec has been refreshed, Rancher checks if the cluster has an update in progress. If it is updating, nothing further is done. If it is not currently updating, any “managed” fields on EKSConfig or GKEConfig are overwritten with their corresponding value from the recently updated UpstreamSpec.
|
||||
|
||||
The effective desired state can be thought of as the UpstreamSpec + all non-nil fields in the EKSConfig or GKEConfig. This is what is displayed in the UI.
|
||||
|
||||
If Rancher and another source attempt to update a cluster at the same time or within the 5 minute refresh window of an update finishing, then it is likely any “managed” fields can be caught in a race condition. To use EKS as an example, a cluster may have PrivateAccess as a managed field. If PrivateAccess is false and then enabled in EKS console, then finishes at 11:01, and then tags are updated from Rancher before 11:05 the value will likely be overwritten. This would also occur if tags were updated while the cluster was processing the update. If the cluster was registered and the PrivateAccess fields was nil then this issue should not occur in the aforementioned case.
|
||||
@@ -0,0 +1,188 @@
|
||||
---
|
||||
title: Nodes and Node Pools
|
||||
weight: 2030
|
||||
---
|
||||
|
||||
After you launch a Kubernetes cluster in Rancher, you can manage individual nodes from the cluster's **Node** tab. Depending on the [option used]({{<baseurl>}}/rancher/v2.6/en/cluster-provisioning/) to provision the cluster, there are different node options available.
|
||||
|
||||
> If you want to manage the _cluster_ and not individual nodes, see [Editing Clusters]({{< baseurl >}}/rancher/v2.6/en/cluster-admin/editing-clusters).
|
||||
|
||||
This section covers the following topics:
|
||||
|
||||
- [Node options available for each cluster creation option](#node-options-available-for-each-cluster-creation-option)
|
||||
- [Nodes hosted by an infrastructure provider](#nodes-hosted-by-an-infrastructure-provider)
|
||||
- [Nodes provisioned by hosted Kubernetes providers](#nodes-provisioned-by-hosted-kubernetes-providers)
|
||||
- [Registered nodes](#registered-nodes)
|
||||
- [Managing and editing individual nodes](#managing-and-editing-individual-nodes)
|
||||
- [Viewing a node in the Rancher API](#viewing-a-node-in-the-rancher-api)
|
||||
- [Deleting a node](#deleting-a-node)
|
||||
- [Scaling nodes](#scaling-nodes)
|
||||
- [SSH into a node hosted by an infrastructure provider](#ssh-into-a-node-hosted-by-an-infrastructure-provider)
|
||||
- [Cordoning a node](#cordoning-a-node)
|
||||
- [Draining a node](#draining-a-node)
|
||||
- [Aggressive and safe draining options](#aggressive-and-safe-draining-options)
|
||||
- [Grace period](#grace-period)
|
||||
- [Timeout](#timeout)
|
||||
- [Drained and cordoned state](#drained-and-cordoned-state)
|
||||
- [Labeling a node to be ignored by Rancher](#labeling-a-node-to-be-ignored-by-rancher)
|
||||
|
||||
# Node Options Available for Each Cluster Creation Option
|
||||
|
||||
The following table lists which node options are available for each type of cluster in Rancher. Click the links in the **Option** column for more detailed information about each feature.
|
||||
|
||||
| Option | [Nodes Hosted by an Infrastructure Provider][1] | [Custom Node][2] | [Hosted Cluster][3] | [Registered EKS Nodes][4] | [All Other Registered Nodes][5] | Description |
|
||||
| ------------------------------------------------ | ------------------------------------------------ | ---------------- | ------------------- | ------------------- | -------------------| ------------------------------------------------------------------ |
|
||||
| [Cordon](#cordoning-a-node) | ✓ | ✓ | ✓ | ✓ | ✓ | Marks the node as unschedulable. |
|
||||
| [Drain](#draining-a-node) | ✓ | ✓ | ✓ | ✓ | ✓ | Marks the node as unschedulable _and_ evicts all pods. |
|
||||
| [Edit](#managing-and-editing-individual-nodes) | ✓ | ✓ | ✓ | ✓ | ✓ | Enter a custom name, description, label, or taints for a node. |
|
||||
| [View API](#viewing-a-node-in-the-rancher-api) | ✓ | ✓ | ✓ | ✓ | ✓ | View API data. |
|
||||
| [Delete](#deleting-a-node) | ✓ | ✓ | | * | * | Deletes defective nodes from the cluster. |
|
||||
| [Download Keys](#ssh-into-a-node-hosted-by-an-infrastructure-provider) | ✓ | | | | | Download SSH key in order to SSH into the node. |
|
||||
| [Node Scaling](#scaling-nodes) | ✓ | | | ✓ | | Scale the number of nodes in the node pool up or down. |
|
||||
|
||||
[1]: {{<baseurl>}}/rancher/v2.6/en/cluster-provisioning/rke-clusters/node-pools/
|
||||
[2]: {{<baseurl>}}/rancher/v2.6/en/cluster-provisioning/rke-clusters/custom-nodes/
|
||||
[3]: {{<baseurl>}}/rancher/v2.6/en/cluster-provisioning/hosted-kubernetes-clusters/
|
||||
[4]: {{<baseurl>}}/rancher/v2.6/en/cluster-provisioning/registered-clusters/
|
||||
[5]: {{<baseurl>}}/rancher/v2.6/en/cluster-provisioning/registered-clusters/
|
||||
|
||||
\* Delete option accessible via View API
|
||||
|
||||
|
||||
### Nodes Hosted by an Infrastructure Provider
|
||||
|
||||
Node pools are available when you provision Rancher-launched Kubernetes clusters on nodes that are [hosted in an infrastructure provider.]({{<baseurl>}}/rancher/v2.6/en/cluster-provisioning/rke-clusters/node-pools/)
|
||||
|
||||
Clusters provisioned using [one of the node pool options]({{<baseurl>}}/rancher/v2.6/en/cluster-provisioning/rke-clusters/node-pools/#node-pools) can be scaled up or down if the node pool is edited.
|
||||
|
||||
A node pool can also automatically maintain the node scale that's set during the initial cluster provisioning if [node auto-replace is enabled.]({{<baseurl>}}/rancher/v2.6/en/cluster-provisioning/rke-clusters/node-pools/#about-node-auto-replace) This scale determines the number of active nodes that Rancher maintains for the cluster.
|
||||
|
||||
Rancher uses [node templates]({{<baseurl>}}/rancher/v2.6/en/cluster-provisioning/rke-clusters/node-pools/#node-templates) to replace nodes in the node pool. Each node template uses cloud provider credentials to allow Rancher to set up the node in the infrastructure provider.
|
||||
|
||||
### Nodes Provisioned by Hosted Kubernetes Providers
|
||||
|
||||
Options for managing nodes [hosted by a Kubernetes provider]({{<baseurl >}}/rancher/v2.6/en/cluster-provisioning/hosted-kubernetes-clusters/) are somewhat limited in Rancher. Rather than using the Rancher UI to make edits such as scaling the number of nodes up or down, edit the cluster directly.
|
||||
|
||||
### Registered Nodes
|
||||
|
||||
Although you can deploy workloads to a [registered cluster]({{< baseurl >}}/rancher/v2.6/en/cluster-provisioning/registered-clusters/) using Rancher, you cannot manage individual cluster nodes. All management of imported cluster nodes must take place outside of Rancher.
|
||||
|
||||
# Managing and Editing Individual Nodes
|
||||
|
||||
Editing a node lets you:
|
||||
|
||||
* Change its name
|
||||
* Change its description
|
||||
* Add [labels](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/)
|
||||
* Add/Remove [taints](https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/)
|
||||
|
||||
To manage individual nodes, browse to the cluster that you want to manage and then select **Nodes** from the main menu. You can open the options menu for a node by clicking its **⋮** icon (**..**.).
|
||||
|
||||
# Viewing a Node in the Rancher API
|
||||
|
||||
Select this option to view the node's [API endpoints]({{< baseurl >}}/rancher/v2.6/en/api/).
|
||||
|
||||
# Deleting a Node
|
||||
|
||||
Use **Delete** to remove defective nodes from the cloud provider.
|
||||
|
||||
When you the delete a defective node, Rancher can automatically replace it with an identically provisioned node if the node is in a node pool and [node auto-replace is enabled.]({{<baseurl>}}/rancher/v2.6/en/cluster-provisioning/rke-clusters/node-pools/#about-node-auto-replace)
|
||||
|
||||
>**Tip:** If your cluster is hosted by an infrastructure provider, and you want to scale your cluster down instead of deleting a defective node, [scale down](#scaling-nodes) rather than delete.
|
||||
|
||||
# Scaling Nodes
|
||||
|
||||
For nodes hosted by an infrastructure provider, you can scale the number of nodes in each [node pool]({{<baseurl>}}/rancher/v2.6/en/cluster-provisioning/rke-clusters/node-pools/#node-pools) by using the scale controls. This option isn't available for other cluster types.
|
||||
|
||||
# SSH into a Node Hosted by an Infrastructure Provider
|
||||
|
||||
For [nodes hosted by an infrastructure provider]({{<baseurl>}}/rancher/v2.6/en/cluster-provisioning/rke-clusters/node-pools/), you have the option of downloading its SSH key so that you can connect to it remotely from your desktop.
|
||||
|
||||
1. In the upper left corner, click **☰ > Cluster Management**.
|
||||
1. On the **Clusters** page, go to the cluster where you want to SSH into a node and click the name of the cluster.
|
||||
1. On the **Machine Pools** tab, find the node that you want to remote into and click **⋮ > Download SSH Key**. A ZIP file containing files used for SSH will be downloaded.
|
||||
1. Extract the ZIP file to any location.
|
||||
1. Open Terminal. Change your location to the extracted ZIP file.
|
||||
1. Enter the following command:
|
||||
|
||||
```
|
||||
ssh -i id_rsa root@<IP_OF_HOST>
|
||||
```
|
||||
|
||||
# Cordoning a Node
|
||||
|
||||
_Cordoning_ a node marks it as unschedulable. This feature is useful for performing short tasks on the node during small maintenance windows, like reboots, upgrades, or decommissions. When you're done, power back on and make the node schedulable again by uncordoning it.
|
||||
|
||||
# Draining a Node
|
||||
|
||||
_Draining_ is the process of first cordoning the node, and then evicting all its pods. This feature is useful for performing node maintenance (like kernel upgrades or hardware maintenance). It prevents new pods from deploying to the node while redistributing existing pods so that users don't experience service interruption.
|
||||
|
||||
- For pods with a replica set, the pod is replaced by a new pod that will be scheduled to a new node. Additionally, if the pod is part of a service, then clients will automatically be redirected to the new pod.
|
||||
|
||||
- For pods with no replica set, you need to bring up a new copy of the pod, and assuming it is not part of a service, redirect clients to it.
|
||||
|
||||
You can drain nodes that are in either a `cordoned` or `active` state. When you drain a node, the node is cordoned, the nodes are evaluated for conditions they must meet to be drained, and then (if it meets the conditions) the node evicts its pods.
|
||||
|
||||
However, you can override the conditions draining when you initiate the drain. You're also given an opportunity to set a grace period and timeout value.
|
||||
|
||||
### Aggressive and Safe Draining Options
|
||||
|
||||
When you configure the upgrade strategy for the cluster, you will be able to enable node draining. If node draining is enabled, you will be able to configure how pods are deleted and rescheduled.
|
||||
|
||||
- **Aggressive Mode**
|
||||
|
||||
In this mode, pods won't get rescheduled to a new node, even if they do not have a controller. Kubernetes expects you to have your own logic that handles the deletion of these pods.
|
||||
|
||||
Kubernetes also expects the implementation to decide what to do with pods using emptyDir. If a pod uses emptyDir to store local data, you might not be able to safely delete it, since the data in the emptyDir will be deleted once the pod is removed from the node. Choosing aggressive mode will delete these pods.
|
||||
|
||||
- **Safe Mode**
|
||||
|
||||
If a node has standalone pods or ephemeral data it will be cordoned but not drained.
|
||||
### Grace Period
|
||||
|
||||
The timeout given to each pod for cleaning things up, so they will have chance to exit gracefully. For example, when pods might need to finish any outstanding requests, roll back transactions or save state to some external storage. If negative, the default value specified in the pod will be used.
|
||||
|
||||
### Timeout
|
||||
|
||||
The amount of time drain should continue to wait before giving up.
|
||||
|
||||
>**Kubernetes Known Issue:** The [timeout setting](https://github.com/kubernetes/kubernetes/pull/64378) was not enforced while draining a node before Kubernetes 1.12.
|
||||
|
||||
### Drained and Cordoned State
|
||||
|
||||
If there's any error related to user input, the node enters a `cordoned` state because the drain failed. You can either correct the input and attempt to drain the node again, or you can abort by uncordoning the node.
|
||||
|
||||
If the drain continues without error, the node enters a `draining` state. You'll have the option to stop the drain when the node is in this state, which will stop the drain process and change the node's state to `cordoned`.
|
||||
|
||||
Once drain successfully completes, the node will be in a state of `drained`. You can then power off or delete the node.
|
||||
|
||||
>**Want to know more about cordon and drain?** See the [Kubernetes documentation](https://kubernetes.io/docs/tasks/administer-cluster/cluster-management/#maintenance-on-a-node).
|
||||
|
||||
# Labeling a Node to be Ignored by Rancher
|
||||
|
||||
Some solutions, such as F5's BIG-IP integration, may require creating a node that is never registered to a cluster.
|
||||
|
||||
Since the node will never finish registering, it will always be shown as unhealthy in the Rancher UI.
|
||||
|
||||
In that case, you may want to label the node to be ignored by Rancher so that Rancher only shows nodes as unhealthy when they are actually failing.
|
||||
|
||||
You can label nodes to be ignored by using a setting in the Rancher UI, or by using `kubectl`.
|
||||
|
||||
> **Note:** There is an [open issue](https://github.com/rancher/rancher/issues/24172) in which nodes labeled to be ignored can get stuck in an updating state.
|
||||
|
||||
|
||||
### Labeling Nodes to be Ignored with kubectl
|
||||
|
||||
To add a node that will be ignored by Rancher, use `kubectl` to create a node that has the following label:
|
||||
|
||||
```
|
||||
cattle.rancher.io/node-status: ignore
|
||||
```
|
||||
|
||||
**Result:** If you add the node to a cluster, Rancher will not attempt to sync with this node. The node can still be part of the cluster and can be listed with `kubectl`.
|
||||
|
||||
If the label is added before the node is added to the cluster, the node will not be shown in the Rancher UI.
|
||||
|
||||
If the label is added after the node is added to a Rancher cluster, the node will not be removed from the UI.
|
||||
|
||||
If you delete the node from the Rancher server using the Rancher UI or API, the node will not be removed from the cluster if the `nodeName` is listed in the Rancher settings in the Rancher API under `v3/settings/ignore-node-name`.
|
||||
+19
@@ -0,0 +1,19 @@
|
||||
---
|
||||
title: Assigning Pod Security Policies
|
||||
weight: 2260
|
||||
---
|
||||
|
||||
_Pod Security Policies_ are objects that control security-sensitive aspects of pod specification (like root privileges).
|
||||
|
||||
## Adding a Default Pod Security Policy
|
||||
|
||||
When you create a new cluster with RKE, you can configure it to apply a PSP immediately. As you create the cluster, use the **Cluster Options** to enable a PSP. The PSP assigned to the cluster will be the default PSP for projects within the cluster.
|
||||
|
||||
>**Prerequisite:**
|
||||
>Create a Pod Security Policy within Rancher. Before you can assign a default PSP to a new cluster, you must have a PSP available for assignment. For instruction, see [Creating Pod Security Policies]({{<baseurl>}}/rancher/v2.6/en/admin-settings/pod-security-policies/).
|
||||
>**Note:**
|
||||
>For security purposes, we recommend assigning a PSP as you create your clusters.
|
||||
|
||||
To enable a default Pod Security Policy, set the **Pod Security Policy Support** option to **Enabled**, and then make a selection from the **Default Pod Security Policy** drop-down.
|
||||
|
||||
When the cluster finishes provisioning, the PSP you selected is applied to all projects within the cluster.
|
||||
@@ -0,0 +1,28 @@
|
||||
---
|
||||
title: Adding a Pod Security Policy
|
||||
weight: 80
|
||||
---
|
||||
|
||||
> **Prerequisite:** The options below are available only for clusters that are [launched using RKE.]({{<baseurl>}}/rancher/v2.6/en/cluster-provisioning/rke-clusters/)
|
||||
|
||||
When your cluster is running pods with security-sensitive configurations, assign it a [pod security policy]({{<baseurl>}}/rancher/v2.6/en/admin-settings/pod-security-policies/), which is a set of rules that monitors the conditions and settings in your pods. If a pod doesn't meet the rules specified in your policy, the policy stops it from running.
|
||||
|
||||
You can assign a pod security policy when you provision a cluster. However, if you need to relax or restrict security for your pods later, you can update the policy while editing your cluster.
|
||||
|
||||
1. Click **☰ > Cluster Management**.
|
||||
1. Go to the cluster to which you want to apply a pod security policy and click **⋮ > Edit Config**.
|
||||
1. From **Pod Security Policy Support**, select **Enabled**.
|
||||
|
||||
>**Note:** This option is only available for clusters [provisioned by RKE]({{<baseurl>}}/rancher/v2.6/en/cluster-provisioning/rke-clusters/).
|
||||
|
||||
4. From the **Default Pod Security Policy** drop-down, select the policy you want to apply to the cluster.
|
||||
|
||||
Rancher ships with [policies]({{<baseurl>}}/rancher/v2.6/en/admin-settings/pod-security-policies/#default-pod-security-policies) of `restricted` and `unrestricted`, although you can [create custom policies]({{<baseurl>}}/rancher/v2.6/en/admin-settings/pod-security-policies/#default-pod-security-policies) as well.
|
||||
|
||||
5. Click **Save**.
|
||||
|
||||
**Result:** The pod security policy is applied to the cluster and any projects within the cluster.
|
||||
|
||||
>**Note:** Workloads already running before assignment of a pod security policy are grandfathered in. Even if they don't meet your pod security policy, workloads running before assignment of the policy continue to run.
|
||||
>
|
||||
>To check if a running workload passes your pod security policy, clone or upgrade it.
|
||||
+183
@@ -0,0 +1,183 @@
|
||||
---
|
||||
title: Projects and Kubernetes Namespaces with Rancher
|
||||
description: Rancher Projects ease the administrative burden of your cluster and support multi-tenancy. Learn to create projects and divide projects into Kubernetes namespaces
|
||||
weight: 2032
|
||||
---
|
||||
|
||||
A namespace is a Kubernetes concept that allows a virtual cluster within a cluster, which is useful for dividing the cluster into separate "virtual clusters" that each have their own access control and resource quotas.
|
||||
|
||||
A project is a group of namespaces, and it is a concept introduced by Rancher. Projects allow you to manage multiple namespaces as a group and perform Kubernetes operations in them. You can use projects to support multi-tenancy, so that a team can access a project within a cluster without having access to other projects in the same cluster.
|
||||
|
||||
>**Note:** As of Rancher v2.6, projects are de-emphasized on the UI because it is no longer required to create any Kubernetes resources within a project scope. However, resources such as [Secrets]({{<baseurl>}}/rancher/v2.6/en/k8s-in-rancher/secrets/#creating-secrets-in-projects) can still be created in a project scope if the legacy feature flag is enabled.
|
||||
|
||||
This section describes how projects and namespaces work with Rancher. It covers the following topics:
|
||||
|
||||
- [About namespaces](#about-namespaces)
|
||||
- [About projects](#about-projects)
|
||||
- [The cluster's default project](#the-cluster-s-default-project)
|
||||
- [The system project](#the-system-project)
|
||||
- [Project authorization](#project-authorization)
|
||||
- [Pod security policies](#pod-security-policies)
|
||||
- [Creating projects](#creating-projects)
|
||||
- [Switching between clusters and projects](#switching-between-clusters-and-projects)
|
||||
|
||||
# About Namespaces
|
||||
|
||||
A namespace is a concept introduced by Kubernetes. According to the [official Kubernetes documentation on namespaces,](https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/)
|
||||
|
||||
> Kubernetes supports multiple virtual clusters backed by the same physical cluster. These virtual clusters are called namespaces. [...] Namespaces are intended for use in environments with many users spread across multiple teams, or projects. For clusters with a few to tens of users, you should not need to create or think about namespaces at all.
|
||||
|
||||
Namespaces provide the following functionality:
|
||||
|
||||
- **Providing a scope for names:** Names of resources need to be unique within a namespace, but not across namespaces. Namespaces can not be nested inside one another and each Kubernetes resource can only be in one namespace.
|
||||
- **Resource quotas:** Namespaces provide a way to divide cluster resources between multiple users.
|
||||
|
||||
You can assign resources at the project level so that each namespace in the project can use them. You can also bypass this inheritance by assigning resources explicitly to a namespace.
|
||||
|
||||
You can assign the following resources directly to namespaces:
|
||||
|
||||
- [Workloads]({{<baseurl>}}/rancher/v2.6/en/k8s-in-rancher/workloads/)
|
||||
- [Load Balancers/Ingress]({{<baseurl>}}/rancher/v2.6/en/k8s-in-rancher/load-balancers-and-ingress/)
|
||||
- [Service Discovery Records]({{<baseurl>}}/rancher/v2.6/en/k8s-in-rancher/service-discovery/)
|
||||
- [Persistent Volume Claims]({{<baseurl>}}/rancher/v2.6/en/cluster-admin/volumes-and-storage/)
|
||||
- [Certificates]({{<baseurl>}}/rancher/v2.6/en/k8s-in-rancher/certificates/)
|
||||
- [ConfigMaps]({{<baseurl>}}/rancher/v2.6/en/k8s-in-rancher/configmaps/)
|
||||
- [Registries]({{<baseurl>}}/rancher/v2.6/en/k8s-in-rancher/registries/)
|
||||
- [Secrets]({{<baseurl>}}/rancher/v2.6/en/k8s-in-rancher/secrets/)
|
||||
|
||||
To manage permissions in a vanilla Kubernetes cluster, cluster admins configure role-based access policies for each namespace. With Rancher, user permissions are assigned on the project level instead, and permissions are automatically inherited by any namespace owned by the particular project.
|
||||
|
||||
For more information on creating and moving namespaces, see [Namespaces]({{<baseurl>}}/rancher/v2.6/en/project-admin/namespaces/).
|
||||
|
||||
### Role-based access control issues with namespaces and kubectl
|
||||
|
||||
Because projects are a concept introduced by Rancher, kubectl does not have the capability to restrict the creation of namespaces to a project the creator has access to.
|
||||
|
||||
This means that when standard users with project-scoped permissions create a namespaces with `kubectl`, it may be unusable because `kubectl` doesn't require the new namespace to be scoped within a certain project.
|
||||
|
||||
If your permissions are restricted to the project level, it is better to [create a namespace through Rancher]({{<baseurl>}}/rancher/v2.6/en/project-admin/namespaces/) to ensure that you will have permission to access the namespace.
|
||||
|
||||
If a standard user is a project owner, the user will be able to create namespaces within that project. The Rancher UI will prevent that user from creating namespaces outside the scope of the projects they have access to.
|
||||
|
||||
# About Projects
|
||||
|
||||
In terms of hierarchy:
|
||||
|
||||
- Clusters contain projects
|
||||
- Projects contain namespaces
|
||||
|
||||
You can use projects to support multi-tenancy, so that a team can access a project within a cluster without having access to other projects in the same cluster.
|
||||
|
||||
In the base version of Kubernetes, features like role-based access rights or cluster resources are assigned to individual namespaces. A project allows you to save time by giving an individual or a team access to multiple namespaces simultaneously.
|
||||
|
||||
You can use projects to perform actions such as:
|
||||
|
||||
- Assign users to a group of namespaces (i.e., [project membership]({{<baseurl>}}/rancher/v2.6/en/admin-settings/rbac/cluster-project-roles)).
|
||||
- Assign users specific roles in a project. A role can be owner, member, read-only, or [custom]({{<baseurl>}}/rancher/v2.6/en/admin-settings/rbac/default-custom-roles/).
|
||||
- Assign resources to the project.
|
||||
- Assign Pod Security Policies.
|
||||
|
||||
When you create a cluster, two projects are automatically created within it:
|
||||
|
||||
- [Default Project](#the-cluster-s-default-project)
|
||||
- [System Project](#the-system-project)
|
||||
|
||||
### The Cluster's Default Project
|
||||
|
||||
When you provision a cluster with Rancher, it automatically creates a `default` project for the cluster. This is a project you can use to get started with your cluster, but you can always delete it and replace it with projects that have more descriptive names.
|
||||
|
||||
If you don't have a need for more than the default namespace, you also do not need more than the **Default** project in Rancher.
|
||||
|
||||
If you require another level of organization beyond the **Default** project, you can create more projects in Rancher to isolate namespaces, applications and resources.
|
||||
|
||||
### The System Project
|
||||
|
||||
When troubleshooting, you can view the `system` project to check if important namespaces in the Kubernetes system are working properly. This easily accessible project saves you from troubleshooting individual system namespace containers.
|
||||
|
||||
To open it, open the cluster view and click **Cluster > Projects/Namespaces**. This view shows all of the namespaces in the `system` project.
|
||||
|
||||
The `system` project:
|
||||
|
||||
- Is automatically created when you provision a cluster.
|
||||
- Lists all namespaces that exist in `v3/settings/system-namespaces`, if they exist.
|
||||
- Allows you to add more namespaces or move its namespaces to other projects.
|
||||
- Cannot be deleted because it's required for cluster operations.
|
||||
|
||||
>**Note:** In RKE clusters where the project network isolation option is enabled, the `system` project overrides the project network isolation option so that it can communicate with other projects, collect logs, and check health.
|
||||
|
||||
# Project Authorization
|
||||
|
||||
Standard users are only authorized for project access in two situations:
|
||||
|
||||
- An administrator, cluster owner or cluster member explicitly adds the standard user to the project's **Members** tab.
|
||||
- Standard users can access projects that they create themselves.
|
||||
|
||||
# Pod Security Policies
|
||||
|
||||
Rancher extends Kubernetes to allow the application of [Pod Security Policies](https://kubernetes.io/docs/concepts/policy/pod-security-policy/) at the project level in addition to the cluster level. However, as a best practice, we recommend applying Pod Security Policies at the cluster level.
|
||||
|
||||
# Creating Projects
|
||||
|
||||
This section describes how to create a new project with a name and with optional pod security policy, members, and resource quotas.
|
||||
|
||||
1. [Name a new project.](#1-name-a-new-project)
|
||||
2. [Optional: Select a pod security policy.](#2-optional-select-a-pod-security-policy)
|
||||
3. [Recommended: Add project members.](#3-recommended-add-project-members)
|
||||
4. [Optional: Add resource quotas.](#4-optional-add-resource-quotas)
|
||||
|
||||
### 1. Name a New Project
|
||||
|
||||
1. In the upper left corner, click **☰ > Cluster Management**.
|
||||
1. On the **Clusters** page, go to the cluster you want to project in and click **Explore**.
|
||||
1. Click **Cluster > Projects/Namespaces**.
|
||||
1. Click **Create Project**.
|
||||
1. Enter a **Project Name**.
|
||||
|
||||
### 2. Optional: Select a Pod Security Policy
|
||||
|
||||
This option is only available if you've already created a Pod Security Policy. For instruction, see [Creating Pod Security Policies]({{<baseurl>}}/rancher/v2.6/en/admin-settings/pod-security-policies/).
|
||||
|
||||
Assigning a PSP to a project will:
|
||||
|
||||
- Override the cluster's default PSP.
|
||||
- Apply the PSP to the project.
|
||||
- Apply the PSP to any namespaces you add to the project later.
|
||||
|
||||
### 3. Recommended: Add Project Members
|
||||
|
||||
Use the **Members** section to provide other users with project access and roles.
|
||||
|
||||
By default, your user is added as the project `Owner`.
|
||||
|
||||
>**Notes on Permissions:**
|
||||
>
|
||||
>- Users assigned the `Owner` or `Member` role for a project automatically inherit the `namespace creation` role. However, this role is a [Kubernetes ClusterRole](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#role-and-clusterrole), meaning its scope extends to all projects in the cluster. Therefore, users explicitly assigned the `Owner` or `Member` role for a project can create namespaces in other projects they're assigned to, even with only the `Read Only` role assigned.
|
||||
>
|
||||
>- By default, the Rancher role of `project-member` inherits from the `Kubernetes-edit` role, and the `project-owner` role inherits from the `Kubernetes-admin` role. As such, both `project-member` and `project-owner` roles will allow for namespace management, including the ability to create and delete namespaces.
|
||||
>
|
||||
>- Choose `Custom` to create a custom role on the fly: [Custom Project Roles]({{<baseurl>}}/rancher/v2.6/en/admin-settings/rbac/cluster-project-roles/#custom-project-roles).
|
||||
|
||||
To add members:
|
||||
|
||||
1. In the **Members** tab, click **Add**.
|
||||
1. From the **Select Member** field, search for a user or group that you want to assign project access. Note: You can only search for groups if external authentication is enabled.
|
||||
1. In the **Project Permissions** section, choose a role. For more information, refer to the [documentation on project roles.]({{<baseurl>}}/rancher/v2.6/en/admin-settings/rbac/cluster-project-roles/)
|
||||
|
||||
### 4. Optional: Add Resource Quotas
|
||||
|
||||
Resource quotas limit the resources that a project (and its namespaces) can consume. For more information, see [Resource Quotas]({{<baseurl>}}/rancher/v2.6/en/cluster-admin/projects-and-namespaces/).
|
||||
|
||||
To add a resource quota,
|
||||
|
||||
1. In the **Resource Quotas** tab, click **Add Resource**.
|
||||
1. Select a **Resource Type**. For more information, see [Resource Quotas.]({{<baseurl>}}/rancher/v2.6/en/cluster-admin/projects-and-namespaces/).
|
||||
1. Enter values for the **Project Limit** and the **Namespace Default Limit**.
|
||||
1. **Optional:** Specify **Container Default Resource Limit**, which will be applied to every container started in the project. The parameter is recommended if you have CPU or Memory limits set by the Resource Quota. It can be overridden on per an individual namespace or a container level. For more information, see [Container Default Resource Limit]({{<baseurl>}}/rancher/v2.6/en/project-admin/resource-quotas/)
|
||||
1. Click **Create**.
|
||||
|
||||
**Result:** Your project is created. You can view it from the cluster's **Projects/Namespaces** view.
|
||||
|
||||
| Field | Description |
|
||||
| ----------------------- | -------------------------------------------------------------------------------------------------------- |
|
||||
| Project Limit | The overall resource limit for the project. |
|
||||
| Namespace Default Limit | The default resource limit available for each namespace. This limit is propagated to each namespace in the project when created. The combined limit of all project namespaces shouldn't exceed the project limit. |
|
||||
@@ -0,0 +1,81 @@
|
||||
---
|
||||
title: Restoring a Cluster from Backup
|
||||
weight: 2050
|
||||
---
|
||||
|
||||
etcd backup and recovery for [Rancher launched Kubernetes clusters]({{<baseurl>}}/rancher/v2.6/en/cluster-provisioning/rke-clusters/) can be easily performed. Snapshots of the etcd database are taken and saved either locally onto the etcd nodes or to a S3 compatible target. The advantages of configuring S3 is that if all etcd nodes are lost, your snapshot is saved remotely and can be used to restore the cluster.
|
||||
|
||||
Rancher recommends enabling the [ability to set up recurring snapshots of etcd]({{<baseurl>}}/rancher/v2.6/en/cluster-admin/backing-up-etcd/#configuring-recurring-snapshots), but [one-time snapshots]({{<baseurl>}}/rancher/v2.6/en/cluster-admin/backing-up-etcd/#one-time-snapshots) can easily be taken as well. Rancher allows restore from [saved snapshots](#restoring-a-cluster-from-a-snapshot) or if you don't have any snapshots, you can still [restore etcd](#recovering-etcd-without-a-snapshot).
|
||||
|
||||
Clusters can also be restored to a prior Kubernetes version and cluster configuration.
|
||||
|
||||
This section covers the following topics:
|
||||
|
||||
- [Viewing Available Snapshots](#viewing-available-snapshots)
|
||||
- [Restoring a Cluster from a Snapshot](#restoring-a-cluster-from-a-snapshot)
|
||||
- [Recovering etcd without a Snapshot](#recovering-etcd-without-a-snapshot)
|
||||
- [Enabling snapshot features for clusters created before Rancher v2.2.0](#enabling-snapshot-features-for-clusters-created-before-rancher-v2-2-0)
|
||||
|
||||
## Viewing Available Snapshots
|
||||
|
||||
The list of all available snapshots for the cluster is available.
|
||||
|
||||
1. In the upper left corner, click **☰ > Cluster Management**.
|
||||
1. In the **Clusters** page, go to the cluster where you want to view the snapshots and click the name of the cluster.
|
||||
1. Click the **Snapshots** tab. The listed snapshots include a timestamp of when they were created.
|
||||
|
||||
## Restoring a Cluster from a Snapshot
|
||||
|
||||
If your Kubernetes cluster is broken, you can restore the cluster from a snapshot.
|
||||
|
||||
Snapshots are composed of the cluster data in etcd, the Kubernetes version, and the cluster configuration in the `cluster.yml.` These components allow you to select from the following options when restoring a cluster from a snapshot:
|
||||
|
||||
- **Restore just the etcd contents:** This restore is similar to restoring to snapshots in Rancher before v2.4.0.
|
||||
- **Restore etcd and Kubernetes version:** This option should be used if a Kubernetes upgrade is the reason that your cluster is failing, and you haven't made any cluster configuration changes.
|
||||
- **Restore etcd, Kubernetes versions and cluster configuration:** This option should be used if you changed both the Kubernetes version and cluster configuration when upgrading.
|
||||
|
||||
When rolling back to a prior Kubernetes version, the [upgrade strategy options]({{<baseurl>}}/rancher/v2.6/en/cluster-admin/upgrading-kubernetes/#configuring-the-upgrade-strategy) are ignored. Worker nodes are not cordoned or drained before being reverted to the older Kubernetes version, so that an unhealthy cluster can be more quickly restored to a healthy state.
|
||||
|
||||
> **Prerequisite:** To restore snapshots from S3, the cluster needs to be configured to [take recurring snapshots on S3.]({{<baseurl>}}/rancher/v2.6/en/cluster-admin/backing-up-etcd/#configuring-recurring-snapshots)
|
||||
|
||||
1. In the upper left corner, click **☰ > Cluster Management**.
|
||||
1. In the **Clusters** page, go to the cluster where you want to view the snapshots and click the name of the cluster.
|
||||
1. Click the **Snapshots** tab to view the list of saved snapshots.
|
||||
1. Go to the snapshot you want to restore and click **⋮ > Restore Snapshot**.
|
||||
1. Click **Restore**.
|
||||
|
||||
**Result:** The cluster will go into `updating` state and the process of restoring the `etcd` nodes from the snapshot will start. The cluster is restored when it returns to an `active` state.
|
||||
|
||||
## Recovering etcd without a Snapshot
|
||||
|
||||
If the group of etcd nodes loses quorum, the Kubernetes cluster will report a failure because no operations, e.g. deploying workloads, can be executed in the Kubernetes cluster. The cluster should have three etcd nodes to prevent a loss of quorum. If you want to recover your set of etcd nodes, follow these instructions:
|
||||
|
||||
1. Keep only one etcd node in the cluster by removing all other etcd nodes.
|
||||
|
||||
2. On the single remaining etcd node, run the following command:
|
||||
|
||||
```
|
||||
$ docker run --rm -v /var/run/docker.sock:/var/run/docker.sock assaflavie/runlike etcd
|
||||
```
|
||||
|
||||
This command outputs the running command for etcd, save this command to use later.
|
||||
|
||||
3. Stop the etcd container that you launched in the previous step and rename it to `etcd-old`.
|
||||
|
||||
```
|
||||
$ docker stop etcd
|
||||
$ docker rename etcd etcd-old
|
||||
```
|
||||
|
||||
4. Take the saved command from Step 2 and revise it:
|
||||
|
||||
- If you originally had more than 1 etcd node, then you need to change `--initial-cluster` to only contain the node that remains.
|
||||
- Add `--force-new-cluster` to the end of the command.
|
||||
|
||||
5. Run the revised command.
|
||||
|
||||
6. After the single nodes is up and running, Rancher recommends adding additional etcd nodes to your cluster. If you have a [custom cluster]({{<baseurl>}}/rancher/v2.6/en/cluster-provisioning/rke-clusters/custom-nodes) and you want to reuse an old node, you are required to [clean up the nodes]({{<baseurl>}}/rancher/v2.6/en/cluster-admin/cleaning-cluster-nodes/) before attempting to add them back into a cluster.
|
||||
|
||||
# Enabling Snapshot Features for Clusters Created Before Rancher v2.2.0
|
||||
|
||||
If you have any Rancher launched Kubernetes clusters that were created before v2.2.0, after upgrading Rancher, you must [edit the cluster]({{<baseurl>}}/rancher/v2.6/en/cluster-admin/editing-clusters/) and _save_ it, in order to enable the updated snapshot features. Even if you were already creating snapshots before v2.2.0, you must do this step as the older snapshots will not be available to use to [back up and restore etcd through the UI]({{<baseurl>}}/rancher/v2.6/en/cluster-admin/restoring-etcd/).
|
||||
@@ -0,0 +1,59 @@
|
||||
---
|
||||
title: Tools for Logging, Monitoring, and Visibility
|
||||
weight: 2033
|
||||
---
|
||||
|
||||
Rancher contains a variety of tools that aren't included in Kubernetes to assist in your DevOps operations. Rancher can integrate with external services to help your clusters run more efficiently. Tools are divided into following categories:
|
||||
|
||||
<!-- TOC -->
|
||||
|
||||
- [Logging](#logging)
|
||||
- [Monitoring and Alerts](#monitoring-and-alerts)
|
||||
- [Istio](#istio)
|
||||
- [OPA Gatekeeper](#opa-gatekeeper)
|
||||
- [CIS Scans](#cis-scans)
|
||||
|
||||
<!-- /TOC -->
|
||||
|
||||
|
||||
# Logging
|
||||
|
||||
Logging is helpful because it allows you to:
|
||||
|
||||
- Capture and analyze the state of your cluster
|
||||
- Look for trends in your environment
|
||||
- Save your logs to a safe location outside of your cluster
|
||||
- Stay informed of events like a container crashing, a pod eviction, or a node dying
|
||||
- More easily debugg and troubleshoot problems
|
||||
|
||||
Rancher can integrate with Elasticsearch, splunk, kafka, syslog, and fluentd.
|
||||
|
||||
For more information, refer to the logging documentation [here.]({{<baseurl>}}/rancher/v2.6/en/logging/)
|
||||
# Monitoring and Alerts
|
||||
|
||||
Using Rancher, you can monitor the state and processes of your cluster nodes, Kubernetes components, and software deployments through integration with [Prometheus](https://prometheus.io/), a leading open-source monitoring solution.
|
||||
|
||||
After monitoring is enabled, you can set up alerts and notifiers that provide the mechanism to receive them.
|
||||
|
||||
Notifiers are services that inform you of alert events. You can configure notifiers to send alert notifications to staff best suited to take corrective action. Notifications can be sent with Slack, email, PagerDuty, WeChat, and webhooks.
|
||||
|
||||
Alerts are rules that trigger those notifications. Before you can receive alerts, you must configure one or more notifier in Rancher. The scope for alerts can be set at either the cluster or project level.
|
||||
|
||||
For more information, refer to the monitoring documentation [here.]({{<baseurl>}}/rancher/v2.6/en/monitoring-alerting/)
|
||||
|
||||
# Istio
|
||||
|
||||
[Istio](https://istio.io/) is an open-source tool that makes it easier for DevOps teams to observe, control, troubleshoot, and secure the traffic within a complex network of microservices.
|
||||
|
||||
Rancher's integration with Istio was improved in Rancher v2.5.
|
||||
|
||||
For more information, refer to the Istio documentation [here.]({{<baseurl>}}/rancher/v2.6/en/istio)
|
||||
# OPA Gatekeeper
|
||||
|
||||
[OPA Gatekeeper](https://github.com/open-policy-agent/gatekeeper) is an open-source project that provides integration between OPA and Kubernetes to provide policy control via admission controller webhooks. For details on how to enable Gatekeeper in Rancher, refer to the [OPA Gatekeeper section.]({{<baseurl>}}/rancher/v2.6/en/opa-gatekeper)
|
||||
|
||||
# CIS Scans
|
||||
|
||||
Rancher can run a security scan to check whether Kubernetes is deployed according to security best practices as defined in the CIS Kubernetes Benchmark.
|
||||
|
||||
For more information, refer to the CIS scan documentation [here.]({{<baseurl>}}/rancher/v2.6/en/cis-scans)
|
||||
+134
@@ -0,0 +1,134 @@
|
||||
---
|
||||
title: Upgrading and Rolling Back Kubernetes
|
||||
weight: 70
|
||||
---
|
||||
|
||||
Following an upgrade to the latest version of Rancher, downstream Kubernetes clusters can be upgraded to use the latest supported version of Kubernetes.
|
||||
|
||||
Rancher calls RKE (Rancher Kubernetes Engine) as a library when provisioning and editing RKE clusters. For more information on configuring the upgrade strategy for RKE clusters, refer to the [RKE documentation]({{<baseurl>}}/rke/latest/en/).
|
||||
|
||||
This section covers the following topics:
|
||||
|
||||
- [New Features](#new-features)
|
||||
- [Tested Kubernetes Versions](#tested-kubernetes-versions)
|
||||
- [How Upgrades Work](#how-upgrades-work)
|
||||
- [Recommended Best Practice for Upgrades](#recommended-best-practice-for-upgrades)
|
||||
- [Upgrading the Kubernetes Version](#upgrading-the-kubernetes-version)
|
||||
- [Rolling Back](#rolling-back)
|
||||
- [Configuring the Upgrade Strategy](#configuring-the-upgrade-strategy)
|
||||
- [Configuring the Maximum Unavailable Worker Nodes in the Rancher UI](#configuring-the-maximum-unavailable-worker-nodes-in-the-rancher-ui)
|
||||
- [Enabling Draining Nodes During Upgrades from the Rancher UI](#enabling-draining-nodes-during-upgrades-from-the-rancher-ui)
|
||||
- [Maintaining Availability for Applications During Upgrades](#maintaining-availability-for-applications-during-upgrades)
|
||||
- [Configuring the Upgrade Strategy in the cluster.yml](#configuring-the-upgrade-strategy-in-the-cluster-yml)
|
||||
- [Troubleshooting](#troubleshooting)
|
||||
|
||||
# Tested Kubernetes Versions
|
||||
|
||||
Before a new version of Rancher is released, it's tested with the latest minor versions of Kubernetes to ensure compatibility. For details on which versions of Kubernetes were tested on each Rancher version, refer to the [support maintenance terms.](https://rancher.com/support-maintenance-terms/all-supported-versions/rancher-v2.6.0/)
|
||||
|
||||
# How Upgrades Work
|
||||
|
||||
RKE v1.1.0 changed the way that clusters are upgraded.
|
||||
|
||||
In this section of the [RKE documentation,]({{<baseurl>}}/rke/latest/en/upgrades/how-upgrades-work) you'll learn what happens when you edit or upgrade your RKE Kubernetes cluster.
|
||||
|
||||
|
||||
# Recommended Best Practice for Upgrades
|
||||
|
||||
When upgrading the Kubernetes version of a cluster, we recommend that you:
|
||||
|
||||
1. Take a snapshot.
|
||||
1. Initiate a Kubernetes upgrade.
|
||||
1. If the upgrade fails, revert the cluster to the pre-upgrade Kubernetes version. This is achieved by selecting the **Restore etcd and Kubernetes version** option. This will return your cluster to the pre-upgrade kubernetes version before restoring the etcd snapshot.
|
||||
|
||||
The restore operation will work on a cluster that is not in a healthy or active state.
|
||||
|
||||
# Upgrading the Kubernetes Version
|
||||
|
||||
> **Prerequisites:**
|
||||
>
|
||||
> - The options below are available only for [Rancher-launched RKE Kubernetes clusters]({{<baseurl>}}/rancher/v2.6/en/cluster-provisioning/rke-clusters/) and [Registered K3s Kubernetes clusters.]({{<baseurl>}}/rancher/v2.6/en/cluster-provisioning/registered-clusters/#additional-features-for-registered-k3s-clusters)
|
||||
> - Before upgrading Kubernetes, [back up your cluster.]({{<baseurl>}}/rancher/v2.6/en/backups)
|
||||
|
||||
1. In the upper left corner, click **☰ > Cluster Management**.
|
||||
1. On the **Clusters** page, go to the cluster you want to upgrade and click **⋮ > Edit Config**.
|
||||
1. From the **Kubernetes Version** drop-down, choose the version of Kubernetes that you want to use for the cluster.
|
||||
1. Click **Save**.
|
||||
|
||||
**Result:** Kubernetes begins upgrading for the cluster.
|
||||
|
||||
# Rolling Back
|
||||
|
||||
A cluster can be restored to a backup in which the previous Kubernetes version was used. For more information, refer to the following sections:
|
||||
|
||||
- [Backing up a cluster]({{<baseurl>}}/rancher/v2.6/en/cluster-admin/backing-up-etcd/#how-snapshots-work)
|
||||
- [Restoring a cluster from backup]({{<baseurl>}}/rancher/v2.6/en/cluster-admin/restoring-etcd/#restoring-a-cluster-from-a-snapshot)
|
||||
|
||||
# Configuring the Upgrade Strategy
|
||||
|
||||
As of RKE v1.1.0, additional upgrade options became available to give you more granular control over the upgrade process. These options can be used to maintain availability of your applications during a cluster upgrade if certain [conditions and requirements]({{<baseurl>}}/rke/latest/en/upgrades/maintaining-availability) are met.
|
||||
|
||||
The upgrade strategy can be configured in the Rancher UI, or by editing the `cluster.yml`. More advanced options are available by editing the `cluster.yml`.
|
||||
|
||||
### Configuring the Maximum Unavailable Worker Nodes in the Rancher UI
|
||||
|
||||
From the Rancher UI, the maximum number of unavailable worker nodes can be configured. During a cluster upgrade, worker nodes will be upgraded in batches of this size.
|
||||
|
||||
By default, the maximum number of unavailable worker is defined as 10 percent of all worker nodes. This number can be configured as a percentage or as an integer. When defined as a percentage, the batch size is rounded down to the nearest node, with a minimum of one node.
|
||||
|
||||
To change the default number or percentage of worker nodes,
|
||||
|
||||
1. In the upper left corner, click **☰ > Cluster Management**.
|
||||
1. On the **Clusters** page, go to the cluster you want to upgrade and click **⋮ > Edit Config**.
|
||||
1. In the **Upgrade Strategy** tab, enter the **Worker Concurrency** as a fixed number or percentage. To get this number, you can take the number of nodes in your cluster and subtract the max unavailable nodes.
|
||||
1. Click **Save**.
|
||||
|
||||
**Result:** The cluster is updated to use the new upgrade strategy.
|
||||
|
||||
### Enabling Draining Nodes During Upgrades from the Rancher UI
|
||||
|
||||
By default, RKE [cordons](https://kubernetes.io/docs/concepts/architecture/nodes/#manual-node-administration) each node before upgrading it. [Draining](https://kubernetes.io/docs/tasks/administer-cluster/safely-drain-node/) is disabled during upgrades by default. If draining is enabled in the cluster configuration, RKE will both cordon and drain the node before it is upgraded.
|
||||
|
||||
To enable draining each node during a cluster upgrade,
|
||||
|
||||
1. In the upper left corner, click **☰ > Cluster Management**.
|
||||
1. On the **Clusters** page, go to the cluster you want to enable node draining and click **⋮ > Edit Config**.
|
||||
1. Click **⋮ > Edit**.
|
||||
1. In the **Upgrade Strategy** tab, go to the **Drain nodes** field and click **Yes**. Node draining is configured separately for control plane and worker nodes.
|
||||
1. Configure the options for how pods are deleted. For more information about each option, refer to [this section.]({{<baseurl>}}/rancher/v2.6/en/cluster-admin/nodes/#aggressive-and-safe-draining-options)
|
||||
1. Optionally, configure a grace period. The grace period is the timeout given to each pod for cleaning things up, so they will have chance to exit gracefully. Pods might need to finish any outstanding requests, roll back transactions or save state to some external storage. If this value is negative, the default value specified in the pod will be used.
|
||||
1. Optionally, configure a timeout, which is the amount of time the drain should continue to wait before giving up.
|
||||
1. Click **Save**.
|
||||
|
||||
**Result:** The cluster is updated to use the new upgrade strategy.
|
||||
|
||||
> **Note:** As of Rancher v2.4.0, there is a [known issue](https://github.com/rancher/rancher/issues/25478) in which the Rancher UI doesn't show state of etcd and controlplane as drained, even though they are being drained.
|
||||
|
||||
### Maintaining Availability for Applications During Upgrades
|
||||
|
||||
_Available as of RKE v1.1.0_
|
||||
|
||||
In [this section of the RKE documentation,]({{<baseurl>}}/rke/latest/en/upgrades/maintaining-availability/) you'll learn the requirements to prevent downtime for your applications when upgrading the cluster.
|
||||
|
||||
### Configuring the Upgrade Strategy in the cluster.yml
|
||||
|
||||
More advanced upgrade strategy configuration options are available by editing the `cluster.yml`.
|
||||
|
||||
For details, refer to [Configuring the Upgrade Strategy]({{<baseurl>}}/rke/latest/en/upgrades/configuring-strategy) in the RKE documentation. The section also includes an example `cluster.yml` for configuring the upgrade strategy.
|
||||
|
||||
# Troubleshooting
|
||||
|
||||
If a node doesn't come up after an upgrade, the `rke up` command errors out.
|
||||
|
||||
No upgrade will proceed if the number of unavailable nodes exceeds the configured maximum.
|
||||
|
||||
If an upgrade stops, you may need to fix an unavailable node or remove it from the cluster before the upgrade can continue.
|
||||
|
||||
A failed node could be in many different states:
|
||||
|
||||
- Powered off
|
||||
- Unavailable
|
||||
- User drains a node while upgrade is in process, so there are no kubelets on the node
|
||||
- The upgrade itself failed
|
||||
|
||||
If the max unavailable number of nodes is reached during an upgrade, Rancher user clusters will be stuck in updating state and not move forward with upgrading any other control plane nodes. It will continue to evaluate the set of unavailable nodes in case one of the nodes becomes available. If the node cannot be fixed, you must remove the node in order to continue the upgrade.
|
||||
+82
@@ -0,0 +1,82 @@
|
||||
---
|
||||
title: Setting up Existing Storage
|
||||
weight: 1
|
||||
---
|
||||
|
||||
This section describes how to set up existing persistent storage for workloads in Rancher.
|
||||
|
||||
> This section assumes that you understand the Kubernetes concepts of persistent volumes and persistent volume claims. For more information, refer to the section on [how storage works.](../how-storage-works)
|
||||
|
||||
To set up storage, follow these steps:
|
||||
|
||||
1. [Set up persistent storage.](#1-set-up-persistent-storage)
|
||||
2. [Add a PersistentVolume that refers to the persistent storage.](#2-add-a-persistentvolume-that-refers-to-the-persistent-storage)
|
||||
3. [Use the PersistentVolume for Pods Deployed with a StatefulSet.](#3-use-the-persistentvolume-for-pods-deployed-with-a-statefulset)
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- To create a persistent volume as a Kubernetes resource, you must have the `Manage Volumes` [role.]({{<baseurl>}}/rancher/v2.6/en/admin-settings/rbac/cluster-project-roles/#project-role-reference)
|
||||
- If you are provisioning storage for a cluster hosted in the cloud, the storage and cluster hosts must have the same cloud provider.
|
||||
|
||||
### 1. Set up persistent storage
|
||||
|
||||
Creating a persistent volume in Rancher will not create a storage volume. It only creates a Kubernetes resource that maps to an existing volume. Therefore, before you can create a persistent volume as a Kubernetes resource, you must have storage provisioned.
|
||||
|
||||
The steps to set up a persistent storage device will differ based on your infrastructure. We provide examples of how to set up storage using [vSphere,](../examples/vsphere) [NFS,](../examples/nfs) or Amazon's [EBS.](../examples/ebs)
|
||||
|
||||
If you have a pool of block storage, and you don't want to use a cloud provider, Longhorn could help you provide persistent storage to your Kubernetes cluster. For more information, see [this page.]({{<baseurl>}}/rancher/v2.6/en/longhorn)
|
||||
|
||||
### 2. Add a PersistentVolume that refers to the persistent storage
|
||||
|
||||
These steps describe how to set up a PersistentVolume at the cluster level in Kubernetes.
|
||||
|
||||
1. Click **☰ > Cluster Management**.
|
||||
1. Go to the cluster where you want to add a persistent volume and click **Explore**.
|
||||
1. In the left navigation bar, click **Storage > Persistent Volumes**.
|
||||
1. Click **Create**.
|
||||
1. Enter a **Name** for the persistent volume.
|
||||
1. Select the **Volume Plugin** for the disk type or service that you're using. When adding storage to a cluster that's hosted by a cloud provider, use the cloud provider's plug-in for cloud storage. For example, if you have a Amazon EC2 cluster and you want to use cloud storage for it, you must use the `Amazon EBS Disk` volume plugin.
|
||||
1. Enter the **Capacity** of your volume in gigabytes.
|
||||
1. Complete the **Plugin Configuration** form. Each plugin type requires information specific to the vendor of disk type. For help regarding each plugin's form and the information that's required, refer to the plug-in's vendor documentation.
|
||||
1. Optional: In the **Customize** form, configure the [access modes.](https://kubernetes.io/docs/concepts/storage/persistent-volumes/#access-modes) This options sets how many nodes can access the volume, along with the node read/write permissions. The [Kubernetes Documentation](https://kubernetes.io/docs/concepts/storage/persistent-volumes/#access-modes) includes a table that lists which access modes are supported by the plugins available.
|
||||
1. Optional: In the **Customize** form, configure the [mount options.](https://kubernetes.io/docs/concepts/storage/persistent-volumes/#mount-options) Each volume plugin allows you to specify additional command line options during the mounting process. Consult each plugin's vendor documentation for the mount options available.
|
||||
1. Click **Create**.
|
||||
|
||||
**Result:** Your new persistent volume is created.
|
||||
|
||||
|
||||
### 3. Use the Storage Class for Pods Deployed with a StatefulSet
|
||||
|
||||
StatefulSets manage the deployment and scaling of Pods while maintaining a sticky identity for each Pod. In this StatefulSet, we will configure a VolumeClaimTemplate. Each Pod managed by the StatefulSet will be deployed with a PersistentVolumeClaim based on this VolumeClaimTemplate. The PersistentVolumeClaim will refer to the PersistentVolume that we created. Therefore, when each Pod managed by the StatefulSet is deployed, it will be bound a PersistentVolume as defined in its PersistentVolumeClaim.
|
||||
|
||||
You can configure storage for the StatefulSet during or after workload creation.
|
||||
|
||||
The following steps describe how to assign existing storage to a new StatefulSet:
|
||||
|
||||
1. Click **☰ > Cluster Management**.
|
||||
1. Go to the cluster where you want to configure storage for the StatefulSet and click **Explore**.
|
||||
1. In the left navigation bar, click **Workload > StatefulSets**.
|
||||
1. Click **Create**.
|
||||
1. Choose the namespace where the workload will be deployed.
|
||||
1. Enter a name for the StatefulSet.
|
||||
1. On the **Volume Claim Templates** tab, click **Add Claim Template**.
|
||||
1. Click **Use an existing Persistent Volume**.
|
||||
1. In the Persistent Volumes field, select the Persistent Volume that you created.
|
||||
1. In the **Mount Point** field, enter the path that the workload will use to access the volume.
|
||||
1. Click **Launch**.
|
||||
|
||||
**Result:** When the workload is deployed, it will make a request for the specified amount of disk space to the Kubernetes master. If a PV with the specified resources is available when the workload is deployed, the Kubernetes master will bind the PV to the PVC.
|
||||
|
||||
The following steps describe how to assign persistent storage to an existing workload:
|
||||
|
||||
1. Click **☰ > Cluster Management**.
|
||||
1. Go to the cluster where you want to configure storage for the StatefulSet and click **Explore**.
|
||||
1. In the left navigation bar, click **Workload > StatefulSets**.
|
||||
1. Go to the workload that you want to add the persistent storage to. Click **⋮ > Edit**.
|
||||
1. On the **Volume Claim Templates** tab, click **Add Claim Template**.
|
||||
1. Click **Use an existing Persistent Volume**.
|
||||
1. In the Persistent Volumes field, select the Persistent Volume that you created.
|
||||
1. In the **Mount Point** field, enter the path that the workload will use to access the volume.
|
||||
1. Click **Launch**.
|
||||
|
||||
**Result:** The workload will make a request for the specified amount of disk space to the Kubernetes master. If a PV with the specified resources is available when the workload is deployed, the Kubernetes master will bind the PV to the PVC.
|
||||
@@ -0,0 +1,433 @@
|
||||
---
|
||||
title: Using an External Ceph Driver
|
||||
weight: 10
|
||||
---
|
||||
|
||||
These instructions are about using the external Ceph driver in an RKE2 cluster. If you are using RKE, additional steps are required. For details, refer to [this section.](#using-the-ceph-driver-with-rke)
|
||||
|
||||
- [Requirements](#requirements)
|
||||
- [Using the Ceph Driver with RKE](#using-the-ceph-driver-with-rke)
|
||||
- [Installing the ceph-csi driver on an RKE2 cluster](#installing-the-ceph-csi-driver-on-an-rke2-cluster)
|
||||
- [Install the ceph-csi driver using Helm](#install-the-ceph-csi-driver-using-helm)
|
||||
- [Creating RBD Ceph Resources](#creating-rbd-ceph-resources)
|
||||
- [Configure RBD Ceph Access Secrets](#configure-rbd-ceph-access-secrets)
|
||||
- [User Account](#user-account)
|
||||
- [Admin Account](#admin-account)
|
||||
- [Create RBD Testing Resources](#create-rbd-testing-resources)
|
||||
- [Using RBD in Pods](#using-rbd-in-pods)
|
||||
- [Using RBD in Persistent Volumes](#using-rbd-in-persistent-volumes)
|
||||
- [Using RBD in Storage Classes](#using-rbd-in-storage-classes)
|
||||
- [RKE2 Server/Master Provisioning](#rke2-server-master-provisioning)
|
||||
- [RKE2 Agent/Worker provisioning](#rke2-agent-worker-provisioning)
|
||||
- [Tested Versions](#tested-versions)
|
||||
- [Troubleshooting](#troubleshooting)
|
||||
|
||||
# Requirements
|
||||
|
||||
Make sure ceph-common and xfsprogs packages are installed on SLE worker nodes.
|
||||
|
||||
# Using the Ceph Driver with RKE
|
||||
|
||||
The resources below are fully compatible with RKE based clusters, but there is a need to do an additional kubelet configuration for RKE.
|
||||
|
||||
On RKE clusters, the kubelet component is running in a Docker container and doesn't have access to the host's kernel modules as rbd and libceph by default.
|
||||
|
||||
To solve this limitation, you can either run `modprobe rbd` on worker nodes, or configure the kubelet containers to automatically mount the `/lib/modules` directory from the host into the container.
|
||||
|
||||
For the kubelet configuration, put the following lines into the `cluster.yml` file prior to RKE cluster provisioning. You can also modify the `cluster.yml` later in the Rancher UI by clicking on **Edit Cluster > Edit as YAML** and restarting the worker nodes.
|
||||
|
||||
```yaml
|
||||
services:
|
||||
kubelet:
|
||||
extra_binds:
|
||||
- '/lib/modules:/lib/modules:ro'
|
||||
```
|
||||
|
||||
For more information about the `extra_binds` directive, refer to [this section.]({{<baseurl>}}/rke/latest/en/config-options/services/services-extras/#extra-binds)
|
||||
|
||||
# Installing the ceph-csi driver on an RKE2 cluster
|
||||
|
||||
> **Note:** These steps are needed for dynamic RBD provisioning only.
|
||||
|
||||
For more information about the `ceph-csi-rbd` chart, refer to [this page.](https://github.com/ceph/ceph-csi/blob/devel/charts/ceph-csi-rbd/README.md)
|
||||
|
||||
To get details about your SES cluster, run:
|
||||
|
||||
```
|
||||
ceph mon dump
|
||||
```
|
||||
|
||||
Read its output:
|
||||
|
||||
```
|
||||
dumped monmap epoch 3
|
||||
epoch 3
|
||||
fsid 79179d9d-98d8-4976-ab2e-58635caa7235
|
||||
last_changed 2021-02-11T10:56:42.110184+0000
|
||||
created 2021-02-11T10:56:22.913321+0000
|
||||
min_mon_release 15 (octopus)
|
||||
0: [v2:10.85.8.118:3300/0,v1:10.85.8.118:6789/0] mon.a
|
||||
1: [v2:10.85.8.123:3300/0,v1:10.85.8.123:6789/0] mon.b
|
||||
2: [v2:10.85.8.124:3300/0,v1:10.85.8.124:6789/0] mon.c
|
||||
```
|
||||
|
||||
Later you'll need the fsid and mon addresses values.
|
||||
|
||||
# Install the ceph-csi Driver Using Helm
|
||||
|
||||
Run these commands:
|
||||
|
||||
```
|
||||
helm repo add ceph-csi https://ceph.github.io/csi-charts
|
||||
helm repo update
|
||||
helm search repo ceph-csi -l
|
||||
helm inspect values ceph-csi/ceph-csi-rbd > ceph-csi-rbd-values.yaml
|
||||
```
|
||||
|
||||
Modify the `ceph-csi-rbd-values.yaml` file and keep there only the required changes:
|
||||
|
||||
```yaml
|
||||
# ceph-csi-rbd-values.yaml
|
||||
csiConfig:
|
||||
- clusterID: "79179d9d-98d8-4976-ab2e-58635caa7235"
|
||||
monitors:
|
||||
- "10.85.8.118:6789"
|
||||
- "10.85.8.123:6789"
|
||||
- "10.85.8.124:6789"
|
||||
provisioner:
|
||||
name: provisioner
|
||||
replicaCount: 2
|
||||
```
|
||||
|
||||
Make sure the ceph monitors are reachable from the RKE2 cluster, for example, by ping.
|
||||
|
||||
```
|
||||
kubectl create namespace ceph-csi-rbd
|
||||
helm install --namespace ceph-csi-rbd ceph-csi-rbd ceph-csi/ceph-csi-rbd --values ceph-csi-rbd-values.yaml
|
||||
kubectl rollout status deployment ceph-csi-rbd-provisioner -n ceph-csi-rbd
|
||||
helm status ceph-csi-rbd -n ceph-csi-rbd
|
||||
```
|
||||
|
||||
in case you'd like to modify the configuration directly via Helm, you may adapt the `ceph-csi-rbd-values.yaml` file and call:
|
||||
|
||||
```
|
||||
helm upgrade \
|
||||
--namespace ceph-csi-rbd ceph-csi-rbd ceph-csi/ceph-csi-rbd --values ceph-csi-rbd-values.yaml
|
||||
```
|
||||
|
||||
# Creating RBD Ceph Resources
|
||||
|
||||
```
|
||||
# Create a ceph pool:
|
||||
ceph osd pool create myPool 64 64
|
||||
|
||||
# Create a block device pool:
|
||||
rbd pool init myPool
|
||||
|
||||
# Create a block device image:
|
||||
rbd create -s 2G myPool/image
|
||||
|
||||
# Create a block device user and record the key:
|
||||
ceph auth get-or-create-key client.myPoolUser mon "allow r" osd "allow class-read object_prefix rbd_children, allow rwx pool=myPool" | tr -d '\n' | base64
|
||||
QVFDZ0R5VmdyRk9KREJBQTJ5b2s5R1E2NUdSWExRQndhVVBwWXc9PQ==
|
||||
|
||||
# Encode the ceph user myPoolUser into a bash64 hash:
|
||||
echo "myPoolUser" | tr -d '\n' | base64
|
||||
bXlQb29sVXNlcg==
|
||||
|
||||
# Create a block device admin user and record the key:
|
||||
ceph auth get-or-create-key client.myPoolAdmin mds 'allow *' mgr 'allow *' mon 'allow *' osd 'allow * pool=myPool' | tr -d '\n' | base64
|
||||
QVFCK0hDVmdXSjQ1T0JBQXBrc0VtcVhlZFpjc0JwaStIcmU5M3c9PQ==
|
||||
|
||||
# Encode the ceph user myPoolAdmin into a bash64 hash:
|
||||
echo "myPoolAdmin" | tr -d '\n' | base64
|
||||
bXlQb29sQWRtaW4=
|
||||
```
|
||||
# Configure RBD Ceph Access Secrets
|
||||
|
||||
### User Account
|
||||
|
||||
For static RBD provisioning (the image within the ceph pool must exist), run these commands:
|
||||
|
||||
```
|
||||
cat > ceph-user-secret.yaml << EOF
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: ceph-user
|
||||
namespace: default
|
||||
type: kubernetes.io/rbd
|
||||
data:
|
||||
userID: bXlQb29sVXNlcg==
|
||||
userKey: QVFDZ0R5VmdyRk9KREJBQTJ5b2s5R1E2NUdSWExRQndhVVBwWXc9PQ==
|
||||
EOF
|
||||
|
||||
kubectl apply -f ceph-user-secret.yaml
|
||||
```
|
||||
|
||||
### Admin Account
|
||||
|
||||
For dynamic RBD provisioning (used for automatic image creation within a given ceph pool), run these commands:
|
||||
|
||||
```
|
||||
cat > ceph-admin-secret.yaml << EOF
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: ceph-admin
|
||||
namespace: default
|
||||
type: kubernetes.io/rbd
|
||||
data:
|
||||
userID: bXlQb29sQWRtaW4=
|
||||
userKey: QVFCK0hDVmdXSjQ1T0JBQXBrc0VtcVhlZFpjc0JwaStIcmU5M3c9PQ==
|
||||
EOF
|
||||
|
||||
kubectl apply -f ceph-admin-secret.yaml
|
||||
```
|
||||
|
||||
# Create RBD Testing Resources
|
||||
|
||||
### Using RBD in Pods
|
||||
|
||||
```
|
||||
# pod
|
||||
cat > ceph-rbd-pod-inline.yaml << EOF
|
||||
apiVersion: v1
|
||||
kind: Pod
|
||||
metadata:
|
||||
name: ceph-rbd-pod-inline
|
||||
spec:
|
||||
containers:
|
||||
- name: ceph-rbd-pod-inline
|
||||
image: busybox
|
||||
command: ["sleep", "infinity"]
|
||||
volumeMounts:
|
||||
- mountPath: /mnt/ceph_rbd
|
||||
name: volume
|
||||
volumes:
|
||||
- name: volume
|
||||
rbd:
|
||||
monitors:
|
||||
- 10.85.8.118:6789
|
||||
- 10.85.8.123:6789
|
||||
- 10.85.8.124:6789
|
||||
pool: myPool
|
||||
image: image
|
||||
user: myPoolUser
|
||||
secretRef:
|
||||
name: ceph-user
|
||||
fsType: ext4
|
||||
readOnly: false
|
||||
EOF
|
||||
|
||||
kubectl apply -f ceph-rbd-pod-inline.yaml
|
||||
kubectl get pod
|
||||
kubectl exec pod/ceph-rbd-pod-inline -- df -k | grep rbd
|
||||
```
|
||||
|
||||
### Using RBD in Persistent Volumes
|
||||
|
||||
```
|
||||
# pod-pvc-pv
|
||||
cat > ceph-rbd-pod-pvc-pv-allinone.yaml << EOF
|
||||
apiVersion: v1
|
||||
kind: PersistentVolume
|
||||
metadata:
|
||||
name: ceph-rbd-pv
|
||||
spec:
|
||||
capacity:
|
||||
storage: 2Gi
|
||||
accessModes:
|
||||
- ReadWriteOnce
|
||||
rbd:
|
||||
monitors:
|
||||
- 10.85.8.118:6789
|
||||
- 10.85.8.123:6789
|
||||
- 10.85.8.124:6789
|
||||
pool: myPool
|
||||
image: image
|
||||
user: myPoolUser
|
||||
secretRef:
|
||||
name: ceph-user
|
||||
fsType: ext4
|
||||
readOnly: false
|
||||
---
|
||||
kind: PersistentVolumeClaim
|
||||
apiVersion: v1
|
||||
metadata:
|
||||
name: ceph-rbd-pvc
|
||||
spec:
|
||||
accessModes:
|
||||
- ReadWriteOnce
|
||||
resources:
|
||||
requests:
|
||||
storage: 2Gi
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Pod
|
||||
metadata:
|
||||
name: ceph-rbd-pod-pvc-pv
|
||||
spec:
|
||||
containers:
|
||||
- name: ceph-rbd-pod-pvc-pv
|
||||
image: busybox
|
||||
command: ["sleep", "infinity"]
|
||||
volumeMounts:
|
||||
- mountPath: /mnt/ceph_rbd
|
||||
name: volume
|
||||
volumes:
|
||||
- name: volume
|
||||
persistentVolumeClaim:
|
||||
claimName: ceph-rbd-pvc
|
||||
EOF
|
||||
|
||||
kubectl apply -f ceph-rbd-pod-pvc-pv-allinone.yaml
|
||||
kubectl get pv,pvc,pod
|
||||
kubectl exec pod/ceph-rbd-pod-pvc-pv -- df -k | grep rbd
|
||||
```
|
||||
|
||||
### Using RBD in Storage Classes
|
||||
|
||||
This example is for dynamic provisioning. The ceph-csi driver is needed.
|
||||
|
||||
```
|
||||
# pod-pvc-sc
|
||||
cat > ceph-rbd-pod-pvc-sc-allinone.yaml <<EOF
|
||||
apiVersion: storage.k8s.io/v1
|
||||
kind: StorageClass
|
||||
metadata:
|
||||
name: ceph-rbd-sc
|
||||
annotations:
|
||||
storageclass.kubernetes.io/is-default-class: "true"
|
||||
provisioner: rbd.csi.ceph.com
|
||||
parameters:
|
||||
clusterID: 79179d9d-98d8-4976-ab2e-58635caa7235
|
||||
pool: myPool
|
||||
imageFeatures: layering
|
||||
csi.storage.k8s.io/provisioner-secret-name: ceph-admin
|
||||
csi.storage.k8s.io/provisioner-secret-namespace: default
|
||||
csi.storage.k8s.io/controller-expand-secret-name: ceph-admin
|
||||
csi.storage.k8s.io/controller-expand-secret-namespace: default
|
||||
csi.storage.k8s.io/node-stage-secret-name: ceph-admin
|
||||
csi.storage.k8s.io/node-stage-secret-namespace: default
|
||||
reclaimPolicy: Delete
|
||||
allowVolumeExpansion: true
|
||||
mountOptions:
|
||||
- discard
|
||||
---
|
||||
kind: PersistentVolumeClaim
|
||||
apiVersion: v1
|
||||
metadata:
|
||||
name: ceph-rbd-sc-pvc
|
||||
spec:
|
||||
accessModes:
|
||||
- ReadWriteOnce
|
||||
resources:
|
||||
requests:
|
||||
storage: 2Gi
|
||||
storageClassName: ceph-rbd-sc
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Pod
|
||||
metadata:
|
||||
name: ceph-rbd-pod-pvc-sc
|
||||
spec:
|
||||
containers:
|
||||
- name: ceph-rbd-pod-pvc-sc
|
||||
image: busybox
|
||||
command: ["sleep", "infinity"]
|
||||
volumeMounts:
|
||||
- mountPath: /mnt/ceph_rbd
|
||||
name: volume
|
||||
volumes:
|
||||
- name: volume
|
||||
persistentVolumeClaim:
|
||||
claimName: ceph-rbd-sc-pvc
|
||||
EOF
|
||||
|
||||
kubectl apply -f ceph-rbd-pod-pvc-sc-allinone.yaml
|
||||
kubectl get pv,pvc,sc,pod
|
||||
kubectl exec pod/ceph-rbd-pod-pvc-sc -- df -k | grep rbd
|
||||
```
|
||||
|
||||
### RKE2 Server/Master Provisioning
|
||||
|
||||
```
|
||||
sudo su
|
||||
curl -sfL https://get.rke2.io | sh -
|
||||
systemctl enable --now rke2-server
|
||||
|
||||
cat > /root/.bashrc << EOF
|
||||
export PATH=$PATH:/var/lib/rancher/rke2/bin/
|
||||
export KUBECONFIG=/etc/rancher/rke2/rke2.yaml
|
||||
EOF
|
||||
|
||||
cat /var/lib/rancher/rke2/server/node-token
|
||||
token: K10ca0c38d4ff90d8b80319ab34092e315a8b732622e6adf97bc9eb0536REDACTED::server:ec0308000b8a6b595da000efREDACTED
|
||||
```
|
||||
|
||||
### RKE2 Agent/Worker provisioning
|
||||
|
||||
```
|
||||
mkdir -p /etc/rancher/rke2/
|
||||
|
||||
cat > /etc/rancher/rke2/config.yaml << EOF
|
||||
server: https://10.100.103.23:9345
|
||||
token: K10ca0c38d4ff90d8b80319ab34092e315a8b732622e6adf97bc9eb0536REDACTED::server:ec0308000b8a6b595da000efREDACTED
|
||||
EOF
|
||||
|
||||
curl -sfL https://get.rke2.io | INSTALL_RKE2_TYPE="agent" sh -
|
||||
systemctl enable --now rke2-agent.service
|
||||
```
|
||||
|
||||
To import the cluster into Rancher, click **☰ > Cluster Management**. Then on the **Clusters** page, click **Import Existing**. Then run the provided kubectl command on the server/master node.
|
||||
|
||||
# Tested Versions
|
||||
|
||||
OS for running RKE2 nodes: JeOS SLE15-SP2 with installed kernel-default-5.3.18-24.49
|
||||
|
||||
```
|
||||
kubectl version
|
||||
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.4", GitCommit:"c96aede7b5205121079932896c4ad89bb93260af", GitTreeState:"clean", BuildDate:"2020-06-22T12:00:00Z", GoVersion:"go1.13.11", Compiler:"gc", Platform:"linux/amd64"}
|
||||
Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.7+rke2r1", GitCommit:"1dd5338295409edcfff11505e7bb246f0d325d15", GitTreeState:"clean", BuildDate:"2021-01-20T01:50:52Z", GoVersion:"go1.15.5b5", Compiler:"gc", Platform:"linux/amd64"}
|
||||
|
||||
helm version
|
||||
version.BuildInfo{Version:"3.4.1", GitCommit:"c4e74854886b2efe3321e185578e6db9be0a6e29", GitTreeState:"clean", GoVersion:"go1.14.12"}
|
||||
```
|
||||
|
||||
Kubernetes version on RKE2 cluster: v1.19.7+rke2r1
|
||||
|
||||
# Troubleshooting
|
||||
|
||||
In case you are using SUSE's ceph-rook based on SES7, it might be useful to expose the monitors on hostNetwork by editing `rook-1.4.5/ceph/cluster.yaml` and setting `spec.network.hostNetwork=true`.
|
||||
|
||||
Also for operating the ceph-rook cluster, it is useful to deploy a toolbox on the Kubernetes cluster where ceph-rook is provisioned by `kubectl apply -f rook-1.4.5/ceph/toolbox.yaml` Then all the ceph related commands can be executed in the toolbox pod, for example, by running `kubectl exec -it -n rook-ceph rook-ceph-tools-686d8b8bfb-2nvqp -- bash`
|
||||
|
||||
Operating with the ceph - basic commands:
|
||||
|
||||
```
|
||||
ceph osd pool stats
|
||||
ceph osd pool delete myPool myPool --yes-i-really-really-mean-it
|
||||
rbd list -p myPool
|
||||
> csi-vol-f5d3766c-7296-11eb-b32a-c2b045952d38
|
||||
> image
|
||||
```
|
||||
|
||||
Delete the image: `rbd rm csi-vol-f5d3766c-7296-11eb-b32a-c2b045952d38 -p myPool`
|
||||
|
||||
CephFS commands in rook toolbox:
|
||||
|
||||
```
|
||||
ceph -s
|
||||
ceph fs ls
|
||||
ceph fs fail cephfs
|
||||
ceph fs rm cephfs --yes-i-really-mean-it
|
||||
ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it
|
||||
ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it
|
||||
```
|
||||
|
||||
To prepare a cephfs filesystem, you can run this command on a rook cluster:
|
||||
|
||||
```
|
||||
kubectl apply -f rook-1.4.5/ceph/filesystem.yaml
|
||||
```
|
||||
@@ -0,0 +1,16 @@
|
||||
---
|
||||
title: Creating Persistent Storage in Amazon's EBS
|
||||
weight: 3053
|
||||
---
|
||||
|
||||
This section describes how to set up Amazon's Elastic Block Store in EC2.
|
||||
|
||||
1. From the EC2 console, go to the **ELASTIC BLOCK STORE** section in the left panel and click **Volumes**.
|
||||
1. Click **Create Volume**.
|
||||
1. Optional: Configure the size of the volume or other options. The volume should be created in the same availability zone as the instance it will be attached to.
|
||||
1. Click **Create Volume**.
|
||||
1. Click **Close**.
|
||||
|
||||
**Result:** Persistent storage has been created.
|
||||
|
||||
For details on how to set up the newly created storage in Rancher, refer to the section on [setting up existing storage.]({{<baseurl>}}/rancher/v2.6/en/cluster-admin/volumes-and-storage/attaching-existing-storage/)
|
||||
@@ -0,0 +1,12 @@
|
||||
---
|
||||
title: Provisioning Storage Examples
|
||||
weight: 3053
|
||||
---
|
||||
|
||||
Rancher supports persistent storage with a variety of volume plugins. However, before you use any of these plugins to bind persistent storage to your workloads, you have to configure the storage itself, whether its a cloud-based solution from a service-provider or an on-prem solution that you manage yourself.
|
||||
|
||||
For your convenience, Rancher offers documentation on how to configure some of the popular storage methods:
|
||||
|
||||
- [NFS](./nfs)
|
||||
- [vSphere](./vsphere)
|
||||
- [EBS](./ebs)
|
||||
@@ -0,0 +1,66 @@
|
||||
---
|
||||
title: NFS Storage
|
||||
weight: 3054
|
||||
---
|
||||
|
||||
Before you can use the NFS storage volume plug-in with Rancher deployments, you need to provision an NFS server.
|
||||
|
||||
>**Note:**
|
||||
>
|
||||
>- If you already have an NFS share, you don't need to provision a new NFS server to use the NFS volume plugin within Rancher. Instead, skip the rest of this procedure and complete [adding storage]({{<baseurl>}}/rancher/v2.6/en/cluster-admin/volumes-and-storage/).
|
||||
>
|
||||
>- This procedure demonstrates how to set up an NFS server using Ubuntu, although you should be able to use these instructions for other Linux distros (e.g. Debian, RHEL, Arch Linux, etc.). For official instruction on how to create an NFS server using another Linux distro, consult the distro's documentation.
|
||||
|
||||
>**Recommended:** To simplify the process of managing firewall rules, use NFSv4.
|
||||
|
||||
1. Using a remote Terminal connection, log into the Ubuntu server that you intend to use for NFS storage.
|
||||
|
||||
1. Enter the following command:
|
||||
|
||||
```
|
||||
sudo apt-get install nfs-kernel-server
|
||||
```
|
||||
|
||||
1. Enter the command below, which sets the directory used for storage, along with user access rights. Modify the command if you'd like to keep storage at a different directory.
|
||||
|
||||
```
|
||||
mkdir -p /nfs && chown nobody:nogroup /nfs
|
||||
```
|
||||
- The `-p /nfs` parameter creates a directory named `nfs` at root.
|
||||
- The `chown nobody:nogroup /nfs` parameter allows all access to the storage directory.
|
||||
|
||||
1. Create an NFS exports table. This table sets the directory paths on your NFS server that are exposed to the nodes that will use the server for storage.
|
||||
|
||||
1. Open `/etc/exports` using your text editor of choice.
|
||||
1. Add the path of the `/nfs` folder that you created in step 3, along with the IP addresses of your cluster nodes. Add an entry for each IP address in your cluster. Follow each address and its accompanying parameters with a single space that is a delimiter.
|
||||
|
||||
```
|
||||
/nfs <IP_ADDRESS1>(rw,sync,no_subtree_check) <IP_ADDRESS2>(rw,sync,no_subtree_check) <IP_ADDRESS3>(rw,sync,no_subtree_check)
|
||||
```
|
||||
|
||||
**Tip:** You can replace the IP addresses with a subnet. For example: `10.212.50.12/24`
|
||||
|
||||
1. Update the NFS table by entering the following command:
|
||||
|
||||
```
|
||||
exportfs -ra
|
||||
```
|
||||
|
||||
1. Open the ports used by NFS.
|
||||
|
||||
1. To find out what ports NFS is using, enter the following command:
|
||||
|
||||
```
|
||||
rpcinfo -p | grep nfs
|
||||
```
|
||||
2. [Open the ports](https://help.ubuntu.com/lts/serverguide/firewall.html.en) that the previous command outputs. For example, the following command opens port 2049:
|
||||
|
||||
```
|
||||
sudo ufw allow 2049
|
||||
```
|
||||
|
||||
**Result:** Your NFS server is configured to be used for storage with your Rancher nodes.
|
||||
|
||||
## What's Next?
|
||||
|
||||
Within Rancher, add the NFS server as a storage volume and/or storage class. After adding the server, you can use it for storage for your deployments.
|
||||
+76
@@ -0,0 +1,76 @@
|
||||
---
|
||||
title: vSphere Storage
|
||||
weight: 3055
|
||||
---
|
||||
|
||||
To provide stateful workloads with vSphere storage, we recommend creating a vSphereVolume StorageClass. This practice dynamically provisions vSphere storage when workloads request volumes through a PersistentVolumeClaim.
|
||||
|
||||
In order to dynamically provision storage in vSphere, the vSphere provider must be [enabled.]({{<baseurl>}}/rancher/v2.6/en/cluster-provisioning/rke-clusters/cloud-providers/vsphere)
|
||||
|
||||
- [Prerequisites](#prerequisites)
|
||||
- [Creating a StorageClass](#creating-a-storageclass)
|
||||
- [Creating a Workload with a vSphere Volume](#creating-a-workload-with-a-vsphere-volume)
|
||||
- [Verifying Persistence of the Volume](#verifying-persistence-of-the-volume)
|
||||
- [Why to Use StatefulSets Instead of Deployments](#why-to-use-statefulsets-instead-of-deployments)
|
||||
|
||||
### Prerequisites
|
||||
|
||||
In order to provision vSphere volumes in a cluster created with the [Rancher Kubernetes Engine (RKE)]({{< baseurl>}}/rancher/v2.6/en/cluster-provisioning/rke-clusters/), the [vSphere cloud provider]({{<baseurl>}}/rke/latest/en/config-options/cloud-providers/vsphere) must be explicitly enabled in the [cluster options]({{<baseurl>}}/rancher/v2.6/en/cluster-provisioning/rke-clusters/options/).
|
||||
|
||||
### Creating a StorageClass
|
||||
|
||||
> **Note:**
|
||||
>
|
||||
> The following steps can also be performed using the `kubectl` command line tool. See [Kubernetes documentation on persistent volumes](https://kubernetes.io/docs/concepts/storage/persistent-volumes/) for details.
|
||||
|
||||
1. Click **☰ > Cluster Management**.
|
||||
1. Go to the cluster where you want to provide vSphere storage.
|
||||
1. In the left navigation bar, click **Storage > StorageClasses**.
|
||||
1. Click **Create**.
|
||||
3. Enter a **Name** for the StorageClass.
|
||||
4. Under **Provisioner**, select **VMWare vSphere Volume**.
|
||||
|
||||
{{< img "/img/rancher/vsphere-storage-class.png" "vsphere-storage-class">}}
|
||||
|
||||
5. Optionally, specify additional properties for this storage class under **Parameters**. Refer to the [vSphere storage documentation](https://vmware.github.io/vsphere-storage-for-kubernetes/documentation/storageclass.html) for details.
|
||||
5. Click **Create**.
|
||||
|
||||
### Creating a Workload with a vSphere Volume
|
||||
|
||||
1. In the left navigation bar, click **Workload**.
|
||||
1. Click **Create**.
|
||||
1. Click **StatefulSet**.
|
||||
1. In the **Volume Claim Templates** tab, click **Add Claim Template**.
|
||||
1. Enter a persistent volume name.
|
||||
1. In the Storage Class field, select the vSphere StorageClass that you created.
|
||||
6. Enter the required **Capacity** for the volume. Then click **Define**.
|
||||
7. Assign a path in the **Mount Point** field. This is the full path where the volume will be mounted in the container file system, e.g. `/persistent`.
|
||||
8. Click **Create**.
|
||||
|
||||
### Verifying Persistence of the Volume
|
||||
|
||||
1. In the left navigation bar, click **Workload > Pods**.
|
||||
1. Go to the workload you just created and click **⋮ > Execute Shell**.
|
||||
2. Note the directory at root where the volume has been mounted to (in this case `/persistent`).
|
||||
3. Create a file in the volume by executing the command `touch /<volumeMountPoint>/data.txt`.
|
||||
4. Close the shell window.
|
||||
5. Click on the name of the workload to reveal detail information.
|
||||
7. Click **⋮ > Delete**.
|
||||
8. Observe that the pod is deleted. Then a new pod is scheduled to replace it so that the workload maintains its configured scale of a single stateful pod.
|
||||
9. Once the replacement pod is running, click **Execute Shell**.
|
||||
10. Inspect the contents of the directory where the volume is mounted by entering `ls -l /<volumeMountPoint>`. Note that the file you created earlier is still present.
|
||||
|
||||
|
||||
|
||||
### Why to Use StatefulSets Instead of Deployments
|
||||
|
||||
You should always use [StatefulSets](https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/) for workloads consuming vSphere storage, as this resource type is designed to address a VMDK block storage caveat.
|
||||
|
||||
Since vSphere volumes are backed by VMDK block storage, they only support an [access mode](https://kubernetes.io/docs/concepts/storage/persistent-volumes/#persistentvolumeclaims) of `ReadWriteOnce`. This setting restricts the volume so that it can only be mounted to a single pod at a time, unless all pods consuming that volume are co-located on the same node. This behavior makes a deployment resource unusable for scaling beyond a single replica if it consumes vSphere volumes.
|
||||
|
||||
Even using a deployment resource with just a single replica may result in a deadlock situation while updating the deployment. If the updated pod is scheduled to a node different from where the existing pod lives, it will fail to start because the VMDK is still attached to the other node.
|
||||
|
||||
### Related Links
|
||||
|
||||
- [vSphere Storage for Kubernetes](https://vmware.github.io/vsphere-storage-for-kubernetes/documentation/)
|
||||
- [Kubernetes Persistent Volumes](https://kubernetes.io/docs/concepts/storage/persistent-volumes/)
|
||||
+32
@@ -0,0 +1,32 @@
|
||||
---
|
||||
title: GlusterFS Volumes
|
||||
weight: 5000
|
||||
---
|
||||
|
||||
> This section only applies to [RKE clusters.]({{<baseurl>}}/rancher/v2.6/en/cluster-provisioning/rke-clusters/)
|
||||
|
||||
In clusters that store data on GlusterFS volumes, you may experience an issue where pods fail to mount volumes after restarting the `kubelet`. The logging of the `kubelet` will show: `transport endpoint is not connected`. To prevent this from happening, you can configure your cluster to mount the `systemd-run` binary in the `kubelet` container. There are two requirements before you can change the cluster configuration:
|
||||
|
||||
- The node needs to have the `systemd-run` binary installed (this can be checked by using the command `which systemd-run` on each cluster node)
|
||||
- The `systemd-run` binary needs to be compatible with Debian OS on which the hyperkube image is based (this can be checked using the following command on each cluster node, replacing the image tag with the Kubernetes version you want to use)
|
||||
|
||||
```
|
||||
docker run -v /usr/bin/systemd-run:/usr/bin/systemd-run --entrypoint /usr/bin/systemd-run rancher/hyperkube:v1.16.2-rancher1 --version
|
||||
```
|
||||
|
||||
>**Note:**
|
||||
>
|
||||
>Before updating your Kubernetes YAML to mount the `systemd-run` binary, make sure the `systemd` package is installed on your cluster nodes. If this package isn't installed _before_ the bind mounts are created in your Kubernetes YAML, Docker will automatically create the directories and files on each node and will not allow the package install to succeed.
|
||||
|
||||
```
|
||||
services:
|
||||
kubelet:
|
||||
extra_binds:
|
||||
- "/usr/bin/systemd-run:/usr/bin/systemd-run"
|
||||
```
|
||||
|
||||
After the cluster has finished provisioning, you can check the `kubelet` container logging to see if the functionality is activated by looking for the following logline:
|
||||
|
||||
```
|
||||
Detected OS with systemd
|
||||
```
|
||||
+76
@@ -0,0 +1,76 @@
|
||||
---
|
||||
title: How Persistent Storage Works
|
||||
weight: 1
|
||||
---
|
||||
|
||||
A persistent volume (PV) is a piece of storage in the Kubernetes cluster, while a persistent volume claim (PVC) is a request for storage.
|
||||
|
||||
There are two ways to use persistent storage in Kubernetes:
|
||||
|
||||
- Use an existing persistent volume
|
||||
- Dynamically provision new persistent volumes
|
||||
|
||||
To use an existing PV, your application will need to use a PVC that is bound to a PV, and the PV should include the minimum resources that the PVC requires.
|
||||
|
||||
For dynamic storage provisioning, your application will need to use a PVC that is bound to a storage class. The storage class contains the authorization to provision new persistent volumes.
|
||||
|
||||
|
||||
|
||||
For more information, refer to the [official Kubernetes documentation on storage](https://kubernetes.io/docs/concepts/storage/volumes/)
|
||||
|
||||
This section covers the following topics:
|
||||
|
||||
- [About persistent volume claims](#about-persistent-volume-claims)
|
||||
- [PVCs are required for both new and existing persistent storage](#pvcs-are-required-for-both-new-and-existing-persistent-storage)
|
||||
- [Setting up existing storage with a PVC and PV](#setting-up-existing-storage-with-a-pvc-and-pv)
|
||||
- [Binding PVs to PVCs](#binding-pvs-to-pvcs)
|
||||
- [Provisioning new storage with a PVC and storage class](#provisioning-new-storage-with-a-pvc-and-storage-class)
|
||||
|
||||
# About Persistent Volume Claims
|
||||
|
||||
Persistent volume claims (PVCs) are objects that request storage resources from your cluster. They're similar to a voucher that your deployment can redeem for storage access. A PVC is mounted into a workloads as a volume so that the workload can claim its specified share of the persistent storage.
|
||||
|
||||
To access persistent storage, a pod must have a PVC mounted as a volume. This PVC lets your deployment application store its data in an external location, so that if a pod fails, it can be replaced with a new pod and continue accessing its data stored externally, as though an outage never occurred.
|
||||
|
||||
Each Rancher project contains a list of PVCs that you've created, available from **Resources > Workloads > Volumes**. You can reuse these PVCs when creating deployments in the future.
|
||||
|
||||
### PVCs are Required for Both New and Existing Persistent Storage
|
||||
|
||||
A PVC is required for pods to use any persistent storage, regardless of whether the workload is intended to use storage that already exists, or the workload will need to dynamically provision new storage on demand.
|
||||
|
||||
If you are setting up existing storage for a workload, the workload mounts a PVC, which refers to a PV, which corresponds to existing storage infrastructure.
|
||||
|
||||
If a workload should request new storage, the workload mounts PVC, which refers to a storage class, which has the capability to create a new PV along with its underlying storage infrastructure.
|
||||
|
||||
Rancher lets you create as many PVCs within a project as you'd like.
|
||||
|
||||
You can mount PVCs to a deployment as you create it, or later, after the deployment is running.
|
||||
|
||||
# Setting up Existing Storage with a PVC and PV
|
||||
|
||||
Your pods can store data in [volumes,](https://kubernetes.io/docs/concepts/storage/volumes/) but if the pod fails, that data is lost. To solve this issue, Kubernetes offers persistent volumes (PVs), which are Kubernetes resources that correspond to external storage disks or file systems that your pods can access. If a pod crashes, its replacement pod can access the data in persistent storage without any data loss.
|
||||
|
||||
PVs can represent a physical disk or file system that you host on premise, or a vendor-hosted storage resource, such as Amazon EBS or Azure Disk.
|
||||
|
||||
Creating a persistent volume in Rancher will not create a storage volume. It only creates a Kubernetes resource that maps to an existing volume. Therefore, before you can create a persistent volume as a Kubernetes resource, you must have storage provisioned.
|
||||
|
||||
> **Important:** PVs are created at the cluster level, which means that in a multi-tenant cluster, teams with access to separate namespaces could have access to the same PV.
|
||||
|
||||
### Binding PVs to PVCs
|
||||
|
||||
When pods are set up to use persistent storage, they mount a persistent volume claim (PVC) that is mounted the same way as any other Kubernetes volume. When each PVC is created, the Kubernetes master considers it to be a request for storage and binds it to a PV that matches the minimum resource requirements of the PVC. Not every PVC is guaranteed to be bound to a PV. According to the Kubernetes [documentation,](https://kubernetes.io/docs/concepts/storage/persistent-volumes/)
|
||||
|
||||
> Claims will remain unbound indefinitely if a matching volume does not exist. Claims will be bound as matching volumes become available. For example, a cluster provisioned with many 50Gi PVs would not match a PVC requesting 100Gi. The PVC can be bound when a 100Gi PV is added to the cluster.
|
||||
|
||||
In other words, you can create unlimited PVCs, but they will only be bound to PVs if the Kubernetes master can find a sufficient PVs that has at least the amount of disk space required by the PVC.
|
||||
|
||||
To dynamically provision new storage, the PVC mounted in the pod would have to correspond to a storage class instead of a persistent volume.
|
||||
|
||||
# Provisioning New Storage with a PVC and Storage Class
|
||||
|
||||
Storage Classes allow you to create PVs dynamically without having to create persistent storage in an infrastructure provider first.
|
||||
|
||||
For example, if a workload is bound to a PVC and the PVC refers to an Amazon EBS Storage Class, the storage class can dynamically create an EBS volume and a corresponding PV.
|
||||
|
||||
The Kubernetes master will then bind the newly created PV to your workload's PVC, allowing your workload to use the persistent storage.
|
||||
|
||||
+33
@@ -0,0 +1,33 @@
|
||||
---
|
||||
title: iSCSI Volumes
|
||||
weight: 6000
|
||||
---
|
||||
|
||||
In [Rancher Launched Kubernetes clusters]({{<baseurl>}}/rancher/v2.6/en/cluster-provisioning/rke-clusters/) that store data on iSCSI volumes, you may experience an issue where kubelets fail to automatically connect with iSCSI volumes. This failure is likely due to an incompatibility issue involving the iSCSI initiator tool. You can resolve this issue by installing the iSCSI initiator tool on each of your cluster nodes.
|
||||
|
||||
Rancher Launched Kubernetes clusters storing data on iSCSI volumes leverage the [iSCSI initiator tool](http://www.open-iscsi.com/), which is embedded in the kubelet's `rancher/hyperkube` Docker image. From each kubelet (i.e., the _initiator_), the tool discovers and launches sessions with an iSCSI volume (i.e., the _target_). However, in some instances, the versions of the iSCSI initiator tool installed on the initiator and the target may not match, resulting in a connection failure.
|
||||
|
||||
If you encounter this issue, you can work around it by installing the initiator tool on each node in your cluster. You can install the iSCSI initiator tool by logging into your cluster nodes and entering one of the following commands:
|
||||
|
||||
| Platform | Package Name | Install Command |
|
||||
| ------------- | ----------------------- | -------------------------------------- |
|
||||
| Ubuntu/Debian | `open-iscsi` | `sudo apt install open-iscsi` |
|
||||
| RHEL | `iscsi-initiator-utils` | `yum install iscsi-initiator-utils -y` |
|
||||
|
||||
|
||||
After installing the initiator tool on your nodes, edit the YAML for your cluster, editing the kubelet configuration to mount the iSCSI binary and configuration, as shown in the sample below.
|
||||
|
||||
>**Notes:**
|
||||
>
|
||||
>- Before updating your Kubernetes YAML to mount the iSCSI binary and configuration, make sure either the `open-iscsi` (deb) or `iscsi-initiator-utils` (yum) package is installed on your cluster nodes. If this package isn't installed _before_ the bind mounts are created in your Kubernetes YAML, Docker will automatically create the directories and files on each node and will not allow the package install to succeed.</br>
|
||||
></br>
|
||||
>
|
||||
>- The example YAML below does not apply to K3s, but only to RKE clusters. Since the K3s kubelet does not run in a container, adding extra binds is not necessary. However, all iSCSI tools must still be installed on your K3s nodes.
|
||||
|
||||
```
|
||||
services:
|
||||
kubelet:
|
||||
extra_binds:
|
||||
- "/etc/iscsi:/etc/iscsi"
|
||||
- "/sbin/iscsiadm:/sbin/iscsiadm"
|
||||
```
|
||||
+88
@@ -0,0 +1,88 @@
|
||||
---
|
||||
title: Dynamically Provisioning New Storage in Rancher
|
||||
weight: 2
|
||||
---
|
||||
|
||||
This section describes how to provision new persistent storage for workloads in Rancher.
|
||||
|
||||
This section assumes that you understand the Kubernetes concepts of storage classes and persistent volume claims. For more information, refer to the section on [how storage works.](../how-storage-works)
|
||||
|
||||
New storage is often provisioned by a cloud provider such as Amazon EBS. However, new storage doesn't have to be in the cloud.
|
||||
|
||||
If you have a pool of block storage, and you don't want to use a cloud provider, Longhorn could help you provide persistent storage to your Kubernetes cluster. For more information, see [this page.]({{<baseurl>}}/rancher/v2.6/en/longhorn)
|
||||
|
||||
To provision new storage for your workloads, follow these steps:
|
||||
|
||||
1. [Add a storage class and configure it to use your storage.](#1-add-a-storage-class-and-configure-it-to-use-your-storage)
|
||||
2. [Use the Storage Class for Pods Deployed with a StatefulSet.](#2-use-the-storage-class-for-pods-deployed-with-a-statefulset)
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- To set up persistent storage, the `Manage Volumes` [role]({{<baseurl>}}/rancher/v2.6/en/admin-settings/rbac/cluster-project-roles/#project-role-reference) is required.
|
||||
- If you are provisioning storage for a cluster hosted in the cloud, the storage and cluster hosts must have the same cloud provider.
|
||||
- The cloud provider must be enabled. For details on enabling cloud providers, refer to [this page.]({{<baseurl>}}/rancher/v2.6/en/cluster-provisioning/rke-clusters/cloud-providers/)
|
||||
- Make sure your storage provisioner is available to be enabled.
|
||||
|
||||
The following storage provisioners are enabled by default:
|
||||
|
||||
Name | Plugin
|
||||
--------|----------
|
||||
Amazon EBS Disk | `aws-ebs`
|
||||
AzureFile | `azure-file`
|
||||
AzureDisk | `azure-disk`
|
||||
Google Persistent Disk | `gce-pd`
|
||||
Longhorn | `flex-volume-longhorn`
|
||||
VMware vSphere Volume | `vsphere-volume`
|
||||
Local | `local`
|
||||
Network File System | `nfs`
|
||||
hostPath | `host-path`
|
||||
|
||||
To use a storage provisioner that is not on the above list, you will need to use a [feature flag to enable unsupported storage drivers.]({{<baseurl>}}/rancher/v2.6/en/installation/resources/feature-flags/enable-not-default-storage-drivers/)
|
||||
|
||||
### 1. Add a storage class and configure it to use your storage
|
||||
|
||||
These steps describe how to set up a storage class at the cluster level.
|
||||
|
||||
1. Click **☰ > Cluster Management**.
|
||||
1. Go to the cluster where you want to dynamically provision persistent storage volumes and click **Explore**.
|
||||
1. Click **Storage > Storage Classes**.
|
||||
1. Click **Create**.
|
||||
1. Enter a name for your storage class.
|
||||
1. From the **Provisioner** drop-down, select the service that you want to use to dynamically provision storage volumes. For example, if you have a Amazon EC2 cluster and you want to use cloud storage for it, use the `Amazon EBS Disk` provisioner.
|
||||
1. In the **Parameters** tab, fill out the information required for the service to dynamically provision storage volumes. Each provisioner requires different information to dynamically provision storage volumes. Consult the service's documentation for help on how to obtain this information.
|
||||
1. Click **Create**.
|
||||
|
||||
**Result:** The storage class is available to be consumed by a PVC.
|
||||
|
||||
For full information about the storage class parameters, refer to the official [Kubernetes documentation.](https://kubernetes.io/docs/concepts/storage/storage-classes/#parameters).
|
||||
|
||||
### 2. Use the Storage Class for Pods Deployed with a StatefulSet
|
||||
|
||||
StatefulSets manage the deployment and scaling of Pods while maintaining a sticky identity for each Pod. In this StatefulSet, we will configure a VolumeClaimTemplate. Each Pod managed by the StatefulSet will be deployed with a PersistentVolumeClaim based on this VolumeClaimTemplate. The PersistentVolumeClaim will refer to the StorageClass that we created. Therefore, when each Pod managed by the StatefulSet is deployed, it will be bound to dynamically provisioned storage using the StorageClass defined in its PersistentVolumeClaim.
|
||||
|
||||
1. Click **☰ > Cluster Management**.
|
||||
1. Go to the cluster where you want to add use the StorageClass for a workload and click **Explore**.
|
||||
1. In the left navigation bar, click **Workload**.
|
||||
1. Click **Create**.
|
||||
1. Click **StatefulSet**.
|
||||
1. In the **Volume Claim Templates** tab, click **Add Claim Template**.
|
||||
1. Enter a name for the persistent volume.
|
||||
1. In the **StorageClass* field, select the StorageClass that will dynamically provision storage for pods managed by this StatefulSet.
|
||||
1. In the **Mount Point** field, enter the path that the workload will use to access the volume.
|
||||
1. Click **Launch**.
|
||||
|
||||
**Result:** When each Pod managed by the StatefulSet is deployed, it will make a request for the specified amount of disk space to the Kubernetes master. If a PV with the specified resources is available when the workload is deployed, the Kubernetes master will bind the PV to Pod with a compatible PVC.
|
||||
|
||||
To attach the PVC to an existing workload,
|
||||
|
||||
1. Click **☰ > Cluster Management**.
|
||||
1. Go to the cluster where you want to add use the StorageClass for a workload and click **Explore**.
|
||||
1. In the left navigation bar, click **Workload**.
|
||||
1. Go to the workload that will use storage provisioned with the StorageClass that you cared at click **⋮ > Edit Config**.
|
||||
1. In the **Volume Claim Templates** section, click **Add Claim Template**.
|
||||
1. Enter a persistent volume name.
|
||||
1. In the **StorageClass* field, select the StorageClass that will dynamically provision storage for pods managed by this StatefulSet.
|
||||
1. In the **Mount Point** field, enter the path that the workload will use to access the volume.
|
||||
1. Click **Save**.
|
||||
|
||||
**Result:** The workload will make a request for the specified amount of disk space to the Kubernetes master. If a PV with the specified resources is available when the workload is deployed, the Kubernetes master will bind the PV to the PVC. If not, Rancher will provision new persistent storage.
|
||||
@@ -0,0 +1,74 @@
|
||||
---
|
||||
title: "Kubernetes Persistent Storage: Volumes and Storage Classes"
|
||||
description: "Learn about the two ways with which you can create persistent storage in Kubernetes: persistent volumes and storage classes"
|
||||
weight: 2031
|
||||
---
|
||||
When deploying an application that needs to retain data, you'll need to create persistent storage. Persistent storage allows you to store application data external from the pod running your application. This storage practice allows you to maintain application data, even if the application's pod fails.
|
||||
|
||||
The documents in this section assume that you understand the Kubernetes concepts of persistent volumes, persistent volume claims, and storage classes. For more information, refer to the section on [how storage works.](./how-storage-works)
|
||||
|
||||
### Prerequisites
|
||||
|
||||
To set up persistent storage, the `Manage Volumes` [role]({{<baseurl>}}/rancher/v2.6/en/admin-settings/rbac/cluster-project-roles/#project-role-reference) is required.
|
||||
|
||||
If you are provisioning storage for a cluster hosted in the cloud, the storage and cluster hosts must have the same cloud provider.
|
||||
|
||||
For provisioning new storage with Rancher, the cloud provider must be enabled. For details on enabling cloud providers, refer to [this page.]({{<baseurl>}}/rancher/v2.6/en/cluster-provisioning/rke-clusters/cloud-providers/)
|
||||
|
||||
For attaching existing persistent storage to a cluster, the cloud provider does not need to be enabled.
|
||||
|
||||
### Setting up Existing Storage
|
||||
|
||||
The overall workflow for setting up existing storage is as follows:
|
||||
|
||||
1. Set up your persistent storage. This may be storage in an infrastructure provider, or it could be your own storage.
|
||||
2. Add a persistent volume (PV) that refers to the persistent storage.
|
||||
3. Add a persistent volume claim (PVC) that refers to the PV.
|
||||
4. Mount the PVC as a volume in your workload.
|
||||
|
||||
For details and prerequisites, refer to [this page.](./attaching-existing-storage)
|
||||
|
||||
### Dynamically Provisioning New Storage in Rancher
|
||||
|
||||
The overall workflow for provisioning new storage is as follows:
|
||||
|
||||
1. Add a StorageClass and configure it to use your storage provider. The StorageClass could refer to storage in an infrastructure provider, or it could refer to your own storage.
|
||||
2. Add a persistent volume claim (PVC) that refers to the storage class.
|
||||
3. Mount the PVC as a volume for your workload.
|
||||
|
||||
For details and prerequisites, refer to [this page.](./provisioning-new-storage)
|
||||
|
||||
### Longhorn Storage
|
||||
|
||||
[Longhorn](https://longhorn.io/) is a lightweight, reliable and easy-to-use distributed block storage system for Kubernetes.
|
||||
|
||||
Longhorn is free, open source software. Originally developed by Rancher Labs, it is now being developed as a sandbox project of the Cloud Native Computing Foundation. It can be installed on any Kubernetes cluster with Helm, with kubectl, or with the Rancher UI.
|
||||
|
||||
If you have a pool of block storage, Longhorn can help you provide persistent storage to your Kubernetes cluster without relying on cloud providers. For more information about Longhorn features, refer to the [documentation.](https://longhorn.io/docs/latest/what-is-longhorn/)
|
||||
|
||||
Rancher v2.5 simplified the process of installing Longhorn on a Rancher-managed cluster. For more information, see [this page.]({{<baseurl>}}/rancher/v2.6/en/longhorn)
|
||||
|
||||
### Provisioning Storage Examples
|
||||
|
||||
We provide examples of how to provision storage with [NFS,](./examples/nfs) [vSphere,](./examples/vsphere) and [Amazon's EBS.](./examples/ebs)
|
||||
|
||||
### GlusterFS Volumes
|
||||
|
||||
In clusters that store data on GlusterFS volumes, you may experience an issue where pods fail to mount volumes after restarting the `kubelet`. For details on preventing this from happening, refer to [this page.](./glusterfs-volumes)
|
||||
|
||||
### iSCSI Volumes
|
||||
|
||||
In [Rancher Launched Kubernetes clusters]({{<baseurl>}}/rancher/v2.6/en/cluster-provisioning/rke-clusters/) that store data on iSCSI volumes, you may experience an issue where kubelets fail to automatically connect with iSCSI volumes. For details on resolving this issue, refer to [this page.](./iscsi-volumes)
|
||||
|
||||
### hostPath Volumes
|
||||
Before you create a hostPath volume, you need to set up an [extra_bind]({{<baseurl>}}/rke/latest/en/config-options/services/services-extras/#extra-binds/) in your cluster configuration. This will mount the path as a volume in your kubelets, which can then be used for hostPath volumes in your workloads.
|
||||
|
||||
### Migrating vSphere Cloud Provider from In-tree to Out-of-tree
|
||||
|
||||
Kubernetes is moving away from maintaining cloud providers in-tree. vSphere has an out-of-tree cloud provider that can be used by installing the vSphere cloud provider and cloud storage plugins.
|
||||
|
||||
For instructions on how to migrate from the in-tree vSphere cloud provider to out-of-tree, and manage the existing VMs post migration, refer to [this page.]({{<baseurl>}}/rancher/v2.6/en/cluster-provisioning/rke-clusters/cloud-providers/vsphere/out-of-tree)
|
||||
|
||||
### Related Links
|
||||
|
||||
- [Kubernetes Documentation: Storage](https://kubernetes.io/docs/concepts/storage/)
|
||||
Reference in New Issue
Block a user