merge conflict resolved

This commit is contained in:
martyav
2023-09-19 12:42:52 -04:00
969 changed files with 15104 additions and 714 deletions
@@ -2,6 +2,10 @@
title: API Tokens
---
<head>
<link rel="canonical" href="https://ranchermanager.docs.rancher.com//reference-guides/about-the-api/api-tokens"/>
</head>
By default, some cluster-level API tokens are generated with infinite time-to-live (`ttl=0`). In other words, API tokens with `ttl=0` never expire unless you invalidate them. Tokens are not invalidated by changing a password.
You can deactivate API tokens by deleting them or by deactivating the user account.
@@ -2,6 +2,10 @@
title: Backup Configuration
---
<head>
<link rel="canonical" href="https://ranchermanager.docs.rancher.com//reference-guides/backup-restore-configuration/backup-configuration"/>
</head>
The Backup Create page lets you configure a schedule, enable encryption and specify the storage location for your backups.
@@ -2,6 +2,10 @@
title: Examples
---
<head>
<link rel="canonical" href="https://ranchermanager.docs.rancher.com//reference-guides/backup-restore-configuration/examples"/>
</head>
This section contains examples of Backup and Restore custom resources.
The default backup storage location is configured when the `rancher-backup` operator is installed or upgraded.
@@ -2,6 +2,10 @@
title: Restore Configuration
---
<head>
<link rel="canonical" href="https://ranchermanager.docs.rancher.com//reference-guides/backup-restore-configuration/restore-configuration"/>
</head>
The Restore Create page lets you provide details of the backup to restore from
![](/img/backup_restore/restore/restore.png)
@@ -2,6 +2,10 @@
title: Backup Storage Location Configuration
---
<head>
<link rel="canonical" href="https://ranchermanager.docs.rancher.com//reference-guides/backup-restore-configuration/storage-configuration"/>
</head>
Configure a storage location where all backups are saved by default. You will have the option to override this with each backup, but will be limited to using an S3-compatible object store.
Only one storage location can be configured at the operator level.
@@ -2,6 +2,10 @@
title: Logging Best Practices
---
<head>
<link rel="canonical" href="https://ranchermanager.docs.rancher.com//reference-guides/best-practices/rancher-managed-clusters/logging-best-practices"/>
</head>
In this guide, we recommend best practices for cluster-level logging and application logging.
- [Cluster-level Logging](#cluster-level-logging)
@@ -2,6 +2,10 @@
title: Monitoring Best Practices
---
<head>
<link rel="canonical" href="https://ranchermanager.docs.rancher.com//reference-guides/best-practices/rancher-managed-clusters/monitoring-best-practices"/>
</head>
Configuring sensible monitoring and alerting rules is vital for running any production workloads securely and reliably. This is not different when using Kubernetes and Rancher. Fortunately the integrated monitoring and alerting functionality makes this whole process a lot easier.
The [Rancher monitoring documentation](../../../pages-for-subheaders/monitoring-and-alerting.md) describes how you can set up a complete Prometheus and Grafana stack. Out of the box this will scrape monitoring data from all system and Kubernetes components in your cluster and provide sensible dashboards and alerts for them to get started. But for a reliable setup, you also need to monitor your own workloads and adapt Prometheus and Grafana to your own specific use cases and cluster sizes. This document aims to give you best practices for this.
@@ -86,7 +90,7 @@ Sometimes it is useful to monitor workloads from the outside. For this, you can
If you have a (micro)service architecture where multiple individual workloads within your cluster are communicating with each other, it is really important to have detailed metrics and traces about this traffic to understand how all these workloads are communicating with each other and where a problem or bottleneck may be.
Of course you can monitor all this internal traffic in all your workloads and expose these metrics to Prometheus. But this can quickly become quite work intensive. Service Meshes like Istio, which can be installed with [a click](https://rancher.com/docs/rancher/v2.6/en/istio/) in Rancher, can do this automatically and provide rich telemetry about the traffic between all services.
Of course you can monitor all this internal traffic in all your workloads and expose these metrics to Prometheus. But this can quickly become quite work intensive. Service Meshes like Istio, which can be installed with [a click](../../../pages-for-subheaders/istio.md) in Rancher, can do this automatically and provide rich telemetry about the traffic between all services.
## Real User Monitoring
@@ -2,6 +2,10 @@
title: Best Practices for Rancher Managed vSphere Clusters
---
<head>
<link rel="canonical" href="https://ranchermanager.docs.rancher.com//reference-guides/best-practices/rancher-managed-clusters/rancher-managed-clusters-in-vsphere"/>
</head>
This guide outlines a reference architecture for provisioning downstream Rancher clusters in a vSphere environment, in addition to standard vSphere best practices as documented by VMware.
- [1. VM Considerations](#1-vm-considerations)
@@ -2,6 +2,10 @@
title: Tips for Setting Up Containers
---
<head>
<link rel="canonical" href="https://ranchermanager.docs.rancher.com//reference-guides/best-practices/rancher-managed-clusters/tips-to-set-up-containers"/>
</head>
Running well-built containers can greatly impact the overall performance and security of your environment.
Below are a few tips for setting up your containers.
@@ -2,6 +2,10 @@
title: Installing Rancher in a vSphere Environment
---
<head>
<link rel="canonical" href="https://ranchermanager.docs.rancher.com//reference-guides/best-practices/rancher-server/on-premises-rancher-in-vsphere"/>
</head>
This guide outlines a reference architecture for installing Rancher on an RKE Kubernetes cluster in a vSphere environment, in addition to standard vSphere best practices as documented by VMware.
@@ -2,6 +2,10 @@
title: Rancher Deployment Strategy
---
<head>
<link rel="canonical" href="https://ranchermanager.docs.rancher.com//reference-guides/best-practices/rancher-server/rancher-deployment-strategy"/>
</head>
There are two recommended deployment strategies for a Rancher instance that manages downstream Kubernetes clusters. Each one has its own pros and cons. Read more about which one would fit best for your use case.
## Hub & Spoke Strategy
@@ -2,6 +2,10 @@
title: Tips for Running Rancher
---
<head>
<link rel="canonical" href="https://ranchermanager.docs.rancher.com//reference-guides/best-practices/rancher-server/tips-for-running-rancher"/>
</head>
This guide is geared toward use cases where Rancher is used to manage downstream Kubernetes clusters. The high-availability setup is intended to prevent losing access to downstream clusters if the Rancher server is not available.
A high-availability Kubernetes installation, defined as an installation of Rancher on a Kubernetes cluster with at least three nodes, should be used in any production installation of Rancher, as well as any installation deemed "important." Multiple Rancher instances running on multiple nodes ensure high availability that cannot be accomplished with a single node environment.
@@ -2,6 +2,10 @@
title: Tips for Scaling Rancher
---
<head>
<link rel="canonical" href="https://ranchermanager.docs.rancher.com//reference-guides/best-practices/rancher-server/tips-for-scaling-rancher"/>
</head>
This guide aims to introduce the approaches that should be considered to scale Rancher setups, and associated challenges with doing so. As systems grow performance will naturally reduce, but there are steps we can take to minimize the load put on Rancher, as well as optimize Rancher's ability to handle these larger setups.
## General Tips on Optimizing Rancher's Performance
@@ -2,6 +2,10 @@
title: kubectl Utility
---
<head>
<link rel="canonical" href="https://ranchermanager.docs.rancher.com//reference-guides/cli-with-rancher/kubectl-utility"/>
</head>
## kubectl
Interact with Rancher using kubectl.
@@ -3,6 +3,10 @@ title: Rancher CLI
description: Interact with Rancher using command line interface (CLI) tools from your workstation.
---
<head>
<link rel="canonical" href="https://ranchermanager.docs.rancher.com//reference-guides/cli-with-rancher/rancher-cli"/>
</head>
The Rancher CLI (Command Line Interface) is a unified tool that you can use to interact with Rancher. With this tool, you can operate Rancher using a command line rather than the GUI.
### Download Rancher CLI
@@ -10,7 +14,7 @@ The Rancher CLI (Command Line Interface) is a unified tool that you can use to i
The binary can be downloaded directly from the UI.
1. In the upper left corner, click **☰**.
1. At the bottom, click **v2.6.x**, where **v2.6.x** is a hyperlinked text indicating the installed Rancher version.
1. At the bottom of the navigation sidebar menu, click **About**.
1. Under the **CLI Downloads section**, there are links to download the binaries for Windows, Mac, and Linux. You can also check the [releases page for our CLI](https://github.com/rancher/cli/releases) for direct downloads of the binary.
### Requirements
@@ -2,6 +2,10 @@
title: EC2 Machine Configuration Reference
---
<head>
<link rel="canonical" href="https://ranchermanager.docs.rancher.com//reference-guides/cluster-configuration/downstream-cluster-configuration/machine-configuration/amazon-ec2"/>
</head>
For more details about EC2 nodes, refer to the official documentation for the [EC2 Management Console](https://aws.amazon.com/ec2).
### Region
@@ -2,6 +2,10 @@
title: Azure Machine Configuration
---
<head>
<link rel="canonical" href="https://ranchermanager.docs.rancher.com//reference-guides/cluster-configuration/downstream-cluster-configuration/machine-configuration/azure"/>
</head>
For more information about Azure, refer to the official [Azure documentation.](https://docs.microsoft.com/en-us/azure/?product=featured)
### Environment
@@ -2,6 +2,10 @@
title: DigitalOcean Machine Configuration
---
<head>
<link rel="canonical" href="https://ranchermanager.docs.rancher.com//reference-guides/cluster-configuration/downstream-cluster-configuration/machine-configuration/digitalocean"/>
</head>
For more details about DigitalOcean, Droplets, refer to the [official documentation](https://docs.digitalocean.com/products/compute/).
### Region
@@ -2,6 +2,10 @@
title: EC2 Node Template Configuration
---
<head>
<link rel="canonical" href="https://ranchermanager.docs.rancher.com//reference-guides/cluster-configuration/downstream-cluster-configuration/node-template-configuration/amazon-ec2"/>
</head>
For more details about EC2, nodes, refer to the official documentation for the [EC2 Management Console](https://aws.amazon.com/ec2).
### Region
@@ -2,6 +2,10 @@
title: Azure Node Template Configuration
---
<head>
<link rel="canonical" href="https://ranchermanager.docs.rancher.com//reference-guides/cluster-configuration/downstream-cluster-configuration/node-template-configuration/azure"/>
</head>
For more information about Azure, refer to the official [Azure documentation.](https://docs.microsoft.com/en-us/azure/?product=featured)
Account access information is stored as a cloud credential. Cloud credentials are stored as Kubernetes secrets. Multiple node templates can use the same cloud credential. You can use an existing cloud credential or create a new one.
@@ -2,6 +2,10 @@
title: DigitalOcean Node Template Configuration
---
<head>
<link rel="canonical" href="https://ranchermanager.docs.rancher.com//reference-guides/cluster-configuration/downstream-cluster-configuration/node-template-configuration/digitalocean"/>
</head>
Account access information is stored as a cloud credential. Cloud credentials are stored as Kubernetes secrets. Multiple node templates can use the same cloud credential. You can use an existing cloud credential or create a new one.
### Droplet Options
@@ -2,6 +2,10 @@
title: Nutanix Node Template Configuration
---
<head>
<link rel="canonical" href="https://ranchermanager.docs.rancher.com//reference-guides/cluster-configuration/downstream-cluster-configuration/node-template-configuration/nutanix"/>
</head>
## Account Access
| Parameter | Required | Description | Default
@@ -2,6 +2,10 @@
title: VSphere Node Template Configuration
---
<head>
<link rel="canonical" href="https://ranchermanager.docs.rancher.com//reference-guides/cluster-configuration/downstream-cluster-configuration/node-template-configuration/vsphere"/>
</head>
## Account Access
| Parameter | Required | Description |
@@ -13,7 +17,7 @@ Your cloud credential has these fields:
| Credential Field | Description |
|-----------------|--------------|
| vCenter or ESXi Server | Enter the vCenter or ESXi hostname/IP. ESXi is the virtualization platform where you create and run virtual machines and virtual appliances. vCenter Server is the service through which you manage multiple hosts connected in a network and pool host resources. |
| Port | Optional: configure configure the port of the vCenter or ESXi server. |
| Port | Configure the port of the vCenter or ESXi server. |
| Username and password | Enter your vSphere login username and password. |
## Scheduling
@@ -92,4 +96,4 @@ In the **Engine Options** section of the node template, you can configure the co
If you're provisioning Red Hat Enterprise Linux (RHEL) or CentOS nodes, leave the **Docker Install URL** field as the default value, or select **none**. This will bypass a check for Docker installation, as Docker is already installed on these node types.
If you set **Docker Install URL** to a value other than the default or **none**, you might see an error message such as the following: `Error creating machine: RHEL ssh command error: command: sudo -E yum install -y curl err: exit status 1 output: Updating Subscription Management repositories.`
:::
:::
@@ -2,6 +2,10 @@
title: AKS Cluster Configuration Reference
---
<head>
<link rel="canonical" href="https://ranchermanager.docs.rancher.com//reference-guides/cluster-configuration/rancher-server-configuration/aks-cluster-configuration"/>
</head>
## Changes in Rancher v2.6
- Support for adding more than one node pool
@@ -3,7 +3,7 @@ title: EKS Cluster Configuration Reference
---
<head>
<link rel="canonical" href="https://ranchermanager.docs.rancher.com/reference-guides/cluster-configuration/rancher-server-configuration/eks-cluster-configuration" />
<link rel="canonical" href="https://ranchermanager.docs.rancher.com/reference-guides/cluster-configuration/rancher-server-configuration/eks-cluster-configuration" />
</head>
### Account Access
@@ -2,6 +2,10 @@
title: Private Clusters
---
<head>
<link rel="canonical" href="https://ranchermanager.docs.rancher.com//reference-guides/cluster-configuration/rancher-server-configuration/gke-cluster-configuration/gke-private-clusters"/>
</head>
In GKE, [private clusters](https://cloud.google.com/kubernetes-engine/docs/concepts/private-cluster-concept) are clusters whose nodes are isolated from inbound and outbound traffic by assigning them internal IP addresses only. Private clusters in GKE have the option of exposing the control plane endpoint as a publicly accessible address or as a private address. This is different from other Kubernetes providers, which may refer to clusters with private control plane endpoints as "private clusters" but still allow traffic to and from nodes. You may want to create a cluster with private nodes, with or without a public control plane endpoint, depending on your organization's networking and security requirements. A GKE cluster provisioned from Rancher can use isolated nodes by selecting "Private Cluster" in the Cluster Options (under "Show advanced options"). The control plane endpoint can optionally be made private by selecting "Enable Private Endpoint".
### Private Nodes
@@ -26,7 +30,7 @@ This scenario is not officially supported, but is described for cases in which u
:::
If restricting both incoming and outgoing traffic to nodes is a requirement, follow the air-gapped installation instructions to set up a private container image [registry](https://rancher.com/docs/rancher/v2.6/en/installation/other-installation-methods/air-gap/) on the VPC where the cluster is going to be, allowing the cluster nodes to access and download the images they need to run the cluster agent. If the control plane endpoint is also private, Rancher will need [direct access](#direct-access) to it.
If restricting both incoming and outgoing traffic to nodes is a requirement, follow the air-gapped installation instructions to set up a private container image [registry](../../../../pages-for-subheaders/air-gapped-helm-cli-install.md) on the VPC where the cluster is going to be, allowing the cluster nodes to access and download the images they need to run the cluster agent. If the control plane endpoint is also private, Rancher will need [direct access](#direct-access) to it.
### Private Control Plane Endpoint
@@ -15,94 +15,158 @@ You can configure the Kubernetes options one of two ways:
- [Rancher UI](#configuration-options-in-the-rancher-ui): Use the Rancher UI to select options that are commonly customized when setting up a Kubernetes cluster.
- [Cluster Config File](#cluster-config-file): Instead of using the Rancher UI to choose Kubernetes options for the cluster, advanced users can create a K3s config file. Using a config file allows you to set any of the [options](https://rancher.com/docs/k3s/latest/en/installation/install-options/) available in an K3s installation.
## Editing Clusters in the Rancher UI
The Rancher UI provides two ways to edit a cluster:
1. With a form.
1. With YAML.
### Editing Clusters with a Form
The form covers the most frequently needed options for clusters.
To edit your cluster,
1. Click **☰ > Cluster Management**.
1. Go to the cluster you want to configure and click **⋮ > Edit Config**.
### Editing Clusters in YAML
For a complete reference of configurable options for K3s clusters in YAML, see the [K3s documentation.](https://rancher.com/docs/k3s/latest/en/installation/install-options/)
To edit your cluster with YAML:
1. Click **☰ > Cluster Management**.
1. Go to the cluster you want to configure and click **⋮ > Edit as YAML**.
1. Edit the RKE options under the `rkeConfig` directive.
## Configuration Options in the Rancher UI
:::tip
### Machine Pool Configuration
Some advanced configuration options are not exposed in the Rancher UI forms, but they can be enabled by editing the K3s cluster configuration file in YAML. For the complete reference of configurable options for K3s clusters in YAML, see the [K3s documentation.](https://rancher.com/docs/k3s/latest/en/installation/install-options/)
This subsection covers generic machine pool configurations. For specific infrastructure provider configurations, refer to the following:
:::
- [Azure](../downstream-cluster-configuration/machine-configuration/azure.md)
- [DigitalOcean](../downstream-cluster-configuration/machine-configuration/digitalocean.md)
- [EC2](../downstream-cluster-configuration/machine-configuration/amazon-ec2.md)
### Basics
#### Kubernetes Version
##### Pool Name
The name of the machine pool.
##### Machine Count
The number of machines in the pool.
##### Roles
Option to assign etcd, control plane, and worker roles to nodes.
#### Advanced
##### Auto Replace
The amount of time nodes can be unreachable before they are automatically deleted and replaced.
##### Drain Before Delete
Enables draining nodes by evicting all pods before the node is deleted.
##### Kubernetes Node Labels
Add [labels](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/) to nodes to help with organization and object selection.
For details on label syntax requirements, see the [Kubernetes documentation.](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#syntax-and-character-set)
##### Taints
Add [taints](https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/) to nodes, to prevent pods from being scheduled to or executed on the nodes, unless the pods have matching tolerations.
### Cluster Configuration
#### Basics
##### Kubernetes Version
The version of Kubernetes installed on your cluster nodes. Rancher packages its own version of Kubernetes based on [hyperkube](https://github.com/rancher/hyperkube).
For more detail, see [Upgrading Kubernetes](../../../getting-started/installation-and-upgrade/upgrade-and-roll-back-kubernetes.md).
#### Encrypt Secrets
##### Pod Security Admission Configuration Template
The default [pod security admission configuration template](../../../how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/psa-config-templates.md) for the cluster.
##### Encrypt Secrets
Option to enable or disable secrets encryption. When enabled, secrets will be encrypted using a AES-CBC key. If disabled, any previously secrets will not be readable until encryption is enabled again. Refer to the [K3s documentation](https://rancher.com/docs/k3s/latest/en/advanced/#secrets-encryption-config-experimental) for details.
#### Project Network Isolation
##### Project Network Isolation
If your network provider allows project network isolation, you can choose whether to enable or disable inter-project communication.
#### SELinux
##### SELinux
Option to enable or disable [SELinux](https://rancher.com/docs/k3s/latest/en/advanced/#selinux-support) support.
#### CoreDNS
##### CoreDNS
By default, [CoreDNS](https://coredns.io/) is installed as the default DNS provider. If CoreDNS is not installed, an alternate DNS provider must be installed yourself. Refer to the [K3s documentation](https://rancher.com/docs/k3s/latest/en/networking/#coredns) for details..
#### Klipper Service LB
##### Klipper Service LB
Option to enable or disable the [Klipper](https://github.com/rancher/klipper-lb) service load balancer. Refer to the [K3s documentation](https://rancher.com/docs/k3s/latest/en/networking/#service-load-balancer) for details.
#### Traefik Ingress
##### Traefik Ingress
Option to enable or disable the [Traefik](https://traefik.io/) HTTP reverse proxy and load balancer. For more details and configuration options, see the [K3s documentation](https://rancher.com/docs/k3s/latest/en/networking/#traefik-ingress-controller).
#### Local Storage
##### Local Storage
Option to enable or disable [local storage](https://rancher.com/docs/k3s/latest/en/storage/) on the node(s).
#### Metrics Server
##### Metrics Server
Option to enable or disable the [metrics server](https://github.com/kubernetes-incubator/metrics-server). If enabled, ensure port 10250 is opened for inbound TCP traffic.
### Add-On Config
#### Add-On Config
Additional Kubernetes manifests, managed as a [Add-on](https://kubernetes.io/docs/concepts/cluster-administration/addons/), to apply to the cluster on startup. Refer to the [K3s documentation](https://rancher.com/docs/k3s/latest/en/helm/#automatically-deploying-manifests-and-helm-charts) for details.
### Agent Environment Vars
#### Agent Environment Vars
Option to set environment variables for [K3s agents](https://rancher.com/docs/k3s/latest/en/architecture/). The environment variables can be set using key value pairs. Refer to the [K3 documentation](https://rancher.com/docs/k3s/latest/en/installation/install-options/agent-config/) for more details.
### etcd
#### etcd
#### Automatic Snapshots
##### Automatic Snapshots
Option to enable or disable recurring etcd snapshots. If enabled, users have the option to configure the frequency of snapshots. For details, refer to the [K3s documentation](https://rancher.com/docs/k3s/latest/en/backup-restore/#creating-snapshots).
#### Metrics
##### Metrics
Option to choose whether to expose etcd metrics to the public or only within the cluster.
### Networking
#### Networking
#### Cluster CIDR
##### Cluster CIDR
IPv4/IPv6 network CIDRs to use for pod IPs (default: 10.42.0.0/16).
#### Service CIDR
##### Service CIDR
IPv4/IPv6 network CIDRs to use for service IPs (default: 10.43.0.0/16).
#### Cluster DNS
##### Cluster DNS
IPv4 Cluster IP for coredns service. Should be in your service-cidr range (default: 10.43.0.10).
#### Cluster Domain
##### Cluster Domain
Select the domain for the cluster. The default is `cluster.local`.
#### NodePort Service Port Range
##### NodePort Service Port Range
Option to change the range of ports that can be used for [NodePort services](https://kubernetes.io/docs/concepts/services-networking/service/#nodeport). The default is `30000-32767`.
#### Truncate Hostnames
##### Truncate Hostnames
Option to truncate hostnames to 15 characters or less. You can only set this field during the initial creation of the cluster. You can't enable or disable the 15 character limit after cluster creation.
@@ -110,11 +174,11 @@ This setting only affects machine-provisioned clusters. Since custom clusters se
Truncating hostnames in a cluster improves compatibility with Windows-based systems. Although Kubernetes allows hostnames up to 63 characters in length, systems that use NetBIOS restrict hostnames to 15 characters or less.
#### TLS Alternate Names
##### TLS Alternate Names
Add additional hostnames or IPv4/IPv6 addresses as Subject Alternative Names on the server TLS cert.
#### Authorized Cluster Endpoint
##### Authorized Cluster Endpoint
Authorized Cluster Endpoint can be used to directly access the Kubernetes API server, without requiring communication through Rancher.
@@ -122,34 +186,249 @@ For more detail on how an authorized cluster endpoint works and why it is used,
We recommend using a load balancer with the authorized cluster endpoint. For details, refer to the [recommended architecture section.](../../rancher-manager-architecture/architecture-recommendations.md#architecture-for-an-authorized-cluster-endpoint-ace)
### Registries
#### Registries
Select the image repository to pull Rancher images from. For more details and configuration options, see the [K3s documentation](https://rancher.com/docs/k3s/latest/en/installation/private-registry/).
### Upgrade Strategy
#### Upgrade Strategy
#### Controle Plane Concurrency
##### Control Plane Concurrency
Select how many nodes can be upgraded at the same time. Can be a fixed number or percentage.
#### Worker Concurrency
##### Worker Concurrency
Select how many nodes can be upgraded at the same time. Can be a fixed number or percentage.
#### Drain Nodes (Control Plane)
##### Drain Nodes (Control Plane)
Option to remove all pods from the node prior to upgrading.
#### Drain Nodes (Worker Nodes)
##### Drain Nodes (Worker Nodes)
Option to remove all pods from the node prior to upgrading.
### Advanced
#### Advanced
Option to set kubelet options for different nodes. For available options, refer to the [Kubernetes documentation](https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/).
## Cluster Config File
## Cluster Config File Reference
Instead of using the Rancher UI forms to choose Kubernetes options for the cluster, advanced users can create an K3s config file. Using a config file allows you to set any of the [options](https://rancher.com/docs/k3s/latest/en/installation/install-options/) available in an K3s installation.
Editing clusters in YAML allows you to set configurations that are already listed in [Configuration Options in the Rancher UI](#configuration-options-in-the-rancher-ui), as well as set Rancher-specific parameters.
To edit an K3s config file directly from the Rancher UI, click **Edit as YAML**.
<details>
<summary>
<b>Example Cluster Config File Snippet</b>
</summary>
```yaml
apiVersion: provisioning.cattle.io/v1
kind: Cluster
spec:
cloudCredentialSecretName: cattle-global-data:cc-fllv6
clusterAgentDeploymentCustomization: {}
fleetAgentDeploymentCustomization: {}
kubernetesVersion: v1.26.7+k3s1
localClusterAuthEndpoint: {}
rkeConfig:
chartValues: {}
etcd:
snapshotRetention: 5
snapshotScheduleCron: 0 */5 * * *
machineGlobalConfig:
disable-apiserver: false
disable-cloud-controller: false
disable-controller-manager: false
disable-etcd: false
disable-kube-proxy: false
disable-network-policy: false
disable-scheduler: false
etcd-expose-metrics: false
kube-apiserver-arg:
- audit-policy-file=/etc/rancher/k3s/user-audit-policy.yaml
- audit-log-path=/etc/rancher/k3s/user-audit.logs
profile: null
secrets-encryption: false
machinePools:
- controlPlaneRole: true
etcdRole: true
machineConfigRef:
kind: Amazonec2Config
name: nc-test-pool1-pwl5h
name: pool1
quantity: 1
unhealthyNodeTimeout: 0s
workerRole: true
machineSelectorConfig:
- config:
docker: false
protect-kernel-defaults: false
selinux: false
machineSelectorFiles:
- fileSources:
- configMap:
name: ''
secret:
name: audit-policy
items:
- key: audit-policy
path: /etc/rancher/k3s/user-audit-policy.yaml
machineLabelSelector:
matchLabels:
rke.cattle.io/control-plane-role: 'true'
registries: {}
upgradeStrategy:
controlPlaneConcurrency: '1'
controlPlaneDrainOptions:
deleteEmptyDirData: true
disableEviction: false
enabled: false
force: false
gracePeriod: -1
ignoreDaemonSets: true
ignoreErrors: false
postDrainHooks: null
preDrainHooks: null
skipWaitForDeleteTimeoutSeconds: 0
timeout: 120
workerConcurrency: '1'
workerDrainOptions:
deleteEmptyDirData: true
disableEviction: false
enabled: false
force: false
gracePeriod: -1
ignoreDaemonSets: true
ignoreErrors: false
postDrainHooks: null
preDrainHooks: null
skipWaitForDeleteTimeoutSeconds: 0
timeout: 120
```
</details>
### chartValues
Specify the values for the system charts installed by K3s.
Example:
```yaml
chartValues:
chart-name:
key: value
```
### machineGlobalConfig
Specify K3s configurations. Any configuration change made here will apply to every node. The configuration options available in the [standalone version of k3s](https://docs.k3s.io/cli/server) can be applied here.
Example:
```yaml
machineGlobalConfig:
etcd-arg:
- key1=value1
- key2=value2
```
### machineSelectorConfig
`machineSelectorConfig` is the same as [`machineGlobalConfig`](#machineglobalconfig) except that a [label](#kubernetes-node-labels) selector can be specified with the configuration. The configuration will only be applied to nodes that match the provided label selector.
Multiple `config` entries are allowed, each specifying their own `machineLabelSelector`. A user can specify `matchExpressions`, `matchLabels`, both, or neither. Omitting the `machineLabelSelector` section of this field has the same effect as putting the config in the `machineGlobalConfig` section.
Example:
```yaml
machineSelectorConfig
- config:
config-key: config-value
machineLabelSelector:
matchExpressions:
- key: example-key
operator: string # Valid operators are In, NotIn, Exists and DoesNotExist.
values:
- example-value1
- example-value2
matchLabels:
key1: value1
key2: value2
```
### machineSelectorFiles
:::note
This feature is available in Rancher v2.7.2 and later.
:::
Deliver files to nodes, so that the files can be in place before initiating K3s server or agent processes.
The content of the file is retrieved from either a secret or a configmap. The target nodes are filtered by the `machineLabelSelector`.
Example :
```yaml
machineSelectorFiles:
- fileSources:
- secret:
items:
- key: example-key
path: path-to-put-the-file-on-nodes
permissions: 644 (optional)
hash: base64-encoded-hash-of-the-content (optional)
name: example-secret-name
machineLabelSelector:
matchExpressions:
- key: example-key
operator: string # Valid operators are In, NotIn, Exists and DoesNotExist.
values:
- example-value1
- example-value2
matchLabels:
key1: value1
key2: value2
- fileSources:
- configMap:
items:
- key: example-key
path: path-to-put-the-file-on-nodes
permissions: 644 (optional)
hash: base64-encoded-hash-of-the-content (optional)
name: example-configmap-name
machineLabelSelector:
matchExpressions:
- key: example-key
operator: string # Valid operators are In, NotIn, Exists and DoesNotExist.
values:
- example-value1
- example-value2
matchLabels:
key1: value1
key2: value2
```
The secret or configmap must meet the following requirements:
1. It must be in the `fleet-default` namespace where the Cluster object exists.
2. It must have the annotation `rke.cattle.io/object-authorized-for-clusters: cluster-name1,cluster-name2`, which permits the target clusters to use it.
:::tip
Rancher Dashboard provides an easy-to-use form for creating the secret or configmap.
:::
Example:
```yaml
apiVersion: v1
data:
audit-policy: >-
IyBMb2cgYWxsIHJlcXVlc3RzIGF0IHRoZSBNZXRhZGF0YSBsZXZlbC4KYXBpVmVyc2lvbjogYXVkaXQuazhzLmlvL3YxCmtpbmQ6IFBvbGljeQpydWxlczoKLSBsZXZlbDogTWV0YWRhdGE=
kind: Secret
metadata:
annotations:
rke.cattle.io/object-authorized-for-clusters: cluster1
name: name1
namespace: fleet-default
```
@@ -119,15 +119,15 @@ Rancher v2.6 introduced the ability to configure [ECR registries for RKE cluster
### Authorized Cluster Endpoint
Authorized Cluster Endpoint can be used to directly access the Kubernetes API server, without requiring communication through Rancher.
Authorized Cluster Endpoint (ACE) can be used to directly access the Kubernetes API server, without requiring communication through Rancher.
:::note
The authorized cluster endpoint only works on Rancher-launched Kubernetes clusters. In other words, it only works in clusters where Rancher [used RKE](../../../reference-guides/rancher-manager-architecture/communicating-with-downstream-user-clusters.md#tools-for-provisioning-kubernetes-clusters) to provision the cluster. It is not available for clusters in a hosted Kubernetes provider, such as Amazon's EKS.
ACE is available on RKE, RKE2, and K3s clusters that are provisioned or registered with Rancher. It's not available on clusters in a hosted Kubernetes provider, such as Amazon's EKS.
:::
This is enabled by default in Rancher-launched Kubernetes clusters, using the IP of the node with the `controlplane` role and the default Kubernetes self signed certificates.
ACE must be set up [manually](../../../how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/register-existing-clusters#authorized-cluster-endpoint-support-for-rke2-and-k3s-clusters.md) on RKE2 and K3s clusters. In RKE, ACE is enabled by default in Rancher-launched Kubernetes clusters, using the IP of the node with the `controlplane` role and the default Kubernetes self-signed certificates.
For more detail on how an authorized cluster endpoint works and why it is used, refer to the [architecture section.](../../../reference-guides/rancher-manager-architecture/communicating-with-downstream-user-clusters.md#4-authorized-cluster-endpoint)
@@ -15,18 +15,26 @@ You can configure the Kubernetes options in one of the two following ways:
- [Rancher UI](#configuration-options-in-the-rancher-ui): Use the Rancher UI to select options that are commonly customized when setting up a Kubernetes cluster.
- [Cluster Config File](#cluster-config-file-reference): Instead of using the Rancher UI to choose Kubernetes options for the cluster, advanced users can create an RKE2 config file. Using a config file allows you to set many additional [options](https://docs.rke2.io/install/configuration) available for an RKE2 installation.
## Editing Clusters with a Form in the Rancher UI
## Editing Clusters in the Rancher UI
The Rancher UI provides two ways to edit a cluster:
1. With a form.
1. With YAML.
### Editing Clusters with a Form
The form covers the most frequently needed options for clusters.
To edit your cluster,
1. In the upper left corner, click **☰ > Cluster Management**.
1. Click **☰ > Cluster Management**.
1. Go to the cluster you want to configure and click **⋮ > Edit Config**.
## Editing Clusters with YAML
### Editing Clusters in YAML
Instead of using the Rancher UI to choose Kubernetes options for the cluster, advanced users can create an RKE2 config file. Using a config file allows you to set any of the options available in an RKE2 installation by specifying them in YAML.
For a complete reference of configurable options for K3s clusters in YAML, see the [K3s documentation.](https://rancher.com/docs/k3s/latest/en/installation/install-options/)
To edit an RKE2 config file directly from the Rancher UI,
To edit your cluster in YAML:
1. Click **☰ > Cluster Management**.
1. Go to the cluster you want to configure and click **⋮ > Edit as YAML**.
@@ -34,62 +42,55 @@ To edit an RKE2 config file directly from the Rancher UI,
## Configuration Options in the Rancher UI
:::tip
### Machine Pool Configuration
Some advanced configuration options are not exposed in the Rancher UI forms, but they can be enabled by editing the RKE2 cluster configuration file in YAML. For the complete reference of configurable options for RKE2 Kubernetes clusters in YAML, see the [RKE2 documentation.](https://docs.rke2.io/install/configuration)
:::
## Machine Pool
This subsection covers the generic machine pool configurations. For infrastructure provider specific, configurations refer to the following pages:
This subsection covers generic machine pool configurations. For specific infrastructure provider configurations, refer to the following:
- [Azure](../downstream-cluster-configuration/machine-configuration/azure.md)
- [DigitalOcean](../downstream-cluster-configuration/machine-configuration/digitalocean.md)
- [EC2](../downstream-cluster-configuration/machine-configuration/amazon-ec2.md)
### Pool Name
##### Pool Name
The name of the machine pool.
### Machine Count
##### Machine Count
The number of machines in the pool.
### Roles
##### Roles
Option to assign etcd, control plane, and worker roles to nodes.
### Advanced
#### Advanced
#### Auto Replace
##### Auto Replace
The duration nodes can be unreachable before they are automatically deleted and replaced.
The amount of time nodes can be unreachable before they are automatically deleted and replaced.
#### Drain Before Delete
##### Drain Before Delete
Enables draining nodes by evicting all pods before the node is deleted.
#### Kubernetes Node Labels
##### Kubernetes Node Labels
Add [labels](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/) to nodes to help with organization and object selection.
For details on label syntax requirements, see the [Kubernetes documentation.](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#syntax-and-character-set)
#### Taints
##### Taints
Add [taints](https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/) to nodes, which can be used to prevent pods from being scheduled to or executed on nodes, unless the pods have matching tolerations.
Add [taints](https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/) to nodes, to prevent pods from being scheduled to or executed on the nodes, unless the pods have matching tolerations.
## Cluster Configuration
### Basics
#### Kubernetes Version
### Cluster Configuration
#### Basics
##### Kubernetes Version
The version of Kubernetes installed on your cluster nodes. Rancher packages its own version of Kubernetes based on [hyperkube](https://github.com/rancher/hyperkube).
For more detail, see [Upgrading Kubernetes](../../../getting-started/installation-and-upgrade/upgrade-and-roll-back-kubernetes.md).
#### Container Network Provider
##### Container Network Provider
The [Network Provider](https://kubernetes.io/docs/concepts/cluster-administration/networking/) that the cluster uses.
@@ -110,7 +111,7 @@ Out of the box, Rancher is compatible with the following network providers:
For more details on the different networking providers and how to configure them, please view our [RKE2 documentation](https://docs.rke2.io/install/network_options).
##### Dual-stack Networking
###### Dual-stack Networking
[Dual-stack](https://docs.rke2.io/install/network_options#dual-stack-configuration) networking is supported for all CNI providers. To configure RKE2 in dual-stack mode, set valid IPv4/IPv6 CIDRs for your [Cluster CIDR](#cluster-cidr) and/or [Service CIDR](#service-cidr).
@@ -118,7 +119,7 @@ For more details on the different networking providers and how to configure them
When using `cilium` or `multus,cilium` as your container network interface provider, ensure the **Enable IPv6 Support** option is also enabled.
#### Cloud Provider
##### Cloud Provider
You can configure a [Kubernetes cloud provider](../../../pages-for-subheaders/set-up-cloud-providers.md). If you want to use dynamically provisioned [volumes and storage](../../../pages-for-subheaders/create-kubernetes-persistent-storage.md) in Kubernetes, typically you must select the specific cloud provider in order to use it. For example, if you want to use Amazon EBS, you would need to select the `aws` cloud provider.
@@ -128,89 +129,93 @@ If the cloud provider you want to use is not listed as an option, you will need
:::
#### Default Pod Security Policy
##### Default Pod Security Policy
Choose the default [pod security policy](../../../how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/create-pod-security-policies.md) for the cluster. Please refer to the [RKE2 documentation](https://docs.rke2.io/security/pod_security_policies) on the specifications of each available policy.
The default [pod security policy](../../../how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/create-pod-security-policies.md) for the cluster. Please refer to the [RKE2 documentation](https://docs.rke2.io/security/pod_security_policies) on the specifications of each available policy.
#### Worker CIS Profile
##### Pod Security Admission Configuration Template
The default [pod security admission configuration template](../../../how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/psa-config-templates.md) for the cluster.
##### Worker CIS Profile
Select a [CIS benchmark](../../../pages-for-subheaders/cis-scan-guides.md) to validate the system configuration against.
#### Project Network Isolation
##### Project Network Isolation
If your network provider allows project network isolation, you can choose whether to enable or disable inter-project communication.
Project network isolation is available if you are using any RKE2 network plugin that supports the enforcement of Kubernetes network policies, such as Canal.
#### CoreDNS
##### CoreDNS
By default, [CoreDNS](https://coredns.io/) is installed as the default DNS provider. If CoreDNS is not installed, an alternate DNS provider must be installed yourself. Refer to the [RKE2 documentation](https://docs.rke2.io/networking#coredns) for additional CoreDNS configurations.
#### NGINX Ingress
##### NGINX Ingress
If you want to publish your applications in a high-availability configuration, and you're hosting your nodes with a cloud-provider that doesn't have a native load-balancing feature, enable this option to use NGINX Ingress within the cluster. Refer to the [RKE2 documentation](https://docs.rke2.io/networking#nginx-ingress-controller) for additional configuration options.
Refer to the [RKE2 documentation](https://docs.rke2.io/networking#nginx-ingress-controller) for additional configuration options.
#### Metrics Server
##### Metrics Server
Option to enable or disable [Metrics Server](https://rancher.com/docs/rke/latest/en/config-options/add-ons/metrics-server/).
Each cloud provider capable of launching a cluster using RKE2 can collect metrics and monitor for your cluster nodes. Enable this option to view your node metrics from your cloud provider's portal.
### Add-On Config
#### Add-On Config
Additional Kubernetes manifests, managed as an [Add-on](https://kubernetes.io/docs/concepts/cluster-administration/addons/), to apply to the cluster on startup. Refer to the [RKE2 documentation](https://docs.rke2.io/helm#automatically-deploying-manifests-and-helm-charts) for details.
### Agent Environment Vars
#### Agent Environment Vars
Option to set environment variables for [Rancher agents](https://rancher.com/docs/rancher/v2.6/en/cluster-provisioning/rke-clusters/rancher-agents/). The environment variables can be set using key value pairs. Refer to the [RKE2 documentation](https://docs.rke2.io/reference/linux_agent_config) for more details.
Option to set environment variables for [Rancher agents](../../../how-to-guides/new-user-guides/launch-kubernetes-with-rancher/about-rancher-agents.md). The environment variables can be set using key value pairs. Refer to the [RKE2 documentation](https://docs.rke2.io/reference/linux_agent_config) for more details.
### etcd
#### etcd
#### Automatic Snapshots
##### Automatic Snapshots
Option to enable or disable recurring etcd snapshots. If enabled, users have the option to configure the frequency of snapshots. For details, refer to the [RKE2 documentation](https://docs.rke2.io/backup_restore#creating-snapshots). Note that with RKE2, snapshots are stored on each etcd node. This varies from RKE1 which only stores one snapshot per cluster.
#### Metrics
##### Metrics
Option to choose whether to expose etcd metrics to the public or only within the cluster.
### Networking
#### Networking
#### Cluster CIDR
##### Cluster CIDR
IPv4 and/or IPv6 network CIDRs to use for pod IPs (default: 10.42.0.0/16).
##### Dual-stack Networking
###### Dual-stack Networking
To configure [dual-stack](https://docs.rke2.io/install/network_options#dual-stack-configuration) mode, enter a valid IPv4/IPv6 CIDR. For example `10.42.0.0/16,2001:cafe:42:0::/56`.
[Additional configuration](#dual-stack-additional-config) is required when using `cilium` or `multus,cilium` as your [container network](#container-network-provider) interface provider.
#### Service CIDR
##### Service CIDR
IPv4/IPv6 network CIDRs to use for service IPs (default: 10.43.0.0/16).
##### Dual-stack Networking
###### Dual-stack Networking
To configure [dual-stack](https://docs.rke2.io/install/network_options#dual-stack-configuration) mode, enter a valid IPv4/IPv6 CIDR. For example `10.42.0.0/16,2001:cafe:42:0::/56`.
[Additional configuration](#dual-stack-additional-config) is required when using `cilium ` or `multus,cilium` as your [container network](#container-network-provider) interface provider.
#### Cluster DNS
##### Cluster DNS
IPv4 Cluster IP for coredns service. Should be in your service-cidr range (default: 10.43.0.10).
#### Cluster Domain
##### Cluster Domain
Select the domain for the cluster. The default is `cluster.local`.
#### NodePort Service Port Range
##### NodePort Service Port Range
Option to change the range of ports that can be used for [NodePort services](https://kubernetes.io/docs/concepts/services-networking/service/#nodeport). The default is `30000-32767`.
#### Truncate Hostnames
##### Truncate Hostnames
Option to truncate hostnames to 15 characters or less. You can only set this field during the initial creation of the cluster. You can't enable or disable the 15 character limit after cluster creation.
@@ -218,11 +223,11 @@ This setting only affects machine-provisioned clusters. Since custom clusters se
Truncating hostnames in a cluster improves compatibility with Windows-based systems. Although Kubernetes allows hostnames up to 63 characters in length, systems that use NetBIOS restrict hostnames to 15 characters or less.
#### TLS Alternate Names
##### TLS Alternate Names
Add additional hostnames or IPv4/IPv6 addresses as Subject Alternative Names on the server TLS cert.
#### Authorized Cluster Endpoint
##### Authorized Cluster Endpoint
Authorized Cluster Endpoint can be used to directly access the Kubernetes API server, without requiring communication through Rancher.
@@ -232,35 +237,35 @@ For more detail on how an authorized cluster endpoint works and why it is used,
We recommend using a load balancer with the authorized cluster endpoint. For details, refer to the [recommended architecture section.](../../rancher-manager-architecture/architecture-recommendations.md#architecture-for-an-authorized-cluster-endpoint-ace)
### Registries
#### Registries
Select the image repository to pull Rancher images from. For more details and configuration options, see the [RKE2 documentation](https://docs.rke2.io/install/containerd_registry_configuration).
### Upgrade Strategy
#### Upgrade Strategy
#### Control Plane Concurrency
##### Control Plane Concurrency
Select how many nodes can be upgraded at the same time. Can be a fixed number or percentage.
#### Worker Concurrency
##### Worker Concurrency
Select how many nodes can be upgraded at the same time. Can be a fixed number or percentage.
#### Drain Nodes (Control Plane)
##### Drain Nodes (Control Plane)
Option to remove all pods from the node prior to upgrading.
#### Drain Nodes (Worker Nodes)
##### Drain Nodes (Worker Nodes)
Option to remove all pods from the node prior to upgrading.
### Advanced
#### Advanced
Option to set kubelet options for different nodes. For available options, refer to the [Kubernetes documentation](https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/).
## Cluster Config File Reference
Instead of using the Rancher UI to choose Kubernetes options for the cluster, advanced users can create a config file. Using a config file allows you to set the [options available](https://docs.rke2.io/install/configuration) in an RKE2 installation, including those already listed in [Configuration Options in the Rancher UI](#configuration-options-in-the-rancher-ui), as well as Rancher-specific parameters.
Editing clusters in YAML allows you to set the [options available](https://docs.rke2.io/install/configuration) in an RKE2 installation, including those already listed in [Configuration Options in the Rancher UI](#configuration-options-in-the-rancher-ui), as well as set Rancher-specific parameters.
<details>
<summary>
@@ -268,9 +273,11 @@ Instead of using the Rancher UI to choose Kubernetes options for the cluster, ad
</summary>
```yaml
apiVersion: provisioning.cattle.io/v1
kind: Cluster
spec:
cloudCredentialSecretName: cattle-global-data:cc-s879v
kubernetesVersion: v1.23.6+rke2r2
kubernetesVersion: v1.25.12+rke2r1
localClusterAuthEndpoint: {}
rkeConfig:
chartValues:
@@ -283,6 +290,9 @@ spec:
disable-kube-proxy: false
etcd-expose-metrics: false
profile: null
kube-apiserver-arg:
- audit-policy-file=/etc/rancher/rke2/user-audit-policy.yaml
- audit-log-path=/etc/rancher/rke2/user-audit.logs
machinePools:
- controlPlaneRole: true
etcdRole: true
@@ -296,6 +306,18 @@ spec:
machineSelectorConfig:
- config:
protect-kernel-defaults: false
machineSelectorFiles:
- fileSources:
- configMap:
name: ''
secret:
name: audit-policy
items:
- key: audit-policy
path: /etc/rancher/rke2/user-audit-policy.yaml
machineLabelSelector:
matchLabels:
rke.cattle.io/control-plane-role: 'true'
registries: {}
upgradeStrategy:
controlPlaneConcurrency: "1"
@@ -317,7 +339,7 @@ spec:
### chartValues
Option to specify the values for the system charts installed by RKE2/k3s.
Specify the values for the system charts installed by RKE2.
Example:
@@ -328,7 +350,7 @@ chartValues:
```
### machineGlobalConfig
The RKE2/K3s configurations are nested under the `machineGlobalConfig` directive. Any configuration change made here will apply to every node. The configuration options available in the [standalone version of RKE2](https://docs.rke2.io/reference/server_config) can be applied here.
Specify RKE2 configurations. Any configuration change made here will apply to every node. The configuration options available in the [standalone version of RKE2](https://docs.rke2.io/reference/server_config) can be applied here.
Example:
@@ -341,9 +363,9 @@ machineGlobalConfig:
### machineSelectorConfig
This is the same as [`machineGlobalConfig`](#machineglobalconfig) except that a [label](#kubernetes-node-labels) selector can be specified with the configuration. The configuration will only be applied to nodes that match the provided label selector.
`machineSelectorConfig` is the same as [`machineGlobalConfig`](#machineglobalconfig) except that a [label](#kubernetes-node-labels) selector can be specified with the configuration. The configuration will only be applied to nodes that match the provided label selector.
Multiple `config` entries are allowed, each specifying their own `machineLabelSelector`. A user can specify `matchExpressions`, `matchLabels`, both, or neither. Omitting the `machineLabelSelector` section of this has the same effect as putting the config in the `machineGlobalConfig` section.
Multiple `config` entries are allowed, each specifying their own `machineLabelSelector`. A user can specify `matchExpressions`, `matchLabels`, both, or neither. Omitting the `machineLabelSelector` section of this field has the same effect as putting the config in the `machineGlobalConfig` section.
Example:
@@ -362,3 +384,81 @@ machineSelectorConfig
key1: value1
key2: value2
```
### machineSelectorFiles
:::note
This feature is available in Rancher v2.7.2 and later.
:::
Deliver files to nodes, so that the files can be in place before initiating RKE2 server or agent processes.
The content of the file is retrieved from either a secret or a configmap. The target nodes are filtered by the `machineLabelSelector`.
Example :
```yaml
machineSelectorFiles:
- fileSources:
- secret:
items:
- key: example-key
path: path-to-put-the-file-on-nodes
permissions: 644 (optional)
hash: base64-encoded-hash-of-the-content (optional)
name: example-secret-name
machineLabelSelector:
matchExpressions:
- key: example-key
operator: string # Valid operators are In, NotIn, Exists and DoesNotExist.
values:
- example-value1
- example-value2
matchLabels:
key1: value1
key2: value2
- fileSources:
- configMap:
items:
- key: example-key
path: path-to-put-the-file-on-nodes
permissions: 644 (optional)
hash: base64-encoded-hash-of-the-content (optional)
name: example-configmap-name
machineLabelSelector:
matchExpressions:
- key: example-key
operator: string # Valid operators are In, NotIn, Exists and DoesNotExist.
values:
- example-value1
- example-value2
matchLabels:
key1: value1
key2: value2
```
The secret or configmap must meet the following requirements:
1. It must be in the `fleet-default` namespace where the Cluster object exists.
2. It must have the annotation `rke.cattle.io/object-authorized-for-clusters: cluster-name1,cluster-name2`, which permits the target clusters to use it.
:::tip
Rancher Dashboard provides an easy-to-use form for creating the secret or configmap.
:::
Example:
```yaml
apiVersion: v1
data:
audit-policy: >-
IyBMb2cgYWxsIHJlcXVlc3RzIGF0IHRoZSBNZXRhZGF0YSBsZXZlbC4KYXBpVmVyc2lvbjogYXVkaXQuazhzLmlvL3YxCmtpbmQ6IFBvbGljeQpydWxlczoKLSBsZXZlbDogTWV0YWRhdGE=
kind: Secret
metadata:
annotations:
rke.cattle.io/object-authorized-for-clusters: cluster1
name: name1
namespace: fleet-default
```
@@ -2,6 +2,10 @@
title: Syncing
---
<head>
<link rel="canonical" href="https://ranchermanager.docs.rancher.com//reference-guides/cluster-configuration/rancher-server-configuration/sync-clusters"/>
</head>
Syncing allows Rancher to update cluster values so that they're up to date with the corresponding cluster object hosted in AKS, EKS or GKE. This enables sources other than Rancher to own a hosted cluster’s state.
:::warning
@@ -2,6 +2,10 @@
title: Rancher Agent Options
---
<head>
<link rel="canonical" href="https://ranchermanager.docs.rancher.com//reference-guides/cluster-configuration/rancher-server-configuration/use-existing-nodes/rancher-agent-options"/>
</head>
Rancher deploys an agent on each node to communicate with the node. This pages describes the options that can be passed to the agent. To use these options, you will need to [create a cluster with custom nodes](../../../../pages-for-subheaders/use-existing-nodes.md) and add the options to the generated `docker run` command when adding a node.
For an overview of how Rancher communicates with downstream clusters using node agents, refer to the [architecture section.](../../../rancher-manager-architecture/communicating-with-downstream-user-clusters.md#3-node-agents)
@@ -2,6 +2,10 @@
title: Examples
---
<head>
<link rel="canonical" href="https://ranchermanager.docs.rancher.com//reference-guides/monitoring-v2-configuration/examples"/>
</head>
### ServiceMonitor
An example ServiceMonitor custom resource can be found [here.](https://github.com/prometheus-operator/prometheus-operator/blob/master/example/prometheus-operator-crd/monitoring.coreos.com_servicemonitors.yaml)
@@ -2,6 +2,10 @@
title: Helm Chart Options
---
<head>
<link rel="canonical" href="https://ranchermanager.docs.rancher.com//reference-guides/monitoring-v2-configuration/helm-chart-options"/>
</head>
## Configuring Resource Limits and Requests
The resource requests and limits can be configured when installing `rancher-monitoring`.
@@ -2,6 +2,10 @@
title: Receiver Configuration
---
<head>
<link rel="canonical" href="https://ranchermanager.docs.rancher.com//reference-guides/monitoring-v2-configuration/receivers"/>
</head>
The [Alertmanager Config](https://prometheus.io/docs/alerting/latest/configuration/#configuration-file) Secret contains the configuration of an Alertmanager instance that sends out notifications based on alerts it receives from Prometheus.
:::note
@@ -2,6 +2,10 @@
title: Route Configuration
---
<head>
<link rel="canonical" href="https://ranchermanager.docs.rancher.com//reference-guides/monitoring-v2-configuration/routes"/>
</head>
The route configuration is the section of the Alertmanager custom resource that controls how the alerts fired by Prometheus are grouped and filtered before they reach the receiver.
When a Route is changed, the Prometheus Operator regenerates the Alertmanager custom resource to reflect the changes.
@@ -2,6 +2,10 @@
title: ServiceMonitor and PodMonitor Configuration
---
<head>
<link rel="canonical" href="https://ranchermanager.docs.rancher.com//reference-guides/monitoring-v2-configuration/servicemonitors-and-podmonitors"/>
</head>
ServiceMonitors and PodMonitors are both pseudo-CRDs that map the scrape configuration of the Prometheus custom resource.
These configuration objects declaratively specify the endpoints that Prometheus will scrape metrics from.
@@ -56,9 +56,9 @@ The `cattle-node-agent` is deployed using a [DaemonSet](https://kubernetes.io/do
### 4. Authorized Cluster Endpoint
An authorized cluster endpoint allows users to connect to the Kubernetes API server of a downstream cluster without having to route their requests through the Rancher authentication proxy.
An authorized cluster endpoint (ACE) allows users to connect to the Kubernetes API server of a downstream cluster without having to route their requests through the Rancher authentication proxy.
> The authorized cluster endpoint only works on Rancher-launched Kubernetes clusters. In other words, it only works in clusters where Rancher [used RKE](../../pages-for-subheaders/launch-kubernetes-with-rancher.md) to provision the cluster. It is not available for imported clusters, or for clusters in a hosted Kubernetes provider, such as Amazon's EKS.
> ACE is available on RKE, RKE2, and K3s clusters that are provisioned or registered with Rancher. It's not available on clusters in a hosted Kubernetes provider, such as Amazon's EKS.
There are two main reasons why a user might need the authorized cluster endpoint:
@@ -129,4 +129,4 @@ The GitHub repositories for Rancher can be found at the following links:
- [Rancher CLI](https://github.com/rancher/cli)
- [Catalog applications](https://github.com/rancher/helm)
This is a partial list of the most important Rancher repositories. For more details about Rancher source code, refer to the section on [contributing to Rancher.](../../contribute-to-rancher.md#repositories) To see all libraries and projects used in Rancher, see the [`go.mod` file](https://github.com/rancher/rancher/blob/master/go.mod) in the `rancher/rancher` repository.
This is a partial list of the most important Rancher repositories. For more details about Rancher source code, refer to the section on [contributing to Rancher.](../../contribute-to-rancher.md#rancher-repositories) To see all libraries and projects used in Rancher, see the [`go.mod` file](https://github.com/rancher/rancher/blob/master/go.mod) in the `rancher/rancher` repository.
@@ -19,10 +19,10 @@ Rancher is committed to informing the community of security issues in our produc
| [CVE-2022-31247](https://github.com/rancher/rancher/security/advisories/GHSA-6x34-89p7-95wg) | An issue was discovered in Rancher versions up to and including 2.5.15 and 2.6.6 where a flaw with authorization logic allows privilege escalation in downstream clusters through cluster role template binding (CRTB) and project role template binding (PRTB). The vulnerability can be exploited by any user who has permissions to create/edit CRTB or PRTB (such as `cluster-owner`, `manage cluster members`, `project-owner`, and `manage project members`) to gain owner permission in another project in the same cluster or in another project on a different downstream cluster. | 18 August 2022 | [Rancher v2.6.7](https://github.com/rancher/rancher/releases/tag/v2.6.7) and [Rancher v2.5.16](https://github.com/rancher/rancher/releases/tag/v2.5.16) |
| [CVE-2021-36783](https://github.com/rancher/rancher/security/advisories/GHSA-8w87-58w6-hfv8) | It was discovered that in Rancher versions up to and including 2.5.12 and 2.6.3, there is a failure to properly sanitize credentials in cluster template answers. This failure can lead to plaintext storage and exposure of credentials, passwords, and API tokens. The exposed credentials are visible in Rancher to authenticated `Cluster Owners`, `Cluster Members`, `Project Owners`, and `Project Members` on the endpoints `/v1/management.cattle.io.clusters`, `/v3/clusters`, and `/k8s/clusters/local/apis/management.cattle.io/v3/clusters`. | 18 August 2022 | [Rancher v2.6.7](https://github.com/rancher/rancher/releases/tag/v2.6.7) and [Rancher v2.5.16](https://github.com/rancher/rancher/releases/tag/v2.5.16) |
| [CVE-2021-36782](https://github.com/rancher/rancher/security/advisories/GHSA-g7j7-h4q8-8w2f) | An issue was discovered in Rancher versions up to and including 2.5.15 and 2.6.6 where sensitive fields like passwords, API keys, and Rancher's service account token (used to provision clusters) were stored in plaintext directly on Kubernetes objects like `Clusters` (e.g., `cluster.management.cattle.io`). Anyone with read access to those objects in the Kubernetes API could retrieve the plaintext version of those sensitive data. The issue was partially found and reported by Florian Struck (from [Continum AG](https://www.continum.net/)) and [Marco Stuurman](https://github.com/fe-ax) (from [Shock Media B.V.](https://www.shockmedia.nl/)). | 18 August 2022 | [Rancher v2.6.7](https://github.com/rancher/rancher/releases/tag/v2.6.7) and [Rancher v2.5.16](https://github.com/rancher/rancher/releases/tag/v2.5.16) |
| [CVE-2022-21951](https://github.com/rancher/rancher/security/advisories/GHSA-vrph-m5jj-c46c) | This vulnerability only affects customers using [Weave](https://rancher.com/docs/rancher/v2.6/en/faq/networking/cni-providers/#weave) Container Network Interface (CNI) when configured through [RKE templates](https://rancher.com/docs/rancher/v2.6/en/admin-settings/rke-templates/). A vulnerability was discovered in Rancher versions 2.5.0 up to and including 2.5.13, and 2.6.0 up to and including 2.6.4, where a user interface (UI) issue with RKE templates does not include a value for the Weave password when Weave is chosen as the CNI. If a cluster is created based on the mentioned template, and Weave is configured as the CNI, no password will be created for [network encryption](https://www.weave.works/docs/net/latest/tasks/manage/security-untrusted-networks/) in Weave; therefore, network traffic in the cluster will be sent unencrypted. | 24 May 2022 | [Rancher v2.6.5](https://github.com/rancher/rancher/releases/tag/v2.6.5) and [Rancher v2.5.14](https://github.com/rancher/rancher/releases/tag/v2.5.14) |
| [CVE-2021-36784](https://github.com/rancher/rancher/security/advisories/GHSA-jwvr-vv7p-gpwq) | A vulnerability was discovered in Rancher versions from 2.5.0 up to and including 2.5.12 and from 2.6.0 up to and including 2.6.3 which allows users who have create or update permissions on [Global Roles](https://rancher.com/docs/rancher/v2.6/en/admin-settings/rbac/) to escalate their permissions, or those of another user, to admin-level permissions. Global Roles grant users Rancher-wide permissions, such as the ability to create clusters. In the identified versions of Rancher, when users are given permission to edit or create Global Roles, they are not restricted to only granting permissions which they already posses. This vulnerability affects customers who utilize non-admin users that are able to create or edit Global Roles. The most common use case for this scenario is the `restricted-admin` role. | 14 Apr 2022 | [Rancher v2.6.4](https://github.com/rancher/rancher/releases/tag/v2.6.4) and [Rancher v2.5.13](https://github.com/rancher/rancher/releases/tag/v2.5.13) |
| [CVE-2022-21951](https://github.com/rancher/rancher/security/advisories/GHSA-vrph-m5jj-c46c) | This vulnerability only affects customers using [Weave](../../faq/container-network-interface-providers.md#weave) Container Network Interface (CNI) when configured through [RKE templates](../../pages-for-subheaders/about-rke1-templates.md). A vulnerability was discovered in Rancher versions 2.5.0 up to and including 2.5.13, and 2.6.0 up to and including 2.6.4, where a user interface (UI) issue with RKE templates does not include a value for the Weave password when Weave is chosen as the CNI. If a cluster is created based on the mentioned template, and Weave is configured as the CNI, no password will be created for [network encryption](https://www.weave.works/docs/net/latest/tasks/manage/security-untrusted-networks/) in Weave; therefore, network traffic in the cluster will be sent unencrypted. | 24 May 2022 | [Rancher v2.6.5](https://github.com/rancher/rancher/releases/tag/v2.6.5) and [Rancher v2.5.14](https://github.com/rancher/rancher/releases/tag/v2.5.14) |
| [CVE-2021-36784](https://github.com/rancher/rancher/security/advisories/GHSA-jwvr-vv7p-gpwq) | A vulnerability was discovered in Rancher versions from 2.5.0 up to and including 2.5.12 and from 2.6.0 up to and including 2.6.3 which allows users who have create or update permissions on [Global Roles](../../pages-for-subheaders/manage-role-based-access-control-rbac.md) to escalate their permissions, or those of another user, to admin-level permissions. Global Roles grant users Rancher-wide permissions, such as the ability to create clusters. In the identified versions of Rancher, when users are given permission to edit or create Global Roles, they are not restricted to only granting permissions which they already posses. This vulnerability affects customers who utilize non-admin users that are able to create or edit Global Roles. The most common use case for this scenario is the `restricted-admin` role. | 14 Apr 2022 | [Rancher v2.6.4](https://github.com/rancher/rancher/releases/tag/v2.6.4) and [Rancher v2.5.13](https://github.com/rancher/rancher/releases/tag/v2.5.13) |
| [CVE-2021-4200](https://github.com/rancher/rancher/security/advisories/GHSA-hx8w-ghh8-r4xf) | This vulnerability only affects customers using the `restricted-admin` role in Rancher. A vulnerability was discovered in Rancher versions from 2.5.0 up to and including 2.5.12 and from 2.6.0 up to and including 2.6.3 where the `global-data` role in `cattle-global-data` namespace grants write access to the Catalogs. Since each user with any level of catalog access was bound to the `global-data` role, this grants write access to templates (`CatalogTemplates`) and template versions (`CatalogTemplateVersions`) for any user with any level of catalog access. New users created in Rancher are by default assigned to the `user` role (standard user), which is not designed to grant write catalog access. This vulnerability effectively elevates the privilege of any user to write access for the catalog template and catalog template version resources. | 14 Apr 2022 | [Rancher v2.6.4](https://github.com/rancher/rancher/releases/tag/v2.6.4) and [Rancher v2.5.13](https://github.com/rancher/rancher/releases/tag/v2.5.13) |
| [GHSA-wm2r-rp98-8pmh](https://github.com/rancher/rancher/security/advisories/GHSA-wm2r-rp98-8pmh) | This vulnerability only affects customers using [Fleet](https://rancher.com/docs/rancher/v2.6/en/deploy-across-clusters/fleet/) for continuous delivery with authenticated Git and/or Helm repositories. An issue was discovered in `go-getter` library in versions prior to [`v1.5.11`](https://github.com/hashicorp/go-getter/releases/tag/v1.5.11) that exposes SSH private keys in base64 format due to a failure in redacting such information from error messages. The vulnerable version of this library is used in Rancher through Fleet in versions of Fleet prior to [`v0.3.9`](https://github.com/rancher/fleet/releases/tag/v0.3.9). This issue affects Rancher versions 2.5.0 up to and including 2.5.12 and from 2.6.0 up to and including 2.6.3. The issue was found and reported by Dagan Henderson from Raft Engineering. | 14 Apr 2022 | [Rancher v2.6.4](https://github.com/rancher/rancher/releases/tag/v2.6.4) and [Rancher v2.5.13](https://github.com/rancher/rancher/releases/tag/v2.5.13) |
| [GHSA-wm2r-rp98-8pmh](https://github.com/rancher/rancher/security/advisories/GHSA-wm2r-rp98-8pmh) | This vulnerability only affects customers using [Fleet](../../how-to-guides/new-user-guides/deploy-apps-across-clusters/fleet.md) for continuous delivery with authenticated Git and/or Helm repositories. An issue was discovered in `go-getter` library in versions prior to [`v1.5.11`](https://github.com/hashicorp/go-getter/releases/tag/v1.5.11) that exposes SSH private keys in base64 format due to a failure in redacting such information from error messages. The vulnerable version of this library is used in Rancher through Fleet in versions of Fleet prior to [`v0.3.9`](https://github.com/rancher/fleet/releases/tag/v0.3.9). This issue affects Rancher versions 2.5.0 up to and including 2.5.12 and from 2.6.0 up to and including 2.6.3. The issue was found and reported by Dagan Henderson from Raft Engineering. | 14 Apr 2022 | [Rancher v2.6.4](https://github.com/rancher/rancher/releases/tag/v2.6.4) and [Rancher v2.5.13](https://github.com/rancher/rancher/releases/tag/v2.5.13) |
| [CVE-2021-36778](https://github.com/rancher/rancher/security/advisories/GHSA-4fc7-hc63-7fjg) | A vulnerability was discovered in Rancher versions from 2.5.0 up to and including 2.5.11 and from 2.6.0 up to and including 2.6.2, where an insufficient check of the same-origin policy when downloading Helm charts from a configured private repository can lead to exposure of the repository credentials to a third-party provider. This issue only happens when the user configures access credentials to a private repository in Rancher inside `Apps & Marketplace > Repositories`. The issue was found and reported by Martin Andreas Ullrich. | 14 Apr 2022 | [Rancher v2.6.3](https://github.com/rancher/rancher/releases/tag/v2.6.3) and [Rancher v2.5.12](https://github.com/rancher/rancher/releases/tag/v2.5.12) |
| [GHSA-hwm2-4ph6-w6m5](https://github.com/rancher/rancher/security/advisories/GHSA-hwm2-4ph6-w6m5) | A vulnerability was discovered in versions of Rancher starting 2.0 up to and including 2.6.3. The `restricted` pod security policy (PSP) provided in Rancher deviated from the upstream `restricted` policy provided in Kubernetes on account of which Rancher's PSP had `runAsUser` set to `runAsAny`, while upstream had `runAsUser` set to `MustRunAsNonRoot`. This allowed containers to run as any user, including a privileged user (`root`), even when Rancher's `restricted` policy was enforced on a project or at the cluster level. | 31 Mar 2022 | [Rancher v2.6.4](https://github.com/rancher/rancher/releases/tag/v2.6.4) |
| [CVE-2021-36775](https://github.com/rancher/rancher/security/advisories/GHSA-28g7-896h-695v) | A vulnerability was discovered in Rancher versions up to and including 2.4.17, 2.5.11 and 2.6.2. After removing a `Project Role` associated with a group from the project, the bindings that granted access to cluster-scoped resources for those subjects were not deleted. This was due to an incomplete authorization logic check. A user who was a member of the affected group with authenticated access to Rancher could exploit this vulnerability to access resources they shouldn't have had access to. The exposure level would depend on the original permission level granted to the affected project role. This vulnerability only affected customers using group based authentication in Rancher. | 31 Mar 2022 | [Rancher v2.6.3](https://github.com/rancher/rancher/releases/tag/v2.6.3), [Rancher v2.5.12](https://github.com/rancher/rancher/releases/tag/v2.5.12) and [Rancher v2.4.18](https://github.com/rancher/rancher/releases/tag/v2.4.18) |
@@ -11,12 +11,47 @@ Rancher-Webhook is an essential component of Rancher that works in conjunction w
It integrates with Kubernetes' extensible admission controllers, as described in the [Kubernetes documentation](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/), which allows Rancher-Webhook to inspect specific requests sent to the Kubernetes API server, and add custom, Rancher-specific validation and mutations to the requests that are specific to Rancher. Rancher-Webhook manages the resources to be validated using the `rancher.cattle.io` `ValidatingWebhookConfiguration` and the `rancher.cattle.io` `MutatingWebhookConfiguration`, and will override any manual edits.
Rancher deploys Rancher-Webhook as a separate deployment and service in both local and downstream clusters. Rancher manages Rancher-Webhook using Helm. It's important to note that Rancher may override modifications made by users to the Helm release.
## Why do we need it?
Each Rancher version is designed to be compatible with a single version of the webhook. The compatible versions are provided below for convenience.
**Note:** Rancher manages deployment and upgrade of the webhook. Under most circumstances, no user intervention should be needed to ensure that the webhook version is compatible with the version of Rancher that you are running.
| Rancher Version | Webhook Version |
|-----------------|:---------------:|
| v2.7.0 | v0.3.0 |
| v2.7.1 | v0.3.0 |
| v2.7.2 | v0.3.2 |
| v2.7.3 | v0.3.3 |
| v2.7.4 | v0.3.4 |
| v2.7.5 | v0.3.5 |
| v2.7.6 | v0.3.5 |
## Why Do We Need It?
Rancher-Webhook is crucial for Rancher to protect clusters against malicious attacks and enable various features.
Rancher relies on the Rancher-Webhook as an integral part of its functionality. Without the webhook, Rancher would not be a complete product.
It provides essential protection for Rancher-managed clusters, preventing security vulnerabilities and ensuring the consistency and stability of the cluster.
## What Resources Does the Webhook Validate?
An in-progress list of the resources that the webhook validates can be found in the [webhook's repo](https://github.com/rancher/webhook/blob/release/v0.4/docs.md). Checks specific to one version can be found by viewing the `docs.md` file associated with a particular tag (note that webhook versions prior to `v0.3.6` won't have this file).
## Bypassing the Webhook
Sometimes, it may be necessary to bypass Rancher's webhook validation to perform emergency restore operations, or fix other critical issues. The bypass operation is exhaustive, meaning that no webhook checks will apply when this is used. It is not possible to bypass some checks and have others still apply - they are either all bypassed, or all active.
:::danger
Rancher's webhook provides critical security protections. Bypassing the webhook should only be done by administrators in specific scenarios, after all other options have been exhausted. In addition, permission to bypass the webhook should be carefully controlled, and never given to users who are not admins.
:::
To bypass the webhook, impersonate both the `rancher-webhook-sudo` service account and the `system:masters` group (both are required):
```bash
kubectl create -f example.yaml --as=system:serviceaccount:cattle-system:rancher-webhook-sudo --as-group=system:masters
```
## Common Issues
### EKS Cluster with Calico CNI
@@ -38,3 +73,34 @@ When using a private GKE cluster, errors may occur that prevent the Kubernetes A
Internal error occurred: failed calling webhook "rancher.cattle.io.namespaces.create-non-kubesystem": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation/namespaces?timeout=10s": context deadline exceeded
```
This issue occurs because firewall rules restrict communication between the API server and the private cluster. To resolve this communication problem, users must add firewall rules to allow the GKE control plane to communicate with the Rancher-Webhook on port 9443. Please refer to the [GKE documentation](https://cloud.google.com/kubernetes-engine/docs/how-to/private-clusters#add_firewall_rules) for detailed information and steps on updating the firewall rules.
### Application Fails to Deploy Due to rancher-webhook Blocking Access
The webhook provides extra validations on [namespaces](https://github.com/rancher/webhook/blob/release/v0.4/docs.md#psa-label-validation). One of these validations ensures that users can only update PSA relevant labels if they have the proper permissions (`updatepsa` for `projects` in `management.cattle.io`). This can result in specific operators, such as Tigera or Trident, failing when they attempt to deploy namespaces with PSA labels. There are several ways to resolve this issue:
- Configure the application to create a namespace with no PSA labels. If users wish to apply a PSA to these namespaces, they can add them to a project with the desired PSA after configuration. See the [docs on PSS and PSA resources](../../how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/pod-security-standards) for instructions on how.
- This is the preferred option, though not all applications can be configured in this fashion.
- Manually grant the operator permissions to manage PSAs for namespaces.
- This option will introduce security risks, since the operator will now be able to set the PSA for the namespaces it has access to. This could allow the operator to deploy a privileged pod, or effect cluster takeover through other means.
- A user account with the proper permissions can pre-create the namespace with the appropriate configuration.
- This option depends on the ability of the application to handle existing resources.
## Issues on Specific Versions
**Note:** The following is an incomplete list of high-severity issues affecting specific Rancher/webhook versions. In most cases, these issues can be resolved by upgrading to a more recent Rancher version.
### Incompatible Webhook Version on Rollback
**Note:** This affects rolling back to Rancher v2.7.5 or earlier.
If you roll back to Rancher v2.7.5 or earlier, you may see webhook versions that are too recent to be compatible with downstream clusters running pre-v2.7.5 version of Rancher. This may cause various incompatibility issues. For example, project members may be unable to create namespaces. In addition, when you roll back to versions before the webhook was installed in downstream clusters, the webhook may remain installed, which can result in similar incompatibility issues.
To help alleviate these issues, you can run the [adjust-downstream-webhook](https://github.com/rancherlabs/support-tools/tree/master/adjust-downstream-webhook) shell script after roll back. This script selects and installs the proper webhook version (or removes the webhook entirely) for the corresponding Rancher version.
### Project Members Can't Create Namespaces
**Note:** This affects Rancher versions `v2.7.2 - v2.7.4`
Project users who aren't owners may not be able to create namespaces in projects. This issue is caused by Rancher automatically upgrading the webhook to a version compatible with a more recent version of Rancher than the one currently installed.
To help alleviate these issues, you can run the [adjust-downstream-webhook](https://github.com/rancherlabs/support-tools/tree/master/adjust-downstream-webhook) shell script after roll back. This script selects and installs the proper webhook version (or removes the webhook entirely) for the corresponding Rancher version.