Sync content from rancher/docs

Start: cb708da8d Jul 7 2022 End: 09d646cfc Sep 2 2022
2026-09-12 04:05:50 +00:00 · 2022-09-09 14:21:51 -07:00
parent 38da4eb019
commit 80b78c98d5
62 changed files with 1868 additions and 467 deletions
@@ -0,0 +1,97 @@
+---
+title: Prerequisites
+weight: 1
+---
+
+### 1. Setting Up License Manager and Purchasing Support
+
+First, complete the [first step](https://docs.aws.amazon.com/license-manager/latest/userguide/getting-started.html) of the license manager one-time setup.
+Next, go to the AWS Marketplace. Locate the "Rancher Premium Support Billing Container Starter Pack". Purchase at least one entitlement.
+
+If you have installed Rancher using the "Rancher Setup" AWS Marketplace offering, skip to [Step 4](#4-create-an-oidc-provider).
+
+> **Note:** Each entitlement grants access to support for a certain amount of nodes. You can purchase more licenses as necessary later on.
+
+### 2. Create an EKS Cluster
+Follow the [Rancher docs](../../install-upgrade-on-a-kubernetes-cluster/rancher-on-amazon-eks.md) to create an EKS cluster. When you get to the [final step to install Rancher](../../install-upgrade-on-a-kubernetes-cluster/rancher-on-amazon-eks.md#8-install-the-rancher-helm-chart), **stop and return to this page**. This cluster will need to meet the following requirements:
+
+- EKS version 1.22.
+- Each node in the cluster has access to the registry containing Rancher and its related images.
+- Each node in the cluster has access to the ECR repo storing the CSP Adapter.
+- Each node in the cluster has access to the license manager service.
+- Each node in the cluster has access to global endpoints for the STS service.
+
+### 3. Install Rancher
+
+In addition to the options specified to install Rancher in the [Rancher docs]({{<baseurl>}}/rancher/v2.6/en/installation/resources/k8s-tutorials/amazon-eks/#8-install-the-rancher-helm-chart), you will also need to enable extra metrics.
+This can be done through the Helm CLI through the following options:
+
+```bash
+--set extraEnv\[0\].name="CATTLE_PROMETHEUS_METRICS" --set-string extraEnv\[0\].value=true
+```
+
+You can also use a values.yaml like the below:
+
+```yaml
+extraEnv:
+  - name: "CATTLE_PROMETHEUS_METRICS"
+    value: "true"
+```
+
+You will also need to install Rancher version 2.6.7 or higher.
+
+### 4. Create an OIDC Provider
+
+Follow the [AWS documentation](https://docs.aws.amazon.com/eks/latest/userguide/enable-iam-roles-for-service-accounts.html) to create an OIDC provider for the cluster specified in the previous section.
+
+### 5. Create an IAM Role
+
+An IAM role is required for the CSP adapter to check-in/check-out entitlements.
+
+First, configure the trust policy as below. Replace `MY_AWS_ACC` with your AWS account number, `MY_AWS_REGION` with your AWS region, and `MY_OIDC_PROVIDER` with the id of your OIDC provider:
+
+```json
+{
+    "Version": "2012-10-17",
+    "Statement": [
+        {
+            "Effect": "Allow",
+            "Principal": {
+                "Federated": "arn:aws:iam::${MY_AWS_ACC}:oidc-provider/oidc.eks.${MY_AWS_REGION}.amazonaws.com/id/${MY_OIDC_PROVIDER}"
+            },
+            "Action": "sts:AssumeRoleWithWebIdentity",
+            "Condition": {
+                "StringEquals": {
+                    "oidc.eks.${MY_AWS_REGION}.amazonaws.com/id/${MY_OIDC_PROVIDER}:sub": "system:serviceaccount:cattle-csp-adapter-system:rancher-csp-adapter",
+                    "oidc.eks.${MY_AWS_REGION}.amazonaws.com/id/${MY_OIDC_PROVIDER}:aud": "sts.amazonaws.com"
+                }
+            }
+        }
+    ]
+}
+```
+
+Next, use a policy for the role which has the following permissions:
+
+```json
+{
+    "Version": "2012-10-17",
+    "Statement": [
+        {
+            "Sid": "RancherCSPAdapterPermissions",
+            "Effect": "Allow",
+            "Action": [
+                "license-manager:ListReceivedLicenses",
+                "license-manager:CheckoutLicense",
+                "license-manager:ExtendLicenseConsumption",
+                "license-manager:CheckInLicense",
+                "license-manager:GetLicense",
+                "license-manager:GetLicenseUsage"
+            ],
+            "Resource": "*"
+        }
+    ]
+}
+```
+
+Save the name of the role. You will need it later on when installing the CSP adapter.
@@ -0,0 +1,29 @@
+---
+title: Common Issues
+weight: 4
+---
+
+**After installing the adapter, a banner message appears in Rancher that says "AWS Marketplace Adapter: Unable to run the adapter, please check the adapter logs"**
+
+This error indicates that while the adapter was installed into the cluster, an error has occurred which prevents it from properly checking-in/checking-out licenses.
+
+This often occurs because the IAM role was not set up properly. Review the [prerequisites](./adapter-requirements.md) and verify that:
+
+- An OIDC provider has been created/associated with the cluster Rancher is running on.
+- The IAM role has been configured to trust this OIDC provider.
+- The IAM role has at least the permissions outlined in the policy.
+
+If all of the above have been configured correctly, reach out to support for assistance.
+
+**I see a banner message that states, "AWS Marketplace Adapter: You have exceeded your licensed node count. At least x more license(s) are required in AWS to become compliant"**
+
+This message indicates that you do not have enough entitlements for the amount of nodes Rancher is currently managing.
+
+Keep in mind the following limitations:
+
+- Each entitlement is valid for a certain number of nodes.
+- Every node currently managed by Rancher counts toward your usage total (with exception of nodes in the cluster rancher is installed on).
+- Each entitlement can be used by at most one Rancher instance. For example, if you have two running Rancher instances in your account (each installed on a separate EKS cluster), then you will need at least two entitlements.
+
+You may also have recently uninstalled/re-installed the adapter. If the adapter loses track of the licenses that it is currently managing, it can take up to an hour to resolve the actual state of the licenses.
+
@@ -0,0 +1,141 @@
+---
+title: Installing the Adapter
+weight: 2
+---
+
+> **Important:** If you are attempting to re-install the adapter, you may experience errant out-of-compliance messages for up to an hour.
+
+### 1. Gain Access to the Local Cluster
+
+> **Note:** Only admin users should have access to the local cluster. Because the CSP adapter must be installed in the local cluster, this installation must be carried out by an admin user.
+
+First, click on the local cluster and download a kubeconfig token. You can then configure your CLI to use this new token with the following command, replacing `$TOKEN_PATH` with the path on your filesystem to the downloaded token:
+
+```bash
+export KUBECONFIG=$TOKEN_PATH
+```
+
+### 2. Create the Adapter Namespace
+
+Create the namespace that the adapter will be installed in.
+
+```bash
+kubectl create ns cattle-csp-adapter-system
+```
+
+### 3. Create Certificate Secrets
+
+The adapter requires access to the root CA that Rancher is using to communicate with the Rancher server. You can read more about which certificate options Rancher supports in the [chart options page](../../../../reference-guides/installation-references/helm-chart-options.md).
+
+If your Rancher install uses a certificate signed by a recognized Certificate Authority such as Let's Encrypt, then you can safely skip to [Step 4](#4-install-the-chart).
+
+However, if your Rancher install uses a custom certificate such as a Rancher-generated certificate or one signed by a private Certificate Authority, you will need to provide the certificate for this authority in PEM-encoded format so that the adapter can communicate with Rancher.
+
+First, retrieve the certificate that Rancher is using and place in a file named `ca-additional.pem`. If you are using the Rancher-generated certs option, this can be done with the following command:
+
+```bash
+kubectl get secret tls-rancher -n cattle-system -o jsonpath="{.data.tls\.crt}" | base64 -d  >> ca-additional.pem
+```
+
+Then, create a secret which uses this cert:
+
+```bash
+kubectl -n cattle-csp-adapter-system create secret generic tls-ca-additional --from-file=ca-additional.pem
+```
+
+> **Important:** Do not change the names of the file or of the created secret. Making changes to these values may result in errors when the adapter runs.
+
+### 4. Install the Chart
+
+First, add the `rancher/charts` repo using the following command:
+
+```bash
+helm repo add rancher-charts https://charts.rancher.io
+```
+
+Next, install the CSP adapter. You must specify several values, including the account number, and the name of the role created in the prerequisites.
+
+For the below instructions, replace `$MY_ACC_NUM` with your AWS account number and `$MY_ROLE_NAME` with the name of the role created in the prerequisites.
+
+> **Note:** If you use shell variables, do not specify quotation marks. For example, MY_ACC_NUM=123456789012 will work, but MY_ACC_NUM="123456789012" will fail.
+
+> **Note:** Accounts using the AWS Marketplace listing for the EU and the UK will need to specify an additional `--set image.repository=rancher/rancher-csp-adapter-eu` option. To see if your account needs this option when installing the adapter, refer to the usage instructions of the marketplace listing.
+
+> **Note:** It is important that you follow the instructions below exactly. In particular, the command to install version 1.0.1 of the adapter (by using --set image.tag=v1.0.1) is key to ensure that node counts are accurate.
+
+{{% tabs %}}
+{{% tab "Let's Encrypt/ Public Certificate Authority" %}}
+
+```bash
+helm install rancher-csp-adapter rancher-charts/rancher-csp-adapter --namespace cattle-csp-adapter-system --set aws.enabled=true --set aws.roleName=$MY_ROLE_NAME --set-string aws.accountNumber=$MY_ACC_NUM --set image.tag=v1.0.1
+```
+
+
+Alternatively, you can use a `values.yaml` and specify options like below:
+
+```yaml
+image:
+  tag: v1.0.1
+aws:
+  enabled: true
+  accountNumber: "$MY_ACC_NUM"
+  roleName: $MY_ROLE_NAME
+```
+
+> **Note:** The account number needs to be specified in a string format, like the above, or the installation will fail.
+
+You can then install the adapter with the following command:
+
+```bash
+helm install rancher-csp-adapter rancher-charts/rancher-csp-adapter -f values.yaml
+```
+
+{{% /tab %}}
+{{% tab "Private CA Authority / Rancher-generated Certificates" %}}
+
+```bash
+helm install rancher-csp-adapter rancher-charts/rancher-csp-adapter --namespace cattle-csp-adapter-system --set aws.enabled=true --set aws.roleName=$MY_ROLE_NAME --set-string aws.accountNumber=$MY_ACC_NUM --set additionalTrustedCAs=true --set image.tag=v1.0.1
+```
+
+Alternatively, you can use a `values.yaml` and specify options the below:
+
+```yaml
+image:
+  tag: v1.0.1
+aws:
+  enabled: true
+  accountNumber: "$MY_ACC_NUM"
+  roleName: $MY_ROLE_NAME
+additionalTrustedCAs: true
+```
+
+> **Note:** The account number needs to be specified in a string format, like the above, or the installation will fail.
+
+You can then install the adapter with the following command:
+
+```bash
+helm install rancher-csp-adapter rancher-charts/rancher-csp-adapter -f values.yaml
+```
+
+{{% /tab %}}
+{{% /tabs %}}
+
+### 5. Managing Certificate Updates
+
+If you had to create a secret storing a custom cert in [Step 3](#3-create-certificate-secrets), you will need to update this secret over time as the certificate is rotated.
+
+First, delete the original secret in the cattle-csp-adapter-system namespace, using the below command:
+
+```bash
+kubectl delete secret tls-ca-additional -n cattle-csp-adapter-system
+```
+
+Then, follow the original installation steps in [Step 3](#3-create-certificate-secrets) to replace the content of the secret with the updated value.
+
+Finally, restart the rancher-csp-adapter deployment to ensure that the updated value is made available to the adapter:
+
+```bash
+kubectl rollout restart deploy rancher-csp-adapter -n cattle-csp-adapter-system
+```
+
+> **Note:** There are methods such as cert-manager's [trust operator](https://cert-manager.io/docs/projects/trust/) which can help reduce the number of manual rotation tasks over time. While these options are not officially supported, they may be useful to users wishing to automate some of these tasks.
@@ -0,0 +1,22 @@
+---
+title: Uninstalling The Adapter
+weight: 3
+---
+
+### 1. Uninstall the adapter chart using helm.
+
+```bash
+helm uninstall rancher-csp-adapter -n cattle-csp-adapter-system
+```
+
+### 2. Remove the namespace created for the adapter.
+
+```bash
+kubectl delete ns cattle-csp-adapter-system
+```
+
+### 3. (Optional) remove any outstanding user notifications.
+
+```bash
+kubectl delete RancherUserNotification csp-compliance
+```
@@ -0,0 +1,54 @@
+---
+title: Supportconfig bundle
+weight: 100
+---
+
+After installing the CSP adapter, you will have the ability to generate a supportconfig bundle. This bundle is a tar file which can be used to quickly provide information to support.
+
+These bundles can be created through Rancher or through direct access to the cluster that Rancher is installed on. Note that accessing through Rancher is preferred.
+
+> **Note:** Only admin users can generate/download supportconfig bundles, regardless of method.
+
+### Accessing through Rancher
+
+First, click on the hamburger menu. Then click the `Get Support` button.
+
+![Get Support](/img/support-help.png)
+
+In the next page, click on the `Generate Support Config` button.
+
+> **Note:** If the adapter is not installed, the option to generate the supportconfig bundle will not be present. You must install the CSP adapter to generate a supportconfig bundle.
+
+![Get Support](/img/generate-support-config.png)
+
+### Accessing without rancher
+
+First, generate a kubeconfig for the cluster that Rancher is installed on.
+
+> **Note:** If Rancher is down, you will not be able to use a kubeconfig token generated by Rancher to access the cluster.
+
+Configure your shell environment to use this kubeconfig token:
+
+```bash
+export KUBECONFIG=$MY_KUBECONFIG_PATH
+```
+
+It is recommended to create a temporary working directory while running this command, like below:
+
+```bash
+mkdir temp && cd temp
+```
+
+Then, retrieve the supportconfig bundle:
+
+```bash
+mkdir rancher && kubectl get configmap csp-config -n cattle-csp-adapter-system -o=jsonpath='{.data.data}' >> rancher/config.json && tar -c -f supportconfig_rancher.tar rancher && rm -rf rancher
+```
+
+This will create a `supportconfig_rancher.tar` file in your current directory.
+
+Users who run these commands on Mac may experience issues due to incompatibilities between gnu-tar and bsd-tar. If support has issues reading a supportconfig that you produce, try using the below command after making gnu-tar accessible as `gtar` on your path:
+
+```bash
+mkdir rancher && kubectl get configmap csp-config -n cattle-csp-adapter-system -o=jsonpath='{.data.data}' >> rancher/config.json && gtar -c -f supportconfig_rancher.tar rancher && rm -rf rancher
+```
@@ -100,6 +100,89 @@ Below are the minimum recommended computing resources for the NeuVector chart in
 \* Minimum 1GB of memory total required for Controller, Manager, and Scanner containers combined.


+### Hardened Cluster Support - Calico and Canal
+
+{{% tabs %}}
+{{% tab "RKE1" %}}
+
+- All NeuVector components are deployable if PSP is set to true.
+
+**_New in v2.6.7_**
+
+You will need to set additional configuration for your hardened cluster environment as follows:
+
+1. Click **☰ > Cluster Management**.
+1. Go to the cluster that you created and click **Explore**.
+1. In the left navigation bar, click **Apps & Marketplace**.
+1. Install (or upgrade to) NeuVector version `100.0.1+up2.2.2`.
+
+   - Under **Edit Options** > **Other Configuration**, enable **Pod Security Policy** by checking the box (no other config needed):
+
+     ![Enable PSP for RKE1 Hardened Cluster](/img/psp-nv-rke.png)
+
+1. Click **Install** at the bottom-right to complete.
+
+
+{{% /tab %}}
+{{% tab "RKE2" %}}
+
+- NeuVector components Controller and Enforcer are deployable if PSP is set to true.
+
+
+**Applicable to NeuVector chart version 100.0.0+up2.2.0 only:**
+
+- For Manager, Scanner, and Updater components, additional configuration is required as shown below:
+
+```
+kubectl patch deploy neuvector-manager-pod -n cattle-neuvector-system --patch '{"spec":{"template":{"spec":{"securityContext":{"runAsUser": 5400}}}}}'
+kubectl patch deploy neuvector-scanner-pod -n cattle-neuvector-system --patch '{"spec":{"template":{"spec":{"securityContext":{"runAsUser": 5400}}}}}'
+kubectl patch cronjob neuvector-updater-pod -n cattle-neuvector-system --patch '{"spec":{"jobTemplate":{"spec":{"template":{"spec":{"securityContext":{"runAsUser": 5400}}}}}}}'
+```
+<br/>
+
+**_New in v2.6.7_**
+
+You will need to set additional configuration for your hardened cluster environment.
+
+>**Note:** You must update your config in both RKE2 and K3s hardened clusters as shown below.
+
+1. Click **☰ > Cluster Management**.
+1. Go to the cluster that you created and click **Explore**.
+1. In the left navigation bar, click **Apps & Marketplace**.
+1. Install (or upgrade to) NeuVector version `100.0.1+up2.2.2`.
+
+   - Under **Edit Options** > **Other Configuration**, enable **Pod Security Policy** by checking the box. Note that you must also enter a value greater than `zero` for `Manager runAsUser ID`, `Scanner runAsUser ID`, and `Updater runAsUser ID`:
+
+     ![Enable PSP for RKE2 and K3s Hardened Clusters](/img/psp-nv-rke2.png)
+
+1. Click **Install** at the bottom-right to complete.
+
+{{% /tab %}}
+{{% /tabs %}}
+
+
+### SELinux-enabled Cluster Support - Calico and Canal
+
+To enable SELinux on RKE2 clusters, follow the steps below:
+
+- NeuVector components Controller and Enforcer are deployable if PSP is set to true.
+
+
+**Applicable to NeuVector chart version 100.0.0+up2.2.0 only:**
+
+- For Manager, Scanner, and Updater components, additional configuration is required as shown below:
+
+```
+kubectl patch deploy neuvector-manager-pod -n cattle-neuvector-system --patch '{"spec":{"template":{"spec":{"securityContext":{"runAsUser": 5400}}}}}'
+kubectl patch deploy neuvector-scanner-pod -n cattle-neuvector-system --patch '{"spec":{"template":{"spec":{"securityContext":{"runAsUser": 5400}}}}}'
+kubectl patch cronjob neuvector-updater-pod -n cattle-neuvector-system --patch '{"spec":{"jobTemplate":{"spec":{"template":{"spec":{"securityContext":{"runAsUser": 5400}}}}}}}'
+```
+
+### Cluster Support in an Air-Gapped Environment
+
+- All NeuVector components are deployable on a cluster in an air-gapped environment without any additional configuration needed.
+
+
 ### Support Limitations

 * Only admins and cluster owners are currently supported.
@@ -108,11 +191,6 @@ Below are the minimum recommended computing resources for the NeuVector chart in

 * NeuVector is not supported on a Windows cluster.

-* NeuVector installation is not supported on hardened clusters.
-
-* NeuVector installation is not supported on SELinux clusters.
-
-* NeuVector installation is not supported on clusters in an air-gapped environment.

 ### Other Limitations

@@ -46,7 +46,7 @@ CNI network providers using this network model include Calico and Cilium. Cilium

 ### RKE Kubernetes clusters

-Out-of-the-box, Rancher provides the following CNI network providers for RKE Kubernetes clusters: Canal, Flannel, and Weave.
+Out-of-the-box, Rancher provides the following CNI network providers for RKE Kubernetes clusters: Canal, Flannel, Calico, and Weave.

 You can choose your CNI network provider when you create new Kubernetes clusters from Rancher.

@@ -22,7 +22,7 @@ services:

 You can follow the recommendations from [the etcd docs](https://etcd.io/docs/v3.4.0/tuning/#disk) on how to tune the disk priority on the host.

-Additionally, to reduce IO contention on the disks for etcd, you can use a dedicated device for the data and wal directory. Based on etcd best practices, mirroring RAID configurations are unnecessary because etcd replicates data between the nodes in the cluster. You can use stripping RAID configurations to increase available IOPS.
+Additionally, to reduce IO contention on the disks for etcd, you can use a dedicated device for the data and wal directory. Based on etcd best practices, mirroring RAID configurations are unnecessary because etcd replicates data between the nodes in the cluster. You can use striping RAID configurations to increase available IOPS.

 To implement this solution in an RKE cluster, the `/var/lib/etcd/data` and `/var/lib/etcd/wal` directories will need to have disks mounted and formatted on the underlying host. In the `extra_args` directive of the `etcd` service, you must include the `wal_dir` directory. Without specifying the `wal_dir`, etcd process will try to manipulate the underlying `wal` mount with insufficient permissions.

@@ -90,7 +90,7 @@ helm upgrade --install \
  ingress-nginx ingress-nginx/ingress-nginx \
  --namespace ingress-nginx \
  --set controller.service.type=LoadBalancer \
-  --version 3.12.0 \
+  --version 4.0.18 \
  --create-namespace
 ```

@@ -105,9 +105,9 @@ kubectl get service ingress-nginx-controller --namespace=ingress-nginx
 The result should look similar to the following:

 ```
-NAME                       TYPE           CLUSTER-IP     EXTERNAL-IP    PORT(S)                     
+NAME                       TYPE           CLUSTER-IP     EXTERNAL-IP    PORT(S)
 AGE
-ingress-nginx-controller   LoadBalancer   10.0.116.18    40.31.180.83   80:31229/TCP,443:31050/TCP  
+ingress-nginx-controller   LoadBalancer   10.0.116.18    40.31.180.83   80:31229/TCP,443:31050/TCP
 67s
 ```

@@ -126,3 +126,13 @@ There are many valid ways to set up the DNS. For help, refer to the [Azure DNS d
 Next, install the Rancher Helm chart by following the instructions on [this page.](../../../pages-for-subheaders/install-upgrade-on-a-kubernetes-cluster.md#install-the-rancher-helm-chart) The Helm instructions are the same for installing Rancher on any Kubernetes distribution.

 Use that DNS name from the previous step as the Rancher server URL when you install Rancher. It can be passed in as a Helm option. For example, if the DNS name is `rancher.my.org`, you could run the Helm installation command with the option `--set hostname=rancher.my.org`.
+
+**_New in v2.6.7_**
+
+When installing Rancher on top of this setup, you will also need to pass the value below into the Rancher Helm install command in order to set the name of the ingress controller to be used with Rancher's ingress resource:
+
+```
+--set ingress.ingressClassName=nginx
+```
+
+Refer [here for the Helm install command](../../../pages-for-subheaders/install-upgrade-on-a-kubernetes-cluster.md#5-install-rancher-with-helm-and-your-chosen-certificate-option) for your chosen certificate option.
@@ -83,10 +83,12 @@ Then enter the following values:

 To create an EKS cluster, run the following command. Use the AWS region that applies to your use case. When choosing a Kubernetes version, be sure to first consult the [support matrix](https://rancher.com/support-matrix/) to find the highest version of Kubernetes that has been validated for your Rancher version.

+**Note:** If the version of Kubernetes is updated to v1.22 or later, the version of ingress-nginx would also need to be [updated](https://kubernetes.github.io/ingress-nginx/#faq-migration-to-apiversion-networkingk8siov1).
+
 ```
 eksctl create cluster \
  --name rancher-server \
-  --version 1.20 \
+  --version <VERSION> \
  --region us-west-2 \
  --nodegroup-name ranchernodes \
  --nodes 3 \
@@ -128,7 +130,7 @@ helm upgrade --install \
  ingress-nginx ingress-nginx/ingress-nginx \
  --namespace ingress-nginx \
  --set controller.service.type=LoadBalancer \
-  --version 3.12.0 \
+  --version 4.0.18 \
  --create-namespace
 ```

@@ -143,9 +145,9 @@ kubectl get service ingress-nginx-controller --namespace=ingress-nginx
 The result should look similar to the following:

 ```
-NAME                       TYPE           CLUSTER-IP     EXTERNAL-IP                                                              PORT(S)                     
+NAME                       TYPE           CLUSTER-IP     EXTERNAL-IP                                                              PORT(S)
 AGE
-ingress-nginx-controller   LoadBalancer   10.100.90.18   a904a952c73bf4f668a17c46ac7c56ab-962521486.us-west-2.elb.amazonaws.com   80:31229/TCP,443:31050/TCP  
+ingress-nginx-controller   LoadBalancer   10.100.90.18   a904a952c73bf4f668a17c46ac7c56ab-962521486.us-west-2.elb.amazonaws.com   80:31229/TCP,443:31050/TCP
 27m
 ```

@@ -164,3 +166,13 @@ There are many valid ways to set up the DNS. For help, refer to the AWS document
 Next, install the Rancher Helm chart by following the instructions on [this page.](../../../pages-for-subheaders/install-upgrade-on-a-kubernetes-cluster.md#install-the-rancher-helm-chart) The Helm instructions are the same for installing Rancher on any Kubernetes distribution.

 Use that DNS name from the previous step as the Rancher server URL when you install Rancher. It can be passed in as a Helm option. For example, if the DNS name is `rancher.my.org`, you could run the Helm installation command with the option `--set hostname=rancher.my.org`.
+
+**_New in v2.6.7_**
+
+When installing Rancher on top of this setup, you will also need to pass the value below into the Rancher Helm install command in order to set the name of the ingress controller to be used with Rancher's ingress resource:
+
+```
+--set ingress.ingressClassName=nginx
+```
+
+Refer [here for the Helm install command](../../../pages-for-subheaders/install-upgrade-on-a-kubernetes-cluster.md#5-install-rancher-with-helm-and-your-chosen-certificate-option) for your chosen certificate option.
@@ -123,8 +123,10 @@ Replace `cluster-name` with the name of your new cluster.

 When choosing a Kubernetes version, be sure to first consult the [support matrix](https://rancher.com/support-matrix/) to find the highest version of Kubernetes that has been validated for your Rancher version.

+**Note:** If the version of Kubernetes is updated to v1.22 or later, the version of ingress-nginx would also need to be [updated](https://kubernetes.github.io/ingress-nginx/#faq-migration-to-apiversion-networkingk8siov1).
+
 ```
-gcloud container clusters create cluster-name --num-nodes=3 --cluster-version=1.20.8-gke.900
+gcloud container clusters create cluster-name --num-nodes=3 --cluster-version=<VERSION>
 ```

 # 6. Get Authentication Credentials
@@ -150,7 +152,7 @@ helm upgrade --install \
  ingress-nginx ingress-nginx/ingress-nginx \
  --namespace ingress-nginx \
  --set controller.service.type=LoadBalancer \
-  --version 3.12.0 \
+  --version 4.0.18 \
  --create-namespace
 ```

@@ -184,3 +186,13 @@ There are many valid ways to set up the DNS. For help, refer to the Google Cloud
 Next, install the Rancher Helm chart by following the instructions on [this page.](../../../pages-for-subheaders/install-upgrade-on-a-kubernetes-cluster.md#install-the-rancher-helm-chart) The Helm instructions are the same for installing Rancher on any Kubernetes distribution.

 Use the DNS name from the previous step as the Rancher server URL when you install Rancher. It can be passed in as a Helm option. For example, if the DNS name is `rancher.my.org`, you could run the Helm installation command with the option `--set hostname=rancher.my.org`.
+
+**_New in v2.6.7_**
+
+When installing Rancher on top of this setup, you will also need to pass the value below into the Rancher Helm install command in order to set the name of the ingress controller to be used with Rancher's ingress resource:
+
+```
+--set ingress.ingressClassName=nginx
+```
+
+Refer [here for the Helm install command](../../../pages-for-subheaders/install-upgrade-on-a-kubernetes-cluster.md#5-install-rancher-with-helm-and-your-chosen-certificate-option) for your chosen certificate option.
@@ -7,31 +7,33 @@ The Dockershim is the CRI compliant layer between the Kubelet and the Docker dae

 RKE clusters now support the external Dockershim to continue leveraging Docker as the CRI runtime. We now implement the upstream open source community external Dockershim announced by [Mirantis and Docker](https://www.mirantis.com/blog/mirantis-to-take-over-support-of-kubernetes-dockershim-2/) to ensure RKE clusters can continue to leverage Docker.

-To enable the external Dockershim, configure the following option.
+To enable the external Dockershim in versions of RKE before 1.24, configure the following option.

 ```
 enable_cri_dockerd: true
 ```

+Starting with version 1.24, the above defaults to true.
+
 For users looking to use another container runtime, Rancher has the edge-focused K3s and datacenter-focused RKE2 Kubernetes distributions that use containerd as the default runtime. Imported RKE2 and K3s Kubernetes clusters can then be upgraded and managed through Rancher going forward.

 ### FAQ

 <br/>

-Q. Do I have to upgrade Rancher to get Rancher’s support of the upstream Dockershim?
+Q: Do I have to upgrade Rancher to get Rancher’s support of the upstream external Dockershim replacement?

-A The upstream support of Dockershim begins for RKE in Kubernetes 1.21. You will need to be on a version of Rancher that supports RKE 1.21. See our support matrix for details.
+A: The upstream support of the Dockershim replacement `cri_dockerd` begins for RKE in Kubernetes 1.21. You will need to be on a version of Rancher that supports RKE 1.21. See our support matrix for details.

 <br/>

-Q. I am currently on RKE with Kubernetes 1.20. Do I need to upgrade to RKE with Kubernetes 1.21 sooner to avoid being out of support for Dockershim?
+Q: I am currently on RKE with Kubernetes 1.23. What happens when upstream finally removes Dockershim in 1.24?

-A. The version of Dockershim in RKE with Kubernetes 1.20 will continue to work and it is not deprecated until a later release. For information on the timeline, see the [Kubernetes Dockershim Deprecation FAQ](https://kubernetes.io/blog/2020/12/02/dockershim-faq/#when-will-dockershim-be-removed). It will only emit a warning of its future deprecation, which Rancher has mitigated in RKE with Kubernetes 1.21. You can plan your upgrade to 1.21 as you would normally.
+A: The version of Dockershim in RKE with Kubernetes will continue to work until 1.23. For information on the timeline, see the [Kubernetes Dockershim Deprecation FAQ](https://kubernetes.io/blog/2020/12/02/dockershim-faq/#when-will-dockershim-be-removed). After this, starting in 1.24, RKE will default to enabling `cri_dockerd` by default and will continue to do for versions afterwards.

 <br/>

-Q: What are my other options if I don’t want to depend on the Dockershim?
+Q: What are my other options if I don’t want to depend on the Dockershim or cri_dockerd?

 A: You can use a runtime like containerd with Kubernetes that does not require Dockershim support. RKE2 or K3s are two options for doing this.

@@ -98,7 +98,7 @@ sudo mv ./kubectl /usr/local/bin/kubectl
 * [helm](https://helm.sh/docs/intro/install/)

 ```
-curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3
+curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3
 chmod +x get_helm.sh
 sudo ./get_helm.sh
 ```
@@ -22,9 +22,13 @@ The full installation requirements are [here](../../../pages-for-subheaders/inst
 Install a K3s cluster by running this command on the Linux machine:

 ```
-curl -sfL https://get.k3s.io | sh -s - server
+curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION="***" sh -s - server --cluster-init
 ```

+Rancher needs to be installed on a supported Kubernetes version. To specify the K3s version, use the INSTALL_K3S_VERSION environment variable when running the K3s installation script. Refer to the [support maintenance terms](https://rancher.com/support-maintenance-terms/).
+
+Using `--cluster-init` allows K3s to use embedded etcd as the datastore and has the ability to convert to an HA setup. Refer to [High Availability with Embedded DB](https://rancher.com/docs/k3s/latest/en/installation/ha-embedded/).
+
 Save the IP of the Linux machine.

 ## Save the kubeconfig to your workstation
@@ -95,7 +99,7 @@ helm repo add rancher-latest https://releases.rancher.com/server-charts/latest

 kubectl create namespace cattle-system

-kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.7.2/cert-manager.crds.yaml
+kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.7.1/cert-manager.crds.yaml

 helm repo add jetstack https://charts.jetstack.io

@@ -115,6 +119,8 @@ helm install cert-manager jetstack/cert-manager `

 The final command to install Rancher is below. The command requires a domain name that forwards traffic to the Linux machine. For the sake of simplicity in this tutorial, you can use a fake domain name to create your proof-of-concept. An example of a fake domain name would be `<IP_OF_LINUX_NODE>.sslip.io`.

+To install a specific Rancher version, use the `--version` flag (e.g., `--version 2.6.6`). Otherwise, the latest Rancher is installed by default. Refer to [Choosing a Rancher Version](../../installation-and-upgrade/resources/choose-a-rancher-version.md).
+
 ```
 helm install rancher rancher-latest/rancher \
  --namespace cattle-system \
@@ -129,7 +135,6 @@ helm install rancher rancher-latest/rancher `
  --set replicas=1 `
  --set bootstrapPassword=<PASSWORD_FOR_RANCHER_ADMIN>
 ```
-```

 Now if you navigate to `<IP_OF_LINUX_NODE>.sslip.io` in a web browser, you should see the Rancher UI.

@@ -32,7 +32,7 @@ The intent of these guides is to quickly launch a sandbox that you can use to ev
 3. **Optional:** Edit `config.yaml` to:

    - Change the number of nodes and the memory allocations, if required. (`node.count`, `node.cpus`, `node.memory`)
-    - Change the password of the `admin` user for logging into Rancher. (`default_password`)
+    - Change the password of the `admin` user for logging into Rancher. (`admin_password`)

 4. To initiate the creation of the environment run, `vagrant up --provider=virtualbox`.

@@ -3,23 +3,28 @@ title: Configure Azure AD
 weight: 1115
 ---

+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+<Tabs>
+  <TabItem value="Rancher v2.6.7">
+
+## Microsoft Graph API
+
+Microsoft Graph API is now the flow through which you will set up Azure AD. The below sections will assist [new users](#new-user-setup) in configuring Azure AD with a new instance as well as assist existing Azure app owners in [migrating to the new flow](#migrating-from-azure-ad-graph-api-to-microsoft-graph-api).
+
+### New User Setup
+
 If you have an instance of Active Directory (AD) hosted in Azure, you can configure Rancher to allow your users to log in using their AD accounts. Configuration of Azure AD external authentication requires you to make configurations in both Azure and Rancher.

-:::note Prerequisite:
-
-Have an instance of Azure AD configured.
-
-:::
-
-:::note Notes 
+:::note Notes

 - Azure AD integration only supports Service Provider initiated logins.
-
 - Most of this procedure takes place from the [Microsoft Azure Portal](https://portal.azure.com/).

 :::

-## Azure Active Directory Configuration Outline
+#### Azure Active Directory Configuration Outline

 Configuring Rancher to allow your users to authenticate with their Azure AD accounts involves multiple procedures. Review the outline below before getting started.

@@ -34,13 +39,12 @@ Before you start, we recommend creating an empty text file. You can use this fil
 - [1. Register Rancher with Azure](#1-register-rancher-with-azure)
 - [2. Create a new client secret](#2-create-a-new-client-secret)
 - [3. Set Required Permissions for Rancher](#3-set-required-permissions-for-rancher)
- [4. Add a Reply URL](#4-add-a-reply-url)
- [5. Copy Azure Application Data](#5-copy-azure-application-data)
- [6. Configure Azure AD in Rancher](#6-configure-azure-ad-in-rancher)
+- [4. Copy Azure Application Data](#5-copy-azure-application-data)
+- [5. Configure Azure AD in Rancher](#6-configure-azure-ad-in-rancher)

 <!-- /TOC -->

-### 1. Register Rancher with Azure
+#### 1. Register Rancher with Azure

 Before enabling Azure AD within Rancher, you must register Rancher with Azure.

@@ -55,14 +59,21 @@ Before enabling Azure AD within Rancher, you must register Rancher with Azure.
    ![New App Registration](/img/new-app-registration.png)

    1. Enter a **Name** (something like `Rancher`).
+    <a id="3.2"></a>

    1. From **Supported account types**, select "Accounts in this organizational directory only (AzureADTest only - Single tenant)" This corresponds to the legacy app registration options.

-    1. In the **Redirect URI** section, make sure **Web** is selected from the dropdown and enter the URL of your Rancher Server in the text box next to the dropdown. This Rancher server URL should be appended with the verification path: `<MY_RANCHER_URL>/verify-auth-azure`.
+        :::note
+
+        In the updated Azure portal, Redirect URIs are synonymous with Reply URLs. In order to use Azure AD with Rancher, you must whitelist Rancher with Azure (previously done through Reply URLs). Therefore, you must ensure to fill in the Redirect URI with your Rancher server URL, to include the verification path as listed below.
+
+        :::
+
+    1. In the [**Redirect URI**](https://docs.microsoft.com/en-us/azure/active-directory/develop/reply-url) section, make sure **Web** is selected from the dropdown and enter the URL of your Rancher Server in the text box next to the dropdown. This Rancher server URL should be appended with the verification path: `<MY_RANCHER_URL>/verify-auth-azure`.

       :::tip
-       
-       You can find your personalized Azure reply URL in Rancher on the Azure AD Authentication page (Global View > Security Authentication > Azure AD).
+
+       You can find your personalized Azure Redirect URI (reply URL) in Rancher on the Azure AD Authentication page (Global View > Authentication > Web).

       :::

@@ -74,19 +85,19 @@ It can take up to five minutes for this change to take affect, so don't be alarm

 :::

-### 2. Create a new client secret
+#### 2. Create a new client secret

 From the Azure portal, create a client secret. Rancher will use this key to authenticate with Azure AD.

 1. Use search to open **App registrations** services. Then open the entry for Rancher that you created in the last procedure.

-     ![Open Rancher Registration](/img/open-rancher-app.png)
+     ![Open Rancher Registration](/img/open-rancher-app-reg.png)

 1. From the navigation pane on left, click **Certificates and Secrets**.

 1. Click **New client secret**.

-     ![Create new client secret](/img/select-client-secret.png)
+     ![Create new client secret](/img/new-client-secret.png)

    1. Enter a **Description** (something like `Rancher`).

@@ -101,112 +112,85 @@ From the Azure portal, create a client secret. Rancher will use this key to auth

    You won't be able to access the key value again within the Azure UI.

-### 3. Set Required Permissions for Rancher
+#### 3. Set Required Permissions for Rancher

 Next, set API permissions for Rancher within Azure.

+:::caution
+
+Ensure that you set the permissions of type Application and NOT Delegated. Otherwise, you may not be able to login to Azure AD. This issue will persist even after you disable/re-enable Azure AD and will require an hour wait, or manual deletion of a cache value to resolve.
+
+:::
+
 1. From the navigation pane on left, select **API permissions**.

-    ![Open Required Permissions](/img/select-required-permissions.png)
+    ![Open Required Permissions](/img/select-req-permissions.png)

 1. Click **Add a permission**.

-1. From the **Azure Active Directory Graph**, select the following **Delegated Permissions**:
+1. From the **Microsoft Graph**, select the following **Application Permissions**:
+    - `Group.Read.All`
+    - `User.Read.All`

-    ![Select API Permissions](/img/select-required-permissions-2.png)
+    ![Select API Permissions](/img/api-permissions.png)

-    <br/>
-    <br/>
-    - **Access the directory as the signed-in user**
-    - **Read directory data**
-    - **Read all groups**
-    - **Read all users' full profiles**
-    - **Read all users' basic profiles**
-    - **Sign in and read user profile**
-
-1. Click **Add permissions**.
-
-1. From **API permissions**, click **Grant admin consent**. Then click **Yes**.
+1. Return to **API permissions** in the left nav bar. From there, click **Grant admin consent**. Then click **Yes**.

    :::note
-    
+
    You must be signed in as an Azure administrator to successfully save your permission settings.

    :::

-
-### 4. Add a Reply URL
-
-To use Azure AD with Rancher you must whitelist Rancher with Azure. You can complete this whitelisting by providing Azure with a reply URL for Rancher, which is your Rancher Server URL followed with a verification path.
-
-
-1. From the **Setting** blade, select **Reply URLs**.
-
-    ![Azure: Enter Reply URL](/img/enter-azure-reply-url.png)
-
-1. From the **Reply URLs** blade, enter the URL of your Rancher Server, appended with the verification path: `<MY_RANCHER_URL>/verify-auth-azure`.
-
-       :::tip
-       
-       You can find your personalized Azure reply URL in Rancher on the Azure AD Authentication page (Global View > Security Authentication > Azure AD).
-
-       :::
-
-1. Click **Save**.
-
-**Result:** Your reply URL is saved.
-
-:::note
-
-It can take up to five minutes for this change to take affect, so don't be alarmed if you can't authenticate immediately after Azure AD configuration.
-
-:::
-
-### 5. Copy Azure Application Data
+#### 4. Copy Azure Application Data

 As your final step in Azure, copy the data that you'll use to configure Rancher for Azure AD authentication and paste it into an empty text file.

 1. Obtain your Rancher **Tenant ID**.

-    1. Use search to open the **Azure Active Directory** service.
-
-         ![Open Azure Active Directory](/img/search-azure-ad.png)
-
-    1. From the left navigation pane, open **Overview**.
-
-    2. Copy the **Directory ID** and paste it into your [text file](#tip).
-
-        You'll paste this value into Rancher as your **Tenant ID**.
-
-1. Obtain your Rancher **Application ID**.
-
    1. Use search to open **App registrations**.

-         ![Open App Registrations](/img/search-app-registrations.png)
+    ![Open App Registrations](/img/search-app-registrations.png)

    1. Find the entry you created for Rancher.

-    1. Copy the **Application ID** and paste it to your [text file](#tip).
+    1. Copy the **Directory ID** and paste it into your [text file](#tip).

-1. Obtain your Rancher **Graph Endpoint**, **Token Endpoint**, and **Auth Endpoint**.
+    ![Tenant ID](/img/tenant-id.png)

-    1. From **App registrations**, click **Endpoints**.
+    - You'll paste this value into Rancher as your **Tenant ID**.

-        ![Click Endpoints](/img/click-endpoints.png)
+1. Obtain your Rancher **Application (Client) ID**.

-    2. Copy the following endpoints to your clipboard and paste them into your [text file](#tip) (these values will be your Rancher endpoint values).
-    
-        - **Microsoft Graph API endpoint** (Graph Endpoint)
-        - **OAuth 2.0 token endpoint (v1)** (Token Endpoint)
-        - **OAuth 2.0 authorization endpoint (v1)** (Auth Endpoint)
-        
-:::note
+    1. Use search to open **App registrations** (if not already there).

-Copy the v1 version of the endpoints
+    2.2. In **Overview**, find the entry you created for Rancher.

-:::
+    2.3. Copy the **Application (Client) ID** and paste it to your [text file](#tip).

-### 6. Configure Azure AD in Rancher
+    ![Application ID](/img/application-client-id.png)
+
+1. Your endpoint options will typically be [Standard](#global) and [China](#china). With these options, you need only enter the **Tenant ID**, **Application ID**, and **Application Secret** (Rancher will take care of the rest).
+
+![Standard Endpoint Options](/img/tenant-application-id-secret.png)
+
+>**For Custom Endpoints:**
+>
+>**Warning:** Custom Endpoints are not supported nor fully tested by Rancher.
+>
+> You will need to also manually enter the Graph, Token, and Auth Endpoints.
+>
+>- From <b>App registrations</b>, click <b>Endpoints</b>:
+>
+>![Click Endpoints](/img/endpoints.png)
+>
+>- Copy the following endpoints to your clipboard and paste them into your [text file](#tip) (these values will be your Rancher endpoint values). Make sure to copy the v1 version of the endpoints.
+>
+>   - **Microsoft Graph API endpoint** (Graph Endpoint)
+>   - **OAuth 2.0 token endpoint (v1)** (Token Endpoint)
+>   - **OAuth 2.0 authorization endpoint (v1)** (Auth Endpoint)
+
+#### 5. Configure Azure AD in Rancher

 From the Rancher UI, enter information about your AD instance hosted in Azure to complete configuration.

@@ -216,17 +200,9 @@ Enter the values that you copied to your [text file](#tip).
 1.	In the top left corner, click **☰ > Users & Authentication**.
 1. In the left navigation menu, click **Auth Provider**.
 1. Click **AzureAD**.
-1. Complete the **Configure Azure AD Account** form using the information you copied while completing [Copy Azure Application Data](#5-copy-azure-application-data).
+1. Complete the **Configure Azure AD Account** form using the information you copied while completing [Copy Azure Application Data](#4-copy-azure-application-data).

-    :::note Important 
-    
-    When entering your Graph Endpoint, remove the tenant ID from the URL, like below.
-    
-    <code>http<span>s://g</span>raph.windows.net/<del>abb5adde-bee8-4821-8b03-e63efdc7701c</del></code>
-
-    :::
-
-    The following table maps the values you copied in the Azure portal to the fields in Rancher.
+    The following table maps the values you copied in the Azure portal to the fields in Rancher:

    | Rancher Field      | Azure Value                           |
    | ------------------ | ------------------------------------- |
@@ -234,10 +210,121 @@ Enter the values that you copied to your [text file](#tip).
    | Application ID     | Application ID                        |
    | Application Secret | Key Value                             |
    | Endpoint           | https://login.microsoftonline.com/    |
-    | Graph Endpoint     | Microsoft Azure AD Graph API Endpoint |
-    | Token Endpoint     | OAuth 2.0 Token Endpoint              |
-    | Auth Endpoint      | OAuth 2.0 Authorization Endpoint      |
+
+
+    >**For Custom Endpoints:**
+    ><br/>
+    >The following table maps the custom config values you copied in the Azure portal to the fields in Rancher:
+    >
+    >| Rancher Field      | Azure Value                           |
+    >| ------------------ | ------------------------------------- |
+    >| Graph Endpoint     | Microsoft Graph API Endpoint          |
+    >| Token Endpoint     | OAuth 2.0 Token Endpoint              |
+    >| Auth Endpoint      | OAuth 2.0 Authorization Endpoint      |
+    ><br/>
+    >**Important:** When entering the Graph Endpoint in a custom config, remove the tenant ID from the URL, like below:
+    >
+    ><code>http<span>s://g</span>raph.microsoft.com/<del>abb5adde-bee8-4821-8b03-e63efdc7701c</del></code>

 1. Click **Enable**.

 **Result:** Azure Active Directory authentication is configured.
+
+
+### Migrating from Azure AD Graph API to Microsoft Graph API
+
+Since [Azure AD Graph API](https://docs.microsoft.com/en-us/graph/migrate-azure-ad-graph-overview) was deprecated in June 2022 and will be retired at the end of 2022, users should update their Azure AD App to use the new [Microsoft Graph API](https://docs.microsoft.com/en-us/graph/use-the-api) in Rancher.
+
+#### Updating Endpoints in the Rancher UI
+
+>**Important:** Admins should create a [backup](../../../../new-user-guides/backup-restore-and-disaster-recovery/back-up-rancher.md) right before they commit to the endpoint migration in Step 4 below.
+
+1. Update the permissions of your Azure AD app registration as described [here](#3-set-required-permissions-for-rancher).
+**This is critical.**
+
+1. Log into Rancher.
+
+1. In the Rancher UI homepage, make note of the banner at the top of screen that advises users to update their Azure AD authentication. Click on the link provided to do so.
+
+    ![Rancher UI Banner](/img/rancher-ui-azure-update.png)
+
+1. To complete the move to the new Microsoft Graph API, click **Update Endpoint**.
+
+    **Note:** Ensure that your Azure app has a [new set of permissions](#3-set-required-permissions-for-rancher) before starting the update.
+
+    ![Update Endpoint](/img/rancher-button-to-update.png)
+
+1. When you receive the pop-up warning message, click **Update**.
+
+    ![Azure Update Pop-up](/img/azure-update-popup.png)
+
+1. Refer to the [tables](#global) below for the full list of endpoint changes that Rancher performs. Admins do not need to do this manually.
+
+#### Air-Gapped Environments
+
+In air-gapped environments, admins should ensure that their endpoints are [whitelisted](#3.2) since the Graph Endpoint URL is changing.
+
+#### Rolling Back the Migration
+
+If you need to roll back your migration, please note the following:
+
+1. Admins are encouraged to use the proper restore process if they want to go back. Please see [backup docs](../../../../new-user-guides/backup-restore-and-disaster-recovery/back-up-rancher.md), [restore docs](../../../../new-user-guides/backup-restore-and-disaster-recovery/restore-rancher.md), and [examples](../../../../../reference-guides/backup-restore-configuration/examples.md) for reference.
+
+1. Azure app owners who want to rotate the Application Secret will need to also rotate it in Rancher as Rancher does not automatically update the Application Secret when it is changed in Azure. In Rancher, note that it is stored in a Kubernetes secret called `azureadconfig-applicationsecret` which is in the `cattle-global-data` namespace.
+
+1. **Caution:** If admins upgrade to Rancher v2.6.7 with an existing Azure AD setup and choose to disable the auth provider, they won't be able to restore the previous setup and also will not be able to set up Azure AD anew using the old flow. Admins will then need to register again with the new auth flow. Rancher now uses the new Graph API and, therefore, users need set up the [proper permissions in the Azure portal](#3-set-required-permissions-for-rancher).
+
+#### Global:
+
+Rancher Field    | Deprecated Endpoints
+---------------- | -------------------------------------------------------------
+Auth Endpoint    | https://login.microsoftonline.com/{tenantID}/oauth2/authorize
+Endpoint         | https://login.microsoftonline.com/
+Graph Endpoint   | https://graph.windows.net/
+Token Endpoint   | https://login.microsoftonline.com/{tenantID}/oauth2/token
+---
+
+Rancher Field    | New Endpoints
+---------------- | ------------------------------------------------------------------
+Auth Endpoint    | https://login.microsoftonline.com/{tenantID}/oauth2/v2.0/authorize
+Endpoint         | https://login.microsoftonline.com/
+Graph Endpoint   | https://graph.microsoft.com
+Token Endpoint   | https://login.microsoftonline.com/{tenantID}/oauth2/v2.0/token
+
+#### China:
+
+Rancher Field    | Deprecated Endpoints
+---------------- | ----------------------------------------------------------
+Auth Endpoint    | https://login.chinacloudapi.cn/{tenantID}/oauth2/authorize
+Endpoint         | https://login.chinacloudapi.cn/
+Graph Endpoint   | https://graph.chinacloudapi.cn/
+Token Endpoint   | https://login.chinacloudapi.cn/{tenantID}/oauth2/token
+---
+
+Rancher Field    | New Endpoints
+---------------- | -------------------------------------------------------------------------
+Auth Endpoint    | https://login.partner.microsoftonline.cn/{tenantID}/oauth2/v2.0/authorize
+Endpoint         | https://login.partner.microsoftonline.cn/
+Graph Endpoint   | https://microsoftgraph.chinacloudapi.cn
+Token Endpoint   | https://login.partner.microsoftonline.cn/{tenantID}/oauth2/v2.0/token
+
+
+</TabItem>
+<TabItem value="Rancher v2.6.0 - v2.6.6">
+
+## Azure AD Graph API
+
+>**Important:**
+>
+>- The [Azure AD Graph API](https://docs.microsoft.com/en-us/graph/migrate-azure-ad-graph-overview) was deprecated in June 2022 and will be retired at the end of 2022. We will update our docs to advise the community when it is retired. Rancher now uses the [Microsoft Graph API](https://docs.microsoft.com/en-us/graph/use-the-api) as the new flow to set up Azure AD as the external auth provider.
+>
+>
+>- For new users, or existing users who wish to migrate, refer to the new flow instructions on the <a href="#microsoft-graph-api/" target="_blank">Rancher v2.6.7</a> tab.
+>
+>
+>- For existing users who do not wish to upgrade to v2.6.7 after the Azure AD Graph API is retired, they will need to either:
+    - Use the built-in Rancher auth or
+    - Use another third-party auth system and set that up in Rancher. Please see the [authentication docs](../../../../../pages-for-subheaders/about-authentication.md) to learn how to configure other open authentication providers.
+
+</TabItem>
+</Tabs>
@@ -13,8 +13,7 @@ This kubeconfig file and its contents are specific to the cluster you are viewin

 After you download the kubeconfig file, you will be able to use the kubeconfig file and its Kubernetes [contexts](https://kubernetes.io/docs/reference/kubectl/cheatsheet/#kubectl-context-and-configuration) to access your downstream cluster.

-If admins have [enforced TTL on kubeconfig tokens](../../../../reference-guides/about-the-api/api-tokens.md#setting-ttl-on-kubeconfig-tokens), the kubeconfig file requires [rancher cli](authorized-cluster-endpoint.md) to be present in your PATH. 
-
+If admins have [kubeconfig token generation turned off](../../../../reference-guides/about-the-api/api-tokens.md#disable-tokens-in-generated-kubeconfigs), the kubeconfig file requires [rancher cli](./authorized-cluster-endpoint.md) to be present in your PATH.

 ### Two Authentication Methods for RKE Clusters

@@ -18,14 +18,14 @@ When removing nodes from your Rancher launched Kubernetes cluster (provided that
 When cleaning nodes provisioned using Rancher, the following components are deleted based on the type of cluster node you're removing.

 | Removed Component                                                              | [Nodes Hosted by Infrastructure Provider][1] | [Custom Nodes][2] | [Hosted Cluster][3] | [Registered Nodes][4] |
-| ------------------------------------------------------------------------------ | --------------- | ----------------- | ------------------- | ------------------- |
-| The Rancher deployment namespace (`cattle-system` by default)                  | ✓               | ✓                 | ✓                   | ✓                   |
-| `serviceAccount`, `clusterRoles`, and `clusterRoleBindings` labeled by Rancher | ✓               | ✓                 | ✓                   | ✓                   |
-| Labels, Annotations, and Finalizers                                            | ✓               | ✓                 | ✓                   | ✓                   |
-| Rancher Deployment                                                             | ✓               | ✓                 | ✓                   |                     |
-| Machines, clusters, projects, and user custom resource definitions (CRDs)      | ✓               | ✓                 | ✓                   |                     |
-| All resources create under the `management.cattle.io` API Group                | ✓               | ✓                 | ✓                   |                     |
-| All CRDs created by Rancher v2.x                                               | ✓               | ✓                 | ✓                   |                     |
+|--------------------------------------------------------------------------------|----------------------------------------------|-------------------|---------------------|-----------------------|
+| The Rancher deployment namespace (`cattle-system` by default)                  | ✓                                            | ✓                 | ✓                   | ✓                     |
+| `serviceAccount`, `clusterRoles`, and `clusterRoleBindings` labeled by Rancher | ✓                                            | ✓                 | ✓                   | ✓                     |
+| Labels, Annotations, and Finalizers                                            | ✓                                            | ✓                 | ✓                   | ✓                     |
+| Rancher Deployment                                                             | ✓                                            | ✓                 | ✓                   |                       |
+| Machines, clusters, projects, and user custom resource definitions (CRDs)      | ✓                                            | ✓                 | ✓                   |                       |
+| All resources create under the `management.cattle.io` API Group                | ✓                                            | ✓                 | ✓                   |                       |
+| All CRDs created by Rancher v2.x                                               | ✓                                            | ✓                 | ✓                   |                       |

 [1]: ../../../pages-for-subheaders/use-new-nodes-in-an-infra-provider.md
 [2]: ../../../pages-for-subheaders/use-existing-nodes.md
@@ -120,6 +120,11 @@ Install [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/).
 </TabItem>
 </Tabs>

+## Cleaning up Nodes
+
+<Tabs>
+<TabItem value="RKE1">
+
 ### Windows Nodes

 To clean up a Windows node, you can run a cleanup script located in `c:\etc\rancher`. The script deletes Kubernetes generated resources and the execution binary. It also drops the firewall rules and network settings.
@@ -150,11 +155,11 @@ docker volume rm $(docker volume ls -q)

 Kubernetes components and secrets leave behind mounts on the system that need to be unmounted.

-Mounts |
--------|
-`/var/lib/kubelet/pods/XXX` (miscellaneous mounts)  |
-`/var/lib/kubelet` |
-`/var/lib/rancher` |
+| Mounts                                             |
+|----------------------------------------------------|
+| `/var/lib/kubelet/pods/XXX` (miscellaneous mounts) |
+| `/var/lib/kubelet`                                 |
+| `/var/lib/rancher`                                 |

 **To unmount all mounts:**

@@ -162,6 +167,57 @@ Mounts |
 for mount in $(mount | grep tmpfs | grep '/var/lib/kubelet' | awk '{ print $3 }') /var/lib/kubelet /var/lib/rancher; do umount $mount; done
 ```

+</TabItem>
+<TabItem value="RKE2">
+
+There are two components that need to be removed on nodes of an RKE2 cluster that was provisioned through Rancher:
+
+* The rancher-system-agent, which connects to Rancher and installs and manages RKE2
+* RKE2 itself
+
+### Removing rancher-system-agent
+
+To remove the rancher-system-agent, run the [system-agent-uninstall.sh](https://github.com/rancher/system-agent/blob/main/system-agent-uninstall.sh) script:
+
+```
+curl https://raw.githubusercontent.com/rancher/system-agent/main/system-agent-uninstall.sh | sudo sh
+```
+
+### Removing RKE2
+
+To remove the RKE2 installation, run the `rke2-uninstall` script that is already present on the node:
+
+```
+sudo rke2-uninstall.sh
+```
+
+</TabItem>
+<TabItem value="K3s">
+
+There are two components that need to be removed on nodes of a K3s cluster that was provisioned through Rancher:
+
+* The rancher-system-agent, which connects to Rancher and installs and manages K3s
+* K3s itself
+
+### Removing rancher-system-agent
+
+To remove the rancher-system-agent, run the [system-agent-uninstall.sh](https://github.com/rancher/system-agent/blob/main/system-agent-uninstall.sh) script:
+
+```
+curl https://raw.githubusercontent.com/rancher/system-agent/main/system-agent-uninstall.sh | sudo sh
+```
+
+### Removing K3s
+
+To remove the K3s installation, run the `k3s-uninstall` script that is already present on the node:
+
+```
+sudo k3s-uninstall.sh
+```
+
+</TabItem>
+</Tabs>
+
 ### Directories and Files

 The following directories are used when adding a node to a cluster, and should be removed. You can remove a directory using `rm -rf /directory_name`.
@@ -172,25 +228,26 @@ Depending on the role you assigned to the node, some of the directories will or

 :::

-Directories |
--------|
-`/etc/ceph` |
-`/etc/cni` |
-`/etc/kubernetes` |
-`/opt/cni` |
-`/opt/rke` |
-`/run/secrets/kubernetes.io` |
-`/run/calico` |
-`/run/flannel` |
-`/var/lib/calico` |
-`/var/lib/etcd` |
-`/var/lib/cni` |
-`/var/lib/kubelet` |
-`/var/lib/rancher/rke/log` |
-`/var/log/containers` |
-`/var/log/kube-audit` |
-`/var/log/pods` |
-`/var/run/calico` |
+| Directories                  |
+|------------------------------|
+| `/etc/ceph`                  |
+| `/etc/cni`                   |
+| `/etc/kubernetes`            |
+| `/etc/rancher`               |
+| `/opt/cni`                   |
+| `/opt/rke`                   |
+| `/run/secrets/kubernetes.io` |
+| `/run/calico`                |
+| `/run/flannel`               |
+| `/var/lib/calico`            |
+| `/var/lib/etcd`              |
+| `/var/lib/cni`               |
+| `/var/lib/kubelet`           |
+| `/var/lib/rancher`           |
+| `/var/log/containers`        |
+| `/var/log/kube-audit`        |
+| `/var/log/pods`              |
+| `/var/run/calico`            |

 **To clean the directories:**

@@ -198,6 +255,7 @@ Directories |
 rm -rf /etc/ceph \
       /etc/cni \
       /etc/kubernetes \
+       /etc/rancher \
       /opt/cni \
       /opt/rke \
       /run/secrets/kubernetes.io \
@@ -207,7 +265,7 @@ rm -rf /etc/ceph \
       /var/lib/etcd \
       /var/lib/cni \
       /var/lib/kubelet \
-       /var/lib/rancher/rke/log \
+       /var/lib/rancher\
       /var/log/containers \
       /var/log/kube-audit \
       /var/log/pods \
@@ -238,13 +296,13 @@ Depending on the network provider configured for the cluster the node was part o

 :::

-Interfaces |
--------|
-`flannel.1` |
-`cni0` |
-`tunl0` |
-`caliXXXXXXXXXXX` (random interface names)  |
-`vethXXXXXXXX` (random interface names)  |
+| Interfaces                                 |
+|--------------------------------------------|
+| `flannel.1`                                |
+| `cni0`                                     |
+| `tunl0`                                    |
+| `caliXXXXXXXXXXX` (random interface names) |
+| `vethXXXXXXXX` (random interface names)    |

 **To list all interfaces:**

@@ -272,32 +330,32 @@ Depending on the network provider configured for the cluster the node was part o

 Iptables rules are used to route traffic from and to containers. The created rules are not persistent, so restarting the node will restore iptables to its original state.

-Chains |
--------|
-`cali-failsafe-in` |
-`cali-failsafe-out` |
-`cali-fip-dnat` |
-`cali-fip-snat` |
-`cali-from-hep-forward` |
-`cali-from-host-endpoint` |
-`cali-from-wl-dispatch` |
-`cali-fw-caliXXXXXXXXXXX` (random chain names) |
-`cali-nat-outgoing` |
-`cali-pri-kns.NAMESPACE` (chain per namespace) |
-`cali-pro-kns.NAMESPACE` (chain per namespace) |
-`cali-to-hep-forward` |
-`cali-to-host-endpoint` |
-`cali-to-wl-dispatch` |
-`cali-tw-caliXXXXXXXXXXX` (random chain names) |
-`cali-wl-to-host` |
-`KUBE-EXTERNAL-SERVICES` |
-`KUBE-FIREWALL` |
-`KUBE-MARK-DROP` |
-`KUBE-MARK-MASQ` |
-`KUBE-NODEPORTS` |
-`KUBE-SEP-XXXXXXXXXXXXXXXX` (random chain names) |
-`KUBE-SERVICES` |
-`KUBE-SVC-XXXXXXXXXXXXXXXX` (random chain names) |
+| Chains                                           |
+|--------------------------------------------------|
+| `cali-failsafe-in`                               |
+| `cali-failsafe-out`                              |
+| `cali-fip-dnat`                                  |
+| `cali-fip-snat`                                  |
+| `cali-from-hep-forward`                          |
+| `cali-from-host-endpoint`                        |
+| `cali-from-wl-dispatch`                          |
+| `cali-fw-caliXXXXXXXXXXX` (random chain names)   |
+| `cali-nat-outgoing`                              |
+| `cali-pri-kns.NAMESPACE` (chain per namespace)   |
+| `cali-pro-kns.NAMESPACE` (chain per namespace)   |
+| `cali-to-hep-forward`                            |
+| `cali-to-host-endpoint`                          |
+| `cali-to-wl-dispatch`                            |
+| `cali-tw-caliXXXXXXXXXXX` (random chain names)   |
+| `cali-wl-to-host`                                |
+| `KUBE-EXTERNAL-SERVICES`                         |
+| `KUBE-FIREWALL`                                  |
+| `KUBE-MARK-DROP`                                 |
+| `KUBE-MARK-MASQ`                                 |
+| `KUBE-NODEPORTS`                                 |
+| `KUBE-SEP-XXXXXXXXXXXXXXXX` (random chain names) |
+| `KUBE-SERVICES`                                  |
+| `KUBE-SVC-XXXXXXXXXXXXXXXX` (random chain names) |

 **To list all iptables rules:**

@@ -27,7 +27,9 @@ These elements are required to follow this guide:

 ### 1. Create a Custom Cluster

-On Rancher server, we should create a custom k8s cluster v1.18.x. Be sure that cloud_provider name is set to `amazonec2`. Once cluster is created we need to get:
+On Rancher server, we should create a custom k8s cluster. Refer [here](https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/) to check for version compatibility.
+
+Be sure that cloud_provider name is set to `amazonec2`. Once cluster is created we need to get:

 * clusterID: `c-xxxxx` will be used on EC2 `kubernetes.io/cluster/<clusterID>` instance tag
 * clusterName: will be used on EC2 `k8s.io/cluster-autoscaler/<clusterName>` instance tag
@@ -481,7 +483,7 @@ spec:
      nodeSelector:
        node-role.kubernetes.io/controlplane: "true"
      containers:
-        - image: eu.gcr.io/k8s-artifacts-prod/autoscaling/cluster-autoscaler:v1.18.1
+        - image: eu.gcr.io/k8s-artifacts-prod/autoscaling/cluster-autoscaler:<VERSION>
          name: cluster-autoscaler
          resources:
            limits:
@@ -0,0 +1,49 @@
+---
+title: Encryption Key Rotation
+weight: 2043
+---
+
+### RKE1 Encryption Key Rotation
+
+1. Enable encryption key rotation with either of the following two options:
+
+    - Select the `Enabled` radio button in the Rancher UI under **Cluster Options > Advanced Options > Secrets Encryption**:
+
+    ![Enable Encryption Key Rotation](/img/rke1-enable-secrets-encryption.png)
+
+    - OR, apply the following YAML:
+
+    ```
+    rancher_kubernetes_engine_config:
+      services:
+        kube_api:
+          secrets_encryption_config:
+            enabled: true
+    ```
+
+2. Rotate keys in the Rancher UI:
+
+    2.1. Click **☰ > Cluster Management**.
+
+    2.2. Select **⋮ > Rotate Encryption Keys** on the far right of the screen next to your chosen cluster:
+
+    ![Encryption Key Rotation](/img/rke1-encryption-key.png)
+
+
+
+### RKE2 Encryption Key Rotation
+
+_**New in v2.6.7**_
+
+>**Important:** Encryption key rotation is enabled by default and cannot be disabled.
+
+To rotate keys in the Rancher UI:
+
+1. Click **☰ > Cluster Management**.
+
+1. Select **⋮ > Rotate Encryption Keys** on the far right of the screen next to your chosen cluster:
+
+    ![Encryption Key Rotation](/img/rke2-encryption-key.png)
+
+
+>**Note:** For more information on RKE2 secrets encryption config, please see the [RKE2 docs](https://docs.rke2.io/security/secrets_encryption/).
@@ -20,8 +20,23 @@ The default configuration should already be compatible with your rancher-monitor

 ## Ensure the cattle-monitoring-system namespace is placed into the System Project (or a similarly locked down Project that has access to other Projects in the cluster)

+![Select Projects-Namespaces](/img/install-in-system-project.png)
+
 Prometheus Operator's security model expects that the namespace it is deployed into (e.g., `cattle-monitoring-system`) has limited access for anyone except Cluster Admins to avoid privilege escalation via execing into Pods (such as the Jobs executing Helm operations). In addition, deploying Prometheus Federator and all Project Prometheus stacks into the System Project ensures that each Project Prometheus is able to reach out to scrape workloads across all Projects, even if Network Policies are defined via Project Network Isolation. It also provides limited access for Project Owners, Project Members, and other users so that they're unable to access data that they shouldn't have access to (i.e., being allowed to exec into pods, set up the ability to scrape namespaces outside of a given Project, etc.).

+1. Open the `System` project to check your namespaces:
+
+    Click **Cluster > Projects/Namespaces** in the Rancher UI. This will display all of the namespaces in the `System` project:
+
+    ![Select Projects-Namespaces](/img/cattle-monitoring-system.png)
+
+1. If you have an existing Monitoring V2 installation within the `cattle-monitoring-system` namespace, but that namespace is not in the `System` project, you may move the `cattle-monitoring-system` namespace into the `System` project or into another project of limited access. To do so, you may either:
+
+    - Drag and drop the namespace into the `System` project or
+    - Select **⋮** to the right of the namespace, click **Move**, then choose `System` from the **Target Project** dropdown
+
+        ![Move to a New Project](/img/move-to-new-project.png)
+
 ## Configure rancher-monitoring to only watch for resources created by the Helm chart itself

 Since each Project Monitoring Stack will watch the other namespaces and collect additional custom workload metrics or dashboards already, it's recommended to configure the following settings on all selectors to ensure that the Cluster Prometheus Stack only monitors resources created by the Helm Chart itself:
@@ -0,0 +1,18 @@
+---
+title: Installing Project Monitors
+weight: 5
+---
+
+Install **Project Monitors** in each project where you want to enable project monitoring.
+
+1. Click **☰ > Cluster Management**.
+
+1. On the **Clusters** page, go to the cluster where you want to enable monitoring and click **Explore**.
+
+1. Click **Monitoring > Project Monitors** on the left nav bar. Then click **Create** in the upper right.
+
+    ![Project Monitors](/img/project-monitors.png)
+
+1. Select your project from the drop-down menu, then click **Create** again.
+
+    ![Create Project Monitors](/img/create-project-monitors.png)
@@ -21,7 +21,7 @@ The monitoring application needs to be installed.

 To create rule groups in the Rancher UI,

-1. Go to the cluster where you want to create rule groups. Click **Monitoring** and click **Prometheus Rules**.
+1. Go to the cluster where you want to create rule groups. Click **Monitoring > Advanced** and click **Prometheus Rules**.
 1. Click **Create**.
 1. Enter a **Group Name**.
 1. Configure the rules. In Rancher's UI, we expect a rule group to contain either alert rules or recording rules, but not both. For help filling out the forms, refer to the configuration options below.
@@ -128,7 +128,7 @@ Once the Restore resource has the status `Completed`, you can continue the Ranch

 ### 3. Install cert-manager

-Follow the steps to [install cert-manager](../../../pages-for-subheaders/install-upgrade-on-a-kubernetes-cluster.md#5-install-cert-manager) in the documentation about installing cert-manager on Kubernetes.
+Follow the steps to [install cert-manager](../../../pages-for-subheaders/install-upgrade-on-a-kubernetes-cluster.md#4-install-cert-manager) in the documentation about installing cert-manager on Kubernetes.

 ### 4. Bring up Rancher with Helm

@@ -17,7 +17,7 @@ This page outlines how to perform a restore with Rancher.

 In Rancher v2.6.4, the cluster-api module has been upgraded from v0.4.4 to v1.0.2 in which the apiVersion of CAPI CRDs are upgraded from `cluster.x-k8s.io/v1alpha4` to `cluster.x-k8s.io/v1beta1`. This has the effect of causing rollbacks from Rancher v2.6.4 to any previous version of Rancher v2.6.x to fail because the previous version the CRDs needed to roll back are no longer available in v1beta1.

-To avoid this, the Rancher resource cleanup scripts should be run **before** the restore or rollback is attempted. Specifically, two scripts have been created to assist you: one to clean up the cluster (`cleanup.sh`), and one to check for any Rancher-related resources in the cluster (`verify.sh`). Details on the cleanup script can be found in the [rancherlabs/support-tools repo](https://github.com/rancherlabs/support-tools/tree/master/rancher-cleanup).
+To avoid this, the Rancher resource cleanup scripts should be run **before** the restore or rollback is attempted. Specifically, two scripts have been created to assist you: one to clean up the cluster (`cleanup.sh`), and one to check for any Rancher-related resources in the cluster (`verify.sh`). Details on the cleanup script can be found in the [rancher/rancher-cleanup repo](https://github.com/rancher/rancher-cleanup).

 :::caution

@@ -27,7 +27,7 @@ Rancher will be down as the `cleanup` script runs as it deletes the resources cr

 The additional preparations:

-1. Follow these [instructions](https://github.com/rancherlabs/support-tools/blob/master/rancher-cleanup/README.md) to run the scripts.
+1. Follow these [instructions](https://github.com/rancher/rancher-cleanup/blob/main/README.md) to run the scripts.
 1. Follow these [instructions](https://rancher.com/docs/rancher/v2.6/en/backups/migrating-rancher/) to install the rancher-backup Helm chart on the existing cluster and restore the previous state.
    1. Omit Step 3.
 	1. When Step 4 is reached, install the required Rancher v2.6.x version on the local cluster you intend to roll back to.
@@ -0,0 +1,321 @@
+---
+title: RKE1 to RKE2 Windows Migration Guidance
+weight: 3
+---
+
+**Caution:** The contents of this document are not covered under SLA by Rancher Support. Please proceed with caution.
+
+This document covers how end users can migrate their Windows workloads from RKE1 to RKE2.
+
+- [RKE1 Windows Scheduling](#rke1-windows-scheduling)
+- [RKE2 Windows Scheduling](#rke2-windows-scheduling)
+- [Example Migrations](#example-migrations)
+  - [RKE1 to RKE2 Windows Workload](#rke1-to-rke2-windows-workload)
+  - [RKE1 Windows Cluster Linux-Only Deployment](#rke1-windows-cluster-linux-only-deployment)
+- [RKE1 Windows-Supported Windows Server Versions](#rke1-windows-supported-windows-server-versions)
+  - [Long-Term Servicing Channel (LTSC)](#long-term-servicing-channel-ltsc)
+  - [Semi-Annual Channel (SAC)](#semi-annual-channel-sac)
+- [RKE2 Windows-Supported Windows Server Versions](#rke2-windows-supported-windows-server-versions)
+  - [Long-Term Servicing Channel in RKE2](#long-term-servicing-channel-in-rke2)
+- [Kubernetes Version Support](#kubernetes-version-support)
+  - [Rancher 2.5 vs. Rancher 2.6 Support Matrix for Windows Clusters](#rancher-2-5-vs-rancher-2-6-support-matrix-for-windows-clusters)
+  - [Rancher 2.5 vs. Rancher 2.6 Supported Kubernetes Versions for Provisioning RKE1 and RKE2 Windows Clusters](#rancher-2-5-vs-rancher-2-6-supported-kubernetes-versions-for-provisioning-rke1-and-rke2-windows-clusters)
+- [Guiding Migrations of Workloads to RKE2 Windows](#guiding-migrations-of-workloads-to-rke2-windows)
+  - [In-Place Upgrade of Rancher 2.5](#in-place-upgrade-of-rancher-2-5)
+  - [Migrating Windows Workloads to a new Rancher environment](#migrating-windows-workloads-to-a-new-rancher-environment)
+
+## RKE1 Windows Scheduling
+
+RKE1 Windows workload scheduling is based on taints and tolerations.
+
+Every Linux node in an RKE1 Windows cluster, regardless of the role assigned to it, will have have a default taint that prevents workloads to be scheduled on it unless the workload has a toleration configured. This is a major design feature for RKE1 Windows clusters which were designed to only run Windows workloads.
+
+- Default RKE1 Linux node `NoSchedule` taint:
+
+```yml
+apiVersion: v1
+kind: Node
+spec:
+  ...
+  taints:
+  - effect: NoSchedule
+    key: cattle.io/os
+    value: linux
+```
+<br/>
+
+- RKE1 Linux `NoSchedule` toleration for workloads
+
+The following toleration would allow an end-user workload to schedule on any Linux node of an RKE1 Windows cluster. These tolerations are used for various core Rancher services and workloads.
+
+```yml
+apiVersion: apps/v1
+kind: Deployment
+spec:
+  ...
+  template:
+    ...
+    spec:
+      tolerations:
+      - effect: NoSchedule
+        key: cattle.io/os
+        operator: Equal
+        value: linux
+```
+<br/>
+
+- Aligning with best practices, any end-user workloads being run on Linux nodes would be scheduled on those with the worker role only:
+
+```yml
+apiVersion: apps/v1
+kind: Deployment
+spec:
+  ...
+  template:
+    ...
+    spec:
+      tolerations:
+      - effect: NoSchedule
+        key: cattle.io/os
+        operator: Equal
+        value: linux
+      affinity:
+        nodeAffinity:
+          requiredDuringSchedulingIgnoredDuringExecution:
+          - preference:
+              matchExpressions:
+              - key: node-role.kubernetes.io/worker
+                operator: In
+                values:
+                - "true"
+            weight: 100
+      ...
+```
+
+## RKE2 Windows Scheduling
+
+Based on feedback and requests for hybrid workload support, RKE2 Windows was designed to support both Linux and Windows workloads by default. RKE2 scheduling relies on node selectors by default. This is a marked change from RKE1 as taints and tolerations were not incorporated into RKE2. Node selectors were a critical part of RKE1 Windows clusters, which makes for an easy migration of your workloads.
+
+## Example Migrations
+
+### RKE1 to RKE2 Windows Workload
+
+- Pre-migration RKE1 Windows deployment:
+
+```yaml
+apiVersion: apps/v1
+kind: Deployment
+spec:
+  ...
+  template:
+    ...
+    spec:
+      affinity:
+        nodeAffinity:
+          requiredDuringSchedulingIgnoredDuringExecution:
+            nodeSelectorTerms:
+            - matchExpressions:
+              - key: kubernetes.io/os
+                operator: NotIn
+                values:
+                - linux
+```
+<br/>
+
+- Migrated RKE2 Windows deployment using `NodeAffinity`:
+
+```yaml
+apiVersion: apps/v1
+kind: Deployment
+...
+spec:
+  ...
+  template:
+    ...
+    spec:
+      ...
+      affinity:
+        nodeAffinity:
+          requiredDuringSchedulingIgnoredDuringExecution:
+            nodeSelectorTerms:
+              - matchExpressions:
+                  - key: kubernetes.io/os
+                    operator: In
+                    values:
+                      - windows
+```
+
+### RKE1 Windows Cluster Linux-Only Deployment
+
+>**Important:** When leveraging node selectors and node affinity, note the following:
+>
+>- If both `nodeSelector` and `nodeAffinity` are specified, both must be satisfied for the `Pod` to be scheduled onto a node.
+>- If you specify multiple `matchExpressions` associated with a single `nodeSelectorTerms`, then the `Pod` can be scheduled onto a node only if all the `matchExpressions` are satisfied.
+
+<br/>
+
+- Pre-migration RKE1 Windows cluster Linux-only deployment targeting RKE1 Linux worker nodes:
+
+```yml
+apiVersion: apps/v1
+kind: Deployment
+spec:
+  ...
+  template:
+    ...
+    spec:
+      tolerations:
+      - effect: NoSchedule
+        key: cattle.io/os
+        operator: Equal
+        value: linux
+      affinity:
+        nodeAffinity:
+          requiredDuringSchedulingIgnoredDuringExecution:
+          - weight: 100
+            preference:
+              matchExpressions:
+              - key: node-role.kubernetes.io/worker
+                operator: In
+                values:
+                - "true"
+
+```
+<br/>
+
+- Migrated RKE2 hybrid cluster Linux-only deployment targeting RKE2 Linux worker nodes using node selectors:
+
+```yml
+apiVersion: apps/v1
+kind: Deployment
+spec:
+  ...
+  template:
+    ...
+    spec:
+      nodeSelector:
+        kubernetes.io/os: "linux"
+        node-role.kubernetes.io/worker: "true"
+ ```
+<br/>
+
+- Migrated RKE2 hybrid cluster Linux-only deployment targeting RKE2 Linux worker nodes using node affinity:
+
+ ```yaml
+ apiVersion: apps/v1
+kind: Deployment
+spec:
+  ...
+  template:
+    ...
+    spec:
+       affinity:
+        nodeAffinity:
+          requiredDuringSchedulingIgnoredDuringExecution:
+          - weight: 100
+            preference:
+              matchExpressions:
+              - key: node-role.kubernetes.io/worker
+                operator: In
+                values:
+                - "true"
+            nodeSelectorTerms:
+              - matchExpressions:
+                  - key: kubernetes.io/os
+                    operator: In
+                    values:
+                      - linux
+ ```
+## RKE1 Windows-Supported Windows Server Versions
+
+### Long-Term Servicing Channel (LTSC)
+
+- Windows Server 2019 LTSC &#9989; Will reach Mainstream EOL on Jan 9, 2024 and Extended EOL on Jan 9, 2029
+
+### Semi-Annual Channel (SAC)
+
+- Windows Server 20H2 SAC &#10071; Will reach EOL on Aug 9, 2022
+- Windows Server 2004 SAC &#10060; EOL Reached on Dec 14, 2021
+- Windows Server 1909 SAC &#10060; EOL Reached on May 11, 2021
+- Windows Server 1903 SAC &#10060; EOL Reached on Dec 8, 2020
+- Windows Server 1809 SAC &#10060; EOL Reached on Nov 10, 2020
+
+## RKE2 Windows-Supported Windows Server Versions
+
+### Long-Term Servicing Channel in RKE2
+
+- Windows Server 2019 LTSC &#9989; Will reach Mainstream EOL on Jan 9, 2024 and Extended EOL on Jan 9, 2029
+- Windows Server 2022 LTSC &#9989; Will reach Mainstream EOL on Oct 13, 2026 and Extended EOL on Oct 13, 2031
+
+>**Note:** SAC is not supported in RKE2.
+
+
+For more information, please see the following references:
+
+- [Windows Server SAC Lifecycle](https://docs.microsoft.com/en-us/lifecycle/products/windows-server)
+
+- [Windows Server 2022 LTSC Lifecycle](https://docs.microsoft.com/en-us/lifecycle/products/windows-server-2022)
+
+- [Windows Server 2019 LTSC Lifecycle](https://docs.microsoft.com/en-us/lifecycle/products/windows-server-2019)
+
+
+## Kubernetes Version Support
+
+>**Note:** All versions listed below are SLA Supported per the [Rancher v2.6.7 Support Matrix](https://www.suse.com/suse-rancher/support-matrix/all-supported-versions/rancher-v2-6-7/). Any version not listed should be assumed as being EOL and not supported under SLA by SUSE.
+
+### Rancher 2.5 vs. Rancher 2.6 Support Matrix for Windows Clusters
+
+**RKE1 vs. RKE2 Windows cluster-supported Kubernetes versions:**
+
+| Kubernetes Versions 	| RKE1 	| RKE2 	|
+|--------------	|:----:	|:----:	|
+| 1.18                	|   &check;   	|      	|
+| 1.19                	|   &check;   	|      	|
+| 1.20                	|   &check;   	|      	|
+| 1.21                	|   &check;   	|      	|
+| 1.22                	|   &check;   	|   &check;   |
+| 1.23                	|      	  |   &check;  	|
+| 1.24                	|      	  |   &check;   |
+| 1.25+               	|      	  |   &check;   |
+
+
+### Rancher 2.5 vs. Rancher 2.6 Supported Kubernetes Versions for Provisioning RKE1 and RKE2 Windows Clusters
+
+| Rancher Versions 	|    Kubernetes Versions   	| RKE1 	| RKE2 	|
+|:-----------------------:	|:------------------------:	|:----:	|:----:	|
+| 2.5 - RKE1 Provisioning 	|      1.18 1.19 1.20      	|   &check;   	|      	|
+| 2.6 - RKE1 Provisioning 	| 1.18 1.19 1.20 1.21 1.22 	|   &check;   	|      	|
+| 2.6 - RKE2 Provisioning 	|   1.22 1.23 1.24 1.25+   	|      	|   &check;   	|
+
+
+## Guiding Migrations of Workloads to RKE2 Windows
+<br/>
+Referencing the tables in [Rancher 2.5 vs. Rancher 2.6 Support Matrix for Windows Clusters](#rancher-2-5-vs-rancher-2-6-support-matrix-for-windows-clusters) and [Rancher 2.5 vs. Rancher 2.6 Supported Kubernetes Versions for Provisioning RKE1 and RKE2 Windows Clusters](#rancher-2-5-vs-rancher-2-6-supported-kubernetes-versions-for-provisioning-rke1-and-rke2-windows-clusters), you will find the overlap in Kubernetes versions between RKE1 and RKE2 occurs in 1.22. This will be the base version required to migrate RKE1 Windows workloads when following the Rancher recommended approach.
+
+### In-Place Upgrade of Rancher 2.5
+
+1. Upgrade the Rancher version to v2.6.5+.
+1. Upgrade the RKE1 Windows downstream cluster(s) to RKE1 v1.22 using the latest available patch version.
+1. Provision a new RKE2 Windows downstream cluster using RKE2 v1.22 using the matching patch version that the RKE1 Windows cluster is at.
+1. Begin the migration of the Windows workloads from RKE1 to RKE2 clusters.
+1. Perform validation tests to ensure that there has been no functionality loss or change when migrating your application from RKE1 to RKE2.
+1. After successful validation tests have occurred, you can opt to upgrade your RKE2 1.22.x cluster to a new minor version such as 1.23 or 1.24.
+
+
+### Migrating Windows Workloads to a New Rancher Environment
+
+>**Important:** To perform either of the following options requires Rancher v2.6.5 or above.
+
+**When using matching Kubernetes patch versions for RKE1 and RKE2:**
+
+1. Provision a new RKE2 Windows downstream cluster using RKE2 v1.22 using the matching patch version that the RKE1 Windows cluster is at.
+1. Begin the migration of the Windows workloads from RKE1 to RKE2 clusters.
+1. Perform validation tests to ensure that there has been no functionality loss or change when migrating your application from RKE1 to RKE2.
+1. After successful validation tests have occurred, you can opt to upgrade your RKE2 1.22.x cluster to a new minor version such as 1.23 or 1.24.
+
+
+**When using a newer Kubernetes patch version for RKE2:**
+
+1. Provision a new RKE2 Windows downstream cluster using RKE2 v1.23 or v1.24.
+1. Begin the migration of the Windows workloads from RKE1 to RKE2 clusters.
+1. Perform validation tests to ensure that there has been no functionality loss or change when migrating your application from RKE1 to RKE2.
@@ -12,8 +12,8 @@ Ingresses can be added for workloads to provide load balancing, SSL termination
 1. Click **Create**.
 1. Select an existing **Namespace** from the drop-down list.
 1. Enter a **Name** for the ingress.
-1. Create ingress forwarding **Rules**. For help configuring the rules, refer to [this section.](#ingress-rule-configuration) If any of your ingress rules handle requests for encrypted ports, add a certificate to encrypt/decrypt communications.
-1. **Optional:** click **Add Rule** to create additional ingress rules. For example, after you create ingress rules to direct requests for your hostname, you'll likely want to create a default backend to handle 404s. 
+1. Create ingress forwarding **Rules**. For help configuring the rules, refer to [this section.](./ingress-configuration.md#ingress-rule-configuration) If any of your ingress rules handle requests for encrypted ports, add a certificate to encrypt/decrypt communications.
+1. **Optional:** click **Add Rule** to create additional ingress rules. For example, after you create ingress rules to direct requests for your hostname, you'll likely want to create a default backend to handle 404s.
 1. Click **Create** at the bottom right.

 **Result:** Your ingress is added to the project. The ingress begins enforcing your ingress rules.
@@ -5,7 +5,6 @@ weight: 9999
 ---

 - [NGINX Ingress controller changes in Kubernetes v1.21](#nginx-ingress-controller-changes-in-Kubernetes-v1-21)
- [Automatically generate a xip.io hostname](#automatically-generate-a-xip-io-hostname)
 - [Specify a hostname to use](#specify-a-hostname-to-use)
 - [Use as the default backend](#use-as-the-default-backend)
 - [Certificates](#certificates)
@@ -0,0 +1,32 @@
+---
+title: AWS Marketplace Integration
+weight: 1
+---
+
+# Overview
+
+Rancher offers an integration with the AWS Marketplace which allows users to purchase a support contract with SUSE. This integration allows you easily adjust your support needs as you start to support more clusters.
+
+# Limitations
+
+- You must be running Rancher v2.6.7 or higher
+- Rancher must be deployed with additional metrics enabled.
+- Rancher must be installed on an EKS cluster.
+- You must purchase at least one entitlement to Rancher support through AWS Marketplace.
+- You may need additional setup to support proxy/airgap use cases. See the [prerequisites](../explanations/integrations-in-rancher/cloud-marketplace/aws-cloud-marketplace/adapter-requirements.md) for more information.
+
+# How to Use
+1. Complete the [prerequisite steps](../explanations/integrations-in-rancher/cloud-marketplace/aws-cloud-marketplace/adapter-requirements.md).
+2. [Install the CSP Adapter](../explanations/integrations-in-rancher/cloud-marketplace/aws-cloud-marketplace/install-adapter.md).
+
+# FAQ
+
+**Can I purchase support for more nodes later on?**
+
+Yes. Simply go to the AWS Marketplace entry that you used to initially purchase support and increase the number of entitlements.
+
+**Can I use multiple instances of Rancher in the same AWS account?**
+
+Yes. However, each cluster that Rancher is installed in will need to adhere to the prerequisites.
+
+In addition, keep in mind that a given entitlement can only be used by one Rancher management server at a time.
@@ -13,7 +13,6 @@ The backup-restore operator needs to be installed in the local cluster, and only
 - [How Backups and Restores Work](#how-backups-and-restores-work)
 - [Installing the rancher-backup Operator](#installing-the-rancher-backup-operator)
  - [Installing rancher-backup with the Rancher UI](#installing-rancher-backup-with-the-rancher-ui)
-  - [Installing rancher-backup with the Helm CLI](#installing-rancher-backup-with-the-helm-cli)
  - [RBAC](#rbac)
 - [Backing up Rancher](#backing-up-rancher)
 - [Restoring Rancher](#restoring-rancher)
@@ -0,0 +1,8 @@
+---
+title: Cloud Marketplace Integration
+weight: 6
+---
+
+Rancher offers integration with cloud marketplaces to easily purchase support for installations hosted on certain cloud providers. In addition, this integration also provides the ability to generate a supportconfig bundle which can be provided to rancher support.
+
+As of Rancher v2.6.7, only AWS is supported for this integration.
@@ -24,19 +24,19 @@ Any major versions that are less than the ones mentioned in the table below are
 | external-ip-webhook | 100.0.0+up1.0.0 | 100.0.1+up1.0.1 |
 | harvester-cloud-provider | 100.0.2+up0.1.12 | 100.0.2+up0.1.12 |
 | harvester-csi-driver | 100.0.2+up0.1.11 | 100.0.2+up0.1.11 |
+| neuvector | 100.0.0+up2.2.0 | 100.0.0+up2.2.0 |
 | rancher-alerting-drivers | 100.0.0 | 100.0.2 |
 | rancher-backup | 2.0.1 | 2.1.2 |
 | rancher-cis-benchmark | 2.0.1 | 2.0.4 |
 | rancher-gatekeeper | 100.0.0+up3.6.0 | 100.1.0+up3.7.1 |
-| rancher-istio | 100.0.0+up1.10.4 | 100.2.0+up1.12.6 |
+| rancher-istio | 100.0.0+up1.10.4 | 100.3.0+up1.13.3 |
 | rancher-logging | 100.0.0+up3.12.0 | 100.1.2+up3.17.4 |
-| rancher-longhorn | 100.0.0+up1.1.2 | 100.1.1+up1.2.3 |
-| rancher-monitoring | 100.0.0+up16.6.0 | 100.1.0+up19.0.3
+| rancher-longhorn | 100.0.0+up1.1.2 | 100.1.2+up1.2.4 |
+| rancher-monitoring | 100.0.0+up16.6.0 | 100.1.2+up19.0.3 |
 | rancher-sriov (experimental) | 100.0.0+up0.1.0 | 100.0.3+up0.1.0 |
 | rancher-vsphere-cpi | 100.3.0+up1.2.1 | 100.3.0+up1.2.1 |
 | rancher-vsphere-csi | 100.3.0+up2.5.1-rancher1 | 100.3.0+up2.5.1-rancher1 |
-| rancher-wins-upgrader | 0.0.100 | 100.0.0+up0.0.1 |
-| neuvector | 100.0.0+up2.2.0 | 100.0.0+up2.2.0 |
+| rancher-wins-upgrader | 0.0.100 | 100.0.1+up0.0.1 |

 <br/>
 **Charts based on upstream:** For charts that are based on upstreams, the +up annotation should inform you of what upstream version the Rancher chart is tracking. Check the upstream version compatibility with Rancher during upgrades also.
@@ -27,7 +27,7 @@ import ClusterCapabilitiesTable from '../shared-files/_cluster-capabilities-tabl

 ## Configuring Tools

-Rancher contains a variety of tools that aren't included in Kubernetes to assist in your DevOps operations. Rancher can integrate with external services to help your clusters run more efficiently. Tools are divided into following categories:
+Rancher contains a variety of tools that aren't included in Kubernetes to assist in your DevOps operations. Rancher can integrate with external services to help your clusters run more efficiently. Tools are divided into the following categories:

 - Alerts
 - Notifiers
@@ -34,6 +34,8 @@ Rancher v2.6 introduces provisioning for [RKE2](https://docs.rke2.io/) clusters

 As of Rancher v2.6.5, provisioning for RKE2 is GA.

+### RKE2 Windows
+
 The RKE2 provisioning feature also includes installing RKE2 on Windows clusters. Windows features for RKE2 include:

 - Windows Containers with RKE2 powered by containerd
@@ -49,6 +51,10 @@ Rancher will allow Windows workload pods to deploy on both Windows and Linux wor

 :::

+**_New in Rancher v2.6.7_**
+
+- HostProcess containers in Windows RKE2 are supported in Kubernetes v1.24.1 and up. See [the upstream documentation](https://kubernetes.io/docs/tasks/configure-pod-container/create-hostprocess-pod/) for more information.
+
 # Requirements for Windows Clusters

 The general node requirements for networking, operating systems, and Docker are the same as the node requirements for a [Rancher installation](installation-requirements.md).
@@ -204,7 +210,7 @@ The instructions for creating a Windows cluster on existing nodes are very simil
 1. Optional: After you enable Windows support, you will be able to choose the Flannel backend. There are two network options: [**Host Gateway (L2bridge)**](https://github.com/coreos/flannel/blob/master/Documentation/backends.md#host-gw) and [**VXLAN (Overlay)**](https://github.com/coreos/flannel/blob/master/Documentation/backends.md#vxlan). The default option is **VXLAN (Overlay)** mode.
 1. Click **Next**.

-:::note Important: 
+:::note Important:

 For <b>Host Gateway (L2bridge)</b> networking, it's best to use the same Layer 2 network for all nodes. Otherwise, you need to configure the route rules for them. For details, refer to the [documentation on configuring cloud-hosted VM routes.](../how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/launch-kubernetes-with-rancher/use-windows-clusters/network-requirements-for-host-gateway.md#cloud-hosted-vm-routes-configuration) You will also need to [disable private IP address checks](../how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/launch-kubernetes-with-rancher/use-windows-clusters/network-requirements-for-host-gateway.md#disabling-private-ip-address-checks) if you are using Amazon EC2, Google GCE, or Azure VM.

@@ -227,13 +233,13 @@ The first node in your cluster should be a Linux host has both the **Control Pla
 1. SSH into your Linux host and run the command that you copied to your clipboard.
 1. When you are finished provisioning your Linux node(s), select **Done**.

-**Result:** 
+**Result:**

 Your cluster is created and assigned a state of **Provisioning**. Rancher is standing up your cluster.

 You can access your cluster after its state is updated to **Active**.

-**Active** clusters are assigned two Projects: 
+**Active** clusters are assigned two Projects:

 - `Default`, containing the `default` namespace
 - `System`, containing the `cattle-system`, `ingress-nginx`, `kube-public`, and `kube-system` namespaces
@@ -257,7 +263,7 @@ After the initial provisioning of your cluster, your cluster only has a single L

 **Result:** The **Worker** role is installed on your Linux host, and the node registers with Rancher. It may take a few minutes for the node to be registered in your cluster.

-:::note 
+:::note

 Taints on Linux Worker Nodes

@@ -15,8 +15,6 @@ Rancher was originally built to work with multiple orchestrators, and it include

 Rancher can provision Kubernetes from a hosted provider, provision compute nodes and then install Kubernetes onto them, or import existing Kubernetes clusters running anywhere.

-One Rancher server installation can manage thousands of Kubernetes clusters and thousands of nodes from the same user interface.
-
 Rancher adds significant value on top of Kubernetes, first by centralizing authentication and role-based access control (RBAC) for all of the clusters, giving global admins the ability to control cluster access from one location.

 It then enables detailed monitoring and alerting for clusters and their resources, ships logs to external providers, and integrates directly with Helm via the Application Catalog. If you have an external CI/CD system, you can plug it into Rancher, but if you don't, Rancher even includes Fleet to help you automatically deploy and upgrade workloads.
@@ -18,43 +18,37 @@ To delete a token,

 Here is the complete list of tokens that are generated with `ttl=0`:

-| Token | Description |
-|-------|-------------|
-| `kubeconfig-*` | Kubeconfig token | 
-| `kubectl-shell-*` | Access to `kubectl` shell in the browser |
-| `agent-*` | Token for agent deployment |
-| `compose-token-*` | Token for compose |
-| `helm-token-*` | Token for Helm chart deployment |
-| `*-pipeline*` | Pipeline token for project |
-| `telemetry-*` | Telemetry token |
-| `drain-node-*` | Token for drain (we use `kubectl` for drain because there is no native Kubernetes API) |
+| Token             | Description                                                                            |
+| ----------------- | -------------------------------------------------------------------------------------- |
+| `kubeconfig-*`    | Kubeconfig token                                                                       |
+| `kubectl-shell-*` | Access to `kubectl` shell in the browser                                               |
+| `agent-*`         | Token for agent deployment                                                             |
+| `compose-token-*` | Token for compose                                                                      |
+| `helm-token-*`    | Token for Helm chart deployment                                                        |
+| `*-pipeline*`     | Pipeline token for project                                                             |
+| `telemetry-*`     | Telemetry token                                                                        |
+| `drain-node-*`    | Token for drain (we use `kubectl` for drain because there is no native Kubernetes API) |


 ### Setting TTL on Kubeconfig Tokens

-Admins can set a global TTL on Kubeconfig tokens. Once the token expires the kubectl command will require the user to authenticate to Rancher.
-
-Go to the global settings and:
-
-1. Set the `kubeconfig-generate-token` setting to `false`. This setting instructs Rancher to no longer automatically generate a token when a user clicks on download a kubeconfig file. The kubeconfig file will now provide a command to login to Rancher.
+Admins can set a global time-to-live (TTL) on Kubeconfig tokens. Changing the default kubeconfig TTL can be done by navigating to global settings and setting [`kubeconfig-default-token-ttl-minutes`](#kubeconfig-default-token-ttl-minutes) to the desired duration in minutes. The default value of [`kubeconfig-default-token-ttl-minutes`](#kubeconfig-default-token-ttl-minutes) is 0, which means tokens never expire.

 :::note

-Once this setting is deactivated, a generated kubeconfig will reference the [Rancher CLI](../../pages-for-subheaders/cli-with-rancher.md) to retrieve a short lived token for the cluster. When you use this kubeconfig in a client, such as `kubectl`, the Rancher CLI needs to be installed as well.
+This setting is used by all kubeconfig tokens except those created by the CLI to [generate kubeconfig tokens](#disable-tokens-in-generated-kubeconfigs).

 :::

+### Disable Tokens in Generated Kubeconfigs
+
+1. Set the `kubeconfig-generate-token` setting to `false`. This setting instructs Rancher to no longer automatically generate a token when a user clicks on download a kubeconfig file. Once this setting is deactivated, a generated kubeconfig will reference the [Rancher CLI](../cli-with-rancher/kubectl-utility.md#authentication-with-kubectl-and-kubeconfig-tokens-with-ttl) to retrieve a short-lived token for the cluster. When this kubeconfig is used in a client, such as `kubectl`, the Rancher CLI needs to be installed to complete the log in request.
+
 2. Set the `kubeconfig-token-ttl-minutes` setting to the desired duration in minutes. By default, `kubeconfig-token-ttl-minutes` is 960 (16 hours).

-:::note
-
-This value cannot exceed max-ttl of API tokens.(`https://<Rancher-Server-IP/v3/settings/auth-token-max-ttl-minutes`). `auth-token-max-ttl-minutes` is set to 1440 (24 hours) by default.  `auth-token-max-ttl-minutes would default to 0 allowing tokens to never expire`.
-
-:::
-
 ### Token Hashing

-Users can enable token hashing, where tokens will undergo a one-way hash using the SHA256 algorithm. This is a non-reversible process, once enabled, this feature cannot be disabled. It is advisable to take backups prior to enabling and/or evaluated in a test environment first.
+Users can enable token hashing, where tokens will undergo a one-way hash using the SHA256 algorithm. This is a non-reversible process, once enabled, this feature cannot be disabled. It is advisable to take backups prior to enabling and/or evaluating in a test environment first.

 To enable token hashing, refer to [this section](../../pages-for-subheaders/enable-experimental-features.md).

@@ -63,4 +57,32 @@ This feature will affect all tokens which include, but are not limited to, the f
 - Kubeconfig tokens
 - Bearer tokens API keys/calls
 - Tokens used by internal operations
- 
+
+### Token Settings
+These global settings affect Rancher token behavior.
+
+| Setting                                                                         | Description                                                                                                                                                                                                                    |
+| ------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
+| [`auth-user-session-ttl-minutes`](#auth-user-session-ttl-minutes)               | TTL in minutes on a user auth session token.                                                                                                                                                                                   |
+| [`kubeconfig-default-token-TTL-minutes`](#kubeconfig-default-token-ttl-minutes) | Default TTL applied to all kubeconfig tokens except those [generated by Rancher CLI](#disable-tokens-in-generated-kubeconfigs). **Introduced in version 2.6.6.**                                                   |
+| [`kubeconfig-token-ttl-minutes`](#kubeconfig-token-ttl-minutes)                 | TTL used for tokens generated via the CLI.  **Deprecated since version 2.6.6, and will be removed in 2.8.0.** This setting will be removed, and `kubeconfig-default-token-TTL-minutes` will be used for all kubeconfig tokens. |
+| [`auth-token-max-ttl-minutes`](#auth-token-max-ttl-minutes)                     | Max TTL for all tokens except those controlled by [`auth-user-session-ttl-minutes`](#auth-user-session-ttl-minutes).                                                                                                           |
+| [`kubeconfig-generate-token`](#kubeconfig-generate-token)                       | If true, automatically generate tokens when a user downloads a kubeconfig.                                                                                                                                                     |
+
+#### auth-user-session-ttl-minutes
+Time to live (TTL) duration in minutes used to determine when a user auth session token expires. When expired, the user will be required to log in and obtain a new token. This setting is not affected by [`auth-token-max-ttl-minutes`](#auth-token-max-ttl-minutes). Session tokens are created when a user logs into Rancher.
+
+#### kubeconfig-default-token-TTL-minutes
+Time to live (TTL) duration in minutes used to determine when a kubeconfig token expires. When the token is expired, the API will reject the token. This setting can not be larger than [`auth-token-max-ttl-minutes`](#auth-token-max-ttl-minutes). This setting applies to a token generated in a requested kubeconfig file. Except those [generated by Rancher CLI](#disable-tokens-in-generated-kubeconfigs).
+**Introduced in version 2.6.6**.
+
+#### kubeconfig-token-ttl-minutes
+Time to live (TTL) duration in minutes used to determine when a kubeconfig token that was generated by the CLI expires. Tokens are generated by the CLI when [`kubeconfig-generate-token`](#kubeconfig-generate-token) is false. When the token is expired, the API will reject the token. This setting can not be larger than [`auth-token-max-ttl-minutes`](#auth-token-max-ttl-minutes).
+**Deprecated since version 2.6.6, and will be removed in 2.8.0: This setting will be replaced with the value of [`kubeconfig-default-token-TTL-minutes`](#kubeconfig-default-token-ttl-minutes).**
+
+#### auth-token-max-ttl-minutes
+Maximum Time to Live (TTL) in minutes allowed for auth tokens. If a user attempts to create a token with a TTL greater than `auth-token-max-ttl-minutes`, Rancher will set the token TTL to the value of `auth-token-max-ttl-minutes`. Auth tokens are tokens created for authenticating API requests.
+**Changed in version 2.6.6: Applies to all kubeconfig tokens and api tokens.**
+
+#### kubeconfig-generate-token
+When true, kubeconfigs requested through the UI will contain a valid token. When false, the kubeconfig will contain a command that uses the Rancher CLI to prompt the user to log in. [The CLI then will retrieve and cache a token for the user](../cli-with-rancher/kubectl-utility.md#authentication-with-kubectl-and-kubeconfig-tokens-with-ttl).
@@ -31,6 +31,8 @@ Encrypted backups can only be restored if the Restore custom resource uses the s

 This section contains example Backup custom resources.

+>**Note:** Refer to the [backup config reference page](./backup-configuration.md) for more information on configuring the options below.
+
 ### Backup in the Default Location with Encryption

 ```yaml
@@ -84,7 +86,7 @@ spec:
      credentialSecretNamespace: default
      bucketName: rancherbackups
      endpoint: minio.xip.io
-      endpointCA: LS0tLS1CRUdJTi3VUFNQkl5UUT.....pbEpWaVzNkRS0tLS0t
+      endpointCA: <base64-encoded-cert>
  resourceSetName: rancher-resource-set
  encryptionConfigSecretName: encryptionconfig
 ```
@@ -155,6 +157,8 @@ spec:

 This section contains example Restore custom resources.

+>**Note:** Refer to the [restore config reference page](./restore-configuration.md) for more information on configuring the options below.
+
 ### Restore Using the Default Backup File Location

 ```yaml
@@ -213,7 +217,7 @@ spec:
      credentialSecretNamespace: default
      bucketName: rancherbackups
      endpoint: minio.xip.io
-      endpointCA: LS0tLS1CRUdJTi3VUFNQkl5UUT.....pbEpWaVzNkRS0tLS0t
+      endpointCA: <base64-encoded-cert>
  encryptionConfigSecretName: test-encryptionconfig
 ```

@@ -14,7 +14,7 @@ When you set up your high-availability Rancher installation, consider the follow
 ### Run Rancher on a Separate Cluster
 Don't run other workloads or microservices in the Kubernetes cluster that Rancher is installed on.

-### Make sure nodes are configured correctly for Kubernetes ###
+### Make sure nodes are configured correctly for Kubernetes
 It's important to follow K8s and etcd best practices when deploying your nodes, including disabling swap, double checking you have full network connectivity between all machines in the cluster, using unique hostnames, MAC addresses, and product_uuids for every node, checking that all correct ports are opened, and deploying with ssd backed etcd. More details can be found in the [kubernetes docs](https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/#before-you-begin) and [etcd's performance op guide](https://etcd.io/docs/v3.4/op-guide/performance/).

 ### When using RKE: Back up the Statefile
@@ -22,7 +22,7 @@ Run `kubectl cluster-info` or `kubectl get pods` successfully.

 _Requirements_

-If admins have [enforced TTL on kubeconfig tokens](../about-the-api/api-tokens.md#setting-ttl-on-kubeconfig-tokens), the kubeconfig file requires the [Rancher CLI](./rancher-cli.md) to be present in your PATH when you run `kubectl`. Otherwise, you’ll see an error like:
+If admins have [kubeconfig token generation turned off](../about-the-api/api-tokens.md##disable-tokens-in-generated-kubeconfigs), the kubeconfig file requires the [Rancher CLI](./rancher-cli.md) to be present in your PATH when you run `kubectl`. Otherwise, you’ll see an error like:
 `Unable to connect to the server: getting credentials: exec: exec: "rancher": executable file not found in $PATH`.

 This feature enables kubectl to authenticate with the Rancher server and get a new kubeconfig token when required. The following auth providers are currently supported:
@@ -33,4 +33,4 @@ This feature enables kubectl to authenticate with the Rancher server and get a n
 4. OpenLDAP
 5. SAML providers: Ping, Okta, ADFS, Keycloak, Shibboleth

-When you first run kubectl, for example, `kubectl get pods`, it will ask you to pick an auth provider and log in with the Rancher server. The kubeconfig token is cached in the path where you run kubectl under `./.cache/token`. This token is valid until [it expires](../about-the-api/api-tokens.md#setting-ttl-on-kubeconfig-tokens-period), or [gets deleted from the Rancher server](../about-the-api/api-tokens.md#deleting-tokens). Upon expiration, the next `kubectl get pods` will ask you to log in with the Rancher server again.
+When you first run kubectl, for example, `kubectl get pods`, it will ask you to pick an auth provider and log in with the Rancher server. The kubeconfig token is cached in the path where you run kubectl under `./.cache/token`. This token is valid until [it expires](../about-the-api/api-tokens.md#disable-tokens-in-generated-kubeconfigs), or [gets deleted from the Rancher server](../about-the-api/api-tokens.md#deleting-tokens). Upon expiration, the next `kubectl get pods` will ask you to log in with the Rancher server again.
@@ -1,7 +1,7 @@
 ---
 title: DigitalOcean Node Template Configuration
 weight: 1
----
+---

 Account access information is stored as a cloud credential. Cloud credentials are stored as Kubernetes secrets. Multiple node templates can use the same cloud credential. You can use an existing cloud credential or create a new one.

@@ -11,7 +11,24 @@ This section covers the configuration options that are available in Rancher for
 You can configure the Kubernetes options in one of the two following ways:

 - [Rancher UI](#configuration-options-in-the-rancher-ui): Use the Rancher UI to select options that are commonly customized when setting up a Kubernetes cluster.
- [Cluster Config File](#cluster-config-file): Instead of using the Rancher UI to choose Kubernetes options for the cluster, advanced users can create an RKE2 config file. Using a config file allows you to set any of the [options](https://docs.rke2.io/install/install_options/install_options) available in an RKE2 installation.
+- [Cluster Config File](#cluster-config-file): Instead of using the Rancher UI to choose Kubernetes options for the cluster, advanced users can create an RKE2 config file. Using a config file allows you to set many additional [options](https://docs.rke2.io/install/install_options/install_options) available for an RKE2 installation.
+
+# Editing Clusters with a Form in the Rancher UI
+
+To edit your cluster,
+
+1. In the upper left corner, click **☰ > Cluster Management**.
+1. Go to the cluster you want to configure and click **⋮ > Edit Config**.
+
+# Editing Clusters with YAML
+
+Instead of using the Rancher UI to choose Kubernetes options for the cluster, advanced users can create an RKE2 config file. Using a config file allows you to set any of the options available in an RKE2 installation by specifying them in YAML.
+
+To edit an RKE2 config file directly from the Rancher UI,
+
+1. Click **☰ > Cluster Management**.
+1. Go to the cluster you want to configure and click **⋮ > Edit as YAML**.
+1. Edit the RKE options under the `rkeConfig` directive.

 # Configuration Options in the Rancher UI

@@ -21,6 +38,49 @@ Some advanced configuration options are not exposed in the Rancher UI forms, but

 :::

+## Machine Pool
+
+This subsection covers the generic machine pool configurations. For infrastructure provider specific, configurations refer to the following pages:
+
+- [Azure](../downstream-cluster-configuration/machine-configuration/azure.md)
+- [DigitalOcean](../downstream-cluster-configuration/machine-configuration/digitalocean.md)
+- [EC2](../downstream-cluster-configuration/machine-configuration/amazon-ec2.md)
+- [vSphere](../downstream-cluster-configuration/machine-configuration/vsphere.md)
+
+### Pool Name
+
+The name of the machine pool.
+
+### Machine Count
+
+The number of machines in the pool.
+
+### Roles
+
+Option to assign etcd, control plane, and worker roles to nodes.
+
+### Advanced
+
+#### Auto Replace
+
+The duration nodes can be unreachable before they are automatically deleted and replaced.
+
+#### Drain Before Delete
+
+Enables draining nodes by evicting all pods before the node is deleted.
+
+#### Kubernetes Node Labels
+
+Add [labels](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/) to nodes to help with organization and object selection.
+
+For details on label syntax requirements, see the [Kubernetes documentation.](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#syntax-and-character-set)
+
+#### Taints
+
+Add [taints](https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/) to nodes, which can be used to prevent pods from being scheduled to or executed on nodes, unless the pods have matching tolerations.
+
+## Cluster Configuration
+
 ### Basics
 #### Kubernetes Version

@@ -81,10 +141,6 @@ If your network provider allows project network isolation, you can choose whethe

 Project network isolation is available if you are using any RKE2 network plugin that supports the enforcement of Kubernetes network policies, such as Canal.

-#### SELinux
-
-Option to enable or disable [SELinux](https://docs.rke2.io/security/selinux) support.
-
 #### CoreDNS

 By default, [CoreDNS](https://coredns.io/) is installed as the default DNS provider. If CoreDNS is not installed, an alternate DNS provider must be installed yourself. Refer to the [RKE2 documentation](https://docs.rke2.io/networking/#coredns) for additional CoreDNS configurations.
@@ -193,8 +249,103 @@ Option to remove all pods from the node prior to upgrading.

 Option to set kubelet options for different nodes. For available options, refer to the [Kubernetes documentation](https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/).

-# Cluster Config File
+# Cluster Config File Reference

-Instead of using the Rancher UI forms to choose Kubernetes options for the cluster, advanced users can create an RKE2 config file. Using a config file allows you to set any of the [options](https://docs.rke2.io/install/install_options/install_options) available in an RKE2 installation.
+Instead of using the Rancher UI to choose Kubernetes options for the cluster, advanced users can create a config file. Using a config file allows you to set the [options available](https://docs.rke2.io/install/install_options/server_config/) in an RKE2 installation, including those already listed in [Configuration Options in the Rancher UI](#configuration-options-in-the-rancher-ui), as well as Rancher-specific parameters.

-To edit an RKE2 config file directly from the Rancher UI, click **Edit as YAML**.
+{{% accordion id="rke2-cluster-config-file" label="Example Cluster Config File Snippet" %}}
+
+```yaml
+spec:
+  cloudCredentialSecretName: cattle-global-data:cc-s879v
+  kubernetesVersion: v1.23.6+rke2r2
+  localClusterAuthEndpoint: {}
+  rkeConfig:
+    chartValues:
+      rke2-calico: {}
+    etcd:
+      snapshotRetention: 5
+      snapshotScheduleCron: 0 */5 * * *
+    machineGlobalConfig:
+      cni: calico
+      disable-kube-proxy: false
+      etcd-expose-metrics: false
+      profile: null
+    machinePools:
+    - controlPlaneRole: true
+      etcdRole: true
+      machineConfigRef:
+        kind: Amazonec2Config
+        name: nc-test-pool1-pwl5h
+      name: pool1
+      quantity: 1
+      unhealthyNodeTimeout: 0s
+      workerRole: true
+    machineSelectorConfig:
+    - config:
+        protect-kernel-defaults: false
+    registries: {}
+    upgradeStrategy:
+      controlPlaneConcurrency: "1"
+      controlPlaneDrainOptions:
+        deleteEmptyDirData: true
+        enabled: true
+        gracePeriod: -1
+        ignoreDaemonSets: true
+        timeout: 120
+      workerConcurrency: "1"
+      workerDrainOptions:
+        deleteEmptyDirData: true
+        enabled: true
+        gracePeriod: -1
+        ignoreDaemonSets: true
+        timeout: 120
+```
+
+### chartValues
+
+Option to specify the values for the system charts installed by RKE2/k3s.
+
+Example:
+
+```yaml
+chartValues:
+    chart-name:
+        key: value
+```
+### machineGlobalConfig
+
+The RKE2/K3s configurations are nested under the `machineGlobalConfig` directive. Any configuration change made here will apply to every node. The configuration options available in the [standalone version of RKE2](https://docs.rke2.io/install/install_options/server_config/) can be applied here.
+
+Example:
+
+```yaml
+machineGlobalConfig:
+    etcd-arg:
+        - key1=value1
+        - key2=value2
+```
+
+### machineSelectorConfig
+
+This is the same as [`machineGlobalConfig`](#machineglobalconfig) except that a [label](#kubernetes-node-labels) selector can be specified with the configuration. The configuration will only be applied to nodes that match the provided label selector.
+
+Multiple `config` entries are allowed, each specifying their own `machineLabelSelector`. A user can specify `matchExpressions`, `matchLabels`, both, or neither. Omitting the `machineLabelSelector` section of this has the same effect as putting the config in the `machineGlobalConfig` section.
+
+Example:
+
+```yaml
+machineSelectorConfig
+  - config:
+      config-key: config-value
+    machineLabelSelector:
+      matchExpressions:
+        - key: example-key
+          operator: string # Valid operators are In, NotIn, Exists and DoesNotExist.
+          values:
+            - example-value1
+            - example-value2
+      matchLabels:
+        key1: value1
+        key2: value2
+```
@@ -45,7 +45,7 @@ To create notification receivers in the Rancher UI,
 <TabItem value="Rancher v2.6.5+">

 1. Go to the cluster where you want to create receivers. Click **Monitoring -> Alerting -> AlertManagerConfigs**.
-1. Ciick **Create**.
+1. Click **Create**.
 1. Click **Add Receiver**.
 1. Enter a **Name** for the receiver.
 1. Configure one or more providers for the receiver. For help filling out the forms, refer to the configuration options below.
@@ -316,4 +316,4 @@ For more information on enabling alerting for `rancher-cis-benchmark`, see [this

 # Trusted CA for Notifiers

-If you need to add a trusted CA to your notifier, follow the steps in [this section.](helm-chart-options.md#trusted-ca-for-notifiers)
+If you need to add a trusted CA to your notifier, follow the steps in [this section.](helm-chart-options.md#trusted-ca-for-notifiers)
@@ -9,6 +9,9 @@ Rancher is committed to informing the community of security issues in our produc

 | ID | Description | Date | Resolution |
 |----|-------------|------|------------|
+| [CVE-2022-31247](https://github.com/rancher/rancher/security/advisories/GHSA-6x34-89p7-95wg) | An issue was discovered in Rancher versions up to and including 2.5.15 and 2.6.6 where a flaw with authorization logic allows privilege escalation in downstream clusters through cluster role template binding (CRTB) and project role template binding (PRTB). The vulnerability can be exploited by any user who has permissions to create/edit CRTB or PRTB (such as `cluster-owner`, `manage cluster members`, `project-owner`, and `manage project members`) to gain owner permission in another project in the same cluster or in another project on a different downstream cluster. | 18 August 2022 | [Rancher v2.6.7](https://github.com/rancher/rancher/releases/tag/v2.6.7) and [Rancher v2.5.16](https://github.com/rancher/rancher/releases/tag/v2.5.16) |
+| [CVE-2021-36783](https://github.com/rancher/rancher/security/advisories/GHSA-8w87-58w6-hfv8) | It was discovered that in Rancher versions up to and including 2.5.12 and 2.6.3, there is a failure to properly sanitize credentials in cluster template answers. This failure can lead to plaintext storage and exposure of credentials, passwords, and API tokens. The exposed credentials are visible in Rancher to authenticated `Cluster Owners`, `Cluster Members`, `Project Owners`, and `Project Members` on the endpoints `/v1/management.cattle.io.clusters`, `/v3/clusters`, and `/k8s/clusters/local/apis/management.cattle.io/v3/clusters`. | 18 August 2022 | [Rancher v2.6.7](https://github.com/rancher/rancher/releases/tag/v2.6.7) and [Rancher v2.5.16](https://github.com/rancher/rancher/releases/tag/v2.5.16) |
+| [CVE-2021-36782](https://github.com/rancher/rancher/security/advisories/GHSA-g7j7-h4q8-8w2f) | An issue was discovered in Rancher versions up to and including 2.5.15 and 2.6.6 where sensitive fields like passwords, API keys, and Rancher's service account token (used to provision clusters) were stored in plaintext directly on Kubernetes objects like `Clusters` (e.g., `cluster.management.cattle.io`). Anyone with read access to those objects in the Kubernetes API could retrieve the plaintext version of those sensitive data. The issue was partially found and reported by Florian Struck (from [Continum AG](https://www.continum.net/)) and [Marco Stuurman](https://github.com/fe-ax) (from [Shock Media B.V.](https://www.shockmedia.nl/)). | 18 August 2022 | [Rancher v2.6.7](https://github.com/rancher/rancher/releases/tag/v2.6.7) and [Rancher v2.5.16](https://github.com/rancher/rancher/releases/tag/v2.5.16) |
 | [CVE-2022-21951](https://github.com/rancher/rancher/security/advisories/GHSA-vrph-m5jj-c46c) | This vulnerability only affects customers using [Weave](https://rancher.com/docs/rancher/v2.6/en/faq/networking/cni-providers/#weave) Container Network Interface (CNI) when configured through [RKE templates](https://rancher.com/docs/rancher/v2.6/en/admin-settings/rke-templates/). A vulnerability was discovered in Rancher versions 2.5.0 up to and including 2.5.13, and 2.6.0 up to and including 2.6.4, where a user interface (UI) issue with RKE templates does not include a value for the Weave password when Weave is chosen as the CNI. If a cluster is created based on the mentioned template, and Weave is configured as the CNI, no password will be created for [network encryption](https://www.weave.works/docs/net/latest/tasks/manage/security-untrusted-networks/) in Weave; therefore, network traffic in the cluster will be sent unencrypted. | 24 May 2022 | [Rancher v2.6.5](https://github.com/rancher/rancher/releases/tag/v2.6.5) and [Rancher v2.5.14](https://github.com/rancher/rancher/releases/tag/v2.5.14) |
 | [CVE-2021-36784](https://github.com/rancher/rancher/security/advisories/GHSA-jwvr-vv7p-gpwq) | A vulnerability was discovered in Rancher versions from 2.5.0 up to and including 2.5.12 and from 2.6.0 up to and including 2.6.3 which allows users who have create or update permissions on [Global Roles](https://rancher.com/docs/rancher/v2.6/en/admin-settings/rbac/) to escalate their permissions, or those of another user, to admin-level permissions. Global Roles grant users Rancher-wide permissions, such as the ability to create clusters. In the identified versions of Rancher, when users are given permission to edit or create Global Roles, they are not restricted to only granting permissions which they already posses. This vulnerability affects customers who utilize non-admin users that are able to create or edit Global Roles. The most common use case for this scenario is the `restricted-admin` role. | 14 Apr 2022 | [Rancher v2.6.4](https://github.com/rancher/rancher/releases/tag/v2.6.4) and [Rancher v2.5.13](https://github.com/rancher/rancher/releases/tag/v2.5.13) |
 | [CVE-2021-4200](https://github.com/rancher/rancher/security/advisories/GHSA-hx8w-ghh8-r4xf) | This vulnerability only affects customers using the `restricted-admin` role in Rancher. A vulnerability was discovered in Rancher versions from 2.5.0 up to and including 2.5.12 and from 2.6.0 up to and including 2.6.3 where the `global-data` role in `cattle-global-data` namespace grants write access to the Catalogs. Since each user with any level of catalog access was bound to the `global-data` role, this grants write access to templates (`CatalogTemplates`) and template versions (`CatalogTemplateVersions`) for any user with any level of catalog access. New users created in Rancher are by default assigned to the `user` role (standard user), which is not designed to grant write catalog access. This vulnerability effectively elevates the privilege of any user to write access for the catalog template and catalog template version resources. | 14 Apr 2022 | [Rancher v2.6.4](https://github.com/rancher/rancher/releases/tag/v2.6.4) and [Rancher v2.5.13](https://github.com/rancher/rancher/releases/tag/v2.5.13) |
@@ -3,9 +3,9 @@ title: Rotation of Expired Webhook Certificates
 weight: 120
 ---

-For Rancher versions that have `rancher-webhook` installed, these certificates will expire after one year. It will be necessary for you to rotate your webhook certificate when this occurs. 
+For Rancher versions that have `rancher-webhook` installed, certain versions created certificates that will expire after one year. It will be necessary for you to rotate your webhook certificate if the certificate did not renew.

-Rancher will advise the community once there is a permanent solution in place for this known issue. Currently, there are two methods to work around this issue:
+In Rancher v2.6.3 and up, rancher-webhook deployments will automatically renew their TLS certificate when it is within 30 or fewer days of its expiration date. If you are using v2.6.2 or below, there are two methods to work around this issue:

 ##### 1. Users with cluster access, run the following commands:
 ```
@@ -126,10 +126,18 @@ kubectl -n ingress-nginx describe pods -l app=ingress-nginx

 ### Pod container logs

+The below command can show the logs of all the pods labeled "app=ingress-nginx", but it will display only 10 lines of log because of the restrictions of the `kubectl logs` command. Refer to `--tail` of `kubectl logs -h` for more information.
+
 ```
 kubectl -n ingress-nginx logs -l app=ingress-nginx
 ```

+If the full log is needed, specify the pod name in the trailing command:
+
+```
+kubectl -n ingress-nginx logs <pod name>
+```
+
 ### Namespace events

 ```
@@ -18,9 +18,9 @@ const sidebars = {

  // But you can create a sidebar manually
  tutorialSidebar: [
-   
-      'rancher-manager',             
-    {    
+
+      'rancher-manager',
+    {
      type: 'category',
      label: 'Getting Started',
      link: {
@@ -38,7 +38,7 @@ const sidebars = {
          items: [
            "getting-started/introduction/overview",
            "getting-started/introduction/what-are-divio-docs",
-                 ], 
+                 ],
        },
        {
          type: 'category',
@@ -74,7 +74,7 @@ const sidebars = {
                type: 'doc',
                id: "pages-for-subheaders/deploy-rancher-workloads",
              },
-              items: [      
+              items: [
                    "getting-started/quick-start-guides/deploy-workloads/workload-ingress",
                    "getting-started/quick-start-guides/deploy-workloads/nodeports",
                     ],
@@ -109,7 +109,7 @@ const sidebars = {
                type: 'doc',
                id: "pages-for-subheaders/install-upgrade-on-a-kubernetes-cluster",
                    },
-              items: [      
+              items: [
                    "getting-started/installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster/rollbacks",
                    "getting-started/installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster/upgrades",
                    "getting-started/installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster/air-gapped-upgrades",
@@ -126,7 +126,7 @@ const sidebars = {
                type: 'doc',
                id: "pages-for-subheaders/other-installation-methods",
              },
-              items: [      
+              items: [
                  {
                    type: 'category',
                    label: 'Air-Gapped Helm CLI Install',
@@ -139,7 +139,7 @@ const sidebars = {
                          "getting-started/installation-and-upgrade/other-installation-methods/air-gapped-helm-cli-install/publish-images",
                          "getting-started/installation-and-upgrade/other-installation-methods/air-gapped-helm-cli-install/install-kubernetes",
                          "getting-started/installation-and-upgrade/other-installation-methods/air-gapped-helm-cli-install/install-rancher-ha",
-                          "getting-started/installation-and-upgrade/other-installation-methods/air-gapped-helm-cli-install/docker-install-commands",    
+                          "getting-started/installation-and-upgrade/other-installation-methods/air-gapped-helm-cli-install/docker-install-commands",
                           ]
                  },
                  {
@@ -155,14 +155,14 @@ const sidebars = {
                          "getting-started/installation-and-upgrade/other-installation-methods/rancher-on-a-single-node-with-docker/certificate-troubleshooting",
                           ]
                  },
-                  { 
+                  {
                    type: 'category',
                    label: 'Rancher Behind an HTTP Proxy',
                    link: {
                      type: 'doc',
                      id: "pages-for-subheaders/rancher-behind-an-http-proxy",
                    },
-                    items: [ 
+                    items: [
                          "getting-started/installation-and-upgrade/other-installation-methods/rancher-behind-an-http-proxy/set-up-infrastructure",
                          "getting-started/installation-and-upgrade/other-installation-methods/rancher-behind-an-http-proxy/install-kubernetes",
                          "getting-started/installation-and-upgrade/other-installation-methods/rancher-behind-an-http-proxy/install-rancher",
@@ -187,11 +187,11 @@ const sidebars = {
                    "getting-started/installation-and-upgrade/resources/bootstrap-password",
                    "getting-started/installation-and-upgrade/resources/local-system-charts",
                     ]
-            }, 
+            },
                    "getting-started/installation-and-upgrade/upgrade-and-roll-back-kubernetes",
-                    
+
                    "getting-started/installation-and-upgrade/upgrade-kubernetes-without-upgrading-rancher",
-            { 
+            {
              type: 'category',
              label: 'Advanced Options',
              link: {
@@ -224,11 +224,11 @@ const sidebars = {
                           ]
                }
                    ]
-                  } 
+                  }
                ]
              }
            ]
-          },          
+          },
    {
      type: 'category',
      label: 'How-to Guides',
@@ -236,7 +236,7 @@ const sidebars = {
        type: 'doc',
        id: "how-to-guides",
      },
-      items: [ 
+      items: [
        {
          type: 'category',
          label: 'New User Guides',
@@ -245,7 +245,7 @@ const sidebars = {
            id: "pages-for-subheaders/new-user-guides",
                },
          items: [
-             {      
+             {
                type: 'category',
                label: 'Kubernetes Cluster Setup',
                link: {
@@ -297,7 +297,7 @@ const sidebars = {
                        "how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/checklist-for-production-ready-clusters/roles-for-nodes-in-kubernetes",
                              ],
                  },
-                  {      
+                  {
                    type: 'category',
                    label: 'Set Up Clusters from Hosted Kubernetes Providers',
                    link: {
@@ -312,7 +312,7 @@ const sidebars = {
                          "how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/set-up-clusters-from-hosted-kubernetes-providers/huawei",
                            ]
                  },
-                  { 
+                  {
                    type: 'category',
                    label: 'Launch Kubernetes with Rancher',
                    link: {
@@ -328,10 +328,10 @@ const sidebars = {
                            id: "pages-for-subheaders/use-new-nodes-in-an-infra-provider",
                                },
                            items: [
-                                    "how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/create-an-amazon-ec2-cluster",  
-                                    
+                                    "how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/create-an-amazon-ec2-cluster",
+
                                    "how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/create-a-digitalocean-cluster",
-                                
+
                                    "how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/create-an-azure-cluster",
                                   {
                                    type: 'category',
@@ -343,7 +343,7 @@ const sidebars = {
                                    items: [
                                      "how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/provision-kubernetes-clusters-in-vsphere",
                                      "how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/create-credentials",
-                                      "how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/create-a-vm-template",     
+                                      "how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/create-a-vm-template",
                                            ]
                                },
                                {
@@ -354,10 +354,10 @@ const sidebars = {
                                      id: "pages-for-subheaders/nutanix",
                                          },
                                    items: [
-                                      "how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/nutanix/provision-kubernetes-clusters-in-aos",          
+                                      "how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/nutanix/provision-kubernetes-clusters-in-aos",
                                          ]
                                        }
-                                      ]   
+                                      ]
                      },
                      {
                          type: 'category',
@@ -370,6 +370,7 @@ const sidebars = {
                                "how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/launch-kubernetes-with-rancher/use-windows-clusters/azure-storageclass-configuration",
                                "how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/launch-kubernetes-with-rancher/use-windows-clusters/windows-linux-cluster-feature-parity",
                                "how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/launch-kubernetes-with-rancher/use-windows-clusters/network-requirements-for-host-gateway",
+                                "how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/launch-kubernetes-with-rancher/use-windows-clusters/workload-migration-guidance"
                                    ]
                      },
                      {
@@ -407,13 +408,13 @@ const sidebars = {
                                          ]
                                  }
                                     ]
-                      },                   
+                      },
                          "how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/launch-kubernetes-with-rancher/rke1-vs-rke2-differences",
-                      
+
                          "how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/launch-kubernetes-with-rancher/about-rancher-agents",
                     ]
                  },
-                  "how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/register-existing-clusters", 
+                  "how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/register-existing-clusters",
                ]
              },
              {
@@ -435,7 +436,7 @@ const sidebars = {
                          "how-to-guides/new-user-guides/kubernetes-resources-setup/workloads-and-pods/deploy-workloads",
                          "how-to-guides/new-user-guides/kubernetes-resources-setup/workloads-and-pods/roll-back-workloads",
                          "how-to-guides/new-user-guides/kubernetes-resources-setup/workloads-and-pods/upgrade-workloads",
-                          "how-to-guides/new-user-guides/kubernetes-resources-setup/workloads-and-pods/add-a-sidecar",                            
+                          "how-to-guides/new-user-guides/kubernetes-resources-setup/workloads-and-pods/add-a-sidecar",
                          ]
                  },
                  {
@@ -449,7 +450,7 @@ const sidebars = {
                          "how-to-guides/new-user-guides/kubernetes-resources-setup/horizontal-pod-autoscaler/about-hpas",
                          "how-to-guides/new-user-guides/kubernetes-resources-setup/horizontal-pod-autoscaler/manage-hpas-with-ui",
                          "how-to-guides/new-user-guides/kubernetes-resources-setup/horizontal-pod-autoscaler/manage-hpas-with-kubectl",
-                          "how-to-guides/new-user-guides/kubernetes-resources-setup/horizontal-pod-autoscaler/test-hpas-with-kubectl",                            
+                          "how-to-guides/new-user-guides/kubernetes-resources-setup/horizontal-pod-autoscaler/test-hpas-with-kubectl",
                          ]
                          },
                  {
@@ -462,18 +463,18 @@ const sidebars = {
                    items: [
                          "how-to-guides/new-user-guides/kubernetes-resources-setup/load-balancer-and-ingress-controller/layer-4-and-layer-7-load-balancing",
                          "how-to-guides/new-user-guides/kubernetes-resources-setup/load-balancer-and-ingress-controller/add-ingresses",
-                          "how-to-guides/new-user-guides/kubernetes-resources-setup/load-balancer-and-ingress-controller/ingress-configuration",                           
+                          "how-to-guides/new-user-guides/kubernetes-resources-setup/load-balancer-and-ingress-controller/ingress-configuration",
                          ]
-                  },                  
-                     "how-to-guides/new-user-guides/kubernetes-resources-setup/create-services", 
-           
-                     "how-to-guides/new-user-guides/kubernetes-resources-setup/encrypt-http-communication", 
-            
-                     "how-to-guides/new-user-guides/kubernetes-resources-setup/configmaps", 
-             
-                     "how-to-guides/new-user-guides/kubernetes-resources-setup/secrets", 
-             
-                     "how-to-guides/new-user-guides/kubernetes-resources-setup/kubernetes-and-docker-registries", 
+                  },
+                     "how-to-guides/new-user-guides/kubernetes-resources-setup/create-services",
+
+                     "how-to-guides/new-user-guides/kubernetes-resources-setup/encrypt-http-communication",
+
+                     "how-to-guides/new-user-guides/kubernetes-resources-setup/configmaps",
+
+                     "how-to-guides/new-user-guides/kubernetes-resources-setup/secrets",
+
+                     "how-to-guides/new-user-guides/kubernetes-resources-setup/kubernetes-and-docker-registries",
                     ],
              },
              {
@@ -484,7 +485,7 @@ const sidebars = {
                  id: "pages-for-subheaders/helm-charts-in-rancher",
                },
                items: [
-                      "how-to-guides/new-user-guides/helm-charts-in-rancher/create-apps",                            
+                      "how-to-guides/new-user-guides/helm-charts-in-rancher/create-apps",
                          ]
              },
              {
@@ -496,7 +497,7 @@ const sidebars = {
                },
                items: [
                      "how-to-guides/new-user-guides/deploy-apps-across-clusters/fleet",
-                      "how-to-guides/new-user-guides/deploy-apps-across-clusters/multi-cluster-apps",                            
+                      "how-to-guides/new-user-guides/deploy-apps-across-clusters/multi-cluster-apps",
                          ]
              },
              {
@@ -513,11 +514,11 @@ const sidebars = {
                  "how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/back-up-docker-installed-rancher",
                  "how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/restore-docker-installed-rancher",
                  "how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/back-up-rancher-launched-kubernetes-clusters",
-                  "how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/restore-rancher-launched-kubernetes-clusters-from-backup",                            
+                  "how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/restore-rancher-launched-kubernetes-clusters-from-backup",
                ]
              }
            ]
-        }, 
+        },
        {
            type: 'category',
            label: 'Advanced User Guides',
@@ -587,7 +588,7 @@ const sidebars = {
                                    ]
                            }
                          ]
-          
+
                      },
                      {
                      type: 'category',
@@ -634,11 +635,11 @@ const sidebars = {
                              ]
                      },
                        "how-to-guides/advanced-user-guides/authentication-permissions-and-global-configuration/manage-cluster-templates",
-                  
+
                        "how-to-guides/advanced-user-guides/authentication-permissions-and-global-configuration/create-pod-security-policies",
-                  
+
                        "how-to-guides/advanced-user-guides/authentication-permissions-and-global-configuration/global-default-private-registry",
-                    
+
                        "how-to-guides/advanced-user-guides/authentication-permissions-and-global-configuration/custom-branding",
                  ]
              },
@@ -695,7 +696,7 @@ const sidebars = {
                          "how-to-guides/advanced-user-guides/manage-clusters/create-kubernetes-persistent-storage/manage-persistent-storage/dynamically-provision-new-storage",
                          "how-to-guides/advanced-user-guides/manage-clusters/create-kubernetes-persistent-storage/manage-persistent-storage/use-external-ceph-driver",
                          "how-to-guides/advanced-user-guides/manage-clusters/create-kubernetes-persistent-storage/manage-persistent-storage/about-glusterfs-volumes",
-                          "how-to-guides/advanced-user-guides/manage-clusters/create-kubernetes-persistent-storage/manage-persistent-storage/install-iscsi-volumes", 
+                          "how-to-guides/advanced-user-guides/manage-clusters/create-kubernetes-persistent-storage/manage-persistent-storage/install-iscsi-volumes",
                                  ]
                          },
                          {
@@ -714,18 +715,19 @@ const sidebars = {
                        ]
                    },
                    "how-to-guides/advanced-user-guides/manage-clusters/projects-and-namespaces",
-                
+
                    "how-to-guides/advanced-user-guides/manage-clusters/clone-cluster-configuration",
-                
+
                    "how-to-guides/advanced-user-guides/manage-clusters/rotate-certificates",
-                
+                    "how-to-guides/advanced-user-guides/manage-clusters/rotate-encryption-key",
+
                    "how-to-guides/advanced-user-guides/manage-clusters/nodes-and-node-pools",
-                
-                    "how-to-guides/advanced-user-guides/manage-clusters/clean-cluster-nodes", 
-                  
-                    "how-to-guides/advanced-user-guides/manage-clusters/add-a-pod-security-policy", 
-                    
-                    "how-to-guides/advanced-user-guides/manage-clusters/assign-pod-security-policies", 
+
+                    "how-to-guides/advanced-user-guides/manage-clusters/clean-cluster-nodes",
+
+                    "how-to-guides/advanced-user-guides/manage-clusters/add-a-pod-security-policy",
+
+                    "how-to-guides/advanced-user-guides/manage-clusters/assign-pod-security-policies",
                ]
              },
              {
@@ -765,12 +767,12 @@ const sidebars = {
                      },
                items: [
                    "how-to-guides/advanced-user-guides/monitoring-alerting-guides/enable-monitoring",
-                    "how-to-guides/advanced-user-guides/monitoring-alerting-guides/uninstall-monitoring", 
+                    "how-to-guides/advanced-user-guides/monitoring-alerting-guides/uninstall-monitoring",
                    "how-to-guides/advanced-user-guides/monitoring-alerting-guides/set-up-monitoring-for-workloads",
                    "how-to-guides/advanced-user-guides/monitoring-alerting-guides/customize-grafana-dashboard",
                    "how-to-guides/advanced-user-guides/monitoring-alerting-guides/create-persistent-grafana-dashboard",
                    "how-to-guides/advanced-user-guides/monitoring-alerting-guides/debug-high-memory-usage",
-                    "how-to-guides/advanced-user-guides/monitoring-alerting-guides/migrate-to-rancher-v2.5+-monitoring", 
+                    "how-to-guides/advanced-user-guides/monitoring-alerting-guides/migrate-to-rancher-v2.5+-monitoring",
                    {
                      type: 'category',
                      label: 'Prometheus Federator Guides',
@@ -783,6 +785,7 @@ const sidebars = {
                            "how-to-guides/advanced-user-guides/monitoring-alerting-guides/prometheus-federator-guides/uninstall-prometheus-federator",
                            "how-to-guides/advanced-user-guides/monitoring-alerting-guides/prometheus-federator-guides/customize-grafana-dashboards",
                            "how-to-guides/advanced-user-guides/monitoring-alerting-guides/prometheus-federator-guides/set-up-workloads",
+                            "how-to-guides/advanced-user-guides/monitoring-alerting-guides/prometheus-federator-guides/project-monitors"
                             ]
                            }
                          ]
@@ -807,7 +810,7 @@ const sidebars = {
                      "how-to-guides/advanced-user-guides/monitoring-v2-configuration-guides/advanced-configuration/prometheus",
                      "how-to-guides/advanced-user-guides/monitoring-v2-configuration-guides/advanced-configuration/prometheusrules",
                     ]
-                    } 
+                    }
                  ]
              },
              {
@@ -848,7 +851,7 @@ const sidebars = {
              ]
            }
          ]
-    },      
+    },
    {
      type: 'category',
      label: 'Reference Guides',
@@ -857,7 +860,7 @@ const sidebars = {
        id: "reference-guides",
      },
      items: [
-        {      
+        {
          type: 'category',
          label: 'Best Practices',
          link: {
@@ -875,7 +878,7 @@ const sidebars = {
              items: [
                    "reference-guides/best-practices/rancher-server/on-premises-rancher-in-vsphere",
                    "reference-guides/best-practices/rancher-server/rancher-deployment-strategy",
-                    "reference-guides/best-practices/rancher-server/tips-for-running-rancher"                            
+                    "reference-guides/best-practices/rancher-server/tips-for-running-rancher"
                    ]
            },
            {
@@ -893,7 +896,7 @@ const sidebars = {
                    ]
                  }
                ]
-           },     
+           },
           {
              type: 'category',
              label: 'Rancher Manager Architecture',
@@ -968,10 +971,10 @@ const sidebars = {
                    type: 'doc',
                    id: "pages-for-subheaders/node-template-configuration",
                  },
-                  items: [ 
+                  items: [
                    "reference-guides/cluster-configuration/downstream-cluster-configuration/node-template-configuration/amazon-ec2",
                    "reference-guides/cluster-configuration/downstream-cluster-configuration/node-template-configuration/digitalocean",
-                    "reference-guides/cluster-configuration/downstream-cluster-configuration/node-template-configuration/azure", 
+                    "reference-guides/cluster-configuration/downstream-cluster-configuration/node-template-configuration/azure",
                    "reference-guides/cluster-configuration/downstream-cluster-configuration/node-template-configuration/vsphere",
                    "reference-guides/cluster-configuration/downstream-cluster-configuration/node-template-configuration/nutanix",
                          ]
@@ -983,10 +986,10 @@ const sidebars = {
                    type: 'doc',
                    id: "pages-for-subheaders/machine-configuration",
                  },
-                  items: [ 
+                  items: [
                    "reference-guides/cluster-configuration/downstream-cluster-configuration/machine-configuration/amazon-ec2",
                    "reference-guides/cluster-configuration/downstream-cluster-configuration/machine-configuration/digitalocean",
-                    "reference-guides/cluster-configuration/downstream-cluster-configuration/machine-configuration/azure", 
+                    "reference-guides/cluster-configuration/downstream-cluster-configuration/machine-configuration/azure",
                         ]
                        }
                      ]
@@ -1024,11 +1027,11 @@ const sidebars = {
            link: {
              type: 'doc',
              id: "pages-for-subheaders/amazon-eks-permissions",
-                      }, 
+                      },
              items: [
                  "reference-guides/amazon-eks-permissions/minimum-eks-permissions",
                      ]
-          },               
+          },
          {
            type: 'category',
            label: 'Backup & Restore Configuration',
@@ -1040,7 +1043,7 @@ const sidebars = {
                  "reference-guides/backup-restore-configuration/backup-configuration",
                  "reference-guides/backup-restore-configuration/restore-configuration",
                  "reference-guides/backup-restore-configuration/storage-configuration",
-                  "reference-guides/backup-restore-configuration/examples", 
+                  "reference-guides/backup-restore-configuration/examples",
                  ],
          },
          {
@@ -1053,7 +1056,7 @@ const sidebars = {
            items: [
                  "reference-guides/configure-openldap/openldap-config-reference",
                      ]
-          },               
+          },
              "reference-guides/kubernetes-concepts",
          {
            type: 'category',
@@ -1119,12 +1122,12 @@ const sidebars = {
                  ]
          },
              "reference-guides/rancher-cluster-tools",
-                  
-              "reference-guides/rancher-project-tools", 
-            
-              "reference-guides/system-tools", 

-              "reference-guides/rke1-template-example-yaml",     
+              "reference-guides/rancher-project-tools",
+
+              "reference-guides/system-tools",
+
+              "reference-guides/rke1-template-example-yaml",
          {
            type: 'category',
            label: 'Pipelines',
@@ -1159,10 +1162,10 @@ const sidebars = {
                "reference-guides/rancher-security/rancher-v2.6-hardening-guides/rke1-hardening-guide-with-cis-v1.6-benchmark",
                "reference-guides/rancher-security/rancher-v2.6-hardening-guides/rke1-self-assessment-guide-with-cis-v1.6-benchmark",
                "reference-guides/rancher-security/rancher-v2.6-hardening-guides/rke2-hardening-guide-with-cis-v1.6-benchmark",
-                "reference-guides/rancher-security/rancher-v2.6-hardening-guides/rke2-self-assessment-guide-with-cis-v1.6-benchmark",   
+                "reference-guides/rancher-security/rancher-v2.6-hardening-guides/rke2-self-assessment-guide-with-cis-v1.6-benchmark",
              ],
            },
-            {  
+            {
              type: 'category',
              label: 'SELinux RPM',
              link: {
@@ -1175,7 +1178,7 @@ const sidebars = {
                     ],
            },
                "reference-guides/rancher-security/kubernetes-security-best-practices",
-            
+
                "reference-guides/rancher-security/security-advisories-and-cves",
                  ],
                }
@@ -1196,7 +1199,32 @@ const sidebars = {
          type: 'doc',
          id: "pages-for-subheaders/integrations-in-rancher",
              },
-          items: [ 
+          items: [
+            {
+              type: 'category',
+              label: 'Cloud Marketplace Integration',
+              link: {
+                type: 'doc',
+                id: "pages-for-subheaders/cloud-marketplace"
+              },
+              items: [
+                {
+                  type: 'category',
+                  label: 'AWS Marketplace Integration',
+                  link: {
+                    type: 'doc',
+                    id: "pages-for-subheaders/aws-cloud-marketplace"
+                  },
+                  items: [
+                    'explanations/integrations-in-rancher/cloud-marketplace/aws-cloud-marketplace/install-adapter',
+                    'explanations/integrations-in-rancher/cloud-marketplace/aws-cloud-marketplace/uninstall-adapter',
+                    'explanations/integrations-in-rancher/cloud-marketplace/aws-cloud-marketplace/common-issues',
+                    'explanations/integrations-in-rancher/cloud-marketplace/aws-cloud-marketplace/adapter-requirements'
+                  ]
+                },
+                'explanations/integrations-in-rancher/cloud-marketplace/supportconfig'
+              ]
+            },
            {
            type: 'category',
            label: 'CIS Scans',
@@ -1214,7 +1242,7 @@ const sidebars = {
            {
            type: 'category',
            label: 'Fleet - GitOps at Scale',
-            link: { 
+            link: {
              type: 'doc',
              id: "pages-for-subheaders/fleet-gitops-at-scale",
                  },
@@ -1250,7 +1278,7 @@ const sidebars = {
                          "explanations/integrations-in-rancher/istio/configuration-options/project-network-isolation",
                           ]
                   }
-                  ]                          
+                  ]
            },
              "explanations/integrations-in-rancher/longhorn",
            {
@@ -1296,12 +1324,12 @@ const sidebars = {
                  ]
            },
              "explanations/integrations-in-rancher/neuvector",
-          
-              "explanations/integrations-in-rancher/opa-gatekeeper", 
+
+              "explanations/integrations-in-rancher/opa-gatekeeper",
                  ]
                }
              ]
-    }, 
+    },
    {
      type: 'category',
      label: 'FAQ',
@@ -1327,7 +1355,7 @@ const sidebars = {
        type: 'doc',
        id: "troubleshooting",
      },
-      items: [ 
+      items: [
        {
          type: 'category',
          label: 'Kubernetes Components',
@@ -24,7 +24,7 @@ services:

 You can follow the recommendations from [the etcd docs](https://etcd.io/docs/v3.4.0/tuning/#disk) on how to tune the disk priority on the host.

-Additionally, to reduce IO contention on the disks for etcd, you can use a dedicated device for the data and wal directory. Based on etcd best practices, mirroring RAID configurations are unnecessary because etcd replicates data between the nodes in the cluster. You can use stripping RAID configurations to increase available IOPS.
+Additionally, to reduce IO contention on the disks for etcd, you can use a dedicated device for the data and wal directory. Based on etcd best practices, mirroring RAID configurations are unnecessary because etcd replicates data between the nodes in the cluster. You can use striping RAID configurations to increase available IOPS.

 To implement this solution in an RKE cluster, the `/var/lib/etcd/data` and `/var/lib/etcd/wal` directories will need to have disks mounted and formatted on the underlying host. In the `extra_args` directive of the `etcd` service, you must include the `wal_dir` directory. Without specifying the `wal_dir`, etcd process will try to manipulate the underlying `wal` mount with insufficient permissions.

@@ -80,7 +80,7 @@ sudo mv ./kubectl /usr/local/bin/kubectl
 * [helm](https://helm.sh/docs/intro/install/)

 ```
-curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3
+curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3
 chmod +x get_helm.sh
 sudo ./get_helm.sh
 ```
@@ -28,7 +28,7 @@ The following steps quickly deploy a Rancher Server with a single node cluster a
 3. **Optional:** Edit `config.yaml` to:

    - Change the number of nodes and the memory allocations, if required. (`node.count`, `node.cpus`, `node.memory`)
-    - Change the password of the `admin` user for logging into Rancher. (`default_password`)
+    - Change the password of the `admin` user for logging into Rancher. (`admin_password`)

 4. To initiate the creation of the environment run, `vagrant up --provider=virtualbox`.

@@ -13,8 +13,6 @@ Rancher was originally built to work with multiple orchestrators, and it include

 Rancher can provision Kubernetes from a hosted provider, provision compute nodes and then install Kubernetes onto them, or import existing Kubernetes clusters running anywhere.

-One Rancher server installation can manage thousands of Kubernetes clusters and thousands of nodes from the same user interface.
-
 Rancher adds significant value on top of Kubernetes, first by centralizing authentication and role-based access control (RBAC) for all of the clusters, giving global admins the ability to control cluster access from one location.

 It then enables detailed monitoring and alerting for clusters and their resources, ships logs to external providers, and integrates directly with Helm via the Application Catalog. If you have an external CI/CD system, you can plug it into Rancher, but if you don't, Rancher even includes a pipeline engine to help you automatically deploy and upgrade workloads.
@@ -37,7 +37,7 @@ _**Available as of v2.4.6**_

 Starting Rancher v2.4.6, admins can set a global TTL on Kubeconfig tokens. Once the token expires the kubectl command will require the user to authenticate to Rancher.

-_**Note:**_:
+_**Note:**_

 Existing kubeconfig tokens won't be updated with the new TTL. Admins can [delete old kubeconfig tokens](#deleting-tokens).

@@ -46,7 +46,7 @@ CNI network providers using this network model include Calico and Romana.

 ### What CNI Providers are Provided by Rancher?

-Out-of-the-box, Rancher provides the following CNI network providers for Kubernetes clusters: Canal, Flannel, Calico and Weave. You can choose your CNI network provider when you create new Kubernetes clusters from Rancher.
+Out-of-the-box, Rancher provides the following CNI network providers for Kubernetes clusters: Canal, Flannel, Calico, and Weave. You can choose your CNI network provider when you create new Kubernetes clusters from Rancher.

 #### Canal

@@ -25,7 +25,7 @@ services:

 You can follow the recommendations from [the etcd docs](https://etcd.io/docs/v3.4.0/tuning/#disk) on how to tune the disk priority on the host.

-Additionally, to reduce IO contention on the disks for etcd, you can use a dedicated device for the data and wal directory. Based on etcd best practices, mirroring RAID configurations are unnecessary because etcd replicates data between the nodes in the cluster. You can use stripping RAID configurations to increase available IOPS.
+Additionally, to reduce IO contention on the disks for etcd, you can use a dedicated device for the data and wal directory. Based on etcd best practices, mirroring RAID configurations are unnecessary because etcd replicates data between the nodes in the cluster. You can use striping RAID configurations to increase available IOPS.

 To implement this solution in an RKE cluster, the `/var/lib/etcd/data` and `/var/lib/etcd/wal` directories will need to have disks mounted and formatted on the underlying host. In the `extra_args` directive of the `etcd` service, you must include the `wal_dir` directory. Without specifying the `wal_dir`, etcd process will try to manipulate the underlying `wal` mount with insufficient permissions.

@@ -82,7 +82,7 @@ sudo mv ./kubectl /usr/local/bin/kubectl
 * [helm](https://helm.sh/docs/intro/install/)

 ```
-curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3
+curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3
 chmod +x get_helm.sh
 sudo ./get_helm.sh
 ```
@@ -30,7 +30,7 @@ The following steps quickly deploy a Rancher Server with a single node cluster a
 3. **Optional:** Edit `config.yaml` to:

    - Change the number of nodes and the memory allocations, if required. (`node.count`, `node.cpus`, `node.memory`)
-    - Change the password of the `admin` user for logging into Rancher. (`default_password`)
+    - Change the password of the `admin` user for logging into Rancher. (`admin_password`)

 4. To initiate the creation of the environment run, `vagrant up --provider=virtualbox`.

@@ -1,23 +1,29 @@
 ---
 title: Configuring Azure AD
 weight: 1115
-aliases:
-  - /rancher/v2.5/en/tasks/global-configuration/authentication/azure-ad/
-  - /rancher/v2.x/en/admin-settings/authentication/azure-ad/
 ---

+{{% tabs %}}
+{{% tab "Rancher v2.5.16+" %}}
+
+## Microsoft Graph API
+
+Microsoft Graph API is now the flow through which you will set up Azure AD. The below sections will assist [new users](#new-user-setup) in configuring Azure AD with a new instance as well as assist existing Azure app owners in [migrating to the new flow](#migrating-from-azure-ad-graph-api-to-microsoft-graph-api).
+
+### New User Setup
+
 If you have an instance of Active Directory (AD) hosted in Azure, you can configure Rancher to allow your users to log in using their AD accounts. Configuration of Azure AD external authentication requires you to make configurations in both Azure and Rancher.

->**Note:** Azure AD integration only supports Service Provider initiated logins.
-
 >**Prerequisite:** Have an instance of Azure AD configured.

->**Note:** Most of this procedure takes place from the [Microsoft Azure Portal](https://portal.azure.com/).
+>**Notes:**
+>
+>- Azure AD integration only supports Service Provider initiated logins.
+>- Most of this procedure takes place from the [Microsoft Azure Portal](https://portal.azure.com/).

-## Azure Active Directory Configuration Outline
+#### Azure Active Directory Configuration Outline

 Configuring Rancher to allow your users to authenticate with their Azure AD accounts involves multiple procedures. Review the outline below before getting started.
-
 <a id="tip"></a>

 >**Tip:** Before you start, we recommend creating an empty text file. You can use this file to copy values from Azure that you'll paste into Rancher later.
@@ -27,13 +33,12 @@ Configuring Rancher to allow your users to authenticate with their Azure AD acco
 - [1. Register Rancher with Azure](#1-register-rancher-with-azure)
 - [2. Create a new client secret](#2-create-a-new-client-secret)
 - [3. Set Required Permissions for Rancher](#3-set-required-permissions-for-rancher)
- [4. Add a Reply URL](#4-add-a-reply-url)
- [5. Copy Azure Application Data](#5-copy-azure-application-data)
- [6. Configure Azure AD in Rancher](#6-configure-azure-ad-in-rancher)
+- [4. Copy Azure Application Data](#5-copy-azure-application-data)
+- [5. Configure Azure AD in Rancher](#6-configure-azure-ad-in-rancher)

 <!-- /TOC -->

-### 1. Register Rancher with Azure
+#### 1. Register Rancher with Azure

 Before enabling Azure AD within Rancher, you must register Rancher with Azure.

@@ -47,37 +52,40 @@ Before enabling Azure AD within Rancher, you must register Rancher with Azure.

    ![New App Registration](/img/new-app-registration.png)

-    1. Enter a **Name** (something like `Rancher`).
+    3.1. Enter a **Name** (something like `Rancher`).
+    <a id="3.2"></a>

-    1. From **Supported account types**, select "Accounts in this organizational directory only (AzureADTest only - Single tenant)" This corresponds to the legacy app registration options.
+    3.2. From **Supported account types**, select "Accounts in this organizational directory only (AzureADTest only - Single tenant)" This corresponds to the legacy app registration options.

-    1. In the **Redirect URI** section, make sure **Web** is selected from the dropdown and enter the URL of your Rancher Server in the text box next to the dropdown. This Rancher server URL should be appended with the verification path: `<MY_RANCHER_URL>/verify-auth-azure`.
+    >**Important:** In the updated Azure portal, Redirect URIs are synonymous with Reply URLs. In order to use Azure AD with Rancher, you must whitelist Rancher with Azure (previously done through Reply URLs). Therefore, you must ensure to fill in the Redirect URI with your Rancher server URL, to include the verification path as listed below.

-       >**Tip:** You can find your personalized Azure reply URL in Rancher on the Azure AD Authentication page (Global View > Security Authentication > Azure AD).
+    3.3. In the [**Redirect URI**](https://docs.microsoft.com/en-us/azure/active-directory/develop/reply-url) section, make sure **Web** is selected from the dropdown and enter the URL of your Rancher Server in the text box next to the dropdown. This Rancher server URL should be appended with the verification path: `<MY_RANCHER_URL>/verify-auth-azure`.

-    1. Click **Register**.
+    >**Tip:** You can find your personalized Azure Redirect URI (reply URL) in Rancher on the Azure AD Authentication page (Global View > Authentication > Web).

->**Note:** It can take up to five minutes for this change to take affect, so don't be alarmed if you can't authenticate immediately after Azure AD configuration.
+    3.4. Click **Register**.

-### 2. Create a new client secret
+>**Important:** It can take up to five minutes for this change to take affect, so don't be alarmed if you can't authenticate immediately after Azure AD configuration.
+
+#### 2. Create a new client secret

 From the Azure portal, create a client secret. Rancher will use this key to authenticate with Azure AD.

 1. Use search to open **App registrations** services. Then open the entry for Rancher that you created in the last procedure.

-     ![Open Rancher Registration](/img/open-rancher-app.png)
+     ![Open Rancher Registration](/img/open-rancher-app-reg.png)

 1. From the navigation pane on left, click **Certificates and Secrets**.

 1. Click **New client secret**.

-     ![Create new client secret](/img/select-client-secret.png)
+     ![Create new client secret](/img//new-client-secret.png)

-    1. Enter a **Description** (something like `Rancher`).
+    3.1. Enter a **Description** (something like `Rancher`).

-    1. Select duration for the key from the options under **Expires**. This drop-down sets the expiration date for the key. Shorter durations are more secure, but require you to create a new key after expiration.
+    3.2. Select duration for the key from the options under **Expires**. This drop-down sets the expiration date for the key. Shorter durations are more secure, but require you to create a new key after expiration.

-    1. Click **Add** (you don't need to enter a value—it will automatically populate after you save).
+    3.3. Click **Add** (you don't need to enter a value—it will automatically populate after you save).
 <a id="secret"></a>

 1.  Copy the key value and save it to an [empty text file](#tip).
@@ -86,110 +94,95 @@ From the Azure portal, create a client secret. Rancher will use this key to auth

    You won't be able to access the key value again within the Azure UI.

-### 3. Set Required Permissions for Rancher
+#### 3. Set Required Permissions for Rancher

 Next, set API permissions for Rancher within Azure.

+:::caution
+
+Ensure that you set the permissions of type Application and NOT Delegated. Otherwise, you may not be able to login to Azure AD. This issue will persist even after you disable/re-enable Azure AD and will require an hour wait, or manual deletion of a cache value to resolve.
+
+:::
+
 1. From the navigation pane on left, select **API permissions**.

-    ![Open Required Permissions](/img/select-required-permissions.png)
+    ![Open Required Permissions](/img/select-req-permissions.png)

 1. Click **Add a permission**.

-1. From the **Azure Active Directory Graph**, select the following **Delegated Permissions**:
+1. From the **Microsoft Graph**, select the following **Application Permissions**:
+    - `Group.Read.All`
+    - `User.Read.All`

-    ![Select API Permissions](/img/select-required-permissions-2.png)
+    ![Select API Permissions]/img/api-permissions.png)

-    <br/>
-    <br/>
-    - **Access the directory as the signed-in user**
-    - **Read directory data**
-    - **Read all groups**
-    - **Read all users' full profiles**
-    - **Read all users' basic profiles**
-    - **Sign in and read user profile**
-
-1. Click **Add permissions**.
-
-1. From **API permissions**, click **Grant admin consent**. Then click **Yes**.
+1. Return to **API permissions** in the left nav bar. From there, click **Grant admin consent**. Then click **Yes**.

    >**Note:** You must be signed in as an Azure administrator to successfully save your permission settings.


-### 4. Add a Reply URL
-
-To use Azure AD with Rancher you must whitelist Rancher with Azure. You can complete this whitelisting by providing Azure with a reply URL for Rancher, which is your Rancher Server URL followed with a verification path.
-
-
-1. From the **Setting** blade, select **Reply URLs**.
-
-    ![Azure: Enter Reply URL](/img/enter-azure-reply-url.png)
-
-1. From the **Reply URLs** blade, enter the URL of your Rancher Server, appended with the verification path: `<MY_RANCHER_URL>/verify-auth-azure`.
-
-       >**Tip:** You can find your personalized Azure reply URL in Rancher on the Azure AD Authentication page (Global View > Security Authentication > Azure AD).
-
-1. Click **Save**.
-
-**Result:** Your reply URL is saved.
-
->**Note:** It can take up to five minutes for this change to take affect, so don't be alarmed if you can't authenticate immediately after Azure AD configuration.
-
-### 5. Copy Azure Application Data
+#### 4. Copy Azure Application Data

 As your final step in Azure, copy the data that you'll use to configure Rancher for Azure AD authentication and paste it into an empty text file.

 1. Obtain your Rancher **Tenant ID**.

-    1. Use search to open the **Azure Active Directory** service.
-
-         ![Open Azure Active Directory](/img/search-azure-ad.png)
-
-    1. From the left navigation pane, open **Overview**.
-
-    2. Copy the **Directory ID** and paste it into your [text file](#tip).
-
-        You'll paste this value into Rancher as your **Tenant ID**.
-
-1. Obtain your Rancher **Application ID**.
-
    1. Use search to open **App registrations**.

-         ![Open App Registrations](/img/search-app-registrations.png)
+    ![Open App Registrations](/img/search-app-registrations.png)

    1. Find the entry you created for Rancher.

-    1. Copy the **Application ID** and paste it to your [text file](#tip).
+    1. Copy the **Directory ID** and paste it into your [text file](#tip).

-1. Obtain your Rancher **Graph Endpoint**, **Token Endpoint**, and **Auth Endpoint**.
+    ![Tenant ID](/img/tenant-id.png)

-    1. From **App registrations**, click **Endpoints**.
+    - You'll paste this value into Rancher as your **Tenant ID**.

-        ![Click Endpoints](/img/click-endpoints.png)
+1. Obtain your Rancher **Application (Client) ID**.

-    2. Copy the following endpoints to your clipboard and paste them into your [text file](#tip) (these values will be your Rancher endpoint values).
-    
-        - **Microsoft Graph API endpoint** (Graph Endpoint)
-        - **OAuth 2.0 token endpoint (v1)** (Token Endpoint)
-        - **OAuth 2.0 authorization endpoint (v1)** (Auth Endpoint)
-        
->**Note:** Copy the v1 version of the endpoints
+    1. Use search to open **App registrations** (if not already there).

-### 6. Configure Azure AD in Rancher
+    1. In **Overview**, find the entry you created for Rancher.
+
+    1. Copy the **Application (Client) ID** and paste it to your [text file](#tip).
+
+    ![Application ID](/img/application-client-id.png)
+
+1. Your endpoint options will typically be [Standard](#global) and [China](#china). With the Standard option, perform the steps below and Rancher will take care of the rest.
+
+    - Enter the **Tenant ID**, **Application ID**, and **Application Secret**
+    - Click **Enable**
+
+![Standard Endpoint Options](/img/tenant-application-id-secret2.png)
+
+>**For Custom Endpoints:**
+>
+>**Warning:** Custom Endpoints are not supported nor fully tested by Rancher.
+>
+> You will need to also manually enter the Graph, Token, and Auth Endpoints.
+>
+>- From <b>App registrations</b>, click <b>Endpoints</b>:
+>
+>![Click Endpoints](/img/endpoints.png)
+>
+>- Copy the following endpoints to your clipboard and paste them into your [text file](#tip) (these values will be your Rancher endpoint values). Make sure to copy the v1 version of the endpoints.
+>
+>   - **Microsoft Graph API endpoint** (Graph Endpoint)
+>   - **OAuth 2.0 token endpoint (v1)** (Token Endpoint)
+>   - **OAuth 2.0 authorization endpoint (v1)** (Auth Endpoint)
+
+#### 5. Configure Azure AD in Rancher

 From the Rancher UI, enter information about your AD instance hosted in Azure to complete configuration.

 Enter the values that you copied to your [text file](#tip).

-1. Log into Rancher. From the **Global** view, select **Security > Authentication**.
-
-1. Select **Azure AD**.
-
-1. Complete the **Configure Azure AD Account** form using the information you copied while completing [Copy Azure Application Data](#5-copy-azure-application-data).
-
-    >**Important:** When entering your Graph Endpoint, remove the tenant ID from the URL, like below.
-    >
-    ><code>http<span>s://g</span>raph.windows.net/<del>abb5adde-bee8-4821-8b03-e63efdc7701c</del></code>
+1. Log into Rancher.
+1.	In the top left corner, click **☰ > Users & Authentication**.
+1. In the left navigation menu, click **Auth Provider**.
+1. Click **AzureAD**.
+1. Complete the **Configure Azure AD Account** form using the information you copied while completing [Copy Azure Application Data](#4-copy-azure-application-data).

    The following table maps the values you copied in the Azure portal to the fields in Rancher.

@@ -199,10 +192,122 @@ Enter the values that you copied to your [text file](#tip).
    | Application ID     | Application ID                        |
    | Application Secret | Key Value                             |
    | Endpoint           | https://login.microsoftonline.com/    |
-    | Graph Endpoint     | Microsoft Azure AD Graph API Endpoint |
-    | Token Endpoint     | OAuth 2.0 Token Endpoint              |
-    | Auth Endpoint      | OAuth 2.0 Authorization Endpoint      |

-1. Click **Authenticate with Azure**.
+
+    >**For Custom Endpoints:**
+    ><br/>
+    >The following table maps the custom config values you copied in the Azure portal to the fields in Rancher:
+    >
+    >| Rancher Field      | Azure Value                           |
+    >| ------------------ | ------------------------------------- |
+    >| Graph Endpoint     | Microsoft Graph API Endpoint          |
+    >| Token Endpoint     | OAuth 2.0 Token Endpoint              |
+    >| Auth Endpoint      | OAuth 2.0 Authorization Endpoint      |
+    ><br/>
+    >**Important:** When entering the Graph Endpoint in a custom config, remove the tenant ID from the URL, like below:
+    >
+    ><code>http<span>s://g</span>raph.microsoft.com/<del>abb5adde-bee8-4821-8b03-e63efdc7701c</del></code>
+
+1. Click **Enable**.

 **Result:** Azure Active Directory authentication is configured.
+
+
+### Migrating from Azure AD Graph API to Microsoft Graph API
+
+Since [Azure AD Graph API](https://docs.microsoft.com/en-us/graph/migrate-azure-ad-graph-overview) was deprecated in June 2022 and will be retired at the end of 2022, users should update their Azure AD App to use the new [Microsoft Graph API](https://docs.microsoft.com/en-us/graph/use-the-api) in Rancher.
+
+#### Updating Endpoints in the Rancher UI
+
+>**Important:** Admins should create a [backup]../../../..//new-user-guides/backup-restore-and-disaster-recovery/back-up-rancher.md) right before they commit to the endpoint migration in Step 4 below.
+
+1. Update the permissions of your Azure AD app registration as described [here](#3-set-required-permissions-for-rancher).
+**This is critical.**
+
+1. Log into Rancher.
+
+1. In the Rancher UI homepage, make note of the banner at the top of screen that advises users to update their Azure AD authentication. Click on the link provided to do so.
+
+    ![Rancher UI Banner](/img/rancher-ui-azure-update2.png)
+
+1. To complete the move to the new Microsoft Graph API, click **Update Endpoint**, then click **Save**.
+
+    **Note:** Ensure that your Azure app has a [new set of permissions](#3-set-required-permissions-for-rancher) before starting the update.
+
+    ![Update Endpoint](/img/rancher-button-to-update2.png)
+
+1. When you receive the pop-up warning message, click **Update**.
+
+    ![Azure Update Pop-up](/img/azure-update-popup2.png)
+
+1. Refer to the [tables](#global) below for the full list of endpoint changes that Rancher performs. Admins do not need to do this manually.
+
+#### Air-Gapped Environments
+
+In air-gapped environments, admins should ensure that their endpoints are [whitelisted](#3.2) since the Graph Endpoint URL is changing.
+
+#### Rolling Back the Migration
+
+If you need to roll back your migration, please note the following:
+
+1. Admins are encouraged to use the proper restore process if they want to go back. Please see [backup docs](../../../../new-user-guides/backup-restore-and-disaster-recovery/back-up-rancher.md), [restore docs](../../../../new-user-guides/backup-restore-and-disaster-recovery/restore-rancher.md), and [examples](../../../../../reference-guides/backup-restore-configuration/examples.md) for reference.
+
+1. Azure app owners who want to rotate the Application Secret will need to also rotate it in Rancher as Rancher does not automatically update the Application Secret when it is changed in Azure. In Rancher, note that it is stored in a Kubernetes secret called `azureadconfig-applicationsecret` which is in the `cattle-global-data` namespace.
+
+1. **Caution:** If admins upgrade to Rancher v2.5.16 with an existing Azure AD setup and choose to disable the auth provider, they won't be able to restore the previous setup and also will not be able to set up Azure AD anew using the old flow. Admins will then need to register again with the new auth flow. Rancher now uses the new Graph API and, therefore, users need set up the [proper permissions in the Azure portal](#3-set-required-permissions-for-rancher).
+
+#### Global:
+
+Rancher Field    | Deprecated Endpoints
+---------------- | -------------------------------------------------------------
+Auth Endpoint    | https://login.microsoftonline.com/{tenantID}/oauth2/authorize
+Endpoint         | https://login.microsoftonline.com/
+Graph Endpoint   | https://graph.windows.net/
+Token Endpoint   | https://login.microsoftonline.com/{tenantID}/oauth2/token
+---
+
+Rancher Field    | New Endpoints
+---------------- | ------------------------------------------------------------------
+Auth Endpoint    | https://login.microsoftonline.com/{tenantID}/oauth2/v2.0/authorize
+Endpoint         | https://login.microsoftonline.com/
+Graph Endpoint   | https://graph.microsoft.com
+Token Endpoint   | https://login.microsoftonline.com/{tenantID}/oauth2/v2.0/token
+
+#### China:
+
+Rancher Field    | Deprecated Endpoints
+---------------- | ----------------------------------------------------------
+Auth Endpoint    | https://login.chinacloudapi.cn/{tenantID}/oauth2/authorize
+Endpoint         | https://login.chinacloudapi.cn/
+Graph Endpoint   | https://graph.chinacloudapi.cn/
+Token Endpoint   | https://login.chinacloudapi.cn/{tenantID}/oauth2/token
+---
+
+Rancher Field    | New Endpoints
+---------------- | -------------------------------------------------------------------------
+Auth Endpoint    | https://login.partner.microsoftonline.cn/{tenantID}/oauth2/v2.0/authorize
+Endpoint         | https://login.partner.microsoftonline.cn/
+Graph Endpoint   | https://microsoftgraph.chinacloudapi.cn
+Token Endpoint   | https://login.partner.microsoftonline.cn/{tenantID}/oauth2/v2.0/token
+
+
+{{% /tab %}}
+{{% tab "Rancher v2.5.0 - v2.5.15" %}}
+
+## Azure AD Graph API
+
+>**Important:**
+>
+>- The [Azure AD Graph API](https://docs.microsoft.com/en-us/graph/migrate-azure-ad-graph-overview) was deprecated in June 2022 and will be retired at the end of 2022. We will update our docs to advise the community when it is retired. Rancher now uses the [Microsoft Graph API](https://docs.microsoft.com/en-us/graph/use-the-api) as the new flow to set up Azure AD as the external auth provider.
+>
+>
+>- For new users, or existing users who wish to migrate, refer to the new flow instructions on the <a href="#microsoft-graph-api" target="_blank">Rancher v2.5.16+</a> tab.
+>
+>
+>- For existing users who do not wish to upgrade to v2.5.16+ after the Azure AD Graph API is retired, they will need to either:
+    - Use the built-in Rancher auth or
+    - Use another third-party auth system and set that up in Rancher. Please see the [authentication docs](../../../../../pages-for-subheaders/about-authentication.md) to learn how to configure other open authentication providers.
+
+
+{{% /tab %}}
+{{% /tabs %}}
@@ -29,7 +29,9 @@ These elements are required to follow this guide:

 ### 1. Create a Custom Cluster

-On Rancher server, we should create a custom k8s cluster v1.18.x. Be sure that cloud_provider name is set to `amazonec2`. Once cluster is created we need to get:
+On Rancher server, we should create a custom k8s cluster. Refer [here](https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/) to check for version compatibility.
+
+Be sure that cloud_provider name is set to `amazonec2`. Once cluster is created we need to get:

 * clusterID: `c-xxxxx` will be used on EC2 `kubernetes.io/cluster/<clusterID>` instance tag
 * clusterName: will be used on EC2 `k8s.io/cluster-autoscaler/<clusterName>` instance tag
@@ -483,7 +485,7 @@ spec:
      nodeSelector:
        node-role.kubernetes.io/controlplane: "true"
      containers:
-        - image: eu.gcr.io/k8s-artifacts-prod/autoscaling/cluster-autoscaler:v1.18.1
+        - image: eu.gcr.io/k8s-artifacts-prod/autoscaling/cluster-autoscaler:<VERSION>
          name: cluster-autoscaler
          resources:
            limits:
@@ -20,7 +20,7 @@ _Available as of v2.5.4_

 To create rule groups in the Rancher UI,

-1. Click **Cluster Explorer > Monitoring** and click **Prometheus Rules.**
+1. Click **Cluster Explorer > Monitoring > Advanced** and click **Prometheus Rules.**
 1. Click **Create.**
 1. Enter a **Group Name.**
 1. Configure the rules. In Rancher's UI, we expect a rule group to contain either alert rules or recording rules, but not both. For help filling out the forms, refer to the configuration options below.
@@ -13,8 +13,6 @@ Rancher was originally built to work with multiple orchestrators, and it include

 Rancher can provision Kubernetes from a hosted provider, provision compute nodes and then install Kubernetes onto them, or import existing Kubernetes clusters running anywhere.

-One Rancher server installation can manage thousands of Kubernetes clusters and thousands of nodes from the same user interface.
-
 Rancher adds significant value on top of Kubernetes, first by centralizing authentication and role-based access control (RBAC) for all of the clusters, giving global admins the ability to control cluster access from one location.

 It then enables detailed monitoring and alerting for clusters and their resources, ships logs to external providers, and integrates directly with Helm via the Application Catalog. If you have an external CI/CD system, you can plug it into Rancher, but if you don't, Rancher even includes [Fleet](http://fleet.rancher.io/) to help you automatically deploy and upgrade workloads.
@@ -7,6 +7,9 @@ Rancher is committed to informing the community of security issues in our produc

 | ID | Description | Date | Resolution |
 |----|-------------|------|------------|
+| [CVE-2022-31247](https://github.com/rancher/rancher/security/advisories/GHSA-6x34-89p7-95wg) | An issue was discovered in Rancher versions up to and including 2.5.15 and 2.6.6 where a flaw with authorization logic allows privilege escalation in downstream clusters through cluster role template binding (CRTB) and project role template binding (PRTB). The vulnerability can be exploited by any user who has permissions to create/edit CRTB or PRTB (such as `cluster-owner`, `manage cluster members`, `project-owner`, and `manage project members`) to gain owner permission in another project in the same cluster or in another project on a different downstream cluster. | 18 August 2022 | [Rancher v2.6.7](https://github.com/rancher/rancher/releases/tag/v2.6.7) and [Rancher v2.5.16](https://github.com/rancher/rancher/releases/tag/v2.5.16) |
+| [CVE-2021-36783](https://github.com/rancher/rancher/security/advisories/GHSA-8w87-58w6-hfv8) | It was discovered that in Rancher versions up to and including 2.5.12 and 2.6.3, there is a failure to properly sanitize credentials in cluster template answers. This failure can lead to plaintext storage and exposure of credentials, passwords, and API tokens. The exposed credentials are visible in Rancher to authenticated `Cluster Owners`, `Cluster Members`, `Project Owners`, and `Project Members` on the endpoints `/v1/management.cattle.io.clusters`, `/v3/clusters`, and `/k8s/clusters/local/apis/management.cattle.io/v3/clusters`. | 18 August 2022 | [Rancher v2.6.7](https://github.com/rancher/rancher/releases/tag/v2.6.7) and [Rancher v2.5.16](https://github.com/rancher/rancher/releases/tag/v2.5.16) |
+| [CVE-2021-36782](https://github.com/rancher/rancher/security/advisories/GHSA-g7j7-h4q8-8w2f) | An issue was discovered in Rancher versions up to and including 2.5.15 and 2.6.6 where sensitive fields like passwords, API keys, and Rancher's service account token (used to provision clusters) were stored in plaintext directly on Kubernetes objects like `Clusters` (e.g., `cluster.management.cattle.io`). Anyone with read access to those objects in the Kubernetes API could retrieve the plaintext version of those sensitive data. The issue was partially found and reported by Florian Struck (from [Continum AG](https://www.continum.net/)) and [Marco Stuurman](https://github.com/fe-ax) (from [Shock Media B.V.](https://www.shockmedia.nl/)). | 18 August 2022 | [Rancher v2.6.7](https://github.com/rancher/rancher/releases/tag/v2.6.7) and [Rancher v2.5.16](https://github.com/rancher/rancher/releases/tag/v2.5.16) |
 | [CVE-2022-21951](https://github.com/rancher/rancher/security/advisories/GHSA-vrph-m5jj-c46c) | This vulnerability only affects customers using [Weave](https://rancher.com/docs/rancher/v2.6/en/faq/networking/cni-providers/#weave) Container Network Interface (CNI) when configured through [RKE templates](https://rancher.com/docs/rancher/v2.6/en/admin-settings/rke-templates/). A vulnerability was discovered in Rancher versions 2.5.0 up to and including 2.5.13, and 2.6.0 up to and including 2.6.4, where a user interface (UI) issue with RKE templates does not include a value for the Weave password when Weave is chosen as the CNI. If a cluster is created based on the mentioned template, and Weave is configured as the CNI, no password will be created for [network encryption](https://www.weave.works/docs/net/latest/tasks/manage/security-untrusted-networks/) in Weave; therefore, network traffic in the cluster will be sent unencrypted. | 24 May 2022 | [Rancher v2.6.5](https://github.com/rancher/rancher/releases/tag/v2.6.5) and [Rancher v2.5.14](https://github.com/rancher/rancher/releases/tag/v2.5.14) |
 | [CVE-2021-36784](https://github.com/rancher/rancher/security/advisories/GHSA-jwvr-vv7p-gpwq) | A vulnerability was discovered in Rancher versions from 2.5.0 up to and including 2.5.12 and from 2.6.0 up to and including 2.6.3 which allows users who have create or update permissions on [Global Roles](https://rancher.com/docs/rancher/v2.6/en/admin-settings/rbac/) to escalate their permissions, or those of another user, to admin-level permissions. Global Roles grant users Rancher-wide permissions, such as the ability to create clusters. In the identified versions of Rancher, when users are given permission to edit or create Global Roles, they are not restricted to only granting permissions which they already posses. This vulnerability affects customers who utilize non-admin users that are able to create or edit Global Roles. The most common use case for this scenario is the `restricted-admin` role. | 14 Apr 2022 | [Rancher v2.6.4](https://github.com/rancher/rancher/releases/tag/v2.6.4) and [Rancher v2.5.13](https://github.com/rancher/rancher/releases/tag/v2.5.13) |
 | [CVE-2021-4200](https://github.com/rancher/rancher/security/advisories/GHSA-hx8w-ghh8-r4xf) | This vulnerability only affects customers using the `restricted-admin` role in Rancher. A vulnerability was discovered in Rancher versions from 2.5.0 up to and including 2.5.12 and from 2.6.0 up to and including 2.6.3 where the `global-data` role in `cattle-global-data` namespace grants write access to the Catalogs. Since each user with any level of catalog access was bound to the `global-data` role, this grants write access to templates (`CatalogTemplates`) and template versions (`CatalogTemplateVersions`) for any user with any level of catalog access. New users created in Rancher are by default assigned to the `user` role (standard user), which is not designed to grant write catalog access. This vulnerability effectively elevates the privilege of any user to write access for the catalog template and catalog template version resources. | 14 Apr 2022 | [Rancher v2.6.4](https://github.com/rancher/rancher/releases/tag/v2.6.4) and [Rancher v2.5.13](https://github.com/rancher/rancher/releases/tag/v2.5.13) |
@@ -0,0 +1,25 @@
+---
+title: Rotation of Expired Webhook Certificates
+weight: 120
+---
+
+For Rancher versions that have `rancher-webhook` installed, certain versions created certificates that will expire after one year. It will be necessary for you to rotate your webhook certificate if the certificate did not renew.
+
+In Rancher v2.5.12 and up, rancher-webhook deployments will automatically renew their TLS certificate when it is within 30 or fewer days of its expiration date. If you are using v2.5.11 or below, there are two methods to work around this issue:
+
+##### 1. Users with cluster access, run the following commands:
+```
+kubectl delete secret -n cattle-system cattle-webhook-tls
+kubectl delete mutatingwebhookconfigurations.admissionregistration.k8s.io --ignore-not-found=true rancher.cattle.io
+kubectl delete pod -n cattle-system -l app=rancher-webhook
+```
+
+##### 2. Users with no cluster access via `kubectl`:
+
+1. Delete the `cattle-webhook-tls` secret in the `cattle-system` namespace in the local cluster.
+
+2. Delete the `rancher.cattle.io` mutating webhook
+
+3. Delete the `rancher-webhook` pod in the `cattle-system` namespace in the local cluster.
+
+**Note:** The webhook certificate expiration issue is not specific to `cattle-webhook-tls` as listed in the examples. You will fill in your expired certificate secret accordingly.
@@ -1259,7 +1259,8 @@
            "troubleshooting/other-troubleshooting-tips/dns",
            "troubleshooting/other-troubleshooting-tips/rancher-ha",
            "troubleshooting/other-troubleshooting-tips/registered-clusters",
-            "troubleshooting/other-troubleshooting-tips/logging"
+            "troubleshooting/other-troubleshooting-tips/logging",
+            "troubleshooting/other-troubleshooting-tips/expired-webhook-certificate-rotation"
                 ]
                }
              ]