Merge branch 'master' into private-CA-Helm

2026-09-25 04:28:15 +00:00 · 2022-01-20 16:56:16 -05:00
parent 847f996251 cfdb58c142
commit ab26e39eba
68 changed files with 851 additions and 7496 deletions
@@ -24,6 +24,7 @@ This section contains advanced information describing the different ways you can
 - [Additional preparation for (Red Hat/CentOS) Enterprise Linux](#additional-preparation-for-red-hat-centos-enterprise-linux)
 - [Enabling Lazy Pulling of eStargz (Experimental)](#enabling-lazy-pulling-of-estargz-experimental)
 - [Additional Logging Sources](#additional-logging-sources)
+- [Server and agent tokens](#server-and-agent-tokens)

 # Certificate Rotation

@@ -440,4 +441,14 @@ helm repo add rancher-charts https://charts.rancher.io
 helm repo update
 helm install --create-namespace -n cattle-logging-system rancher-logging-crd rancher-charts/rancher-logging-crd
 helm install --create-namespace -n cattle-logging-system rancher-logging --set additionalLoggingSources.k3s.enabled=true rancher-charts/rancher-logging
-```
+```
+
+# Server and agent tokens
+
+In K3s, there are two types of tokens: K3S_TOKEN and K3S_AGENT_TOKEN.
+
+K3S_TOKEN: Defines the key required by the server to offer the HTTP config resources. These resources are requested by the other servers before joining the K3s HA cluster. If the K3S_AGENT_TOKEN is not defined, the agents use this token as well to access the required HTTP resources to join the cluster. Note that this token is also used to generate the encryption key for important content in the database (e.g., bootstrap data).
+
+K3S_AGENT_TOKEN: Optional. Defines the key required by the server to offer HTTP config resources to the agents. If not defined, agents will require K3S_TOKEN. Defining K3S_AGENT_TOKEN is encouraged to avoid agents having to know K3S_TOKEN, which is also used to encrypt data.
+
+If no K3S_TOKEN is defined, the first K3s server will generate a random one. The result is part of the content in `/var/lib/rancher/k3s/server/token`. For example, `K1070878408e06a827960208f84ed18b65fa10f27864e71a57d9e053c4caff8504b::server:df54383b5659b9280aa1e73e60ef78fc`, where `df54383b5659b9280aa1e73e60ef78fc` is the K3S_TOKEN.
@@ -18,7 +18,7 @@ For details on taking database snapshots and restoring your database from them,

 - [Official MySQL documentation](https://dev.mysql.com/doc/refman/8.0/en/replication-snapshot-method.html)
 - [Official PostgreSQL documentation](https://www.postgresql.org/docs/8.3/backup-dump.html)
- [Official etcd documentation](https://github.com/etcd-io/etcd/blob/master/Documentation/op-guide/recovery.md)
+- [Official etcd documentation](https://etcd.io/docs/latest/op-guide/recovery/)

 # Backup and Restore with Embedded etcd Datastore (Experimental)

@@ -152,6 +152,7 @@ the agent options are there because the server has the agent process embedded wi

 | Flag |  Description |
 |------|--------------|
+| `--etcd-arg` value | Customized flag for etcd process |
 | `--kube-apiserver-arg` value | Customized flag for kube-apiserver process |
 | `--kube-scheduler-arg` value | Customized flag for kube-scheduler process |
 | `--kube-controller-manager-arg` value  | Customized flag for kube-controller-manager process    |
@@ -234,6 +235,7 @@ OPTIONS:
   --token-file value                         (cluster) File containing the cluster-secret/token [$K3S_TOKEN_FILE]
   --write-kubeconfig value, -o value         (client) Write kubeconfig for admin client to this file [$K3S_KUBECONFIG_OUTPUT]
   --write-kubeconfig-mode value              (client) Write kubeconfig with this mode [$K3S_KUBECONFIG_MODE]
+   --etcd-arg value                           (flags) Customized flag for etcd process
   --kube-apiserver-arg value                 (flags) Customized flag for kube-apiserver process
   --kube-scheduler-arg value                 (flags) Customized flag for kube-scheduler process
   --kube-controller-manager-arg value        (flags) Customized flag for kube-controller-manager process
@@ -51,6 +51,7 @@ If you wish to utilize the metrics server, you will need to open port 10250 on e
 If you plan on achieving high availability with embedded etcd, server nodes must be accessible to each other on ports 2379 and 2380.

 > **Important:** The VXLAN port on nodes should not be exposed to the world as it opens up your cluster network to be accessed by anyone. Run your nodes behind a firewall/security group that disables access to port 8472.
+> **Warning:** Flannel relies on the [Bridge CNI plugin](https://www.cni.dev/plugins/current/main/bridge/) to create a L2 network that switches traffic. Rogue pods with NET_RAW capabilities can abuse that L2 network to launch attacks such as [ARP spoofing](https://static.sched.com/hosted_files/kccncna19/72/ARP%20DNS%20spoof.pdf). Therefore, as documented in the [kubernetes docs](https://kubernetes.io/docs/concepts/security/pod-security-standards/), please set a restricted profile that disables NET_RAW on non-trustable pods.

 <figcaption>Inbound Rules for K3s Server Nodes</figcaption>

@@ -87,7 +87,11 @@ The steps are the same for both embedded DB and external DB clusters.

 To rotate secrets encryption keys on HA setups:

->**Note:** While not required, it is recommended that you pick one server node from which to run the `secrets-encrypt` commands.
+>**Notes:** 
+>
+> - Starting K3s without encryption and enabling it at a later time is currently *not* supported.
+>
+> - While not required, it is recommended that you pick one server node from which to run the `secrets-encrypt` commands.

 - Start up all three K3s servers with the `--secrets-encryption` flag. For brevity, the servers will be referred to as S1, S2, S3.

@@ -3,12 +3,22 @@ title: "Upgrades"
 weight: 25
 ---

-This section describes how to upgrade your K3s cluster.
+### Upgrading your K3s cluster

 [Upgrade basics]({{< baseurl >}}/k3s/latest/en/upgrades/basic/) describes several techniques for upgrading your cluster manually. It can also be used as a basis for upgrading through third-party Infrastructure-as-Code tools like [Terraform](https://www.terraform.io/).

 [Automated upgrades]({{< baseurl >}}/k3s/latest/en/upgrades/automated/) describes how to perform Kubernetes-native automated upgrades using Rancher's [system-upgrade-controller](https://github.com/rancher/system-upgrade-controller).

-> If Traefik is not disabled K3s versions 1.20 and earlier will have installed Traefik v1, while K3s versions 1.21 and later will install Traefik v2 if v1 is not already present.  To upgrade Traefik, please refer to the [Traefik documentation](https://doc.traefik.io/traefik/migration/v1-to-v2/) and use the [migration tool](https://github.com/traefik/traefik-migration-tool) to migrate from the older Traefik v1 to Traefik v2.
+### Version-specific caveats

-> The experimental embedded Dqlite data store was deprecated in K3s v1.19.1. Please note that upgrades from experimental Dqlite to experimental embedded etcd are not supported. If you attempt an upgrade it will not succeed and data will be lost.
+- **Traefik:** If Traefik is not disabled, K3s versions 1.20 and earlier will install Traefik v1, while K3s versions 1.21 and later will install Traefik v2, if v1 is not already present. To upgrade from the older Traefik v1 to Traefik v2, please refer to the [Traefik documentation](https://doc.traefik.io/traefik/migration/v1-to-v2/) and use the [migration tool](https://github.com/traefik/traefik-migration-tool).
+
+- **K3s bootstrap data:** If you are using K3s in an HA configuration with an external SQL datastore, and your server (control-plane) nodes were not started with the `--token` CLI flag, you will no longer be able to add additional K3s servers to the cluster without specifying the token. Ensure that you retain a copy of this token, as it is required when restoring from backup. Previously, K3s did not enforce the use of a token when using external SQL datastores. 
+    - The affected versions are <= v1.19.12+k3s1, v1.20.8+k3s1, v1.21.2+k3s1; the patched versions are v1.19.13+k3s1, v1.20.9+k3s1, v1.21.3+k3s1.
+
+    - You may retrieve the token value from any server already joined to the cluster as follows:
+```
+cat /var/lib/rancher/k3s/server/token
+```
+
+- **Experimental Dqlite:** The experimental embedded Dqlite data store was deprecated in K3s v1.19.1. Please note that upgrades from experimental Dqlite to experimental embedded etcd are not supported. If you attempt an upgrade, it will not succeed, and data will be lost.
@@ -4,7 +4,7 @@ weight: 400
 aliases:
 ---

-You might want to use a private Docker registry to share your custom base images within your organization. With a private registry, you can keep a private, consistent, and centralized source of truth for the Docker images that are used in your clusters.
+You might want to use a private container registry to share your custom base images within your organization. With a private registry, you can keep a private, consistent, and centralized source of truth for the container images that are used in your clusters.

 There are two main ways to set up private registries in Rancher: by setting up the global default registry through the **Settings** tab in the global view, and by setting up a private registry in the advanced options in the cluster-level settings. The global default registry is intended to be used for air-gapped setups, for registries that do not require credentials. The cluster-level private registry is intended to be used in all setups in which the private registry requires credentials.

@@ -231,7 +231,7 @@ helm install rancher rancher-<CHART_REPO>/rancher \
 If you are using a Private CA signed certificate , add `--set privateCA=true` to the command:

 ```
-helm install rancher rancher-latest/rancher \
+helm install rancher rancher-<CHART_REPO>/rancher \
  --namespace cattle-system \
  --set hostname=rancher.my.org \
  --set ingress.tls.source=secret \
@@ -15,7 +15,7 @@ Deploy a workload to run an application in one or more containers.

 1. Enter a **Name** for the workload.

-1. Select a [workload type]({{<baseurl>}}/rancher/v2.0-v2.4/en/k8s-in-rancher/workloads/). The workload defaults to a scalable deployment, by can change the workload type by clicking **More options.**
+1. Select a [workload type]({{<baseurl>}}/rancher/v2.0-v2.4/en/k8s-in-rancher/workloads/). The workload defaults to a scalable deployment, but you can change the workload type by clicking **More options.**

 1. From the **Docker Image** field, enter the name of the Docker image that you want to deploy to the project, optionally prefacing it with the registry host (e.g. `quay.io`, `registry.gitlab.com`, etc.). During deployment, Rancher pulls this image from the specified public or private registry. If no registry host is provided, Rancher will pull the image from [Docker Hub](https://hub.docker.com/explore/). Enter the name exactly as it appears in the registry server, including any required path, and optionally including the desired tag (e.g. `registry.gitlab.com/user/path/image:tag`). If no tag is provided, the `latest` tag will be automatically used.

@@ -45,4 +45,4 @@ Edit [resource quotas]({{<baseurl>}}/rancher/v2.0-v2.4/en/k8s-in-rancher/project

 1. Click **Create**.

-**Result:** The resource quota is applied to your project and namespaces. When you add more namespaces in the future, Rancher validates that the project can accommodate the namespace. If the project can't allocate the resources, Rancher won't let you save your changes.
+**Result:** The resource quota is applied to your project and namespaces. When you add more namespaces in the future, Rancher validates that the project can accommodate the namespace. If the project can't allocate the resources, you may still create namespaces, but they will be given a resource quota of 0. Subsequently, Rancher will not allow you to create any resources restricted by this quota.
@@ -158,7 +158,7 @@ kubectl run -i --restart=Never --rm test-${RANDOM} --image=ubuntu --overrides='{
 Enabling query logging can be done by enabling the [log plugin](https://coredns.io/plugins/log/) in the Corefile configuration in the configmap `coredns`. You can do so by using `kubectl -n kube-system edit configmap coredns` or use the command below to replace the configuration in place:

 ```
-kubectl get configmap -n kube-system coredns -o json |  kubectl get configmap -n kube-system coredns -o json | sed -e 's_loadbalance_log\\n    loadbalance_g' | kubectl apply -f -
+kubectl get configmap -n kube-system coredns -o json | sed -e 's_loadbalance_log\\n    loadbalance_g' | kubectl apply -f -
 ```

 All queries will now be logged and can be checked using the command in [Check CoreDNS logging](#check-coredns-logging).
@@ -5,7 +5,7 @@ aliases:
  - /rancher/v2.x/en/admin-settings/config-private-registry/
 ---

-You might want to use a private Docker registry to share your custom base images within your organization. With a private registry, you can keep a private, consistent, and centralized source of truth for the Docker images that are used in your clusters.
+You might want to use a private container registry to share your custom base images within your organization. With a private registry, you can keep a private, consistent, and centralized source of truth for the container images that are used in your clusters.

 There are two main ways to set up private registries in Rancher: by setting up the global default registry through the **Settings** tab in the global view, and by setting up a private registry in the advanced options in the cluster-level settings. The global default registry is intended to be used for air-gapped setups, for registries that do not require credentials. The cluster-level private registry is intended to be used in all setups in which the private registry requires credentials.

@@ -11,7 +11,9 @@ In this section, you'll learn how to back up Rancher running on any Kubernetes c

 The backup-restore operator needs to be installed in the local cluster, and only backs up the Rancher app. The backup and restore operations are performed only in the local Kubernetes cluster.

-> When restoring a backup into a new Rancher setup, the version of the new setup should be the same as the one where the backup is made.
+Note that the rancher-backup operator version 1.x.x is for Rancher v2.5.x.
+
+> When restoring a backup into a new Rancher setup, the version of the new setup should be the same as the one where the backup is made. The Kubernetes version should also be considered when restoring a backup, since the supported apiVersion in the cluster and in the backup file could be different.

 ### Prerequisites

@@ -70,20 +70,19 @@ spec:
      endpoint: s3.us-west-2.amazonaws.com
 ```

-> **Important:** The field `encryptionConfigSecretName` must be set only if your backup was created with encryption enabled. Provide the name of the Secret containing the encryption config file. If you only have the encryption config file, but don't have a secret created with it in this cluster, use the following steps to create the secret:  
-1. The encryption configuration file must be named `encryption-provider-config.yaml`, and the `--from-file` flag must be used to create this secret. So save your `EncryptionConfiguration` in a file called `encryption-provider-config.yaml` and run this command:
+>**Important:** The field `encryptionConfigSecretName` must be set only if your backup was created with encryption enabled. Provide the name of the Secret containing the encryption config file. If you only have the encryption config file, but don't have a secret created with it in this cluster, use the following steps to create the secret:  

-```
-kubectl create secret generic encryptionconfig \
-  --from-file=./encryption-provider-config.yaml \
-  -n cattle-resources-system
-```
-
-Then apply the resource:
-
-```
-kubectl apply -f migrationResource.yaml 
-```
+1. The encryption configuration file must be named `encryption-provider-config.yaml`, and the `--from-file` flag must be used to create this secret. So save your `EncryptionConfiguration` in a file called `encryption-provider-config.yaml` and run this command:   
+    ```
+    kubectl create secret generic encryptionconfig \
+      --from-file=./encryption-provider-config.yaml \
+      -n cattle-resources-system
+    ```
+  
+1. Then apply the resource:
+    ```
+    kubectl apply -f migrationResource.yaml 
+    ```

 ### 3. Install cert-manager

@@ -53,7 +53,7 @@ https://rancher.com/docs/rancher/v2.5/en/installation/requirements/

 ### Leverage VM Templates to Construct the Environment

-To facilitate consistency across the deployed Virtual Machines across the environment, consider the use of "Golden Images" in the form of VM templates. Packer can be used to accomplish this, adding greater customisation options.
+To facilitate the consistency of Virtual Machines deployed across the environment, consider the use of "Golden Images" in the form of VM templates. Packer can be used to accomplish this, adding greater customization options.

 ### Leverage DRS Anti-Affinity Rules (Where Possible) to Separate Rancher Cluster Nodes Across ESXi Hosts

@@ -24,6 +24,7 @@ Resource targeting uses `*` as the ARN of many of the resources created cannot b
                "ec2:RunInstances",
                "ec2:RevokeSecurityGroupIngress",
                "ec2:RevokeSecurityGroupEgress",
+                "ec2:DescribeRegions",
                "ec2:DescribeVpcs",
                "ec2:DescribeTags",
                "ec2:DescribeSubnets",
@@ -123,31 +124,6 @@ Resource targeting uses `*` as the ARN of many of the resources created cannot b

 ### Service Role Permissions

-Rancher will create a service role with the following trust policy:
-
-```json
-{
-  "Version": "2012-10-17",
-  "Statement": [
-    {
-      "Action": "sts:AssumeRole",
-      "Principal": {
-        "Service": "eks.amazonaws.com"
-      },
-      "Effect": "Allow",
-      "Sid": ""
-    }
-  ]
-}
-```
-
-This role will also have two role policy attachments with the following policies ARNs:
-
-```
-arn:aws:iam::aws:policy/AmazonEKSClusterPolicy
-arn:aws:iam::aws:policy/AmazonEKSServicePolicy
-```
-
 Permissions required for Rancher to create service role on users behalf during the EKS cluster creation process.

 ```json
@@ -182,36 +158,66 @@ Permissions required for Rancher to create service role on users behalf during t
 }
 ```

+When an EKS cluster is created, Rancher will create a service role with the following trust policy:
+
+```json
+{
+  "Version": "2012-10-17",
+  "Statement": [
+    {
+      "Action": "sts:AssumeRole",
+      "Principal": {
+        "Service": "eks.amazonaws.com"
+      },
+      "Effect": "Allow",
+      "Sid": ""
+    }
+  ]
+}
+```
+
+This role will also have two role policy attachments with the following policies ARNs:
+
+```
+arn:aws:iam::aws:policy/AmazonEKSClusterPolicy
+arn:aws:iam::aws:policy/AmazonEKSServicePolicy
+```
+
 ### VPC Permissions

 Permissions required for Rancher to create VPC and associated resources.

 ```json
 {
-  "Sid": "VPCPermissions",
-  "Effect": "Allow",
-  "Action": [
-     "ec2:ReplaceRoute",
-     "ec2:ModifyVpcAttribute",
-     "ec2:ModifySubnetAttribute",
-     "ec2:DisassociateRouteTable",
-     "ec2:DetachInternetGateway",
-     "ec2:DescribeVpcs",
-     "ec2:DeleteVpc",
-     "ec2:DeleteTags",
-     "ec2:DeleteSubnet",
-     "ec2:DeleteRouteTable",
-     "ec2:DeleteRoute",
-     "ec2:DeleteInternetGateway",
-     "ec2:CreateVpc",
-     "ec2:CreateSubnet",
-     "ec2:CreateSecurityGroup",
-     "ec2:CreateRouteTable",
-     "ec2:CreateRoute",
-     "ec2:CreateInternetGateway",
-     "ec2:AttachInternetGateway",
-     "ec2:AssociateRouteTable"
-  ],
-  "Resource": "*"
+  "Version": "2012-10-17",
+  "Statement": [
+    {
+      "Sid": "VPCPermissions",
+      "Effect": "Allow",
+      "Action": [
+        "ec2:ReplaceRoute",
+        "ec2:ModifyVpcAttribute",
+        "ec2:ModifySubnetAttribute",
+        "ec2:DisassociateRouteTable",
+        "ec2:DetachInternetGateway",
+        "ec2:DescribeVpcs",
+        "ec2:DeleteVpc",
+        "ec2:DeleteTags",
+        "ec2:DeleteSubnet",
+        "ec2:DeleteRouteTable",
+        "ec2:DeleteRoute",
+        "ec2:DeleteInternetGateway",
+        "ec2:CreateVpc",
+        "ec2:CreateSubnet",
+        "ec2:CreateSecurityGroup",
+        "ec2:CreateRouteTable",
+        "ec2:CreateRoute",
+        "ec2:CreateInternetGateway",
+        "ec2:AttachInternetGateway",
+        "ec2:AssociateRouteTable"
+      ],
+      "Resource": "*"
+    }
+  ]
 }
-```
+```
@@ -32,6 +32,13 @@ For information on how to install Docker, refer to the official [Docker document

 Some distributions of Linux derived from RHEL, including Oracle Linux, may have default firewall rules that block communication with Helm. We recommend disabling firewalld. For Kubernetes 1.19, firewalld must be turned off.

+>**Note:** In RHEL 8.4, two extra services are included on the NetworkManager: `nm-cloud-setup.service` and `nm-cloud-setup.timer`. These services add a routing table that interferes with the CNI plugin's configuration. If these services are enabled, you must disable them using the command below, and then reboot the node to restore connectivity:
+>  
+>  ```
+   systemctl disable nm-cloud-setup.service nm-cloud-setup.timer
+   reboot
+   ```
+
 ### SUSE Linux Nodes

 SUSE Linux may have a firewall that blocks all ports by default. In that situation, follow [these steps]({{<baseurl>}}/rancher/v2.5/en/installation/requirements/ports/#opening-suse-linux-ports) to open the ports needed for adding a host to a custom cluster.
@@ -85,7 +85,7 @@ If you are configuring DHCP options sets for an AWS virtual private cloud, note

 > Some Linux operating systems accept multiple domain names separated by spaces. However, other Linux operating systems and Windows treat the value as a single domain, which results in unexpected behavior. If your DHCP options set is associated with a VPC that has instances with multiple operating systems, specify only one domain name.

-#### Rancher on vSphere with ESXi 6.7u2 and above
+### Rancher on vSphere with ESXi 6.7u2 and above

 If you are using Rancher on VMware vSphere with ESXi 6.7u2 or later with Red Hat Enterprise Linux 8.3, CentOS 8.3, or SUSE Enterprise Linux 15 SP2 or later, it is necessary to disable the `vmxnet3` virtual network adapter hardware offloading feature. Failure to do so will result in all network connections between pods on different cluster nodes to fail with timeout errors. All connections from Windows pods to critical services running on Linux nodes, such as CoreDNS, will fail as well. It is also possible that external connections may fail. This issue is the result of Linux distributions enabling the hardware offloading feature in `vmxnet3` and a bug in the `vmxnet3` hardware offloading feature that results in the discarding of packets for guest overlay traffic. To address this issue, it is necessary disable the `vmxnet3` hardware offloading feature. This setting does not survive reboot, so it is necessary to disable on every boot. The recommended course of action is to create a systemd unit file at `/etc/systemd/system/disable_hw_offloading.service`, which disables the `vmxnet3` hardware offloading feature on boot. A sample systemd unit file which disables the `vmxnet3` hardware offloading feature is as follows. Note that `<VM network interface>` must be customized to the host `vmxnet3` network interface, e.g., `ens192`:

@@ -245,7 +245,7 @@ helm install rancher rancher-<CHART_REPO>/rancher \
 If you are using a Private CA signed certificate , add `--set privateCA=true` to the command:

 ```
-helm install rancher rancher-latest/rancher \
+helm install rancher rancher-<CHART_REPO>/rancher \
  --namespace cattle-system \
  --set hostname=rancher.my.org \
  --set ingress.tls.source=secret \
@@ -16,7 +16,7 @@ Deploy a workload to run an application in one or more containers.

 1. Enter a **Name** for the workload.

-1. Select a [workload type]({{<baseurl>}}/rancher/v2.5/en/k8s-in-rancher/workloads/). The workload defaults to a scalable deployment, by can change the workload type by clicking **More options.**
+1. Select a [workload type]({{<baseurl>}}/rancher/v2.5/en/k8s-in-rancher/workloads/). The workload defaults to a scalable deployment, but you can change the workload type by clicking **More options.**

 1. From the **Docker Image** field, enter the name of the Docker image that you want to deploy to the project, optionally prefacing it with the registry host (e.g. `quay.io`, `registry.gitlab.com`, etc.). During deployment, Rancher pulls this image from the specified public or private registry. If no registry host is provided, Rancher will pull the image from [Docker Hub](https://hub.docker.com/explore/). Enter the name exactly as it appears in the registry server, including any required path, and optionally including the desired tag (e.g. `registry.gitlab.com/user/path/image:tag`). If no tag is provided, the `latest` tag will be automatically used.

@@ -15,6 +15,8 @@ For information on configuring custom scrape targets and rules for Prometheus, p

 The resource requests and limits for the monitoring application can be configured when installing `rancher-monitoring`. For more information about the default limits, see [this page.](./helm-chart-options/#configuring-resource-limits-and-requests)

+>**Note:** On an idle cluster, Monitoring V2 has significantly higher CPU usage (up to 70%) as compared to Monitoring V1. To improve performance and achieve similar results as in Monitoring V1, turn off the Prometheus adapter.
+
 # Prometheus Configuration

 It is usually not necessary to directly edit the Prometheus custom resource.
@@ -26,6 +26,6 @@ For more information about how ServiceMonitors work, refer to the [Prometheus Op

 This pseudo-CRD maps to a section of the Prometheus custom resource configuration. It declaratively specifies how group of pods should be monitored. 

-When a PodMonitor is created, the Prometheus Operator updates the Prometheus scrape configuration to include the PodMonitor configuration. Then Prometheus begins scraping metrics from the endpoint defined in the ServiceMonitor.
+When a PodMonitor is created, the Prometheus Operator updates the Prometheus scrape configuration to include the PodMonitor configuration. Then Prometheus begins scraping metrics from the endpoint defined in the PodMonitor.

 Any Pods in your cluster that match the labels located within the PodMonitor `selector` field will be monitored based on the `podMetricsEndpoints` specified on the PodMonitor. For more information on what fields can be specified, please look at the [spec](https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/api.md#podmonitorspec) provided by Prometheus Operator.
@@ -42,4 +42,4 @@ Edit [resource quotas]({{<baseurl>}}/rancher/v2.5/en/cluster-admin/projects-and-

 1. Click **Create**.

-**Result:** The resource quota is applied to your project and namespaces. When you add more namespaces in the future, Rancher validates that the project can accommodate the namespace. If the project can't allocate the resources, Rancher won't let you save your changes.
+**Result:** The resource quota is applied to your project and namespaces. When you add more namespaces in the future, Rancher validates that the project can accommodate the namespace. If the project can't allocate the resources, you may still create namespaces, but they will be given a resource quota of 0. Subsequently, Rancher will not allow you to create any resources restricted by this quota.
@@ -1,11 +1,9 @@
 ---
 title: Rancher AWS Quick Start Guide
-description: Read this step by step Rancher AWS guide to quickly deploy a Rancher Server with a single node cluster attached.
+description: Read this step by step Rancher AWS guide to quickly deploy a Rancher server with a single-node downstream Kubernetes cluster attached.
 weight: 100
-aliases:
-  - /rancher/v2.x/en/quick-start-guide/deployment/amazon-aws-qs/
 ---
-The following steps will quickly deploy a Rancher Server on AWS with a single node cluster attached.
+The following steps will quickly deploy a Rancher server on AWS in a single-node K3s Kubernetes cluster, with a single-node downstream Kubernetes cluster attached.

 ## Prerequisites

@@ -21,25 +19,26 @@ The following steps will quickly deploy a Rancher Server on AWS with a single no

 1. Clone [Rancher Quickstart](https://github.com/rancher/quickstart) to a folder using `git clone https://github.com/rancher/quickstart`.

-1. Go into the AWS folder containing the terraform files by executing `cd quickstart/aws`.
+2. Go into the AWS folder containing the terraform files by executing `cd quickstart/aws`.

-1. Rename the `terraform.tfvars.example` file to `terraform.tfvars`.
+3. Rename the `terraform.tfvars.example` file to `terraform.tfvars`.

-1. Edit `terraform.tfvars` and customize the following variables:
-    - `aws_access_key` - Amazon AWS Access Key
+4. Edit `terraform.tfvars` and customize the following variables:
+    - `aws_access_key` - Amazon AWS Access Key 
    - `aws_secret_key` - Amazon AWS Secret Key
    - `rancher_server_admin_password` - Admin password for created Rancher server

-1. **Optional:** Modify optional variables within `terraform.tfvars`.
+5. **Optional:** Modify optional variables within `terraform.tfvars`.
 See the [Quickstart Readme](https://github.com/rancher/quickstart) and the [AWS Quickstart Readme](https://github.com/rancher/quickstart/tree/master/aws) for more information.
 Suggestions include:
-    - `aws_region` - Amazon AWS region, choose the closest instead of the default
+    - `aws_region` - Amazon AWS region, choose the closest instead of the default (`us-east-1`)
    - `prefix` - Prefix for all created resources
    - `instance_type` - EC2 instance size used, minimum is `t3a.medium` but `t3a.large` or `t3a.xlarge` could be used if within budget
+    - `add_windows_node` - If true, an additional Windows worker node is added to the workload cluster

-1. Run `terraform init`.
+6. Run `terraform init`.

-1. To initiate the creation of the environment, run `terraform apply --auto-approve`. Then wait for output similar to the following:
+7. To initiate the creation of the environment, run `terraform apply --auto-approve`. Then wait for output similar to the following:

    ```
    Apply complete! Resources: 16 added, 0 changed, 0 destroyed.
@@ -51,11 +50,12 @@ Suggestions include:
    workload_node_ip = yy.yy.yy.yy
    ```

-1. Paste the `rancher_server_url` from the output above into the browser. Log in when prompted (default username is `admin`, use the password set in `rancher_server_admin_password`).
+8. Paste the `rancher_server_url` from the output above into the browser. Log in when prompted (default username is `admin`, use the password set in `rancher_server_admin_password`).
+9. ssh to the Rancher server using the `id_rsa` key generated in `quickstart/aws`.

 #### Result

-Two Kubernetes clusters are deployed into your AWS account, one running Rancher Server and the other ready for experimentation deployments. Please note that while this setup is a great way to explore Rancher functionality, a production setup should follow our high availability setup guidelines.
+Two Kubernetes clusters are deployed into your AWS account, one running Rancher Server and the other ready for experimentation deployments. Please note that while this setup is a great way to explore Rancher functionality, a production setup should follow our high availability setup guidelines. SSH keys for the VMs are auto-generated and stored in the module directory.

 ### What's Next?

@@ -1,11 +1,9 @@
 ---
 title: Rancher DigitalOcean Quick Start Guide
-description: Read this step by step Rancher DigitalOcean guide to quickly deploy a Rancher Server with a single node cluster attached.
+description: Read this step by step Rancher DigitalOcean guide to quickly deploy a Rancher server with a single-node downstream Kubernetes cluster attached.
 weight: 100
-aliases:
-  - /rancher/v2.x/en/quick-start-guide/deployment/digital-ocean-qs/
 ---
-The following steps will quickly deploy a Rancher Server on DigitalOcean with a single node cluster attached.
+The following steps will quickly deploy a Rancher server on DigitalOcean in a single-node K3s Kubernetes cluster, with a single-node downstream Kubernetes cluster attached.

 ## Prerequisites

@@ -21,25 +19,24 @@ The following steps will quickly deploy a Rancher Server on DigitalOcean with a

 1. Clone [Rancher Quickstart](https://github.com/rancher/quickstart) to a folder using `git clone https://github.com/rancher/quickstart`.

-1. Go into the DigitalOcean folder containing the terraform files by executing `cd quickstart/do`.
+2. Go into the DigitalOcean folder containing the terraform files by executing `cd quickstart/do`.

-1. Rename the `terraform.tfvars.example` file to `terraform.tfvars`.
+3. Rename the `terraform.tfvars.example` file to `terraform.tfvars`.

-1. Edit `terraform.tfvars` and customize the following variables:
+4. Edit `terraform.tfvars` and customize the following variables:
    - `do_token` - DigitalOcean access key
    - `rancher_server_admin_password` - Admin password for created Rancher server

-1. **Optional:** Modify optional variables within `terraform.tfvars`.
+5. **Optional:** Modify optional variables within `terraform.tfvars`.
 See the [Quickstart Readme](https://github.com/rancher/quickstart) and the [DO Quickstart Readme](https://github.com/rancher/quickstart/tree/master/do) for more information.
 Suggestions include:
-    - `do_region` - DigitalOcean region, choose the closest instead of the default
+    - `do_region` - DigitalOcean region, choose the closest instead of the default (`nyc1`)
    - `prefix` - Prefix for all created resources
    - `droplet_size` - Droplet size used, minimum is `s-2vcpu-4gb` but `s-4vcpu-8gb` could be used if within budget
-    - `ssh_key_file_name` - Use a specific SSH key instead of `~/.ssh/id_rsa` (public key is assumed to be `${ssh_key_file_name}.pub`)

-1. Run `terraform init`.
+6. Run `terraform init`.

-1. To initiate the creation of the environment, run `terraform apply --auto-approve`. Then wait for output similar to the following:
+7. To initiate the creation of the environment, run `terraform apply --auto-approve`. Then wait for output similar to the following:

    ```
    Apply complete! Resources: 15 added, 0 changed, 0 destroyed.
@@ -51,11 +48,12 @@ Suggestions include:
    workload_node_ip = yy.yy.yy.yy
    ```

-1. Paste the `rancher_server_url` from the output above into the browser. Log in when prompted (default username is `admin`, use the password set in `rancher_server_admin_password`).
+8. Paste the `rancher_server_url` from the output above into the browser. Log in when prompted (default username is `admin`, use the password set in `rancher_server_admin_password`).
+9. ssh to the Rancher Server using the `id_rsa` key generated in `quickstart/do`.

 #### Result

-Two Kubernetes clusters are deployed into your DigitalOcean account, one running Rancher Server and the other ready for experimentation deployments.
+Two Kubernetes clusters are deployed into your DigitalOcean account, one running Rancher Server and the other ready for experimentation deployments. Please note that while this setup is a great way to explore Rancher functionality, a production setup should follow our high availability setup guidelines. SSH keys for the VMs are auto-generated and stored in the module directory.

 ### What's Next?

@@ -1,69 +1,68 @@
---
-title: Rancher GCP Quick Start Guide
-description: Read this step by step Rancher GCP guide to quickly deploy a Rancher Server with a single node cluster attached.
-weight: 100
-aliases:
-  - /rancher/v2.x/en/quick-start-guide/deployment/google-gcp-qs/
---
-The following steps will quickly deploy a Rancher server on GCP in a single-node RKE Kubernetes cluster, with a single-node downstream Kubernetes cluster attached.
-
-## Prerequisites
-
->**Note**
->Deploying to Google GCP will incur charges.
-
- [Google GCP Account](https://console.cloud.google.com/): A Google GCP Account is required to create resources for deploying Rancher and Kubernetes.
- [Google GCP Project](https://cloud.google.com/appengine/docs/standard/nodejs/building-app/creating-project): Use this link to follow a tutorial to create a GCP Project if you don't have one yet.
- [Google GCP Service Account](https://cloud.google.com/iam/docs/creating-managing-service-account-keys): Use this link and follow instructions to create a GCP service account and token file.
- [Terraform](https://www.terraform.io/downloads.html): Used to provision the server and cluster in Google GCP.
-
-
-## Getting Started
-
-1. Clone [Rancher Quickstart](https://github.com/rancher/quickstart) to a folder using `git clone https://github.com/rancher/quickstart`.
-
-1. Go into the GCP folder containing the terraform files by executing `cd quickstart/gcp`.
-
-1. Rename the `terraform.tfvars.example` file to `terraform.tfvars`.
-
-1. Edit `terraform.tfvars` and customize the following variables:
-    - `gcp_account_json` - GCP service account file path and file name
-    - `rancher_server_admin_password` - Admin password for created Rancher server
-
-1. **Optional:** Modify optional variables within `terraform.tfvars`.
-See the [Quickstart Readme](https://github.com/rancher/quickstart) and the [GCP Quickstart Readme](https://github.com/rancher/quickstart/tree/master/gcp) for more information.
-Suggestions include:
-    - `gcp_region` - Google GCP region, choose the closest instead of the default
-    - `prefix` - Prefix for all created resources
-    - `machine_type` - Compute instance size used, minimum is `n1-standard-1` but `n1-standard-2` or `n1-standard-4` could be used if within budget
-    - `ssh_key_file_name` - Use a specific SSH key instead of `~/.ssh/id_rsa` (public key is assumed to be `${ssh_key_file_name}.pub`)
-
-1. Run `terraform init`.
-
-1. To initiate the creation of the environment, run `terraform apply --auto-approve`. Then wait for output similar to the following:
-
-    ```
-    Apply complete! Resources: 16 added, 0 changed, 0 destroyed.
-
-    Outputs:
-
-    rancher_node_ip = xx.xx.xx.xx
-    rancher_server_url = https://rancher.xx.xx.xx.xx.sslip.io
-    workload_node_ip = yy.yy.yy.yy
-    ```
-
-1. Paste the `rancher_server_url` from the output above into the browser. Log in when prompted (default username is `admin`, use the password set in `rancher_server_admin_password`).
-
-#### Result
-
-Two Kubernetes clusters are deployed into your GCP account, one running Rancher Server and the other ready for experimentation deployments.
-
-### What's Next?
-
-Use Rancher to create a deployment. For more information, see [Creating Deployments]({{< baseurl >}}/rancher/v2.5/en/quick-start-guide/workload).
-
-## Destroying the Environment
-
-1. From the `quickstart/gcp` folder, execute `terraform destroy --auto-approve`.
-
-2. Wait for confirmation that all resources have been destroyed.
+---
+title: Rancher GCP Quick Start Guide
+description: Read this step by step Rancher GCP guide to quickly deploy a Rancher server with a single-node downstream Kubernetes cluster attached.
+weight: 100
+---
+The following steps will quickly deploy a Rancher server on GCP in a single-node K3s Kubernetes cluster, with a single-node downstream Kubernetes cluster attached.
+
+## Prerequisites
+
+>**Note**
+>Deploying to Google GCP will incur charges.
+
+- [Google GCP Account](https://console.cloud.google.com/): A Google GCP Account is required to create resources for deploying Rancher and Kubernetes.
+- [Google GCP Project](https://cloud.google.com/appengine/docs/standard/nodejs/building-app/creating-project): Use this link to follow a tutorial to create a GCP Project if you don't have one yet.
+- [Google GCP Service Account](https://cloud.google.com/iam/docs/creating-managing-service-account-keys): Use this link and follow instructions to create a GCP service account and token file.
+- [Terraform](https://www.terraform.io/downloads.html): Used to provision the server and cluster in Google GCP.
+
+
+## Getting Started
+
+1. Clone [Rancher Quickstart](https://github.com/rancher/quickstart) to a folder using `git clone https://github.com/rancher/quickstart`.
+
+2. Go into the GCP folder containing the terraform files by executing `cd quickstart/gcp`.
+
+3. Rename the `terraform.tfvars.example` file to `terraform.tfvars`.
+
+4. Edit `terraform.tfvars` and customize the following variables:
+    - `gcp_account_json` - GCP service account file path and file name 
+    - `rancher_server_admin_password` - Admin password for created Rancher server
+
+5. **Optional:** Modify optional variables within `terraform.tfvars`.
+See the [Quickstart Readme](https://github.com/rancher/quickstart) and the [GCP Quickstart Readme](https://github.com/rancher/quickstart/tree/master/gcp) for more information.
+Suggestions include:
+    - `gcp_region` - Google GCP region, choose the closest instead of the default (`us-east4`)
+    - `gcp_zone` - Google GCP zone, choose the closest instead of the default (`us-east4-a`)
+    - `prefix` - Prefix for all created resources
+    - `machine_type` - Compute instance size used, minimum is `n1-standard-1` but `n1-standard-2` or `n1-standard-4` could be used if within budget
+
+6. Run `terraform init`.
+
+7. To initiate the creation of the environment, run `terraform apply --auto-approve`. Then wait for output similar to the following:
+
+    ```
+    Apply complete! Resources: 16 added, 0 changed, 0 destroyed.
+
+    Outputs:
+
+    rancher_node_ip = xx.xx.xx.xx
+    rancher_server_url = https://rancher.xx.xx.xx.xx.sslip.io
+    workload_node_ip = yy.yy.yy.yy
+    ```
+
+8. Paste the `rancher_server_url` from the output above into the browser. Log in when prompted (default username is `admin`, use the password set in `rancher_server_admin_password`).
+9. ssh to the Rancher Server using the `id_rsa` key generated in `quickstart/gcp`.
+
+#### Result
+
+Two Kubernetes clusters are deployed into your GCP account, one running Rancher Server and the other ready for experimentation deployments. Please note that while this setup is a great way to explore Rancher functionality, a production setup should follow our high availability setup guidelines. SSH keys for the VMs are auto-generated and stored in the module directory.
+
+### What's Next?
+
+Use Rancher to create a deployment. For more information, see [Creating Deployments]({{< baseurl >}}/rancher/v2.5/en/quick-start-guide/workload).
+
+## Destroying the Environment
+
+1. From the `quickstart/gcp` folder, execute `terraform destroy --auto-approve`.
+
+2. Wait for confirmation that all resources have been destroyed.
@@ -1,73 +1,74 @@
---
-title: Rancher Azure Quick Start Guide
-description: Read this step by step Rancher Azure guide to quickly deploy a Rancher Server with a single node cluster attached.
-weight: 100
-aliases:
-  - /rancher/v2.x/en/quick-start-guide/deployment/microsoft-azure-qs/
---
-
-The following steps will quickly deploy a Rancher server on Azure in a single-node RKE Kubernetes cluster, with a single-node downstream Kubernetes cluster attached.
-
-## Prerequisites
-
->**Note**
->Deploying to Microsoft Azure will incur charges.
-
- [Microsoft Azure Account](https://azure.microsoft.com/en-us/free/): A Microsoft Azure Account is required to create resources for deploying Rancher and Kubernetes.
- [Microsoft Azure Subscription](https://docs.microsoft.com/en-us/azure/cost-management-billing/manage/create-subscription#create-a-subscription-in-the-azure-portal): Use this link to follow a tutorial to create a Microsoft Azure subscription if you don't have one yet.
- [Micsoroft Azure Tenant](https://docs.microsoft.com/en-us/azure/active-directory/develop/quickstart-create-new-tenant): Use this link and follow instructions to create a Microsoft Azure tenant.
- [Microsoft Azure Client ID/Secret](https://docs.microsoft.com/en-us/azure/active-directory/develop/howto-create-service-principal-portal): Use this link and follow instructions to create a Microsoft Azure client and secret.
- [Terraform](https://www.terraform.io/downloads.html): Used to provision the server and cluster in Microsoft Azure.
-
-
-## Getting Started
-
-1. Clone [Rancher Quickstart](https://github.com/rancher/quickstart) to a folder using `git clone https://github.com/rancher/quickstart`.
-
-1. Go into the Azure folder containing the terraform files by executing `cd quickstart/azure`.
-
-1. Rename the `terraform.tfvars.example` file to `terraform.tfvars`.
-
-1. Edit `terraform.tfvars` and customize the following variables:
-    - `azure_subscription_id` - Microsoft Azure Subscription ID
-    - `azure_client_id` - Microsoft Azure Client ID
-    - `azure_client_secret` - Microsoft Azure Client Secret
-    - `azure_tenant_id` - Microsoft Azure Tenant ID
-    - `rancher_server_admin_password` - Admin password for created Rancher server
-
-2. **Optional:** Modify optional variables within `terraform.tfvars`.
-See the [Quickstart Readme](https://github.com/rancher/quickstart) and the [Azure Quickstart Readme](https://github.com/rancher/quickstart/tree/master/azure) for more information.
-Suggestions include:
-    - `azure_location` - Microsoft Azure region, choose the closest instead of the default
-    - `prefix` - Prefix for all created resources
-    - `instance_type` - Compute instance size used, minimum is `Standard_DS2_v2` but `Standard_DS2_v3` or `Standard_DS3_v2` could be used if within budget
-
-1. Run `terraform init`.
-
-1. To initiate the creation of the environment, run `terraform apply --auto-approve`. Then wait for output similar to the following:
-
-    ```
-    Apply complete! Resources: 16 added, 0 changed, 0 destroyed.
-
-    Outputs:
-
-    rancher_node_ip = xx.xx.xx.xx
-    rancher_server_url = https://rancher.xx.xx.xx.xx.sslip.io
-    workload_node_ip = yy.yy.yy.yy
-    ```
-
-1. Paste the `rancher_server_url` from the output above into the browser. Log in when prompted (default username is `admin`, use the password set in `rancher_server_admin_password`).
-2. ssh to the Rancher Server using the `id_rsa` key generated in `quickstart/azure`.
-#### Result
-
-Two Kubernetes clusters are deployed into your Azure account, one running Rancher Server and the other ready for experimentation deployments.
-
-### What's Next?
-
-Use Rancher to create a deployment. For more information, see [Creating Deployments]({{< baseurl >}}/rancher/v2.5/en/quick-start-guide/workload).
-
-## Destroying the Environment
-
-1. From the `quickstart/azure` folder, execute `terraform destroy --auto-approve`.
-
-2. Wait for confirmation that all resources have been destroyed.
+---
+title: Rancher Azure Quick Start Guide
+description: Read this step by step Rancher Azure guide to quickly deploy a Rancher server with a single-node downstream Kubernetes cluster attached.
+weight: 100
+---
+
+The following steps will quickly deploy a Rancher server on Azure in a single-node K3s Kubernetes cluster, with a single-node downstream Kubernetes cluster attached.
+
+## Prerequisites
+
+>**Note**
+>Deploying to Microsoft Azure will incur charges.
+
+- [Microsoft Azure Account](https://azure.microsoft.com/en-us/free/): A Microsoft Azure Account is required to create resources for deploying Rancher and Kubernetes.
+- [Microsoft Azure Subscription](https://docs.microsoft.com/en-us/azure/cost-management-billing/manage/create-subscription#create-a-subscription-in-the-azure-portal): Use this link to follow a tutorial to create a Microsoft Azure subscription if you don't have one yet.
+- [Micsoroft Azure Tenant](https://docs.microsoft.com/en-us/azure/active-directory/develop/quickstart-create-new-tenant): Use this link and follow instructions to create a Microsoft Azure tenant.
+- [Microsoft Azure Client ID/Secret](https://docs.microsoft.com/en-us/azure/active-directory/develop/howto-create-service-principal-portal): Use this link and follow instructions to create a Microsoft Azure client and secret.
+- [Terraform](https://www.terraform.io/downloads.html): Used to provision the server and cluster in Microsoft Azure.
+
+
+## Getting Started
+
+1. Clone [Rancher Quickstart](https://github.com/rancher/quickstart) to a folder using `git clone https://github.com/rancher/quickstart`.
+
+2. Go into the Azure folder containing the terraform files by executing `cd quickstart/azure`.
+
+3. Rename the `terraform.tfvars.example` file to `terraform.tfvars`.
+
+4. Edit `terraform.tfvars` and customize the following variables:
+    - `azure_subscription_id` - Microsoft Azure Subscription ID 
+    - `azure_client_id` - Microsoft Azure Client ID
+    - `azure_client_secret` - Microsoft Azure Client Secret
+    - `azure_tenant_id` - Microsoft Azure Tenant ID
+    - `rancher_server_admin_password` - Admin password for created Rancher server
+
+5. **Optional:** Modify optional variables within `terraform.tfvars`.
+See the [Quickstart Readme](https://github.com/rancher/quickstart) and the [Azure Quickstart Readme](https://github.com/rancher/quickstart/tree/master/azure) for more information.
+Suggestions include:
+    - `azure_location` - Microsoft Azure region, choose the closest instead of the default (`East US`)
+    - `prefix` - Prefix for all created resources
+    - `instance_type` - Compute instance size used, minimum is `Standard_DS2_v2` but `Standard_DS2_v3` or `Standard_DS3_v2` could be used if within budget
+    - `add_windows_node` - If true, an additional Windows worker node is added to the workload cluster
+    - `windows_admin_password` - The admin password of the windows worker node
+    
+6. Run `terraform init`.
+
+7. To initiate the creation of the environment, run `terraform apply --auto-approve`. Then wait for output similar to the following:
+
+    ```
+    Apply complete! Resources: 16 added, 0 changed, 0 destroyed.
+
+    Outputs:
+
+    rancher_node_ip = xx.xx.xx.xx
+    rancher_server_url = https://rancher.xx.xx.xx.xx.sslip.io
+    workload_node_ip = yy.yy.yy.yy
+    ```
+
+8. Paste the `rancher_server_url` from the output above into the browser. Log in when prompted (default username is `admin`, use the password set in `rancher_server_admin_password`).
+9. ssh to the Rancher Server using the `id_rsa` key generated in `quickstart/azure`.
+
+#### Result
+
+Two Kubernetes clusters are deployed into your Azure account, one running Rancher Server and the other ready for experimentation deployments. Please note that while this setup is a great way to explore Rancher functionality, a production setup should follow our high availability setup guidelines. SSH keys for the VMs are auto-generated and stored in the module directory.
+
+### What's Next?
+
+Use Rancher to create a deployment. For more information, see [Creating Deployments]({{< baseurl >}}/rancher/v2.5/en/quick-start-guide/workload).
+
+## Destroying the Environment
+
+1. From the `quickstart/azure` folder, execute `terraform destroy --auto-approve`.
+
+2. Wait for confirmation that all resources have been destroyed.
@@ -33,6 +33,7 @@ On this page, we provide security-related documentation along with resources to
 - [The CIS Benchmark and self-assessment](#the-cis-benchmark-and-self-assessment)
 - [Third-party penetration test reports](#third-party-penetration-test-reports)
 - [Rancher CVEs and resolutions](#rancher-cves-and-resolutions)
+- [Kubernetes Security Best Practices](#kubernetes-security-best-practices)

 ### Running a CIS Security Scan on a Kubernetes Cluster

@@ -86,3 +87,7 @@ Results:
 ### Rancher CVEs and Resolutions

 Rancher is committed to informing the community of security issues in our products. For the list of CVEs (Common Vulnerabilities and Exposures) for issues we have resolved, refer to [this page.](./cve)
+
+### Kubernetes Security Best Practices
+
+For recommendations on securing your Kubernetes cluster, refer to the [Best Practices](./best-practices) guide.
@@ -0,0 +1,8 @@
+---
+title: Kubernetes Security Best Practices
+weight: 5
+---
+
+# Restricting cloud metadata API access
+
+Cloud providers such as AWS, Azure, or GCP often expose metadata services locally to instances. By default, this endpoint is accessible by pods running on a cloud instance, including pods in hosted Kubernetes providers such as EKS, AKS or GKE, and can contain cloud credentials for that node, provisioning data such as kubelet credentials, or other sensitive data. To mitigate this risk when running on a cloud platform, follow the [Kubernetes security recommendations](https://kubernetes.io/docs/tasks/administer-cluster/securing-a-cluster/#restricting-cloud-metadata-api-access): limit permissions given to instance credentials, use network policies to restrict pod access to the metadata API, and avoid using provisioning data to deliver secrets.
@@ -160,7 +160,7 @@ kubectl run -i --restart=Never --rm test-${RANDOM} --image=ubuntu --overrides='{
 Enabling query logging can be done by enabling the [log plugin](https://coredns.io/plugins/log/) in the Corefile configuration in the configmap `coredns`. You can do so by using `kubectl -n kube-system edit configmap coredns` or use the command below to replace the configuration in place:

 ```
-kubectl get configmap -n kube-system coredns -o json |  kubectl get configmap -n kube-system coredns -o json | sed -e 's_loadbalance_log\\n    loadbalance_g' | kubectl apply -f -
+kubectl get configmap -n kube-system coredns -o json | sed -e 's_loadbalance_log\\n    loadbalance_g' | kubectl apply -f -
 ```

 All queries will now be logged and can be checked using the command in [Check CoreDNS logging](#check-coredns-logging).
@@ -3,7 +3,7 @@ title: Configuring a Global Default Private Registry
 weight: 40
 ---

-You might want to use a private Docker registry to share your custom base images within your organization. With a private registry, you can keep a private, consistent, and centralized source of truth for the container images that are used in your clusters.
+You might want to use a private container registry to share your custom base images within your organization. With a private registry, you can keep a private, consistent, and centralized source of truth for the container images that are used in your clusters.

 There are two main ways to set up private registries in Rancher: by setting up the global default registry through the **Settings** tab in the global view, and by setting up a private registry in the advanced options in the cluster-level settings. The global default registry is intended to be used for air-gapped setups, for registries that do not require credentials. The cluster-level private registry is intended to be used in all setups in which the private registry requires credentials.

@@ -35,4 +35,4 @@ You can follow these steps to configure a private registry when you create a clu

 **Result:** The new cluster will be able to pull images from the private registry.

-The private registry cannot be configured after the cluster is created.
+The private registry cannot be configured after the cluster is created.
@@ -5,7 +5,7 @@ weight: 24

 ## How to use the API

-The API has its own user interface accessible from a web browser.  This is an easy way to see resources, perform actions, and see the equivalent cURL or HTTP request & response.  To access it, click on your user avatar in the upper right corner. Under **API & Keys**, you can find the URL endpoint as well as create [API keys]({{<baseurl>}}/rancher/v2.6/en/user-settings/api-keys/).
+The API has its own user interface accessible from a web browser.  This is an easy way to see resources, perform actions, and see the equivalent cURL or HTTP request & response.  To access it, click on your user avatar in the upper right corner. Under **API & Keys**, you can find the URL endpoint (`https://<rancher_fqdn>/v3`) as well as create [API keys]({{<baseurl>}}/rancher/v2.6/en/user-settings/api-keys/).

 ## Authentication

@@ -63,4 +63,4 @@ You can use browser developer tools to capture how the Rancher API is called. Fo
 1. On the **Network** tab, make sure **Fetch/XHR** is selected.
 1. In the Rancher UI, click **Create**. In the developer tools, you should see a new network request with the name `cluster?_replace=true`.
 1. Right-click `cluster?_replace=true` and click **Copy > Copy as cURL.**
-1. Paste the result into any text editor. You will be able to see the POST request, including the URL it was sent to, all of the headers, and the full body of the request. This command can be used to create a cluster from the command line. Note: The request should be stored in a safe place because it contains credentials.
+1. Paste the result into any text editor. You will be able to see the POST request, including the URL it was sent to, all of the headers, and the full body of the request. This command can be used to create a cluster from the command line. Note: The request should be stored in a safe place because it contains credentials.
@@ -43,6 +43,8 @@ When a Restore custom resource is created, the operator accesses the backup .tar

 The Backup and Restore custom resources can be created in the Rancher UI, or by using `kubectl apply`.

+>**Note:** Refer [here]({{<baseurl>}}/rancher/v2.6/en/backups/migrating-rancher/#2-restore-from-backup-using-a-restore-custom-resource) for help on restoring an existing backup file into a v1.22 cluster in Rancher v2.6.3.
+
 # Installing the rancher-backup Operator

 The `rancher-backup` operator can be installed from the Rancher UI, or with the Helm CLI. In both cases, the `rancher-backup` Helm chart is installed on the Kubernetes cluster running the Rancher server. It is a cluster-admin only feature and available only for the **local** cluster.  (*If you do not see `rancher-backup` in the Rancher UI, you may have selected the wrong cluster.*)
@@ -5,12 +5,18 @@ weight: 1

 In this section, you'll learn how to back up Rancher running on any Kubernetes cluster. To backup Rancher installed with Docker, refer the instructions for [single node backups]({{<baseurl>}}/rancher/v2.6/en/backups/docker-installs/docker-backups)

-The backup-restore operator needs to be installed in the local cluster, and only backs up the Rancher app. The backup and restore operations are performed only in the local Kubernetes cluster.
+The backup-restore operator needs to be installed in the local cluster, and only backs up the Rancher app. The backup and restore operations are performed only in the local Kubernetes cluster. 
+
+Note that the rancher-backup operator version 2.x.x is for Rancher v2.6.x.
+
+> When restoring a backup into a new Rancher setup, the version of the new setup should be the same as the one where the backup is made. The Kubernetes version should also be considered when restoring a backup, since the supported apiVersion in the cluster and in the backup file could be different.

 ### Prerequisites

 The Rancher version must be v2.5.0 and up.

+Refer [here]({{<baseurl>}}/rancher/v2.6/en/backups/migrating-rancher/#2-restore-from-backup-using-a-restore-custom-resource) for help on restoring an existing backup file into a v1.22 cluster in Rancher v2.6.3.
+
 ### 1. Install the Rancher Backups operator

 The backup storage location is an operator-level setting, so it needs to be configured when the Rancher Backups application is installed or upgraded.
@@ -29,6 +29,13 @@ helm install rancher-backup rancher-charts/rancher-backup -n cattle-resources-sy

 ### 2. Restore from backup using a Restore custom resource

+>**Important:** Kubernetes v1.22, available as an experimental feature of v2.6.3, does not support restoring from backup files containing CRDs with the apiVersion `apiextensions.k8s.io/v1beta1`. In v1.22, the default `resourceSet` in the rancher-backup app is updated to collect only CRDs that use `apiextensions.k8s.io/v1`. There are currently two ways to work around this issue:
+>
+1. Update the default `resourceSet` to collect the CRDs with the apiVersion v1.
+1. Update the default `resourceSet` and the client to use the new APIs internally, with `apiextensions.k8s.io/v1` as the replacement.
+>
+> - Note that when making or restoring backups for v1.22, the Rancher version and the local cluster's Kubernetes version should be the same. The Kubernetes version should be considered when restoring a backup since the supported apiVersion in the cluster and in the backup file could be different.
+
 If you are using an S3 store as the backup source, and need to use your S3 credentials for restore, create a secret in this cluster using your S3 credentials. The Secret data must have two keys, `accessKey` and `secretKey` containing the s3 credentials like this:

 ```yaml
@@ -68,20 +75,19 @@ spec:
      endpoint: s3.us-west-2.amazonaws.com
 ```

-> **Important:** The field `encryptionConfigSecretName` must be set only if your backup was created with encryption enabled. Provide the name of the Secret containing the encryption config file. If you only have the encryption config file, but don't have a secret created with it in this cluster, use the following steps to create the secret:  
-1. The encryption configuration file must be named `encryption-provider-config.yaml`, and the `--from-file` flag must be used to create this secret. So save your `EncryptionConfiguration` in a file called `encryption-provider-config.yaml` and run this command:
+>**Important:** The field `encryptionConfigSecretName` must be set only if your backup was created with encryption enabled. Provide the name of the Secret containing the encryption config file. If you only have the encryption config file, but don't have a secret created with it in this cluster, use the following steps to create the secret:  

-```
-kubectl create secret generic encryptionconfig \
-  --from-file=./encryption-provider-config.yaml \
-  -n cattle-resources-system
-```
-
-Then apply the resource:
-
-```
-kubectl apply -f migrationResource.yaml 
-```
+1. The encryption configuration file must be named `encryption-provider-config.yaml`, and the `--from-file` flag must be used to create this secret. So save your `EncryptionConfiguration` in a file called `encryption-provider-config.yaml` and run this command:   
+    ```
+    kubectl create secret generic encryptionconfig \
+      --from-file=./encryption-provider-config.yaml \
+      -n cattle-resources-system
+    ```
+ 
+1. Then apply the resource:
+    ```
+    kubectl apply -f migrationResource.yaml 
+    ```

 ### 3. Install cert-manager

@@ -12,9 +12,7 @@ Use the navigation bar on the left to find the current best practices for managi
 For more guidance on best practices, you can consult these resources:

 - [Security]({{<baseurl>}}/rancher/v2.6/en/security/)
- [Rancher Blog](https://rancher.com/blog/)
-    - [Articles about best practices on the Rancher blog](https://rancher.com/tags/best-practices/)
-    - [101 More Security Best Practices for Kubernetes](https://rancher.com/blog/2019/2019-01-17-101-more-kubernetes-security-best-practices/)
+- [Rancher Blog](https://www.suse.com/c/rancherblog/)
 - [Rancher Forum](https://forums.rancher.com/)
 - [Rancher Users Slack](https://slack.rancher.io/)
 - [Rancher Labs YouTube Channel - Online Meetups, Demos, Training, and Webinars](https://www.youtube.com/channel/UCh5Xtp82q8wjijP8npkVTBA/featured)
@@ -50,7 +50,7 @@ https://rancher.com/docs/rancher/v2.6/en/installation/requirements/

 ### Leverage VM Templates to Construct the Environment

-To facilitate consistency across the deployed Virtual Machines across the environment, consider the use of "Golden Images" in the form of VM templates. Packer can be used to accomplish this, adding greater customisation options.
+To facilitate the consistency of Virtual Machines deployed across the environment, consider the use of "Golden Images" in the form of VM templates. Packer can be used to accomplish this, adding greater customization options.

 ### Leverage DRS Anti-Affinity Rules (Where Possible) to Separate Rancher Cluster Nodes Across ESXi Hosts

@@ -24,6 +24,7 @@ Resource targeting uses `*` as the ARN of many of the resources created cannot b
                "ec2:RunInstances",
                "ec2:RevokeSecurityGroupIngress",
                "ec2:RevokeSecurityGroupEgress",
+                "ec2:DescribeRegions",
                "ec2:DescribeVpcs",
                "ec2:DescribeTags",
                "ec2:DescribeSubnets",
@@ -123,31 +124,6 @@ Resource targeting uses `*` as the ARN of many of the resources created cannot b

 ### Service Role Permissions

-Rancher will create a service role with the following trust policy:
-
-```json
-{
-  "Version": "2012-10-17",
-  "Statement": [
-    {
-      "Action": "sts:AssumeRole",
-      "Principal": {
-        "Service": "eks.amazonaws.com"
-      },
-      "Effect": "Allow",
-      "Sid": ""
-    }
-  ]
-}
-```
-
-This role will also have two role policy attachments with the following policies ARNs:
-
-```
-arn:aws:iam::aws:policy/AmazonEKSClusterPolicy
-arn:aws:iam::aws:policy/AmazonEKSServicePolicy
-```
-
 Permissions required for Rancher to create service role on users behalf during the EKS cluster creation process.

 ```json
@@ -182,36 +158,66 @@ Permissions required for Rancher to create service role on users behalf during t
 }
 ```

+When an EKS cluster is created, Rancher will create a service role with the following trust policy:
+
+```json
+{
+  "Version": "2012-10-17",
+  "Statement": [
+    {
+      "Action": "sts:AssumeRole",
+      "Principal": {
+        "Service": "eks.amazonaws.com"
+      },
+      "Effect": "Allow",
+      "Sid": ""
+    }
+  ]
+}
+```
+
+This role will also have two role policy attachments with the following policies ARNs:
+
+```
+arn:aws:iam::aws:policy/AmazonEKSClusterPolicy
+arn:aws:iam::aws:policy/AmazonEKSServicePolicy
+```
+
 ### VPC Permissions

 Permissions required for Rancher to create VPC and associated resources.

 ```json
 {
-  "Sid": "VPCPermissions",
-  "Effect": "Allow",
-  "Action": [
-     "ec2:ReplaceRoute",
-     "ec2:ModifyVpcAttribute",
-     "ec2:ModifySubnetAttribute",
-     "ec2:DisassociateRouteTable",
-     "ec2:DetachInternetGateway",
-     "ec2:DescribeVpcs",
-     "ec2:DeleteVpc",
-     "ec2:DeleteTags",
-     "ec2:DeleteSubnet",
-     "ec2:DeleteRouteTable",
-     "ec2:DeleteRoute",
-     "ec2:DeleteInternetGateway",
-     "ec2:CreateVpc",
-     "ec2:CreateSubnet",
-     "ec2:CreateSecurityGroup",
-     "ec2:CreateRouteTable",
-     "ec2:CreateRoute",
-     "ec2:CreateInternetGateway",
-     "ec2:AttachInternetGateway",
-     "ec2:AssociateRouteTable"
-  ],
-  "Resource": "*"
+  "Version": "2012-10-17",
+  "Statement": [
+    {
+      "Sid": "VPCPermissions",
+      "Effect": "Allow",
+      "Action": [
+        "ec2:ReplaceRoute",
+        "ec2:ModifyVpcAttribute",
+        "ec2:ModifySubnetAttribute",
+        "ec2:DisassociateRouteTable",
+        "ec2:DetachInternetGateway",
+        "ec2:DescribeVpcs",
+        "ec2:DeleteVpc",
+        "ec2:DeleteTags",
+        "ec2:DeleteSubnet",
+        "ec2:DeleteRouteTable",
+        "ec2:DeleteRoute",
+        "ec2:DeleteInternetGateway",
+        "ec2:CreateVpc",
+        "ec2:CreateSubnet",
+        "ec2:CreateSecurityGroup",
+        "ec2:CreateRouteTable",
+        "ec2:CreateRoute",
+        "ec2:CreateInternetGateway",
+        "ec2:AttachInternetGateway",
+        "ec2:AssociateRouteTable"
+      ],
+      "Resource": "*"
+    }
+  ]
 }
-```
+```
@@ -30,6 +30,13 @@ For information on how to install Docker, refer to the official [Docker document

 Some distributions of Linux derived from RHEL, including Oracle Linux, may have default firewall rules that block communication with Helm. We recommend disabling firewalld. For Kubernetes 1.19, firewalld must be turned off.

+>**Note:** In RHEL 8.4, two extra services are included on the NetworkManager: `nm-cloud-setup.service` and `nm-cloud-setup.timer`. These services add a routing table that interferes with the CNI plugin's configuration. If these services are enabled, you must disable them using the command below, and then reboot the node to restore connectivity:
+>  
+>  ```
+   systemctl disable nm-cloud-setup.service nm-cloud-setup.timer
+   reboot
+   ```
+
 ### SUSE Linux Nodes

 SUSE Linux may have a firewall that blocks all ports by default. In that situation, follow [these steps]({{<baseurl>}}/rancher/v2.6/en/installation/requirements/ports/#opening-suse-linux-ports) to open the ports needed for adding a host to a custom cluster.
@@ -12,6 +12,7 @@ The control that Rancher has to manage a registered cluster depends on the type
 - [Management Capabilities for Registered Clusters](#management-capabilities-for-registered-clusters)
 - [Configuring K3s Cluster Upgrades](#configuring-k3s-cluster-upgrades)
 - [Debug Logging and Troubleshooting for Registered K3s Clusters](#debug-logging-and-troubleshooting-for-registered-k3s-clusters)
+- [Authorized Cluster Endpoint Support for RKE2 and K3s Clusters](#authorized-cluster-endpoint-support-for-rke2-and-k3s-clusters)
 - [Annotating Registered Clusters](#annotating-registered-clusters)

 # Prerequisites
@@ -155,8 +156,51 @@ If the cluster becomes stuck in upgrading, restart the `system-upgrade-controlle

 To prevent issues when upgrading, the [Kubernetes upgrade best practices](https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/) should be followed.

+# Authorized Cluster Endpoint Support for RKE2 and K3s Clusters

+_Available as of v2.6.3_

+Authorized Cluster Endpoint (ACE) support has been added for registered RKE2 and K3s clusters. This support includes manual steps you will perform on the downstream cluster to enable the ACE. For additional information on the authorized cluster endpoint, click [here]({{<baseurl>}}/rancher/v2.6/en/cluster-admin/cluster-access/ace/).
+
+> **Note:**
+>
+> - These steps only need to be performed on the control plane nodes of the downstream cluster. You must configure each control plane node individually. 
+>
+> - The following steps will work on both RKE2 and K3s clusters registered in v2.6.x as well as those registered (or imported) from a previous version of Rancher with an upgrade to v2.6.x.
+>
+> - These steps will alter the configuration of the downstream RKE2 and K3s clusters and deploy the `kube-api-authn-webhook`. If a future implementation of ACE requires an update to the `kube-api-authn-webhook`, then this would also have to be done manually. For more information on this webhook, click [here]({{<baseurl>}}/rancher/v2.6/en/cluster-admin/cluster-access/ace/#about-the-kube-api-auth-authentication-webhook).
+
+###### **Manual steps to be taken on the control plane of each downstream cluster to enable ACE:**
+
+1. Create a file at `/var/lib/rancher/{rke2,k3s}/kube-api-authn-webhook.yaml` with the following contents:
+
+        apiVersion: v1
+        kind: Config
+        clusters:
+        - name: Default
+          cluster:
+            insecure-skip-tls-verify: true
+            server: http://127.0.0.1:6440/v1/authenticate
+        users:
+        - name: Default
+          user:
+            insecure-skip-tls-verify: true
+        current-context: webhook
+        contexts:
+        - name: webhook
+          context:
+            user: Default
+            cluster: Default
+        
+1. Add the following to the config file (or create one if it doesn’t exist); note that the default location is `/etc/rancher/{rke2,k3s}/config.yaml`:
+
+        kube-apiserver-arg:
+            - authentication-token-webhook-config-file=/var/lib/rancher/{rke2,k3s}/kube-api-authn-webhook.yaml
+        
+1. Finally, run the following commands:
+
+        sudo systemctl stop {rke2,k3s}-server
+        sudo systemctl start {rke2,k3s}-server

 # Annotating Registered Clusters

@@ -85,6 +85,32 @@ If you are configuring DHCP options sets for an AWS virtual private cloud, note

 > Some Linux operating systems accept multiple domain names separated by spaces. However, other Linux operating systems and Windows treat the value as a single domain, which results in unexpected behavior. If your DHCP options set is associated with a VPC that has instances with multiple operating systems, specify only one domain name.

+### Rancher on vSphere with ESXi 6.7u2 and above
+
+If you are using Rancher on VMware vSphere with ESXi 6.7u2 or later with Red Hat Enterprise Linux 8.3, CentOS 8.3, or SUSE Enterprise Linux 15 SP2 or later, it is necessary to disable the `vmxnet3` virtual network adapter hardware offloading feature. Failure to do so will result in all network connections between pods on different cluster nodes to fail with timeout errors. All connections from Windows pods to critical services running on Linux nodes, such as CoreDNS, will fail as well. It is also possible that external connections may fail. This issue is the result of Linux distributions enabling the hardware offloading feature in `vmxnet3` and a bug in the `vmxnet3` hardware offloading feature that results in the discarding of packets for guest overlay traffic. To address this issue, it is necessary disable the `vmxnet3` hardware offloading feature. This setting does not survive reboot, so it is necessary to disable on every boot. The recommended course of action is to create a systemd unit file at `/etc/systemd/system/disable_hw_offloading.service`, which disables the `vmxnet3` hardware offloading feature on boot. A sample systemd unit file which disables the `vmxnet3` hardware offloading feature is as follows. Note that `<VM network interface>` must be customized to the host `vmxnet3` network interface, e.g., `ens192`:
+
+```
+[Unit]
+Description=Disable vmxnet3 hardware offloading feature
+
+[Service]
+Type=oneshot
+ExecStart=ethtool -K <VM network interface> tx-udp_tnl-segmentation off
+ExecStart=ethtool -K <VM network interface> tx-udp_tnl-csum-segmentation off
+StandardOutput=journal
+
+[Install]
+WantedBy=multi-user.target
+```
+Then set the appropriate permissions on the systemd unit file:
+```
+chmod 0644 /etc/systemd/system/disable_hw_offloading.service
+```
+Finally, enable the systemd service:
+```
+systemctl enable disable_hw_offloading.service
+```
+
 ### Architecture Requirements

 The Kubernetes cluster management nodes (`etcd` and `controlplane`) must be run on Linux nodes.
@@ -14,11 +14,11 @@ Kubernetes uses CNI as an interface between network providers and Kubernetes pod

 For more information visit [CNI GitHub project](https://github.com/containernetworking/cni).

-### What Network Models are Used in CNI?
+## What Network Models are Used in CNI?

-CNI network providers implement their network fabric using either an encapsulated network model such as Virtual Extensible Lan ([VXLAN](https://github.com/coreos/flannel/blob/master/Documentation/backends.md#vxlan)) or an unencapsulated network model such as Border Gateway Protocol ([BGP](https://en.wikipedia.org/wiki/Border_Gateway_Protocol)).
+CNI network providers implement their network fabric using either an encapsulated network model such as Virtual Extensible Lan ([VXLAN](https://github.com/flannel-io/flannel/blob/master/Documentation/backends.md#vxlan)) or an unencapsulated network model such as Border Gateway Protocol ([BGP](https://en.wikipedia.org/wiki/Border_Gateway_Protocol)).

-#### What is an Encapsulated Network?
+### What is an Encapsulated Network?

 This network model provides a logical Layer 2 (L2) network encapsulated over the existing Layer 3 (L3) network topology that spans the Kubernetes cluster nodes. With this model you have an isolated L2 network for containers without needing routing distribution, all at the cost of minimal overhead in terms of processing and increased IP package size, which comes from an IP header generated by overlay encapsulation. Encapsulation information is distributed by UDP ports between Kubernetes workers, interchanging network control plane information about how MAC addresses can be reached. Common encapsulation used in this kind of network model is VXLAN, Internet Protocol Security (IPSec), and IP-in-IP.

@@ -26,11 +26,11 @@ In simple terms, this network model generates a kind of network bridge extended

 This network model is used when an extended L2 bridge is preferred. This network model is sensitive to L3 network latencies of the Kubernetes workers. If datacenters are in distinct geolocations, be sure to have low latencies between them to avoid eventual network segmentation.

-CNI network providers using this network model include Flannel, Canal, and Weave.
+CNI network providers using this network model include Flannel, Canal, Weave, and Cilium. By default, Calico is not using this model, but it can be configured to do so.

 ![Encapsulated Network]({{<baseurl>}}/img/rancher/encapsulated-network.png)

-#### What is an Unencapsulated Network?
+### What is an Unencapsulated Network?

 This network model provides an L3 network to route packets between containers. This model doesn't generate an isolated l2 network, nor generates overhead. These benefits come at the cost of Kubernetes workers having to manage any route distribution that's needed. Instead of using IP headers for encapsulation, this network model uses a network protocol between Kubernetes workers to distribute routing information to reach pods, such as [BGP](https://en.wikipedia.org/wiki/Border_Gateway_Protocol).

@@ -38,13 +38,17 @@ In simple terms, this network model generates a kind of network router extended

 This network model is used when a routed L3 network is preferred. This mode dynamically updates routes at the OS level for Kubernetes workers. It's less sensitive to latency.

-CNI network providers using this network model include Calico and Romana.
+CNI network providers using this network model include Calico and Cilium. Cilium may be configured with this model although it is not the default mode.

 ![Unencapsulated Network]({{<baseurl>}}/img/rancher/unencapsulated-network.png)

-### What CNI Providers are Provided by Rancher?
+## What CNI Providers are Provided by Rancher?

-Out-of-the-box, Rancher provides the following CNI network providers for Kubernetes clusters: Canal, Flannel, Calico and Weave. You can choose your CNI network provider when you create new Kubernetes clusters from Rancher.
+### RKE Kubernetes clusters
+
+Out-of-the-box, Rancher provides the following CNI network providers for RKE Kubernetes clusters: Canal, Flannel, and Weave. 
+
+You can choose your CNI network provider when you create new Kubernetes clusters from Rancher.

 #### Canal

@@ -64,33 +68,18 @@ For more information, see the [Canal GitHub Page.](https://github.com/projectcal

 ![Flannel Logo]({{<baseurl>}}/img/rancher/flannel-logo.png)

-Flannel is a simple and easy way to configure L3 network fabric designed for Kubernetes. Flannel runs a single binary agent named flanneld on each host, which is responsible for allocating a subnet lease to each host out of a larger, preconfigured address space. Flannel uses either the Kubernetes API or etcd directly to store the network configuration, the allocated subnets, and any auxiliary data (such as the host's public IP). Packets are forwarded using one of several backend mechanisms, with the default encapsulation being [VXLAN](https://github.com/coreos/flannel/blob/master/Documentation/backends.md#vxlan).
+Flannel is a simple and easy way to configure L3 network fabric designed for Kubernetes. Flannel runs a single binary agent named flanneld on each host, which is responsible for allocating a subnet lease to each host out of a larger, preconfigured address space. Flannel uses either the Kubernetes API or etcd directly to store the network configuration, the allocated subnets, and any auxiliary data (such as the host's public IP). Packets are forwarded using one of several backend mechanisms, with the default encapsulation being [VXLAN](https://github.com/flannel-io/flannel/blob/master/Documentation/backends.md#vxlan).

-Encapsulated traffic is unencrypted by default. Therefore, flannel provides an experimental backend for encryption, [IPSec](https://github.com/coreos/flannel/blob/master/Documentation/backends.md#ipsec), which makes use of [strongSwan](https://www.strongswan.org/) to establish encrypted IPSec tunnels between Kubernetes workers.
+Encapsulated traffic is unencrypted by default. Flannel provides two solutions for encryption:
+
+* [IPSec](https://github.com/flannel-io/flannel/blob/master/Documentation/backends.md#ipsec), which makes use of [strongSwan](https://www.strongswan.org/) to establish encrypted IPSec tunnels between Kubernetes workers. It is an experimental backend for encryption.
+* [WireGuard](https://github.com/flannel-io/flannel/blob/master/Documentation/backends.md#wireguard), which is a more faster-performing alternative to strongSwan.

 Kubernetes workers should open UDP port `8472` (VXLAN) and TCP port `9099` (healthcheck). See [the port requirements for user clusters]({{<baseurl>}}/rancher/v2.6/en/cluster-provisioning/node-requirements/#networking-requirements) for more details.

 ![Flannel Diagram]({{<baseurl>}}/img/rancher/flannel-diagram.png)

-For more information, see the [Flannel GitHub Page](https://github.com/coreos/flannel).
-
-#### Calico
-
-![Calico Logo]({{<baseurl>}}/img/rancher/calico-logo.png)
-
-Calico enables networking and network policy in Kubernetes clusters across the cloud. Calico uses a pure, unencapsulated IP network fabric and policy engine to provide networking for your Kubernetes workloads. Workloads are able to communicate over both cloud infrastructure and on-prem using BGP.
-
-Calico also provides a stateless IP-in-IP encapsulation mode that can be used, if necessary. Calico also offers policy isolation, allowing you to secure and govern your Kubernetes workloads using advanced ingress and egress policies.
-
-Kubernetes workers should open TCP port `179` (BGP). See [the port requirements for user clusters]({{<baseurl>}}/rancher/v2.6/en/cluster-provisioning/node-requirements/#networking-requirements) for more details.
-
-![Calico Diagram]({{<baseurl>}}/img/rancher/calico-diagram.svg)
-
-For more information, see the following pages:
-
- [Project Calico Official Site](https://www.projectcalico.org/)
- [Project Calico GitHub Page](https://github.com/projectcalico/calico)
-
+For more information, see the [Flannel GitHub Page](https://github.com/flannel-io/flannel).

 #### Weave

@@ -104,16 +93,48 @@ For more information, see the following pages:

 - [Weave Net Official Site](https://www.weave.works/)

-### CNI Features by Provider
+### RKE2 Kubernetes clusters
+
+Out-of-the-box, Rancher provides the following CNI network providers for RKE2 Kubernetes clusters: [Canal](#canal) (see above section), Calico, and Cilium. 
+
+You can choose your CNI network provider when you create new Kubernetes clusters from Rancher.
+
+#### Calico
+
+![Calico Logo]({{<baseurl>}}/img/rancher/calico-logo.png)
+
+Calico enables networking and network policy in Kubernetes clusters across the cloud. By default, Calico uses a pure, unencapsulated IP network fabric and policy engine to provide networking for your Kubernetes workloads. Workloads are able to communicate over both cloud infrastructure and on-prem using BGP.
+
+Calico also provides a stateless IP-in-IP or VXLAN encapsulation mode that can be used, if necessary. Calico also offers policy isolation, allowing you to secure and govern your Kubernetes workloads using advanced ingress and egress policies.
+
+Kubernetes workers should open TCP port `179` (BGP). See [the port requirements for user clusters]({{<baseurl>}}/rancher/v2.6/en/cluster-provisioning/node-requirements/#networking-requirements) for more details.
+
+![Calico Diagram]({{<baseurl>}}/img/rancher/calico-diagram.svg)
+
+For more information, see the following pages:
+
+- [Project Calico Official Site](https://www.projectcalico.org/)
+- [Project Calico GitHub Page](https://github.com/projectcalico/calico)
+
+#### Cilium
+
+![Cilium Logo]({{<baseurl>}}/img/rancher/cilium-logo.png)
+
+Cilium enables networking and network policies (L3, L4, and L7) in Kubernetes. By default, Cilium uses eBPF technologies to route packets inside the node and VXLAN to send packets to other nodes. Unencapsulated techniques can also be configured. 
+
+Cilium recommends kernel versions greater than 5.2 to be able to leverage the full potential of eBPF. Kubernetes workers should open TCP port `8472` for VXLAN and TCP port `4140` for health checks. In addition, ICMP 8/0 must be enabled for health checks. For more information, check [Cilium System Requirements](https://docs.cilium.io/en/latest/operations/system_requirements/#firewall-requirements).
+
+## CNI Features by Provider

 The following table summarizes the different features available for each CNI network provider provided by Rancher.

 | Provider | Network Model | Route Distribution | Network Policies | Mesh | External Datastore | Encryption | Ingress/Egress Policies |
 | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- |
-| Canal | Encapsulated (VXLAN) | No | Yes | No | K8S API | No | Yes |
-| Flannel | Encapsulated (VXLAN) | No | No | No | K8S API | No | No |
-| Calico | Encapsulated (VXLAN,IPIP) OR Unencapsulated | Yes | Yes | Yes | Etcd and K8S API | No | Yes |
+| Canal | Encapsulated (VXLAN) | No | Yes | No | K8s API | Yes | Yes |
+| Flannel | Encapsulated (VXLAN) | No | No | No | K8s API | Yes | No |
+| Calico | Encapsulated (VXLAN,IPIP) OR Unencapsulated | Yes | Yes | Yes | Etcd and K8s API | Yes | Yes |
 | Weave | Encapsulated | Yes | Yes | Yes | No | Yes | Yes |
+| Cilium | Encapsulated (VXLAN) | Yes | Yes | Yes | Etcd and K8s API | Yes | Yes |

 - Network Model: Encapsulated or unencapsulated. For more information, see [What Network Models are Used in CNI?](#what-network-models-are-used-in-cni)

@@ -129,25 +150,27 @@ The following table summarizes the different features available for each CNI net

 - Ingress/Egress Policies: This feature allows you to manage routing control for both Kubernetes and non-Kubernetes communications.

-#### CNI Community Popularity

-The following table summarizes different GitHub metrics to give you an idea of each project's popularity and activity. This data was collected in January 2020.
+## CNI Community Popularity
+
+The following table summarizes different GitHub metrics to give you an idea of each project's popularity and activity. This data was collected in January 2022.

 | Provider | Project | Stars | Forks | Contributors |
 | ---- | ---- | ---- | ---- | ---- |
-| Canal | https://github.com/projectcalico/canal | 614 | 89 | 19 |
-| flannel | https://github.com/coreos/flannel | 4977 | 1.4k | 140 |
-| Calico | https://github.com/projectcalico/calico | 1534 | 429 | 135 |
-| Weave | https://github.com/weaveworks/weave/ | 5737 | 559 | 73 |
+| Canal | https://github.com/projectcalico/canal | 679 | 100 | 21 |
+| Flannel | https://github.com/flannel-io/flannel | 7k | 2.5k | 185 |
+| Calico | https://github.com/projectcalico/calico | 3.1k | 741 | 224 |
+| Weave | https://github.com/weaveworks/weave/ | 6.2k | 635 | 84 |
+| Cilium | https://github.com/cilium/cilium | 10.6k | 1.3k | 352 |

 <br/>

-### Which CNI Provider Should I Use?
+## Which CNI Provider Should I Use?

 It depends on your project needs. There are many different providers, which each have various features and options. There isn't one provider that meets everyone's needs.

 Canal is the default CNI network provider. We recommend it for most use cases. It provides encapsulated networking for containers with Flannel, while adding Calico network policies that can provide project/namespace isolation in terms of networking.

-### How can I configure a CNI network provider?
+## How can I configure a CNI network provider?

 Please see [Cluster Options]({{<baseurl>}}/rancher/v2.6/en/cluster-provisioning/rke-clusters/options/) on how to configure a network provider for your cluster. For more advanced configuration options, please see how to configure your cluster using a [Config File]({{<baseurl>}}/rancher/v2.6/en/cluster-provisioning/rke-clusters/options/#cluster-config-file) and the options for [Network Plug-ins]({{<baseurl>}}/rke/latest/en/config-options/add-ons/network-plugins/).
@@ -5,6 +5,42 @@ weight: 11

 In this section, you'll learn how to manage Helm chart repositories and applications in Rancher. Helm chart repositories are managed using **Apps & Marketplace**. It uses a catalog-like system to import bundles of charts from repositories and then uses those charts to either deploy custom Helm applications or Rancher's tools such as Monitoring or Istio. Rancher tools come as pre-loaded repositories which deploy as standalone Helm charts. Any additional repositories are only added to the current cluster.

+### Changes in Rancher v2.6
+
+Starting in Rancher v2.6.0, a new versioning scheme for Rancher feature charts was implemented. The changes are centered around the major version of the charts and the +up annotation for upstream charts, where applicable.
+
+**Major Version:** The major version of the charts is tied to Rancher minor versions. When you upgrade to a new Rancher minor version, you should ensure that all of your **Apps & Marketplace** charts are also upgraded to the correct release line for the chart. 
+
+>**Note:** Any major versions that are less than the ones mentioned in the table below are meant for 2.5 and below only. For example, you are advised to not use <100.x.x versions of Monitoring in 2.6.x+.
+
+**Feature Charts:**
+
+| **Name** | **Supported Minimum Version** | **Supported Maximum Version** |
+| ---------------- | ------------ | ------------ |
+| external-ip-webhook | 100.0.0+up1.0.0 | 100.0.1+up1.0.1 |
+| harvester-cloud-provider | 100.0.0+up0.1.8 | 100.0.0+up0.1.8 |
+| harvester-csi-driver | 100.0.0+up0.1.9 | 100.0.0+up0.1.9 |
+| rancher-alerting-drivers | 100.0.0 | 100.0.1 |                                                                  
+| rancher-backups | 2.0.0 | 2.1.0 |
+| rancher-cis-benchmark | 2.0.0 | 2.0.2 | 
+| rancher-gatekeeper | 100.0.0+up3.5.1 | 100.0.1+up3.6.0 | 
+| rancher-istio | 100.0.0+up1.10.4 | 100.1.0+up1.11.4 |
+| rancher-logging | 100.0.0+up3.12.0 | 100.0.1+up3.15.0 |
+| rancher-longhorn | 100.0.0+up1.1.2 | 100.1.1+up1.2.3 |                                 
+| rancher-monitoring | 100.0.0+up16.6.0 | 100.1.0+up19.0.3
+| rancher-sriov (experimental) | 100.0.0+up0.1.0 | 100.0.1+up0.1.0 |                                
+| rancher-vsphere-cpi | 100.0.0 | 100.1.0+up1.0.100
+| rancher-vsphere-csi | 100.0.0 | 100.1.0+up2.3.0 |
+| rancher-wins-upgrader | 100.0.0+up0.0.1 | 100.0.0+up0.0.1 |                                                   
+
+</br>
+**Charts based on upstream:** For charts that are based on upstreams, the +up annotation should inform you of what upstream version the Rancher chart is tracking. Check the upstream version compatibility with Rancher during upgrades also.
+
+- As an example, `100.x.x+up16.6.0` for Monitoring tracks upstream kube-prometheus-stack `16.6.0` with some Rancher patches added to it.
+
+- On upgrades, ensure that you are not downgrading the version of the chart that you are using. For example, if you are using a version of Monitoring > `16.6.0` in Rancher 2.5, you should not upgrade to `100.x.x+up16.6.0`. Instead, you should upgrade to the appropriate version in the next release. 
+
+
 ### Charts

 From the top-left menu select _"Apps & Marketplace"_ and you will be taken to the Charts page.
@@ -45,6 +81,19 @@ To add a private CA for Helm Chart repositories:
      insecureSkipTLSVerify: true
    [...]
    ```
+    
+> **Note:** Helm chart repositories with authentication
+>
+> As of Rancher v2.6.3, a new value `disableSameOriginCheck` has been added to the Repo.Spec. This allows users to bypass the same origin checks, sending the repository Authentication information as a Basic Auth Header with all API calls. This is not recommended but can be used as a temporary solution in cases of non-standard Helm chart repositories such as those that have redirects to a different origin URL. 
+>
+> To use this feature for an existing Helm chart repository, click <b>⋮ > Edit YAML</b>. On the `spec` portion of the YAML file, add `disableSameOriginCheck` and set it to `true`.
+>
+> ```yaml
+[...]
+spec:
+  disableSameOriginCheck: true
+[...]
+```

 ### Helm Compatibility

@@ -66,3 +115,13 @@ Most Rancher tools have additional pages located in the toolbar below the _"Apps
 >  * use the default upgrade option ( i.e do not use _"--force"_ option )
 >  * uninstall the existing chart and install the upgraded chart
 >  * delete the resources with immutable fields from the cluster before performing the _"--force"_ upgrade
+
+#### Changes in Rancher v2.6.3
+
+The upgrade button has been removed for legacy apps from the **Apps & Marketplace > Installed Apps** page.
+
+If you have a legacy app installed and want to upgrade it:
+
+- The legacy [feature flag]({{<baseurl>}}/rancher/v2.6/en/installation/resources/feature-flags/) must be turned on (if it's not turned on automatically because of having a legacy app before upgrading)
+- You can upgrade the app from cluster explorer, from the left nav section **Legacy > Project > Apps**
+- For multi-cluster apps, you can go to **≡ > Multi-cluster Apps** and upgrade the app from there
@@ -234,7 +234,7 @@ helm install rancher rancher-<CHART_REPO>/rancher \
 If you are using a Private CA signed certificate , add `--set privateCA=true` to the command:

 ```
-helm install rancher rancher-latest/rancher \
+helm install rancher rancher-<CHART_REPO>/rancher \
  --namespace cattle-system \
  --set hostname=rancher.my.org \
  --set bootstrapPassword=admin \
@@ -16,6 +16,6 @@ When a new version of an application image is released on Docker Hub, you can up

    These options control how the upgrade rolls out to containers that are currently running. For example, for scalable deployments, you can choose whether you want to stop old pods before deploying new ones, or vice versa, as well as the upgrade batch size.

-1. Click **Upgrade**.
+1. Click **Save**.

 **Result:** The workload begins upgrading its containers, per your specifications. Note that scaling up the deployment or updating the upgrade/scaling policy won't result in the pods recreation.
@@ -11,6 +11,8 @@ For information on configuring custom scrape targets and rules for Prometheus, p

 The resource requests and limits for the monitoring application can be configured when installing `rancher-monitoring`. For more information about the default limits, see [this page.](./helm-chart-options/#configuring-resource-limits-and-requests)

+>**Note:** On an idle cluster, Monitoring V2 has significantly higher CPU usage (up to 70%) as compared to Monitoring V1. To improve performance and achieve similar results as in Monitoring V1, turn off the Prometheus adapter. 
+
 # Prometheus Configuration

 It is usually not necessary to directly edit the Prometheus custom resource.
@@ -26,6 +26,6 @@ For more information about how ServiceMonitors work, refer to the [Prometheus Op

 This pseudo-CRD maps to a section of the Prometheus custom resource configuration. It declaratively specifies how group of pods should be monitored. 

-When a PodMonitor is created, the Prometheus Operator updates the Prometheus scrape configuration to include the PodMonitor configuration. Then Prometheus begins scraping metrics from the endpoint defined in the ServiceMonitor.
+When a PodMonitor is created, the Prometheus Operator updates the Prometheus scrape configuration to include the PodMonitor configuration. Then Prometheus begins scraping metrics from the endpoint defined in the PodMonitor.

 Any Pods in your cluster that match the labels located within the PodMonitor `selector` field will be monitored based on the `podMetricsEndpoints` specified on the PodMonitor. For more information on what fields can be specified, please look at the [spec](https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/api.md#podmonitorspec) provided by Prometheus Operator.
@@ -12,7 +12,7 @@ This section covers the following topics:
 - [Recommended Load Balancer Configuration for Kubernetes Installations](#recommended-load-balancer-configuration-for-kubernetes-installations)
 - [Environment for Kubernetes Installations](#environment-for-kubernetes-installations)
 - [Recommended Node Roles for Kubernetes Installations](#recommended-node-roles-for-kubernetes-installations)
- [Architecture for an Authorized Cluster Endpoint](#architecture-for-an-authorized-cluster-endpoint)
+- [Architecture for an Authorized Cluster Endpoint (ACE)](#architecture-for-an-authorized-cluster-endpoint-ace)

 # Separation of Rancher and User Clusters

@@ -104,8 +104,10 @@ Because no additional workloads will be deployed on the Rancher server cluster,

 For more best practices for downstream clusters, refer to the [production checklist]({{<baseurl>}}/rancher/v2.6/en/cluster-provisioning/production) or our [best practices guide.]({{<baseurl>}}/rancher/v2.6/en/best-practices/)

-# Architecture for an Authorized Cluster Endpoint 
+# Architecture for an Authorized Cluster Endpoint (ACE)

-If you are using an [authorized cluster endpoint,]({{<baseurl>}}/rancher/v2.6/en/overview/architecture/#4-authorized-cluster-endpoint) we recommend creating an FQDN pointing to a load balancer which balances traffic across your nodes with the `controlplane` role.
+If you are using an [authorized cluster endpoint (ACE),]({{<baseurl>}}/rancher/v2.6/en/overview/architecture/#4-authorized-cluster-endpoint) we recommend creating an FQDN pointing to a load balancer which balances traffic across your nodes with the `controlplane` role.

 If you are using private CA signed certificates on the load balancer, you have to supply the CA certificate, which will be included in the generated kubeconfig file to validate the certificate chain. See the documentation on [kubeconfig files]({{<baseurl>}}/rancher/v2.6/en/cluster-admin/cluster-access/kubectl/) and [API keys]({{<baseurl>}}/rancher/v2.6/en/user-settings/api-keys/#creating-an-api-key) for more information.
+
+As of Rancher v2.6.3, ACE support is available for registered RKE2 and K3s clusters. To view the manual steps to perform on the downstream cluster to enable the ACE, click [here]({{<baseurl>}}/v2.6/en/cluster-provisioning/registered-clusters/#authorized-cluster-endpoint-support-for-rke2-and-k3s-clusters).
@@ -20,7 +20,7 @@ This section covers the following topics:
  - [The authentication proxy](#1-the-authentication-proxy)
  - [Cluster controllers and cluster agents](#2-cluster-controllers-and-cluster-agents)
  - [Node agents](#3-node-agents)
-  - [Authorized cluster endpoint](#4-authorized-cluster-endpoint)
+  - [Authorized cluster endpoint (ACE)](#4-authorized-cluster-endpoint-ace)
 - [Important files](#important-files)
 - [Tools for provisioning Kubernetes clusters](#tools-for-provisioning-kubernetes-clusters)
 - [Rancher server components and source code](#rancher-server-components-and-source-code)
@@ -103,11 +103,13 @@ If the cluster agent (also called `cattle-cluster-agent`) is not available, one

 The `cattle-node-agent` is deployed using a [DaemonSet](https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/) resource to make sure it runs on every node in a Rancher-launched Kubernetes cluster. It is used to interact with the nodes when performing cluster operations. Examples of cluster operations include upgrading the Kubernetes version and creating or restoring etcd snapshots.

-### 4. Authorized Cluster Endpoint
+### 4. Authorized Cluster Endpoint (ACE)

 An authorized cluster endpoint allows users to connect to the Kubernetes API server of a downstream cluster without having to route their requests through the Rancher authentication proxy.

-> The authorized cluster endpoint only works on Rancher-launched Kubernetes clusters. In other words, it only works in clusters where Rancher [used RKE]({{<baseurl>}}/rancher/v2.6/en/cluster-provisioning/rke-clusters) to provision the cluster. It is not available for registered clusters, or for clusters in a hosted Kubernetes provider, such as Amazon's EKS.
+> The authorized cluster endpoint only works on Rancher-launched Kubernetes clusters. In other words, it only works in clusters where Rancher [used RKE]({{<baseurl>}}/rancher/v2.6/en/cluster-provisioning/rke-clusters) to provision the cluster. The ACE is not available for clusters in a hosted Kubernetes provider, such as Amazon's EKS.
+
+> The [ACE is available for registered RKE2 and K3s clusters]({{<baseurl>}}/v2.6/en/cluster-provisioning/registered-clusters/#authorized-cluster-endpoint-support-for-rke2-and-k3s-clusters) as of Rancher v2.6.3. 

 There are two main reasons why a user might need the authorized cluster endpoint:

@@ -38,4 +38,4 @@ Edit resource quotas when:

 1. Click **Create**.

-**Result:** The resource quota is applied to your project and namespaces. When you add more namespaces in the future, Rancher validates that the project can accommodate the namespace. If the project can't allocate the resources, Rancher won't let you save your changes.
+**Result:** The resource quota is applied to your project and namespaces. When you add more namespaces in the future, Rancher validates that the project can accommodate the namespace. If the project can't allocate the resources, you may still create namespaces, but they will be given a resource quota of 0. Subsequently, Rancher will not allow you to create any resources restricted by this quota.
@@ -1,9 +1,9 @@
 ---
 title: Rancher AWS Quick Start Guide
-description: Read this step by step Rancher AWS guide to quickly deploy a Rancher Server with a single node cluster attached.
+description: Read this step by step Rancher AWS guide to quickly deploy a Rancher server with a single-node downstream Kubernetes cluster attached.
 weight: 100
 ---
-The following steps will quickly deploy a Rancher Server on AWS with a single node cluster attached.
+The following steps will quickly deploy a Rancher server on AWS in a single-node K3s Kubernetes cluster, with a single-node downstream Kubernetes cluster attached.

 ## Prerequisites

@@ -19,25 +19,26 @@ The following steps will quickly deploy a Rancher Server on AWS with a single no

 1. Clone [Rancher Quickstart](https://github.com/rancher/quickstart) to a folder using `git clone https://github.com/rancher/quickstart`.

-1. Go into the AWS folder containing the terraform files by executing `cd quickstart/aws`.
+2. Go into the AWS folder containing the terraform files by executing `cd quickstart/aws`.

-1. Rename the `terraform.tfvars.example` file to `terraform.tfvars`.
+3. Rename the `terraform.tfvars.example` file to `terraform.tfvars`.

-1. Edit `terraform.tfvars` and customize the following variables:
+4. Edit `terraform.tfvars` and customize the following variables:
    - `aws_access_key` - Amazon AWS Access Key 
    - `aws_secret_key` - Amazon AWS Secret Key
    - `rancher_server_admin_password` - Admin password for created Rancher server

-1. **Optional:** Modify optional variables within `terraform.tfvars`.
+5. **Optional:** Modify optional variables within `terraform.tfvars`.
 See the [Quickstart Readme](https://github.com/rancher/quickstart) and the [AWS Quickstart Readme](https://github.com/rancher/quickstart/tree/master/aws) for more information.
 Suggestions include:
-    - `aws_region` - Amazon AWS region, choose the closest instead of the default
+    - `aws_region` - Amazon AWS region, choose the closest instead of the default (`us-east-1`)
    - `prefix` - Prefix for all created resources
    - `instance_type` - EC2 instance size used, minimum is `t3a.medium` but `t3a.large` or `t3a.xlarge` could be used if within budget
+    - `add_windows_node` - If true, an additional Windows worker node is added to the workload cluster

-1. Run `terraform init`.
+6. Run `terraform init`.

-1. To initiate the creation of the environment, run `terraform apply --auto-approve`. Then wait for output similar to the following:
+7. To initiate the creation of the environment, run `terraform apply --auto-approve`. Then wait for output similar to the following:

    ```
    Apply complete! Resources: 16 added, 0 changed, 0 destroyed.
@@ -45,15 +46,16 @@ Suggestions include:
    Outputs:

    rancher_node_ip = xx.xx.xx.xx
-    rancher_server_url = https://rancher.xx.xx.xx.xx.xip.io
+    rancher_server_url = https://rancher.xx.xx.xx.xx.sslip.io
    workload_node_ip = yy.yy.yy.yy
    ```

-1. Paste the `rancher_server_url` from the output above into the browser. Log in when prompted (default username is `admin`, use the password set in `rancher_server_admin_password`).
+8. Paste the `rancher_server_url` from the output above into the browser. Log in when prompted (default username is `admin`, use the password set in `rancher_server_admin_password`).
+9. ssh to the Rancher Server using the `id_rsa` key generated in `quickstart/aws`.

 #### Result

-Two Kubernetes clusters are deployed into your AWS account, one running Rancher Server and the other ready for experimentation deployments. Please note that while this setup is a great way to explore Rancher functionality, a production setup should follow our high availability setup guidelines.
+Two Kubernetes clusters are deployed into your AWS account, one running Rancher Server and the other ready for experimentation deployments. Please note that while this setup is a great way to explore Rancher functionality, a production setup should follow our high availability setup guidelines. SSH keys for the VMs are auto-generated and stored in the module directory.

 ### What's Next?

@@ -1,9 +1,9 @@
 ---
 title: Rancher DigitalOcean Quick Start Guide
-description: Read this step by step Rancher DigitalOcean guide to quickly deploy a Rancher Server with a single node cluster attached.
+description: Read this step by step Rancher DigitalOcean guide to quickly deploy a Rancher server with a single-node downstream Kubernetes cluster attached.
 weight: 100
 ---
-The following steps will quickly deploy a Rancher Server on DigitalOcean with a single node cluster attached.
+The following steps will quickly deploy a Rancher server on DigitalOcean in a single-node K3s Kubernetes cluster, with a single-node downstream Kubernetes cluster attached.

 ## Prerequisites

@@ -19,25 +19,24 @@ The following steps will quickly deploy a Rancher Server on DigitalOcean with a

 1. Clone [Rancher Quickstart](https://github.com/rancher/quickstart) to a folder using `git clone https://github.com/rancher/quickstart`.

-1. Go into the DigitalOcean folder containing the terraform files by executing `cd quickstart/do`.
+2. Go into the DigitalOcean folder containing the terraform files by executing `cd quickstart/do`.

-1. Rename the `terraform.tfvars.example` file to `terraform.tfvars`.
+3. Rename the `terraform.tfvars.example` file to `terraform.tfvars`.

-1. Edit `terraform.tfvars` and customize the following variables:
+4. Edit `terraform.tfvars` and customize the following variables:
    - `do_token` - DigitalOcean access key
    - `rancher_server_admin_password` - Admin password for created Rancher server

-1. **Optional:** Modify optional variables within `terraform.tfvars`.
+5. **Optional:** Modify optional variables within `terraform.tfvars`.
 See the [Quickstart Readme](https://github.com/rancher/quickstart) and the [DO Quickstart Readme](https://github.com/rancher/quickstart/tree/master/do) for more information.
 Suggestions include:
-    - `do_region` - DigitalOcean region, choose the closest instead of the default
+    - `do_region` - DigitalOcean region, choose the closest instead of the default (`nyc1`)
    - `prefix` - Prefix for all created resources
    - `droplet_size` - Droplet size used, minimum is `s-2vcpu-4gb` but `s-4vcpu-8gb` could be used if within budget
-    - `ssh_key_file_name` - Use a specific SSH key instead of `~/.ssh/id_rsa` (public key is assumed to be `${ssh_key_file_name}.pub`)

-1. Run `terraform init`.
+6. Run `terraform init`.

-1. To initiate the creation of the environment, run `terraform apply --auto-approve`. Then wait for output similar to the following:
+7. To initiate the creation of the environment, run `terraform apply --auto-approve`. Then wait for output similar to the following:

    ```
    Apply complete! Resources: 15 added, 0 changed, 0 destroyed.
@@ -45,15 +44,16 @@ Suggestions include:
    Outputs:

    rancher_node_ip = xx.xx.xx.xx
-    rancher_server_url = https://rancher.xx.xx.xx.xx.xip.io
+    rancher_server_url = https://rancher.xx.xx.xx.xx.sslip.io
    workload_node_ip = yy.yy.yy.yy
    ```

-1. Paste the `rancher_server_url` from the output above into the browser. Log in when prompted (default username is `admin`, use the password set in `rancher_server_admin_password`).
+8. Paste the `rancher_server_url` from the output above into the browser. Log in when prompted (default username is `admin`, use the password set in `rancher_server_admin_password`).
+9. ssh to the Rancher Server using the `id_rsa` key generated in `quickstart/do`.

 #### Result

-Two Kubernetes clusters are deployed into your DigitalOcean account, one running Rancher Server and the other ready for experimentation deployments.
+Two Kubernetes clusters are deployed into your DigitalOcean account, one running Rancher Server and the other ready for experimentation deployments. Please note that while this setup is a great way to explore Rancher functionality, a production setup should follow our high availability setup guidelines. SSH keys for the VMs are auto-generated and stored in the module directory.

 ### What's Next?

@@ -1,67 +1,68 @@
---
-title: Rancher GCP Quick Start Guide
-description: Read this step by step Rancher GCP guide to quickly deploy a Rancher Server with a single node cluster attached.
-weight: 100
---
-The following steps will quickly deploy a Rancher server on GCP in a single-node RKE Kubernetes cluster, with a single-node downstream Kubernetes cluster attached.
-
-## Prerequisites
-
->**Note**
->Deploying to Google GCP will incur charges.
-
- [Google GCP Account](https://console.cloud.google.com/): A Google GCP Account is required to create resources for deploying Rancher and Kubernetes.
- [Google GCP Project](https://cloud.google.com/appengine/docs/standard/nodejs/building-app/creating-project): Use this link to follow a tutorial to create a GCP Project if you don't have one yet.
- [Google GCP Service Account](https://cloud.google.com/iam/docs/creating-managing-service-account-keys): Use this link and follow instructions to create a GCP service account and token file.
- [Terraform](https://www.terraform.io/downloads.html): Used to provision the server and cluster in Google GCP.
-
-
-## Getting Started
-
-1. Clone [Rancher Quickstart](https://github.com/rancher/quickstart) to a folder using `git clone https://github.com/rancher/quickstart`.
-
-1. Go into the GCP folder containing the terraform files by executing `cd quickstart/gcp`.
-
-1. Rename the `terraform.tfvars.example` file to `terraform.tfvars`.
-
-1. Edit `terraform.tfvars` and customize the following variables:
-    - `gcp_account_json` - GCP service account file path and file name 
-    - `rancher_server_admin_password` - Admin password for created Rancher server
-
-1. **Optional:** Modify optional variables within `terraform.tfvars`.
-See the [Quickstart Readme](https://github.com/rancher/quickstart) and the [GCP Quickstart Readme](https://github.com/rancher/quickstart/tree/master/gcp) for more information.
-Suggestions include:
-    - `gcp_region` - Google GCP region, choose the closest instead of the default
-    - `prefix` - Prefix for all created resources
-    - `machine_type` - Compute instance size used, minimum is `n1-standard-1` but `n1-standard-2` or `n1-standard-4` could be used if within budget
-    - `ssh_key_file_name` - Use a specific SSH key instead of `~/.ssh/id_rsa` (public key is assumed to be `${ssh_key_file_name}.pub`)
-
-1. Run `terraform init`.
-
-1. To initiate the creation of the environment, run `terraform apply --auto-approve`. Then wait for output similar to the following:
-
-    ```
-    Apply complete! Resources: 16 added, 0 changed, 0 destroyed.
-
-    Outputs:
-
-    rancher_node_ip = xx.xx.xx.xx
-    rancher_server_url = https://rancher.xx.xx.xx.xx.xip.io
-    workload_node_ip = yy.yy.yy.yy
-    ```
-
-1. Paste the `rancher_server_url` from the output above into the browser. Log in when prompted (default username is `admin`, use the password set in `rancher_server_admin_password`).
-
-#### Result
-
-Two Kubernetes clusters are deployed into your GCP account, one running Rancher Server and the other ready for experimentation deployments.
-
-### What's Next?
-
-Use Rancher to create a deployment. For more information, see [Creating Deployments]({{< baseurl >}}/rancher/v2.6/en/quick-start-guide/workload).
-
-## Destroying the Environment
-
-1. From the `quickstart/gcp` folder, execute `terraform destroy --auto-approve`.
-
-2. Wait for confirmation that all resources have been destroyed.
+---
+title: Rancher GCP Quick Start Guide
+description: Read this step by step Rancher GCP guide to quickly deploy a Rancher server with a single-node downstream Kubernetes cluster attached.
+weight: 100
+---
+The following steps will quickly deploy a Rancher server on GCP in a single-node K3s Kubernetes cluster, with a single-node downstream Kubernetes cluster attached.
+
+## Prerequisites
+
+>**Note**
+>Deploying to Google GCP will incur charges.
+
+- [Google GCP Account](https://console.cloud.google.com/): A Google GCP Account is required to create resources for deploying Rancher and Kubernetes.
+- [Google GCP Project](https://cloud.google.com/appengine/docs/standard/nodejs/building-app/creating-project): Use this link to follow a tutorial to create a GCP Project if you don't have one yet.
+- [Google GCP Service Account](https://cloud.google.com/iam/docs/creating-managing-service-account-keys): Use this link and follow instructions to create a GCP service account and token file.
+- [Terraform](https://www.terraform.io/downloads.html): Used to provision the server and cluster in Google GCP.
+
+
+## Getting Started
+
+1. Clone [Rancher Quickstart](https://github.com/rancher/quickstart) to a folder using `git clone https://github.com/rancher/quickstart`.
+
+2. Go into the GCP folder containing the terraform files by executing `cd quickstart/gcp`.
+
+3. Rename the `terraform.tfvars.example` file to `terraform.tfvars`.
+
+4. Edit `terraform.tfvars` and customize the following variables:
+    - `gcp_account_json` - GCP service account file path and file name 
+    - `rancher_server_admin_password` - Admin password for created Rancher server
+
+5. **Optional:** Modify optional variables within `terraform.tfvars`.
+See the [Quickstart Readme](https://github.com/rancher/quickstart) and the [GCP Quickstart Readme](https://github.com/rancher/quickstart/tree/master/gcp) for more information.
+Suggestions include:
+    - `gcp_region` - Google GCP region, choose the closest instead of the default (`us-east4`)
+    - `gcp_zone` - Google GCP zone, choose the closest instead of the default (`us-east4-a`)
+    - `prefix` - Prefix for all created resources
+    - `machine_type` - Compute instance size used, minimum is `n1-standard-1` but `n1-standard-2` or `n1-standard-4` could be used if within budget
+
+6. Run `terraform init`.
+
+7. To initiate the creation of the environment, run `terraform apply --auto-approve`. Then wait for output similar to the following:
+
+    ```
+    Apply complete! Resources: 16 added, 0 changed, 0 destroyed.
+
+    Outputs:
+
+    rancher_node_ip = xx.xx.xx.xx
+    rancher_server_url = https://rancher.xx.xx.xx.xx.sslip.io
+    workload_node_ip = yy.yy.yy.yy
+    ```
+
+8. Paste the `rancher_server_url` from the output above into the browser. Log in when prompted (default username is `admin`, use the password set in `rancher_server_admin_password`).
+9. ssh to the Rancher Server using the `id_rsa` key generated in `quickstart/gcp`.
+
+#### Result
+
+Two Kubernetes clusters are deployed into your GCP account, one running Rancher Server and the other ready for experimentation deployments. Please note that while this setup is a great way to explore Rancher functionality, a production setup should follow our high availability setup guidelines. SSH keys for the VMs are auto-generated and stored in the module directory.
+
+### What's Next?
+
+Use Rancher to create a deployment. For more information, see [Creating Deployments]({{< baseurl >}}/rancher/v2.6/en/quick-start-guide/workload).
+
+## Destroying the Environment
+
+1. From the `quickstart/gcp` folder, execute `terraform destroy --auto-approve`.
+
+2. Wait for confirmation that all resources have been destroyed.
@@ -1,9 +1,9 @@
 ---
 title: Rancher Hetzner Cloud Quick Start Guide
-description: Read this step by step Rancher Hetzner Cloud guide to quickly deploy a Rancher Server with a single node cluster attached.
+description: Read this step by step Rancher Hetzner Cloud guide to quickly deploy a Rancher server with a single-node downstream Kubernetes cluster attached.
 weight: 100
 ---
-The following steps will quickly deploy a Rancher Server on Hetzner Cloud with a single node cluster attached.
+The following steps will quickly deploy a Rancher server on Hetzner Cloud in a single-node K3s Kubernetes cluster, with a single-node downstream Kubernetes cluster attached.

 ## Prerequisites

@@ -19,24 +19,24 @@ The following steps will quickly deploy a Rancher Server on Hetzner Cloud with a

 1. Clone [Rancher Quickstart](https://github.com/rancher/quickstart) to a folder using `git clone https://github.com/rancher/quickstart`.

-1. Go into the Hetzner folder containing the terraform files by executing `cd quickstart/hcloud`.
+2. Go into the Hetzner folder containing the terraform files by executing `cd quickstart/hcloud`.

-1. Rename the `terraform.tfvars.example` file to `terraform.tfvars`.
+3. Rename the `terraform.tfvars.example` file to `terraform.tfvars`.

-1. Edit `terraform.tfvars` and customize the following variables:
+4. Edit `terraform.tfvars` and customize the following variables:
    - `hcloud_token` - Hetzner API access key
    - `rancher_server_admin_password` - Admin password for created Rancher server

-1. **Optional:** Modify optional variables within `terraform.tfvars`.
+5. **Optional:** Modify optional variables within `terraform.tfvars`.
 See the [Quickstart Readme](https://github.com/rancher/quickstart) and the [Hetzner Quickstart Readme](https://github.com/rancher/quickstart/tree/master/hcloud) for more information.
 Suggestions include:
    - `prefix` - Prefix for all created resources
    - `instance_type` - Instance type, minimum required is `cx21`
-    - `ssh_key_file_name` - Use a specific SSH key instead of `~/.ssh/id_rsa` (public key is assumed to be `${ssh_key_file_name}.pub`)
+    - `hcloud_location` - Hetzner Cloud location, choose the closest instead of the default (`fsn1`)

-1. Run `terraform init`.
+6. Run `terraform init`.

-2. To initiate the creation of the environment, run `terraform apply --auto-approve`. Then wait for output similar to the following:
+7. To initiate the creation of the environment, run `terraform apply --auto-approve`. Then wait for output similar to the following:

    ```
    Apply complete! Resources: 15 added, 0 changed, 0 destroyed.
@@ -44,15 +44,16 @@ Suggestions include:
    Outputs:

    rancher_node_ip = xx.xx.xx.xx
-    rancher_server_url = https://rancher.xx.xx.xx.xx.xip.io
+    rancher_server_url = https://rancher.xx.xx.xx.xx.sslip.io
    workload_node_ip = yy.yy.yy.yy
    ```

-3. Paste the `rancher_server_url` from the output above into the browser. Log in when prompted (default username is `admin`, use the password set in `rancher_server_admin_password`).
+8. Paste the `rancher_server_url` from the output above into the browser. Log in when prompted (default username is `admin`, use the password set in `rancher_server_admin_password`).
+9. ssh to the Rancher Server using the `id_rsa` key generated in `quickstart/hcloud`.

 #### Result

-Two Kubernetes clusters are deployed into your Hetzner account, one running Rancher Server and the other ready for experimentation deployments.
+Two Kubernetes clusters are deployed into your Hetzner account, one running Rancher Server and the other ready for experimentation deployments. Please note that while this setup is a great way to explore Rancher functionality, a production setup should follow our high availability setup guidelines. SSH keys for the VMs are auto-generated and stored in the module directory.

 ### What's Next?

@@ -1,71 +1,74 @@
---
-title: Rancher Azure Quick Start Guide
-description: Read this step by step Rancher Azure guide to quickly deploy a Rancher Server with a single node cluster attached.
-weight: 100
---
-
-The following steps will quickly deploy a Rancher server on Azure in a single-node RKE Kubernetes cluster, with a single-node downstream Kubernetes cluster attached.
-
-## Prerequisites
-
->**Note**
->Deploying to Microsoft Azure will incur charges.
-
- [Microsoft Azure Account](https://azure.microsoft.com/en-us/free/): A Microsoft Azure Account is required to create resources for deploying Rancher and Kubernetes.
- [Microsoft Azure Subscription](https://docs.microsoft.com/en-us/azure/cost-management-billing/manage/create-subscription#create-a-subscription-in-the-azure-portal): Use this link to follow a tutorial to create a Microsoft Azure subscription if you don't have one yet.
- [Micsoroft Azure Tenant](https://docs.microsoft.com/en-us/azure/active-directory/develop/quickstart-create-new-tenant): Use this link and follow instructions to create a Microsoft Azure tenant.
- [Microsoft Azure Client ID/Secret](https://docs.microsoft.com/en-us/azure/active-directory/develop/howto-create-service-principal-portal): Use this link and follow instructions to create a Microsoft Azure client and secret.
- [Terraform](https://www.terraform.io/downloads.html): Used to provision the server and cluster in Microsoft Azure.
-
-
-## Getting Started
-
-1. Clone [Rancher Quickstart](https://github.com/rancher/quickstart) to a folder using `git clone https://github.com/rancher/quickstart`.
-
-1. Go into the Azure folder containing the terraform files by executing `cd quickstart/azure`.
-
-1. Rename the `terraform.tfvars.example` file to `terraform.tfvars`.
-
-1. Edit `terraform.tfvars` and customize the following variables:
-    - `azure_subscription_id` - Microsoft Azure Subscription ID 
-    - `azure_client_id` - Microsoft Azure Client ID
-    - `azure_client_secret` - Microsoft Azure Client Secret
-    - `azure_tenant_id` - Microsoft Azure Tenant ID
-    - `rancher_server_admin_password` - Admin password for created Rancher server
-
-2. **Optional:** Modify optional variables within `terraform.tfvars`.
-See the [Quickstart Readme](https://github.com/rancher/quickstart) and the [Azure Quickstart Readme](https://github.com/rancher/quickstart/tree/master/azure) for more information.
-Suggestions include:
-    - `azure_location` - Microsoft Azure region, choose the closest instead of the default
-    - `prefix` - Prefix for all created resources
-    - `instance_type` - Compute instance size used, minimum is `Standard_DS2_v2` but `Standard_DS2_v3` or `Standard_DS3_v2` could be used if within budget
-
-1. Run `terraform init`.
-
-1. To initiate the creation of the environment, run `terraform apply --auto-approve`. Then wait for output similar to the following:
-
-    ```
-    Apply complete! Resources: 16 added, 0 changed, 0 destroyed.
-
-    Outputs:
-
-    rancher_node_ip = xx.xx.xx.xx
-    rancher_server_url = https://rancher.xx.xx.xx.xx.xip.io
-    workload_node_ip = yy.yy.yy.yy
-    ```
-
-1. Paste the `rancher_server_url` from the output above into the browser. Log in when prompted (default username is `admin`, use the password set in `rancher_server_admin_password`).
-2. ssh to the Rancher Server using the `id_rsa` key generated in `quickstart/azure`.
-#### Result
-
-Two Kubernetes clusters are deployed into your Azure account, one running Rancher Server and the other ready for experimentation deployments.
-
-### What's Next?
-
-Use Rancher to create a deployment. For more information, see [Creating Deployments]({{< baseurl >}}/rancher/v2.6/en/quick-start-guide/workload).
-
-## Destroying the Environment
-
-1. From the `quickstart/azure` folder, execute `terraform destroy --auto-approve`.
-
-2. Wait for confirmation that all resources have been destroyed.
+---
+title: Rancher Azure Quick Start Guide
+description: Read this step by step Rancher Azure guide to quickly deploy a Rancher server with a single-node downstream Kubernetes cluster attached.
+weight: 100
+---
+
+The following steps will quickly deploy a Rancher server on Azure in a single-node K3s Kubernetes cluster, with a single-node downstream Kubernetes cluster attached.
+
+## Prerequisites
+
+>**Note**
+>Deploying to Microsoft Azure will incur charges.
+
+- [Microsoft Azure Account](https://azure.microsoft.com/en-us/free/): A Microsoft Azure Account is required to create resources for deploying Rancher and Kubernetes.
+- [Microsoft Azure Subscription](https://docs.microsoft.com/en-us/azure/cost-management-billing/manage/create-subscription#create-a-subscription-in-the-azure-portal): Use this link to follow a tutorial to create a Microsoft Azure subscription if you don't have one yet.
+- [Micsoroft Azure Tenant](https://docs.microsoft.com/en-us/azure/active-directory/develop/quickstart-create-new-tenant): Use this link and follow instructions to create a Microsoft Azure tenant.
+- [Microsoft Azure Client ID/Secret](https://docs.microsoft.com/en-us/azure/active-directory/develop/howto-create-service-principal-portal): Use this link and follow instructions to create a Microsoft Azure client and secret.
+- [Terraform](https://www.terraform.io/downloads.html): Used to provision the server and cluster in Microsoft Azure.
+
+
+## Getting Started
+
+1. Clone [Rancher Quickstart](https://github.com/rancher/quickstart) to a folder using `git clone https://github.com/rancher/quickstart`.
+
+2. Go into the Azure folder containing the terraform files by executing `cd quickstart/azure`.
+
+3. Rename the `terraform.tfvars.example` file to `terraform.tfvars`.
+
+4. Edit `terraform.tfvars` and customize the following variables:
+    - `azure_subscription_id` - Microsoft Azure Subscription ID 
+    - `azure_client_id` - Microsoft Azure Client ID
+    - `azure_client_secret` - Microsoft Azure Client Secret
+    - `azure_tenant_id` - Microsoft Azure Tenant ID
+    - `rancher_server_admin_password` - Admin password for created Rancher server
+
+5. **Optional:** Modify optional variables within `terraform.tfvars`.
+See the [Quickstart Readme](https://github.com/rancher/quickstart) and the [Azure Quickstart Readme](https://github.com/rancher/quickstart/tree/master/azure) for more information.
+Suggestions include:
+    - `azure_location` - Microsoft Azure region, choose the closest instead of the default (`East US`)
+    - `prefix` - Prefix for all created resources
+    - `instance_type` - Compute instance size used, minimum is `Standard_DS2_v2` but `Standard_DS2_v3` or `Standard_DS3_v2` could be used if within budget
+    - `add_windows_node` - If true, an additional Windows worker node is added to the workload cluster
+    - `windows_admin_password` - The admin password of the windows worker node
+    
+6. Run `terraform init`.
+
+7. To initiate the creation of the environment, run `terraform apply --auto-approve`. Then wait for output similar to the following:
+
+    ```
+    Apply complete! Resources: 16 added, 0 changed, 0 destroyed.
+
+    Outputs:
+
+    rancher_node_ip = xx.xx.xx.xx
+    rancher_server_url = https://rancher.xx.xx.xx.xx.sslip.io
+    workload_node_ip = yy.yy.yy.yy
+    ```
+
+8. Paste the `rancher_server_url` from the output above into the browser. Log in when prompted (default username is `admin`, use the password set in `rancher_server_admin_password`).
+9. ssh to the Rancher Server using the `id_rsa` key generated in `quickstart/azure`.
+
+#### Result
+
+Two Kubernetes clusters are deployed into your Azure account, one running Rancher Server and the other ready for experimentation deployments. Please note that while this setup is a great way to explore Rancher functionality, a production setup should follow our high availability setup guidelines. SSH keys for the VMs are auto-generated and stored in the module directory.
+
+### What's Next?
+
+Use Rancher to create a deployment. For more information, see [Creating Deployments]({{< baseurl >}}/rancher/v2.6/en/quick-start-guide/workload).
+
+## Destroying the Environment
+
+1. From the `quickstart/azure` folder, execute `terraform destroy --auto-approve`.
+
+2. Wait for confirmation that all resources have been destroyed.
@@ -11,7 +11,7 @@ weight: 20
 </td>
 <td width="30%" style="border: none;">
 <h4>Reporting process</h4>
-<p style="padding: 8px">Please submit possible security issues by emailing <a href="mailto:security@rancher.com">security@rancher.com</a></p>
+<p style="padding: 8px">Please submit possible security issues by emailing <a href="mailto:security@rancher.com">security@rancher.com</a> .</p>
 </td>
 <td width="30%" style="border: none;">
 <h4>Announcements</h4>
@@ -20,24 +20,25 @@ weight: 20
 </tr>
 </table>

-Security is at the heart of all Rancher features. From integrating with all the popular authentication tools and services, to an enterprise grade [RBAC capability,]({{<baseurl>}}/rancher/v2.6/en/admin-settings/rbac) Rancher makes your Kubernetes clusters even more secure.
+Security is at the heart of all Rancher features. From integrating with all the popular authentication tools and services, to an enterprise grade [RBAC capability]({{<baseurl>}}/rancher/v2.6/en/admin-settings/rbac), Rancher makes your Kubernetes clusters even more secure.

-On this page, we provide security-related documentation along with resources to help you secure your Rancher installation and your downstream Kubernetes clusters:
+On this page, we provide security related documentation along with resources to help you secure your Rancher installation and your downstream Kubernetes clusters:

 - [Running a CIS security scan on a Kubernetes cluster](#running-a-cis-security-scan-on-a-kubernetes-cluster)
 - [SELinux RPM](#selinux-rpm)
 - [Guide to hardening Rancher installations](#rancher-hardening-guide)
 - [The CIS Benchmark and self-assessment](#the-cis-benchmark-and-self-assessment)
 - [Third-party penetration test reports](#third-party-penetration-test-reports)
- [Rancher CVEs and resolutions](#rancher-cves-and-resolutions)
+- [Rancher Security Advisories and CVEs](#rancher-security-advisories-and-cves)
+- [Kubernetes Security Best Practices](#kubernetes-security-best-practices)

 ### Running a CIS Security Scan on a Kubernetes Cluster

-Rancher leverages [kube-bench](https://github.com/aquasecurity/kube-bench) to run a security scan to check whether Kubernetes is deployed according to security best practices as defined in the CIS (Center for Internet Security) Kubernetes Benchmark.
+Rancher leverages [kube-bench](https://github.com/aquasecurity/kube-bench) to run a security scan to check whether Kubernetes is deployed according to security best practices as defined in the [CIS](https://www.cisecurity.org/cis-benchmarks/) (Center for Internet Security) Kubernetes Benchmark.

 The CIS Kubernetes Benchmark is a reference document that can be used to establish a secure configuration baseline for Kubernetes.

-The Center for Internet Security (CIS) is a 501(c\)(3) non-profit organization, formed in October 2000, with a mission to "identify, develop, validate, promote, and sustain best practice solutions for cyber defense and build and lead communities to enable an environment of trust in cyberspace."
+The Center for Internet Security (CIS) is a 501(c\)(3) non-profit organization, formed in October 2000, with a mission to "identify, develop, validate, promote, and sustain best practice solutions for cyber defense and build and lead communities to enable an environment of trust in cyberspace".

 CIS Benchmarks are best practices for the secure configuration of a target system. CIS Benchmarks are developed through the generous volunteer efforts of subject matter experts, technology vendors, public and private community members, and the CIS Benchmark Development team.

@@ -45,13 +46,13 @@ The Benchmark provides recommendations of two types: Automated and Manual. We ru

 When Rancher runs a CIS security scan on a cluster, it generates a report showing the results of each test, including a summary with the number of passed, skipped and failed tests. The report also includes remediation steps for any failed tests.

-For details, refer to the section on [security scans.]({{<baseurl>}}/rancher/v2.6/en/cis-scans)
+For details, refer to the section on [security scans]({{<baseurl>}}/rancher/v2.6/en/cis-scans).

 ### SELinux RPM

 [Security-Enhanced Linux (SELinux)](https://en.wikipedia.org/wiki/Security-Enhanced_Linux) is a security enhancement to Linux. After being historically used by government agencies, SELinux is now industry standard and is enabled by default on CentOS 7 and 8.

-We provide two RPMs (Red Hat packages) that enable Rancher products to function properly on SELinux-enforcing hosts: `rancher-selinux` and `rke2-selinux`. For details, see [this page.]({{<baseurl>}}/rancher/v2.6/en/security/selinux)
+We provide two RPMs (Red Hat packages) that enable Rancher products to function properly on SELinux-enforcing hosts: `rancher-selinux` and `rke2-selinux`. For details, see [this page]({{<baseurl>}}/rancher/v2.6/en/security/selinux).

 ### Rancher Hardening Guide

@@ -77,9 +78,13 @@ Rancher periodically hires third parties to perform security audits and penetrat

 Results:

- [Cure53 Pen Test - 7/2019](https://releases.rancher.com/documents/security/pen-tests/2019/RAN-01-cure53-report.final.pdf)
- [Untamed Theory Pen Test- 3/2019](https://releases.rancher.com/documents/security/pen-tests/2019/UntamedTheory-Rancher_SecurityAssessment-20190712_v5.pdf)
+- [Cure53 Pen Test - July 2019](https://releases.rancher.com/documents/security/pen-tests/2019/RAN-01-cure53-report.final.pdf)
+- [Untamed Theory Pen Test- March 2019](https://releases.rancher.com/documents/security/pen-tests/2019/UntamedTheory-Rancher_SecurityAssessment-20190712_v5.pdf)

-### Rancher CVEs and Resolutions
+### Rancher Security Advisories and CVEs

 Rancher is committed to informing the community of security issues in our products. For the list of CVEs (Common Vulnerabilities and Exposures) for issues we have resolved, refer to [this page.](./cve)
+
+### Kubernetes Security Best Practices
+
+For recommendations on securing your Kubernetes cluster, refer to the [Kubernetes Security Best Practices](./best-practices) guide.
@@ -0,0 +1,12 @@
+---
+title: Kubernetes Security Best Practices
+weight: 5
+---
+
+### Restricting cloud metadata API access
+
+Cloud providers such as AWS, Azure, DigitalOcean or GCP often expose metadata services locally to instances. By default, this endpoint is accessible by pods running on a cloud instance, including pods in hosted Kubernetes providers such as EKS, AKS, DigitalOcean Kubernetes or GKE, and can contain cloud credentials for that node, provisioning data such as kubelet credentials, or other sensitive data. To mitigate this risk when running on a cloud platform, follow the [Kubernetes security recommendations](https://kubernetes.io/docs/tasks/administer-cluster/securing-a-cluster/#restricting-cloud-metadata-api-access): limit permissions given to instance credentials, use network policies to restrict pod access to the metadata API, and avoid using provisioning data to deliver secrets.
+
+It is advised to consult your cloud provider's security best practices for further recommendations and specific details on how to restrict access to cloud instance metadata API.
+
+Further references: MITRE ATT&CK knowledge base on - [Unsecured Credentials: Cloud Instance Metadata API](https://attack.mitre.org/techniques/T1552/005/).
@@ -1,9 +1,9 @@
 ---
-title: Rancher CVEs and Resolutions
+title: Security Advisories and CVEs
 weight: 300
 ---

-Rancher is committed to informing the community of security issues in our products. Rancher will publish CVEs (Common Vulnerabilities and Exposures) for issues we have resolved.
+Rancher is committed to informing the community of security issues in our products. Rancher will publish security advisories and CVEs (Common Vulnerabilities and Exposures) for issues we have resolved. New security advisories are also published in Rancher's GitHub [security page](https://github.com/rancher/rancher/security/advisories).

 | ID | Description | Date | Resolution |
 |----|-------------|------|------------|
@@ -18,4 +18,4 @@ Rancher is committed to informing the community of security issues in our produc
 | [CVE-2019-12274](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-12274) | Nodes using the built-in node drivers using a file path option allows the machine to read arbitrary files including sensitive ones from inside the Rancher server container. | 5 Jun 2019  |  [Rancher v2.2.4](https://github.com/rancher/rancher/releases/tag/v2.2.4), [Rancher v2.1.10](https://github.com/rancher/rancher/releases/tag/v2.1.10) and [Rancher v2.0.15](https://github.com/rancher/rancher/releases/tag/v2.0.15)  |
 | [CVE-2019-11202](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-11202) | The default admin, that is shipped with Rancher, will be re-created upon restart of Rancher despite being explicitly deleted. | 16 Apr 2019  |  [Rancher v2.2.2](https://github.com/rancher/rancher/releases/tag/v2.2.2), [Rancher v2.1.9](https://github.com/rancher/rancher/releases/tag/v2.1.9) and [Rancher v2.0.14](https://github.com/rancher/rancher/releases/tag/v2.0.14)  |
 | [CVE-2019-6287](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-6287) | Project members continue to get access to namespaces from projects that they were removed from if they were added to more than one project. | 29 Jan 2019  |  [Rancher v2.1.6](https://github.com/rancher/rancher/releases/tag/v2.1.6) and [Rancher v2.0.11](https://github.com/rancher/rancher/releases/tag/v2.0.11)  |
-| [CVE-2018-20321](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2018-20321) |  Any project member with access to the `default` namespace can mount the `netes-default` service account in a pod and then use that pod to execute administrative privileged commands against the Kubernetes cluster.  | 29 Jan 2019 | [Rancher v2.1.6](https://github.com/rancher/rancher/releases/tag/v2.1.6) and [Rancher v2.0.11](https://github.com/rancher/rancher/releases/tag/v2.0.11) - Rolling back from these versions or greater have specific [instructions]({{<baseurl>}}/rancher/v2.6/en/installation/install-rancher-on-k8s/rollbacks).  |
+| [CVE-2018-20321](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2018-20321) |  Any project member with access to the `default` namespace can mount the `netes-default` service account in a pod and then use that pod to execute administrative privileged commands against the Kubernetes cluster.  | 29 Jan 2019 | [Rancher v2.1.6](https://github.com/rancher/rancher/releases/tag/v2.1.6) and [Rancher v2.0.11](https://github.com/rancher/rancher/releases/tag/v2.0.11) - Rolling back from these versions or greater have specific [instructions]({{<baseurl>}}/rancher/v2.6/en/installation/install-rancher-on-k8s/rollbacks).  |
@@ -0,0 +1,13 @@
+---
+title: Self-Assessment and Hardening Guides for Rancher v2.6
+shortTitle: Rancher v2.6 Guides
+weight: 1
+aliases:
+  - /rancher/v2.6/en/security/rancher-2.5/
+  - /rancher/v2.6/en/security/rancher-2.5/1.5-hardening-2.5/
+  - /rancher/v2.6/en/security/rancher-2.5/1.5-benchmark-2.5/
+  - /rancher/v2.6/en/security/rancher-2.5/1.6-hardening-2.5/
+  - /rancher/v2.6/en/security/rancher-2.5/1.6-benchmark-2.5/
+---
+
+Rancher v2.6 hardening guides are currently being updated. For the time being, please consult [Rancher v2.5 self-assessment and hardening guides]({{<baseurl>}}/rancher/v2.5/en/security/rancher-2.5) for more information.
@@ -1,720 +0,0 @@
---
-title: Hardening Guide with CIS 1.5 Benchmark
-weight: 200
---
-
-This document provides prescriptive guidance for hardening a production installation of a RKE cluster to be used with Rancher v2.5. It outlines the configurations and controls required to address Kubernetes benchmark controls from the Center for Information Security (CIS).
-
-> This hardening guide describes how to secure the nodes in your cluster, and it is recommended to follow this guide before installing Kubernetes.
-
-This hardening guide is intended to be used for RKE clusters and associated with specific versions of the CIS Kubernetes Benchmark, Kubernetes, and Rancher:
-
- Rancher Version | CIS Benchmark Version | Kubernetes Version
----------------|-----------------------|------------------
- Rancher v2.5 | Benchmark v1.5 | Kubernetes 1.15
-
-[Click here to download a PDF version of this document](https://releases.rancher.com/documents/security/2.5/Rancher_Hardening_Guide_CIS_1.5.pdf)
-
-### Overview
-
-This document provides prescriptive guidance for hardening a RKE cluster to be used for installing Rancher v2.5 with Kubernetes v1.15 or provisioning a RKE cluster with Kubernetes 1.15 to be used within Rancher v2.5. It outlines the configurations required to address Kubernetes benchmark controls from the Center for Information Security (CIS).
-
-For more detail about evaluating a hardened cluster against the official CIS benchmark, refer to the [CIS 1.5 Benchmark - Self-Assessment Guide - Rancher v2.5]({{< baseurl >}}/rancher/v2.6/en/security/rancher-2.5/1.5-benchmark-2.5/).
-
-#### Known Issues
-
- Rancher **exec shell** and **view logs** for pods are **not** functional in a CIS 1.5 hardened setup when only public IP is provided when registering custom nodes. This functionality requires a private IP to be provided when registering the custom nodes.
- When setting the `default_pod_security_policy_template_id:` to `restricted` Rancher creates **RoleBindings** and **ClusterRoleBindings** on the default service accounts. The CIS 1.5 5.1.5 check requires the default service accounts have no roles or cluster roles bound to it apart from the defaults. In addition the default service accounts should be configured such that it does not provide a service account token and does not have any explicit rights assignments.
-
-### Configure Kernel Runtime Parameters
-
-The following `sysctl` configuration is recommended for all nodes type in the cluster. Set the following parameters in `/etc/sysctl.d/90-kubelet.conf`:
-
-```
-vm.overcommit_memory=1
-vm.panic_on_oom=0
-kernel.panic=10
-kernel.panic_on_oops=1
-kernel.keys.root_maxbytes=25000000
-```
-
-Run `sysctl -p /etc/sysctl.d/90-kubelet.conf` to enable the settings.
-
-### Configure `etcd` user and group
-A user account and group for the **etcd** service is required to be setup before installing RKE. The **uid** and **gid** for the **etcd** user will be used in the RKE **config.yml** to set the proper permissions for files and directories during installation time.
-
-#### create `etcd` user and group
-To create the **etcd** group run the following console commands.
-
-The commands below use `52034` for **uid** and **gid** are for example purposes. Any valid unused **uid** or **gid** could also be used in lieu of `52034`.
-
-```
-groupadd --gid 52034 etcd
-useradd --comment "etcd service account" --uid 52034 --gid 52034 etcd
-```
-
-Update the RKE **config.yml** with the **uid** and **gid** of the **etcd** user:
-
-``` yaml
-services:
-  etcd:
-    gid: 52034
-    uid: 52034
-```
-
-#### Set `automountServiceAccountToken` to `false` for `default` service accounts
-Kubernetes provides a default service account which is used by cluster workloads where no specific service account is assigned to the pod. Where access to the Kubernetes API from a pod is required, a specific service account should be created for that pod, and rights granted to that service account. The default service account should be configured such that it does not provide a service account token and does not have any explicit rights assignments.  
-
-For each namespace including **default** and **kube-system** on a standard RKE install the **default** service account must include this value:
-
-```
-automountServiceAccountToken: false
-```
-
-Save the following yaml to a file called `account_update.yaml`
-
-``` yaml
-apiVersion: v1
-kind: ServiceAccount
-metadata:
-  name: default
-automountServiceAccountToken: false
-```
-
-Create a bash script file called `account_update.sh`. Be sure to `chmod +x account_update.sh` so the script has execute permissions.
-
-```
-#!/bin/bash -e
-
-for namespace in $(kubectl get namespaces -A -o json | jq -r '.items[].metadata.name'); do
-  kubectl patch serviceaccount default -n ${namespace} -p "$(cat account_update.yaml)"
-done
-```
-
-### Ensure that all Namespaces have Network Policies defined
-
-Running different applications on the same Kubernetes cluster creates a risk of one
-compromised application attacking a neighboring application. Network segmentation is
-important to ensure that containers can communicate only with those they are supposed
-to. A network policy is a specification of how selections of pods are allowed to
-communicate with each other and other network endpoints.
-
-Network Policies are namespace scoped. When a network policy is introduced to a given
-namespace, all traffic not allowed by the policy is denied. However, if there are no network
-policies in a namespace all traffic will be allowed into and out of the pods in that
-namespace. To enforce network policies, a CNI (container network interface) plugin must be enabled.
-This guide uses [canal](https://github.com/projectcalico/canal) to provide the policy enforcement.
-Additional information about CNI providers can be found
-[here](https://rancher.com/blog/2019/2019-03-21-comparing-kubernetes-cni-providers-flannel-calico-canal-and-weave/)
-
-Once a CNI provider is enabled on a cluster a default network policy can be applied. For reference purposes a
-**permissive** example is provide below. If you want to allow all traffic to all pods in a namespace
-(even if policies are added that cause some pods to be treated as “isolated”),
-you can create a policy that explicitly allows all traffic in that namespace. Save the following `yaml` as
-`default-allow-all.yaml`. Additional [documentation](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
-about network policies can be found on the Kubernetes site.
-
-> This `NetworkPolicy` is not recommended for production use
-
-``` yaml
---
-apiVersion: networking.k8s.io/v1
-kind: NetworkPolicy
-metadata:
-  name: default-allow-all
-spec:
-  podSelector: {}
-  ingress:
-  - {}
-  egress:
-  - {}
-  policyTypes:
-  - Ingress
-  - Egress
-```
-
-Create a bash script file called `apply_networkPolicy_to_all_ns.sh`. Be sure to
-`chmod +x apply_networkPolicy_to_all_ns.sh` so the script has execute permissions.
-
-```
-#!/bin/bash -e
-
-for namespace in $(kubectl get namespaces -A -o json | jq -r '.items[].metadata.name'); do
-  kubectl apply -f default-allow-all.yaml -n ${namespace}
-done
-```
-Execute this script to apply the `default-allow-all.yaml` the **permissive** `NetworkPolicy` to all namespaces.
-
-### Reference Hardened RKE `cluster.yml` configuration
-
-The reference `cluster.yml` is used by the RKE CLI that provides the configuration needed to achieve a hardened install
-of Rancher Kubernetes Engine (RKE). Install [documentation](https://rancher.com/docs/rke/latest/en/installation/) is
-provided with additional details about the configuration items. This reference `cluster.yml` does not include the required **nodes** directive which will vary depending on your environment. Documentation for node configuration can be found here: https://rancher.com/docs/rke/latest/en/config-options/nodes
-
-
-``` yaml
-# If you intend to deploy Kubernetes in an air-gapped environment,
-# please consult the documentation on how to configure custom RKE images.
-kubernetes_version: "v1.15.9-rancher1-1"
-enable_network_policy: true
-default_pod_security_policy_template_id: "restricted"
-# the nodes directive is required and will vary depending on your environment
-# documentation for node configuration can be found here:
-#  https://rancher.com/docs/rke/latest/en/config-options/nodes
-nodes:
-services:
-  etcd:
-    uid: 52034
-    gid: 52034
-  kube-api:
-    pod_security_policy: true
-    secrets_encryption_config:
-      enabled: true
-    audit_log:
-      enabled: true
-    admission_configuration:
-    event_rate_limit:
-      enabled: true
-  kube-controller:
-    extra_args:
-      feature-gates: "RotateKubeletServerCertificate=true"
-  scheduler:
-    image: ""
-    extra_args: {}
-    extra_binds: []
-    extra_env: []
-  kubelet:
-    generate_serving_certificate: true
-    extra_args:
-      feature-gates: "RotateKubeletServerCertificate=true"
-      protect-kernel-defaults: "true"
-      tls-cipher-suites: "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256"
-    extra_binds: []
-    extra_env: []
-    cluster_domain: ""
-    infra_container_image: ""
-    cluster_dns_server: ""
-    fail_swap_on: false
-  kubeproxy:
-    image: ""
-    extra_args: {}
-    extra_binds: []
-    extra_env: []
-network:
-  plugin: ""
-  options: {}
-  mtu: 0
-  node_selector: {}
-authentication:
-  strategy: ""
-  sans: []
-  webhook: null
-addons: |
-  ---
-  apiVersion: v1
-  kind: Namespace
-  metadata:
-    name: ingress-nginx
-  ---
-  apiVersion: rbac.authorization.k8s.io/v1
-  kind: Role
-  metadata:
-    name: default-psp-role
-    namespace: ingress-nginx
-  rules:
-  - apiGroups:
-    - extensions
-    resourceNames:
-    - default-psp
-    resources:
-    - podsecuritypolicies
-    verbs:
-    - use
-  ---
-  apiVersion: rbac.authorization.k8s.io/v1
-  kind: RoleBinding
-  metadata:
-    name: default-psp-rolebinding
-    namespace: ingress-nginx
-  roleRef:
-    apiGroup: rbac.authorization.k8s.io
-    kind: Role
-    name: default-psp-role
-  subjects:
-  - apiGroup: rbac.authorization.k8s.io
-    kind: Group
-    name: system:serviceaccounts
-  - apiGroup: rbac.authorization.k8s.io
-    kind: Group
-    name: system:authenticated
-  ---
-  apiVersion: v1
-  kind: Namespace
-  metadata:
-    name: cattle-system
-  ---
-  apiVersion: rbac.authorization.k8s.io/v1
-  kind: Role
-  metadata:
-    name: default-psp-role
-    namespace: cattle-system
-  rules:
-  - apiGroups:
-    - extensions
-    resourceNames:
-    - default-psp
-    resources:
-    - podsecuritypolicies
-    verbs:
-    - use
-  ---
-  apiVersion: rbac.authorization.k8s.io/v1
-  kind: RoleBinding
-  metadata:
-    name: default-psp-rolebinding
-    namespace: cattle-system
-  roleRef:
-    apiGroup: rbac.authorization.k8s.io
-    kind: Role
-    name: default-psp-role
-  subjects:
-  - apiGroup: rbac.authorization.k8s.io
-    kind: Group
-    name: system:serviceaccounts
-  - apiGroup: rbac.authorization.k8s.io
-    kind: Group
-    name: system:authenticated
-  ---
-  apiVersion: policy/v1beta1
-  kind: PodSecurityPolicy
-  metadata:
-    name: restricted
-  spec:
-    requiredDropCapabilities:
-    - NET_RAW
-    privileged: false
-    allowPrivilegeEscalation: false
-    defaultAllowPrivilegeEscalation: false
-    fsGroup:
-      rule: RunAsAny
-    runAsUser:
-      rule: MustRunAsNonRoot
-    seLinux:
-      rule: RunAsAny
-    supplementalGroups:
-      rule: RunAsAny
-    volumes:
-    - emptyDir
-    - secret
-    - persistentVolumeClaim
-    - downwardAPI
-    - configMap
-    - projected
-  ---
-  apiVersion: rbac.authorization.k8s.io/v1
-  kind: ClusterRole
-  metadata:
-    name: psp:restricted
-  rules:
-  - apiGroups:
-    - extensions
-    resourceNames:
-    - restricted
-    resources:
-    - podsecuritypolicies
-    verbs:
-    - use
-  ---
-  apiVersion: rbac.authorization.k8s.io/v1
-  kind: ClusterRoleBinding
-  metadata:
-    name: psp:restricted
-  roleRef:
-    apiGroup: rbac.authorization.k8s.io
-    kind: ClusterRole
-    name: psp:restricted
-  subjects:
-  - apiGroup: rbac.authorization.k8s.io
-    kind: Group
-    name: system:serviceaccounts
-  - apiGroup: rbac.authorization.k8s.io
-    kind: Group
-    name: system:authenticated
-  ---
-  apiVersion: v1
-  kind: ServiceAccount
-  metadata:
-    name: tiller
-    namespace: kube-system
-  ---
-  apiVersion: rbac.authorization.k8s.io/v1
-  kind: ClusterRoleBinding
-  metadata:
-    name: tiller
-  roleRef:
-    apiGroup: rbac.authorization.k8s.io
-    kind: ClusterRole
-    name: cluster-admin
-  subjects:
-  - kind: ServiceAccount
-    name: tiller
-    namespace: kube-system
-
-addons_include: []
-system_images:
-  etcd: ""
-  alpine: ""
-  nginx_proxy: ""
-  cert_downloader: ""
-  kubernetes_services_sidecar: ""
-  kubedns: ""
-  dnsmasq: ""
-  kubedns_sidecar: ""
-  kubedns_autoscaler: ""
-  coredns: ""
-  coredns_autoscaler: ""
-  kubernetes: ""
-  flannel: ""
-  flannel_cni: ""
-  calico_node: ""
-  calico_cni: ""
-  calico_controllers: ""
-  calico_ctl: ""
-  calico_flexvol: ""
-  canal_node: ""
-  canal_cni: ""
-  canal_flannel: ""
-  canal_flexvol: ""
-  weave_node: ""
-  weave_cni: ""
-  pod_infra_container: ""
-  ingress: ""
-  ingress_backend: ""
-  metrics_server: ""
-  windows_pod_infra_container: ""
-ssh_key_path: ""
-ssh_cert_path: ""
-ssh_agent_auth: false
-authorization:
-  mode: ""
-  options: {}
-ignore_docker_version: false
-private_registries: []
-ingress:
-  provider: ""
-  options: {}
-  node_selector: {}
-  extra_args: {}
-  dns_policy: ""
-  extra_envs: []
-  extra_volumes: []
-  extra_volume_mounts: []
-cluster_name: ""
-prefix_path: ""
-addon_job_timeout: 0
-bastion_host:
-  address: ""
-  port: ""
-  user: ""
-  ssh_key: ""
-  ssh_key_path: ""
-  ssh_cert: ""
-  ssh_cert_path: ""
-monitoring:
-  provider: ""
-  options: {}
-  node_selector: {}
-restore:
-  restore: false
-  snapshot_name: ""
-dns: null
-```
-
-### Reference Hardened RKE Template configuration
-
-The reference RKE Template provides the configuration needed to achieve a hardened install of Kubenetes.
-RKE Templates are used to provision Kubernetes and define Rancher settings. Follow the Rancher
-[documentaion](https://rancher.com/docs/rancher/v2.6/en/installation) for additional installation and RKE Template details.
-
-``` yaml
-#
-# Cluster Config
-#
-default_pod_security_policy_template_id: restricted
-docker_root_dir: /var/lib/docker
-enable_cluster_alerting: false
-enable_cluster_monitoring: false
-enable_network_policy: true
-#
-# Rancher Config
-#
-rancher_kubernetes_engine_config:
-  addon_job_timeout: 30
-  addons: |-
-    ---
-    apiVersion: v1
-    kind: Namespace
-    metadata:
-      name: ingress-nginx
-    ---
-    apiVersion: rbac.authorization.k8s.io/v1
-    kind: Role
-    metadata:
-      name: default-psp-role
-      namespace: ingress-nginx
-    rules:
-    - apiGroups:
-      - extensions
-      resourceNames:
-      - default-psp
-      resources:
-      - podsecuritypolicies
-      verbs:
-      - use
-    ---
-    apiVersion: rbac.authorization.k8s.io/v1
-    kind: RoleBinding
-    metadata:
-      name: default-psp-rolebinding
-      namespace: ingress-nginx
-    roleRef:
-      apiGroup: rbac.authorization.k8s.io
-      kind: Role
-      name: default-psp-role
-    subjects:
-    - apiGroup: rbac.authorization.k8s.io
-      kind: Group
-      name: system:serviceaccounts
-    - apiGroup: rbac.authorization.k8s.io
-      kind: Group
-      name: system:authenticated
-    ---
-    apiVersion: v1
-    kind: Namespace
-    metadata:
-      name: cattle-system
-    ---
-    apiVersion: rbac.authorization.k8s.io/v1
-    kind: Role
-    metadata:
-      name: default-psp-role
-      namespace: cattle-system
-    rules:
-    - apiGroups:
-      - extensions
-      resourceNames:
-      - default-psp
-      resources:
-      - podsecuritypolicies
-      verbs:
-      - use
-    ---
-    apiVersion: rbac.authorization.k8s.io/v1
-    kind: RoleBinding
-    metadata:
-      name: default-psp-rolebinding
-      namespace: cattle-system
-    roleRef:
-      apiGroup: rbac.authorization.k8s.io
-      kind: Role
-      name: default-psp-role
-    subjects:
-    - apiGroup: rbac.authorization.k8s.io
-      kind: Group
-      name: system:serviceaccounts
-    - apiGroup: rbac.authorization.k8s.io
-      kind: Group
-      name: system:authenticated
-    ---
-    apiVersion: policy/v1beta1
-    kind: PodSecurityPolicy
-    metadata:
-      name: restricted
-    spec:
-      requiredDropCapabilities:
-      - NET_RAW
-      privileged: false
-      allowPrivilegeEscalation: false
-      defaultAllowPrivilegeEscalation: false
-      fsGroup:
-        rule: RunAsAny
-      runAsUser:
-        rule: MustRunAsNonRoot
-      seLinux:
-        rule: RunAsAny
-      supplementalGroups:
-        rule: RunAsAny
-      volumes:
-      - emptyDir
-      - secret
-      - persistentVolumeClaim
-      - downwardAPI
-      - configMap
-      - projected
-    ---
-    apiVersion: rbac.authorization.k8s.io/v1
-    kind: ClusterRole
-    metadata:
-      name: psp:restricted
-    rules:
-    - apiGroups:
-      - extensions
-      resourceNames:
-      - restricted
-      resources:
-      - podsecuritypolicies
-      verbs:
-      - use
-    ---
-    apiVersion: rbac.authorization.k8s.io/v1
-    kind: ClusterRoleBinding
-    metadata:
-      name: psp:restricted
-    roleRef:
-      apiGroup: rbac.authorization.k8s.io
-      kind: ClusterRole
-      name: psp:restricted
-    subjects:
-    - apiGroup: rbac.authorization.k8s.io
-      kind: Group
-      name: system:serviceaccounts
-    - apiGroup: rbac.authorization.k8s.io
-      kind: Group
-      name: system:authenticated
-    ---
-    apiVersion: v1
-    kind: ServiceAccount
-    metadata:
-      name: tiller
-      namespace: kube-system
-    ---
-    apiVersion: rbac.authorization.k8s.io/v1
-    kind: ClusterRoleBinding
-    metadata:
-      name: tiller
-    roleRef:
-      apiGroup: rbac.authorization.k8s.io
-      kind: ClusterRole
-      name: cluster-admin
-    subjects:
-    - kind: ServiceAccount
-      name: tiller
-      namespace: kube-system
-  ignore_docker_version: true
-  kubernetes_version: v1.15.9-rancher1-1
-#
-#   If you are using calico on AWS
-#
-#    network:
-#      plugin: calico
-#      calico_network_provider:
-#        cloud_provider: aws
-#
-# # To specify flannel interface
-#
-#    network:
-#      plugin: flannel
-#      flannel_network_provider:
-#      iface: eth1
-#
-# # To specify flannel interface for canal plugin
-#
-#    network:
-#      plugin: canal
-#      canal_network_provider:
-#        iface: eth1
-#
-  network:
-    mtu: 0
-    plugin: canal
-#
-#    services:
-#      kube-api:
-#        service_cluster_ip_range: 10.43.0.0/16
-#      kube-controller:
-#        cluster_cidr: 10.42.0.0/16
-#        service_cluster_ip_range: 10.43.0.0/16
-#      kubelet:
-#        cluster_domain: cluster.local
-#        cluster_dns_server: 10.43.0.10
-#
-  services:
-    etcd:
-      backup_config:
-        enabled: false
-        interval_hours: 12
-        retention: 6
-        safe_timestamp: false
-      creation: 12h
-      extra_args:
-        election-timeout: '5000'
-        heartbeat-interval: '500'
-      gid: 52034
-      retention: 72h
-      snapshot: false
-      uid: 52034
-    kube_api:
-      always_pull_images: false
-      audit_log:
-        enabled: true
-      event_rate_limit:
-        enabled: true
-      pod_security_policy: true
-      secrets_encryption_config:
-        enabled: true
-      service_node_port_range: 30000-32767
-    kube_controller:
-      extra_args:
-        address: 127.0.0.1
-        feature-gates: RotateKubeletServerCertificate=true
-        profiling: 'false'
-        terminated-pod-gc-threshold: '1000'
-    kubelet:
-      extra_args:
-        anonymous-auth: 'false'
-        event-qps: '0'
-        feature-gates: RotateKubeletServerCertificate=true
-        make-iptables-util-chains: 'true'
-        protect-kernel-defaults: 'true'
-        streaming-connection-idle-timeout: 1800s
-        tls-cipher-suites: >-
-          TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256
-      fail_swap_on: false
-      generate_serving_certificate: true
-    scheduler:
-      extra_args:
-        address: 127.0.0.1
-        profiling: 'false'
-  ssh_agent_auth: false
-windows_prefered_cluster: false
-```
-
-### Hardened Reference Ubuntu 18.04 LTS **cloud-config**:
-
-The reference **cloud-config** is generally used in cloud infrastructure environments to allow for
-configuration management of compute instances. The reference config configures Ubuntu operating system level settings
-needed before installing kubernetes.
-
-``` yaml
-#cloud-config
-packages:
-  - curl
-  - jq
-runcmd:
-  - sysctl -w vm.overcommit_memory=1
-  - sysctl -w kernel.panic=10
-  - sysctl -w kernel.panic_on_oops=1
-  - curl https://releases.rancher.com/install-docker/18.09.sh | sh
-  - usermod -aG docker ubuntu
-  - return=1; while [ $return != 0 ]; do sleep 2; docker ps; return=$?; done
-  - addgroup --gid 52034 etcd
-  - useradd --comment "etcd service account" --uid 52034 --gid 52034 etcd
-write_files:
-  - path: /etc/sysctl.d/kubelet.conf
-    owner: root:root
-    permissions: "0644"
-    content: |
-      vm.overcommit_memory=1
-      kernel.panic=10
-      kernel.panic_on_oops=1
-```
@@ -1,570 +0,0 @@
---
-title: Hardening Guide with CIS 1.6 Benchmark
-weight: 100
---
-
-This document provides prescriptive guidance for hardening a production installation of a RKE cluster to be used with Rancher v2.5.4. It outlines the configurations and controls required to address Kubernetes benchmark controls from the Center for Information Security (CIS).
-
-> This hardening guide describes how to secure the nodes in your cluster, and it is recommended to follow this guide before installing Kubernetes.
-
-This hardening guide is intended to be used for RKE clusters and associated with specific versions of the CIS Kubernetes Benchmark, Kubernetes, and Rancher:
-
- Rancher Version | CIS Benchmark Version | Kubernetes Version
----------------|-----------------------|------------------
- Rancher v2.5.4 | Benchmark 1.6 | Kubernetes v1.18
-
-[Click here to download a PDF version of this document](https://releases.rancher.com/documents/security/2.5/Rancher_Hardening_Guide_CIS_1.6.pdf)
-
-### Overview
-
-This document provides prescriptive guidance for hardening a RKE cluster to be used for installing Rancher v2.5.4 with Kubernetes v1.18 or provisioning a RKE cluster with Kubernetes v1.18 to be used within Rancher v2.5.4. It outlines the configurations required to address Kubernetes benchmark controls from the Center for Information Security (CIS).
-
-For more detail about evaluating a hardened cluster against the official CIS benchmark, refer to the [CIS 1.6 Benchmark - Self-Assessment Guide - Rancher v2.5.4]({{< baseurl >}}/rancher/v2.6/en/security/rancher-2.5/1.6-benchmark-2.5/).
-
-#### Known Issues
-
- Rancher **exec shell** and **view logs** for pods are **not** functional in a CIS 1.6 hardened setup when only public IP is provided when registering custom nodes. This functionality requires a private IP to be provided when registering the custom nodes.
- When setting the `default_pod_security_policy_template_id:` to `restricted` Rancher creates **RoleBindings** and **ClusterRoleBindings** on the default service accounts. The CIS 1.6 5.1.5 check requires the default service accounts have no roles or cluster roles bound to it apart from the defaults. In addition the default service accounts should be configured such that it does not provide a service account token and does not have any explicit rights assignments.
-
-Migration Rancher from 2.4 to 2.5. Addons were removed in HG 2.5, and therefore namespaces on migration may be not created on the downstream clusters. Pod may fail to run because of missing namesapce like ingress-nginx, cattle-system.
-
-### Configure Kernel Runtime Parameters
-
-The following `sysctl` configuration is recommended for all nodes type in the cluster. Set the following parameters in `/etc/sysctl.d/90-kubelet.conf`:
-
-```ini
-vm.overcommit_memory=1
-vm.panic_on_oom=0
-kernel.panic=10
-kernel.panic_on_oops=1
-kernel.keys.root_maxbytes=25000000
-```
-
-Run `sysctl -p /etc/sysctl.d/90-kubelet.conf` to enable the settings.
-
-### Configure `etcd` user and group
-A user account and group for the **etcd** service is required to be setup before installing RKE. The **uid** and **gid** for the **etcd** user will be used in the RKE **config.yml** to set the proper permissions for files and directories during installation time.
-
-#### create `etcd` user and group
-To create the **etcd** group run the following console commands.
-
-The commands below use `52034` for **uid** and **gid** are for example purposes. Any valid unused **uid** or **gid** could also be used in lieu of `52034`.
-
-```bash
-groupadd --gid 52034 etcd
-useradd --comment "etcd service account" --uid 52034 --gid 52034 etcd
-```
-
-Update the RKE **config.yml** with the **uid** and **gid** of the **etcd** user:
-
-```yaml
-services:
-  etcd:
-    gid: 52034
-    uid: 52034
-```
-
-#### Set `automountServiceAccountToken` to `false` for `default` service accounts
-Kubernetes provides a default service account which is used by cluster workloads where no specific service account is assigned to the pod. Where access to the Kubernetes API from a pod is required, a specific service account should be created for that pod, and rights granted to that service account. The default service account should be configured such that it does not provide a service account token and does not have any explicit rights assignments.  
-
-For each namespace including **default** and **kube-system** on a standard RKE install the **default** service account must include this value:
-
-```yaml
-automountServiceAccountToken: false
-```
-
-Save the following yaml to a file called `account_update.yaml`
-
-```yaml
-apiVersion: v1
-kind: ServiceAccount
-metadata:
-  name: default
-automountServiceAccountToken: false
-```
-
-Create a bash script file called `account_update.sh`. Be sure to `chmod +x account_update.sh` so the script has execute permissions.
-
-```bash
-#!/bin/bash -e
-
-for namespace in $(kubectl get namespaces -A -o json | jq -r '.items[].metadata.name'); do
-  kubectl patch serviceaccount default -n ${namespace} -p "$(cat account_update.yaml)"
-done
-```
-
-### Ensure that all Namespaces have Network Policies defined
-
-Running different applications on the same Kubernetes cluster creates a risk of one
-compromised application attacking a neighboring application. Network segmentation is
-important to ensure that containers can communicate only with those they are supposed
-to. A network policy is a specification of how selections of pods are allowed to
-communicate with each other and other network endpoints.
-
-Network Policies are namespace scoped. When a network policy is introduced to a given
-namespace, all traffic not allowed by the policy is denied. However, if there are no network
-policies in a namespace all traffic will be allowed into and out of the pods in that
-namespace. To enforce network policies, a CNI (container network interface) plugin must be enabled.
-This guide uses [canal](https://github.com/projectcalico/canal) to provide the policy enforcement.
-Additional information about CNI providers can be found
-[here](https://rancher.com/blog/2019/2019-03-21-comparing-kubernetes-cni-providers-flannel-calico-canal-and-weave/)
-
-Once a CNI provider is enabled on a cluster a default network policy can be applied. For reference purposes a
-**permissive** example is provide below. If you want to allow all traffic to all pods in a namespace
-(even if policies are added that cause some pods to be treated as “isolated”),
-you can create a policy that explicitly allows all traffic in that namespace. Save the following `yaml` as
-`default-allow-all.yaml`. Additional [documentation](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
-about network policies can be found on the Kubernetes site.
-
-> This `NetworkPolicy` is not recommended for production use
-
-```yaml
---
-apiVersion: networking.k8s.io/v1
-kind: NetworkPolicy
-metadata:
-  name: default-allow-all
-spec:
-  podSelector: {}
-  ingress:
-  - {}
-  egress:
-  - {}
-  policyTypes:
-  - Ingress
-  - Egress
-```
-
-Create a bash script file called `apply_networkPolicy_to_all_ns.sh`. Be sure to
-`chmod +x apply_networkPolicy_to_all_ns.sh` so the script has execute permissions.
-
-```bash
-#!/bin/bash -e
-
-for namespace in $(kubectl get namespaces -A -o json | jq -r '.items[].metadata.name'); do
-  kubectl apply -f default-allow-all.yaml -n ${namespace}
-done
-```
-
-Execute this script to apply the `default-allow-all.yaml` the **permissive** `NetworkPolicy` to all namespaces.
-
-### Reference Hardened RKE `cluster.yml` configuration
-
-The reference `cluster.yml` is used by the RKE CLI that provides the configuration needed to achieve a hardened install
-of Rancher Kubernetes Engine (RKE). Install [documentation](https://rancher.com/docs/rke/latest/en/installation/) is
-provided with additional details about the configuration items. This reference `cluster.yml` does not include the required **nodes** directive which will vary depending on your environment. Documentation for node configuration can be found here: https://rancher.com/docs/rke/latest/en/config-options/nodes
-
-
-```yaml
-# If you intend to deploy Kubernetes in an air-gapped environment,
-# please consult the documentation on how to configure custom RKE images.
-# https://rancher.com/docs/rke/latest/en/installation/
-
-# the nodes directive is required and will vary depending on your environment
-# documentation for node configuration can be found here:
-# https://rancher.com/docs/rke/latest/en/config-options/nodes
-nodes: []
-services:
-  etcd:
-    image: ""
-    extra_args: {}
-    extra_binds: []
-    extra_env: []
-    win_extra_args: {}
-    win_extra_binds: []
-    win_extra_env: []
-    external_urls: []
-    ca_cert: ""
-    cert: ""
-    key: ""
-    path: ""
-    uid: 52034
-    gid: 52034
-    snapshot: false
-    retention: ""
-    creation: ""
-    backup_config: null
-  kube-api:
-    image: ""
-    extra_args: {}
-    extra_binds: []
-    extra_env: []
-    win_extra_args: {}
-    win_extra_binds: []
-    win_extra_env: []
-    service_cluster_ip_range: ""
-    service_node_port_range: ""
-    pod_security_policy: true
-    always_pull_images: false
-    secrets_encryption_config:
-      enabled: true
-      custom_config: null
-    audit_log:
-      enabled: true
-      configuration: null
-    admission_configuration: null
-    event_rate_limit:
-      enabled: true
-      configuration: null
-  kube-controller:
-    image: ""
-    extra_args:
-      feature-gates: RotateKubeletServerCertificate=true
-    extra_binds: []
-    extra_env: []
-    win_extra_args: {}
-    win_extra_binds: []
-    win_extra_env: []
-    cluster_cidr: ""
-    service_cluster_ip_range: ""
-  scheduler:
-    image: ""
-    extra_args: {}
-    extra_binds: []
-    extra_env: []
-    win_extra_args: {}
-    win_extra_binds: []
-    win_extra_env: []
-  kubelet:
-    image: ""
-    extra_args:
-      feature-gates: RotateKubeletServerCertificate=true
-      protect-kernel-defaults: "true"
-      tls-cipher-suites: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256
-    extra_binds: []
-    extra_env: []
-    win_extra_args: {}
-    win_extra_binds: []
-    win_extra_env: []
-    cluster_domain: cluster.local
-    infra_container_image: ""
-    cluster_dns_server: ""
-    fail_swap_on: false
-    generate_serving_certificate: true
-  kubeproxy:
-    image: ""
-    extra_args: {}
-    extra_binds: []
-    extra_env: []
-    win_extra_args: {}
-    win_extra_binds: []
-    win_extra_env: []
-network:
-  plugin: ""
-  options: {}
-  mtu: 0
-  node_selector: {}
-  update_strategy: null
-authentication:
-  strategy: ""
-  sans: []
-  webhook: null
-addons: |
-  apiVersion: policy/v1beta1
-  kind: PodSecurityPolicy
-  metadata:
-    name: restricted
-  spec:
-    requiredDropCapabilities:
-    - NET_RAW
-    privileged: false
-    allowPrivilegeEscalation: false
-    defaultAllowPrivilegeEscalation: false
-    fsGroup:
-      rule: RunAsAny
-    runAsUser:
-      rule: MustRunAsNonRoot
-    seLinux:
-      rule: RunAsAny
-    supplementalGroups:
-      rule: RunAsAny
-    volumes:
-    - emptyDir
-    - secret
-    - persistentVolumeClaim
-    - downwardAPI
-    - configMap
-    - projected
-  ---
-  apiVersion: rbac.authorization.k8s.io/v1
-  kind: ClusterRole
-  metadata:
-    name: psp:restricted
-  rules:
-  - apiGroups:
-    - extensions
-    resourceNames:
-    - restricted
-    resources:
-    - podsecuritypolicies
-    verbs:
-    - use
-  ---
-  apiVersion: rbac.authorization.k8s.io/v1
-  kind: ClusterRoleBinding
-  metadata:
-    name: psp:restricted
-  roleRef:
-    apiGroup: rbac.authorization.k8s.io
-    kind: ClusterRole
-    name: psp:restricted
-  subjects:
-  - apiGroup: rbac.authorization.k8s.io
-    kind: Group
-    name: system:serviceaccounts
-  - apiGroup: rbac.authorization.k8s.io
-    kind: Group
-    name: system:authenticated
-  ---
-  apiVersion: networking.k8s.io/v1
-  kind: NetworkPolicy
-  metadata:
-    name: default-allow-all
-  spec:
-    podSelector: {}
-    ingress:
-    - {}
-    egress:
-    - {}
-    policyTypes:
-    - Ingress
-    - Egress
-  ---
-  apiVersion: v1
-  kind: ServiceAccount
-  metadata:
-    name: default
-  automountServiceAccountToken: false
-addons_include: []
-system_images:
-  etcd: ""
-  alpine: ""
-  nginx_proxy: ""
-  cert_downloader: ""
-  kubernetes_services_sidecar: ""
-  kubedns: ""
-  dnsmasq: ""
-  kubedns_sidecar: ""
-  kubedns_autoscaler: ""
-  coredns: ""
-  coredns_autoscaler: ""
-  nodelocal: ""
-  kubernetes: ""
-  flannel: ""
-  flannel_cni: ""
-  calico_node: ""
-  calico_cni: ""
-  calico_controllers: ""
-  calico_ctl: ""
-  calico_flexvol: ""
-  canal_node: ""
-  canal_cni: ""
-  canal_controllers: ""
-  canal_flannel: ""
-  canal_flexvol: ""
-  weave_node: ""
-  weave_cni: ""
-  pod_infra_container: ""
-  ingress: ""
-  ingress_backend: ""
-  metrics_server: ""
-  windows_pod_infra_container: ""
-ssh_key_path: ""
-ssh_cert_path: ""
-ssh_agent_auth: false
-authorization:
-  mode: ""
-  options: {}
-ignore_docker_version: false
-kubernetes_version: v1.18.12-rancher1-1
-private_registries: []
-ingress:
-  provider: ""
-  options: {}
-  node_selector: {}
-  extra_args: {}
-  dns_policy: ""
-  extra_envs: []
-  extra_volumes: []
-  extra_volume_mounts: []
-  update_strategy: null
-  http_port: 0
-  https_port: 0
-  network_mode: ""
-cluster_name:
-cloud_provider:
-  name: ""
-prefix_path: ""
-win_prefix_path: ""
-addon_job_timeout: 0
-bastion_host:
-  address: ""
-  port: ""
-  user: ""
-  ssh_key: ""
-  ssh_key_path: ""
-  ssh_cert: ""
-  ssh_cert_path: ""
-monitoring:
-  provider: ""
-  options: {}
-  node_selector: {}
-  update_strategy: null
-  replicas: null
-restore:
-  restore: false
-  snapshot_name: ""
-dns: null
-upgrade_strategy:
-  max_unavailable_worker: ""
-  max_unavailable_controlplane: ""
-  drain: null
-  node_drain_input: null
-```
-
-### Reference Hardened RKE Template configuration
-
-The reference RKE Template provides the configuration needed to achieve a hardened install of Kubenetes.
-RKE Templates are used to provision Kubernetes and define Rancher settings. Follow the Rancher
-[documentaion](https://rancher.com/docs/rancher/v2.6/en/installation) for additional installation and RKE Template details.
-
-```yaml
-# 
-# Cluster Config
-# 
-default_pod_security_policy_template_id: restricted
-docker_root_dir: /var/lib/docker
-enable_cluster_alerting: false
-enable_cluster_monitoring: false
-enable_network_policy: true
-# 
-# Rancher Config
-# 
-rancher_kubernetes_engine_config:
-  addon_job_timeout: 45
-  ignore_docker_version: true
-  kubernetes_version: v1.18.12-rancher1-1
-# 
-#   If you are using calico on AWS
-# 
-#    network:
-#      plugin: calico
-#      calico_network_provider:
-#        cloud_provider: aws
-# 
-# # To specify flannel interface
-# 
-#    network:
-#      plugin: flannel
-#      flannel_network_provider:
-#      iface: eth1
-# 
-# # To specify flannel interface for canal plugin
-# 
-#    network:
-#      plugin: canal
-#      canal_network_provider:
-#        iface: eth1
-# 
-  network:
-    mtu: 0
-    plugin: canal
-  rotate_encryption_key: false
-# 
-#    services:
-#      kube-api:
-#        service_cluster_ip_range: 10.43.0.0/16
-#      kube-controller:
-#        cluster_cidr: 10.42.0.0/16
-#        service_cluster_ip_range: 10.43.0.0/16
-#      kubelet:
-#        cluster_domain: cluster.local
-#        cluster_dns_server: 10.43.0.10
-# 
-  services:
-    etcd:
-      backup_config:
-        enabled: false
-        interval_hours: 12
-        retention: 6
-        safe_timestamp: false
-      creation: 12h
-      extra_args:
-        election-timeout: '5000'
-        heartbeat-interval: '500'
-      gid: 52034
-      retention: 72h
-      snapshot: false
-      uid: 52034
-    kube_api:
-      always_pull_images: false
-      audit_log:
-        enabled: true
-      event_rate_limit:
-        enabled: true
-      pod_security_policy: true
-      secrets_encryption_config:
-        enabled: true
-      service_node_port_range: 30000-32767
-    kube_controller:
-      extra_args:
-        feature-gates: RotateKubeletServerCertificate=true
-    kubelet:
-      extra_args:
-        feature-gates: RotateKubeletServerCertificate=true
-        protect-kernel-defaults: 'true'
-        tls-cipher-suites: >-
-          TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256
-      fail_swap_on: false
-      generate_serving_certificate: true
-  ssh_agent_auth: false
-  upgrade_strategy:
-    max_unavailable_controlplane: '1'
-    max_unavailable_worker: 10%
-windows_prefered_cluster: false
-```
-
-### Hardened Reference Ubuntu 20.04 LTS **cloud-config**:
-
-The reference **cloud-config** is generally used in cloud infrastructure environments to allow for
-configuration management of compute instances. The reference config configures Ubuntu operating system level settings
-needed before installing kubernetes.
-
-```yaml
-#cloud-config
-apt:
-  sources:
-    docker.list:
-      source: deb [arch=amd64] http://download.docker.com/linux/ubuntu $RELEASE stable
-      keyid: 9DC858229FC7DD38854AE2D88D81803C0EBFCD88
-system_info:
-  default_user:
-    groups:
-    - docker
-write_files:
- path: "/etc/apt/preferences.d/docker"
-  owner: root:root
-  permissions: '0600'
-  content: |
-    Package: docker-ce
-    Pin: version 5:19*
-    Pin-Priority: 800
- path: "/etc/sysctl.d/90-kubelet.conf"
-  owner: root:root
-  permissions: '0644'
-  content: |
-    vm.overcommit_memory=1
-    vm.panic_on_oom=0
-    kernel.panic=10
-    kernel.panic_on_oops=1
-    kernel.keys.root_maxbytes=25000000
-package_update: true
-packages:
- docker-ce
- docker-ce-cli
- containerd.io
-runcmd:
- sysctl -p /etc/sysctl.d/90-kubelet.conf
- groupadd --gid 52034 etcd
- useradd --comment "etcd service account" --uid 52034 --gid 52034 etcd
-```
@@ -1,55 +0,0 @@
---
-title: Self-Assessment and Hardening Guides for Rancher v2.5
-shortTitle: Rancher v2.5 Guides
-weight: 1
---
-
-Rancher v2.5 introduced the capability to deploy Rancher on any Kubernetes cluster. For that reason, we now provide separate security hardening guides for Rancher deployments on each of Rancher's Kubernetes distributions.
-
- [Rancher Kubernetes Distributions](#rancher-kubernetes-distributions)
- [Hardening Guides and Benchmark Versions](#hardening-guides-and-benchmark-versions)
-  - [RKE Guides](#rke-guides)
-  - [RKE2 Guides](#rke2-guides)
-  - [K3s Guides](#k3s)
- [Rancher with SELinux](#rancher-with-selinux)
-
-# Rancher Kubernetes Distributions
-
-Rancher has the following Kubernetes distributions:
-
- [**RKE,**]({{<baseurl>}}/rke/latest/en/) Rancher Kubernetes Engine, is a CNCF-certified Kubernetes distribution that runs entirely within Docker containers. 
- [**K3s,**]({{<baseurl>}}/k3s/latest/en/) is a fully conformant, lightweight Kubernetes distribution. It is easy to install, with half the memory of upstream Kubernetes, all in a binary of less than 100 MB.
- [**RKE2**](https://docs.rke2.io/) is a fully conformant Kubernetes distribution that focuses on security and compliance within the U.S. Federal Government sector.
-
-To harden a Kubernetes cluster outside of Rancher's distributions, refer to your Kubernetes provider docs.
-
-# Hardening Guides and Benchmark Versions
-
-These guides have been tested along with the Rancher v2.5 release. Each self-assessment guide is accompanied with a hardening guide and tested on a specific Kubernetes version and CIS benchmark version. If a CIS benchmark has not been validated for your Kubernetes version, you can choose to use the existing guides until a newer version is added.
-
-### RKE Guides
-
-Kubernetes Version | CIS Benchmark Version | Self Assessment Guide | Hardening Guides
---|---|---|---
-Kubernetes v1.15+ | CIS v1.5 | [Link](./1.5-benchmark-2.5) | [Link](./1.5-hardening-2.5)
-Kubernetes v1.18+ | CIS v1.6 | [Link](./1.6-benchmark-2.5) | [Link](./1.6-hardening-2.5)
-
-### RKE2 Guides
-
-Kubernetes Version | CIS Benchmark Version | Self Assessment Guide | Hardening Guides
---|---|---|---
-Kubernetes v1.18 | CIS v1.5 | [Link](https://docs.rke2.io/security/cis_self_assessment15/) | [Link](https://docs.rke2.io/security/hardening_guide/)
-Kubernetes v1.20 | CIS v1.6 | [Link](https://docs.rke2.io/security/cis_self_assessment16/) | [Link](https://docs.rke2.io/security/hardening_guide/)
-
-### K3s Guides
-
-Kubernetes Version | CIS Benchmark Version | Self Assessment Guide | Hardening Guide
---|---|---|---
-Kubernetes v1.17, v1.18, & v1.19 | CIS v1.5 | [Link]({{<baseurl>}}/k3s/latest/en/security/self_assessment/) | [Link]({{<baseurl>}}/k3s/latest/en/security/hardening_guide/)
-
-
-# Rancher with SELinux
-
-[Security-Enhanced Linux (SELinux)](https://en.wikipedia.org/wiki/Security-Enhanced_Linux) is a security enhancement to Linux. After being historically used by government agencies, SELinux is now industry standard and is enabled by default on CentOS 7 and 8.
-
-To use Rancher with SELinux, we recommend installing the `rancher-selinux` RPM according to the instructions on [this page.]({{<baseurl>}}/rancher/v2.6/en/security/selinux/#installing-the-rancher-selinux-rpm)
@@ -158,7 +158,7 @@ kubectl run -i --restart=Never --rm test-${RANDOM} --image=ubuntu --overrides='{
 Enabling query logging can be done by enabling the [log plugin](https://coredns.io/plugins/log/) in the Corefile configuration in the configmap `coredns`. You can do so by using `kubectl -n kube-system edit configmap coredns` or use the command below to replace the configuration in place:

 ```
-kubectl get configmap -n kube-system coredns -o json |  kubectl get configmap -n kube-system coredns -o json | sed -e 's_loadbalance_log\\n    loadbalance_g' | kubectl apply -f -
+kubectl get configmap -n kube-system coredns -o json | sed -e 's_loadbalance_log\\n    loadbalance_g' | kubectl apply -f -
 ```

 All queries will now be logged and can be checked using the command in [Check CoreDNS logging](#check-coredns-logging).
@@ -34,7 +34,7 @@ Additional volume binds can be added to services using the `extra_binds` argumen
 services:
    kubelet:
      extra_binds:
-        - "/host/dev:/dev"
+        - "/dev:/host/dev"
        - "/usr/libexec/kubernetes/kubelet-plugins:/usr/libexec/kubernetes/kubelet-plugins:z"
 ```

@@ -148,6 +148,13 @@ https://kubic.opensuse.org/blog/2021-02-08-MicroOS-Kubic-Rancher-RKE/

 If using Red Hat Enterprise Linux, Oracle Linux or CentOS, you cannot use the `root` user as [SSH user]({{<baseurl>}}/rke/latest/en/config-options/nodes/#ssh-user) due to [Bugzilla 1527565](https://bugzilla.redhat.com/show_bug.cgi?id=1527565). Please follow the instructions below how to setup Docker correctly, based on the way you installed Docker on the node.

+>**Note:** In RHEL 8.4, two extra services are included on the NetworkManager: `nm-cloud-setup.service` and `nm-cloud-setup.timer`. These services add a routing table that interferes with the CNI plugin's configuration. If these services are enabled, you must disable them using the command below, and then reboot the node to restore connectivity:
+>  
+>  ```
+   systemctl disable nm-cloud-setup.service nm-cloud-setup.timer
+   reboot
+   ```
+
 #### Using upstream Docker
 If you are using upstream Docker, the package name is `docker-ce` or `docker-ee`. You can check the installed package by executing: