mirror of
https://github.com/rancher/rancher-docs.git
synced 2026-09-25 20:48:11 +00:00
Merge branch 'main' into add-rancher-security-best-practices
This commit is contained in:
Marty Hernandez Avedon
+2
-2
@@ -154,7 +154,7 @@ helm upgrade --install \
|
||||
--namespace ingress-nginx \
|
||||
--set controller.service.type=LoadBalancer \
|
||||
--version 4.0.18 \
|
||||
--create-namespace
|
||||
--create-namespace
|
||||
```
|
||||
|
||||
## 8. Get the Load Balancer IP
|
||||
@@ -202,4 +202,4 @@ In Rancher v2.7.5, if you intend to use the default GKE ingress on your cluster
|
||||
--set service.type=NodePort
|
||||
```
|
||||
|
||||
This is necessary because of compatibility issues between this setup and ClusterIP, the default type for `cattle-system/rancher`.
|
||||
This is necessary because of compatibility issues between this setup and ClusterIP, the default type for `cattle-system/rancher`.
|
||||
|
||||
+3
-1
@@ -17,6 +17,7 @@ plugins:
|
||||
- cattle-system
|
||||
- cattle-epinio-system
|
||||
- cattle-fleet-system
|
||||
- cattle-fleet-local-system
|
||||
- longhorn-system
|
||||
- cattle-neuvector-system
|
||||
- cattle-monitoring-system
|
||||
@@ -32,6 +33,7 @@ plugins:
|
||||
- cattle-sriov-system
|
||||
- cattle-ui-plugin-system
|
||||
- tigera-operator
|
||||
- cattle-provisioning-capi-system
|
||||
kind: PodSecurityConfiguration
|
||||
name: PodSecurity
|
||||
path: ""
|
||||
path: ""
|
||||
|
||||
+43
-18
@@ -6,29 +6,54 @@ title: Rollbacks
|
||||
<link rel="canonical" href="https://ranchermanager.docs.rancher.com/getting-started/installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster/rollbacks"/>
|
||||
</head>
|
||||
|
||||
## Additional Steps for Rollbacks with Rancher v2.6.4+
|
||||
This page outlines how to rollback Rancher to a previous version after an upgrade.
|
||||
|
||||
Rancher v2.6.4 upgrades the cluster-api module from v0.4.4 to v1.0.2. Version v1.0.2 of the cluster-api, in turn, upgrades the Cluster API's Custom Resource Definitions (CRDs) from `cluster.x-k8s.io/v1alpha4` to `cluster.x-k8s.io/v1beta1`. The CRDs upgrade to v1beta1 causes rollbacks to fail when you attempt to move from Rancher v2.6.4 to any previous version of Rancher v2.6.x. This is because CRDs that use the older apiVersion (v1alpha4) are incompatible with v1beta1.
|
||||
Follow the instructions from this page when:
|
||||
- The running Rancher instance has been upgraded to a newer version after the backup was made.
|
||||
- The upstream (local) cluster is the same as where the backup was made.
|
||||
|
||||
To avoid rollback failure, the following Rancher scripts should be run **before** you attempt a restore operation or rollback:
|
||||
:::tip
|
||||
|
||||
* `verify.sh`: Checks for any Rancher-related resources in the cluster.
|
||||
* `cleanup.sh`: Cleans up the cluster.
|
||||
|
||||
See the [rancher/rancher-cleanup repo](https://github.com/rancher/rancher-cleanup) for more details and source code.
|
||||
|
||||
:::caution
|
||||
|
||||
There will be downtime while `cleanup.sh` runs, since the script deletes resources created by Rancher.
|
||||
* Follow these steps to [migrate Rancher](../../../how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/migrate-rancher-to-new-cluster.md).
|
||||
* If you need to restore Rancher to its previous state at the same Rancher version, see the [restore documentation]( ../../../how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/restore-rancher.md).
|
||||
|
||||
:::
|
||||
|
||||
### Rolling back from v2.6.4+ to lower versions of v2.6.x
|
||||
## Alternative Steps for Special Scenarios
|
||||
|
||||
1. Follow these [instructions](https://github.com/rancher/rancher-cleanup/blob/main/README.md) to run the scripts.
|
||||
1. Follow these [instructions](../../../how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/migrate-rancher-to-new-cluster.md) to install the rancher-backup Helm chart on the existing cluster and restore the previous state.
|
||||
1. Omit Step 3.
|
||||
1. When you reach Step 4, install the Rancher v2.6.x version on the local cluster you intend to roll back to.
|
||||
Alternative steps need to be performed for rollbacks in the following scenarios:
|
||||
- Rolling back from v2.6.4 and later to an earlier version of v2.6.x.
|
||||
- Rolling back from v2.7.7 and later to an earlier version of v2.7.x.
|
||||
|
||||
In Rancher v2.6.4, the cluster-api module is upgraded from v0.4.4 to v1.0.2. The cluster-api v1.0.2, in turn, upgrades the apiVersions of its Custom Resource Definitions (CRDs) from `cluster.x-k8s.io/v1alpha4` to `cluster.x-k8s.io/v1beta1`. Custom Resources (CRs) that use the older apiVersion (v1alpha4) are incompatible with v1beta1, which causes rollbacks to fail when you attempt to move from Rancher v2.6.4 to any previous version of Rancher v2.6.x.
|
||||
|
||||
In Rancher v2.7.7, the app `rancher-provisioning-capi` is installed on the upstream (local) cluster automatically as a replacement for the embedded cluster-api controllers. Conflicts and unexpected errors will occur if the upstream cluster contains both the app, and Rancher v2.7.6 and earlier. Therefore, alternative steps are needed if you attempt to move from Rancher v2.7.7 to any previous version of Rancher v2.7.x.
|
||||
|
||||
### Step 1: Clean Up the Upstream (Local) Cluster
|
||||
|
||||
To avoid rollback failure, follow these [instructions](https://github.com/rancher/rancher-cleanup/blob/main/README.md) to run the scripts **before** you attempt a restore operation or rollback:
|
||||
|
||||
* `cleanup.sh`: Cleans up the cluster.
|
||||
* `verify.sh`: Checks for any Rancher-related resources in the cluster.
|
||||
|
||||
:::caution
|
||||
|
||||
There will be downtime while `cleanup.sh` runs, since the script deletes resources created by Rancher.
|
||||
|
||||
:::
|
||||
|
||||
**Result:** all Rancher-related resources should be cleaned up on the upstream (local) cluster.
|
||||
|
||||
See the [rancher/rancher-cleanup repo](https://github.com/rancher/rancher-cleanup) for more details and source code.
|
||||
|
||||
### Step 2: Restore the Backup and Bring Up Rancher
|
||||
|
||||
At this point, there should be no Rancher-related resources on the upstream cluster. Therefore, the next step will be the same as if you were migrating Rancher to a new cluster that contains no Rancher resources.
|
||||
|
||||
Follow these [instructions](./migrate-rancher-to-new-cluster.md) to install the Rancher-Backup Helm chart and restore Rancher to its previous state.
|
||||
Please keep in mind that:
|
||||
1. Step 3 can be skipped, because the Cert-Manager app should still exist on the upstream (local) cluster if it was installed before.
|
||||
2. At Step 4, install the Rancher version you intend to roll back to.
|
||||
|
||||
## Rolling Back to Rancher v2.5.0+
|
||||
|
||||
@@ -46,7 +71,7 @@ A restore is performed by creating a Restore custom resource.
|
||||
|
||||
:::
|
||||
|
||||
### Create the Restore Custom Resource
|
||||
### Step 1: Create the Restore Custom Resource
|
||||
|
||||
1. Click **☰ > Cluster Management**.
|
||||
1. Go to the local cluster and click **Explore**.
|
||||
@@ -95,7 +120,7 @@ kubectl get pods -n cattle-resources-system
|
||||
kubectl logs -n cattle-resources-system -f
|
||||
```
|
||||
|
||||
### Roll back to a previous Rancher version
|
||||
### Step 2: Roll Back to a Previous Rancher Version
|
||||
|
||||
Rancher can be rolled back using the Helm CLI. To roll back to the previous version:
|
||||
|
||||
|
||||
+1
-1
@@ -62,7 +62,7 @@ For information on enabling experimental features, refer to [this page.](../../.
|
||||
| `systemDefaultRegistry` | "" | `string` - private registry to be used for all system container images, e.g., http://registry.example.com/ |
|
||||
| `tls` | "ingress" | `string` - See [External TLS Termination](#external-tls-termination) for details. - "ingress, external" |
|
||||
| `useBundledSystemChart` | `false` | `bool` - select to use the system-charts packaged with Rancher server. This option is used for air gapped installations. |
|
||||
| `global.cattle.psp.enabled` | `true` | `bool` - select 'false' to disable PSPs for Kubernetes v1.25 and above when using Rancher v2.7.2-v2.7.4. When using Rancher v2.7.5 and above, Rancher attempts to detect if a cluster is running a Kubernetes version where PSPs are not supported, and will default it's usage of PSPs to false if it can determine that PSPs are not supported in the cluster. Users can still manually override this by explicitly providing `true` or `false` for this value. Rancher will still use PSPs by default in clusters which support PSPs (such as clusters running Kubernetes v1.24 or lower).|
|
||||
| `global.cattle.psp.enabled` | `true` | `bool` - select 'false' to disable PSPs for Kubernetes v1.25 and above when using Rancher v2.7.2-v2.7.4. When using Rancher v2.7.5 and above, Rancher attempts to detect if a cluster is running a Kubernetes version where PSPs are not supported, and will default it's usage of PSPs to false if it can determine that PSPs are not supported in the cluster. Users can still manually override this by explicitly providing `true` or `false` for this value. Rancher will still use PSPs by default in clusters which support PSPs (such as clusters running Kubernetes v1.24 or lower). |
|
||||
|
||||
|
||||
### Bootstrap Password
|
||||
|
||||
+2
-2
@@ -71,7 +71,7 @@ In a Kubernetes Install, if you elect to use the Rancher default self-signed TLS
|
||||
```plain
|
||||
helm repo add jetstack https://charts.jetstack.io
|
||||
helm repo update
|
||||
helm fetch jetstack/cert-manager --version v1.11.0
|
||||
helm fetch jetstack/cert-manager
|
||||
helm template ./cert-manager-<version>.tgz | awk '$1 ~ /image:/ {print $2}' | sed s/\"//g >> ./rancher-images.txt
|
||||
```
|
||||
|
||||
@@ -250,7 +250,7 @@ The workstation must have Docker 18.02+ in order to support manifests, which are
|
||||
```plain
|
||||
helm repo add jetstack https://charts.jetstack.io
|
||||
helm repo update
|
||||
helm fetch jetstack/cert-manager --version v1.11.0
|
||||
helm fetch jetstack/cert-manager
|
||||
helm template ./cert-manager-<version>.tgz | awk '$1 ~ /image:/ {print $2}' | sed s/\"//g >> ./rancher-images.txt
|
||||
```
|
||||
|
||||
|
||||
+2
-2
@@ -35,7 +35,7 @@ kubectl create namespace cert-manager
|
||||
Install the CustomResourceDefinitions of cert-manager:
|
||||
|
||||
```
|
||||
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.11.0/cert-manager.crds.yaml
|
||||
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/<VERSION>/cert-manager.crds.yaml
|
||||
```
|
||||
|
||||
And install it with Helm. Note that cert-manager also needs your proxy configured in case it needs to communicate with Let's Encrypt or other external certificate issuers:
|
||||
@@ -48,7 +48,7 @@ To see options on how to customize the cert-manager install (including for cases
|
||||
|
||||
```
|
||||
helm upgrade --install cert-manager jetstack/cert-manager \
|
||||
--namespace cert-manager --version v1.11.0 \
|
||||
--namespace cert-manager \
|
||||
--set http_proxy=http://${proxy_host} \
|
||||
--set https_proxy=http://${proxy_host} \
|
||||
--set no_proxy=127.0.0.0/8\\,10.0.0.0/8\\,cattle-system.svc\\,172.16.0.0/12\\,192.168.0.0/16\\,.svc\\,.cluster.local
|
||||
|
||||
@@ -6,6 +6,8 @@ title: Upgrading Cert-Manager
|
||||
<link rel="canonical" href="https://ranchermanager.docs.rancher.com/getting-started/installation-and-upgrade/resources/upgrade-cert-manager"/>
|
||||
</head>
|
||||
|
||||
Rancher is compatible with the API version cert-manager.io/v1 and was last tested with cert-manager version v1.13.1.
|
||||
|
||||
Rancher uses cert-manager to automatically generate and renew TLS certificates for HA deployments of Rancher. As of Fall 2019, three important changes to cert-manager are set to occur that you need to take action on if you have an HA deployment of Rancher:
|
||||
|
||||
1. [Let's Encrypt will be blocking cert-manager instances older than 0.8.0 starting November 1st 2019.](https://community.letsencrypt.org/t/blocking-old-cert-manager-versions/98753)
|
||||
@@ -96,8 +98,7 @@ In order to upgrade cert-manager, follow these instructions:
|
||||
```plain
|
||||
helm install \
|
||||
cert-manager jetstack/cert-manager \
|
||||
--namespace cert-manager \
|
||||
--version v1.11.0
|
||||
--namespace cert-manager
|
||||
```
|
||||
|
||||
1. [Restore back up resources](https://cert-manager.io/docs/tutorials/backup/#restoring-resources)
|
||||
@@ -129,7 +130,7 @@ Before you can perform the upgrade, you must prepare your air gapped environment
|
||||
1. Fetch the latest cert-manager chart available from the [Helm chart repository](https://artifacthub.io/packages/helm/cert-manager/cert-manager).
|
||||
|
||||
```plain
|
||||
helm fetch jetstack/cert-manager --version v1.11.0
|
||||
helm fetch jetstack/cert-manager
|
||||
```
|
||||
|
||||
1. Render the cert manager template with the options you would like to use to install the chart. Remember to set the `image.repository` option to pull the image from your private registry. This will create a `cert-manager` directory with the Kubernetes manifest files.
|
||||
|
||||
@@ -96,7 +96,7 @@ To enable draining each node during a cluster upgrade,
|
||||
|
||||
:::note
|
||||
|
||||
There is a [known issue](https://github.com/rancher/rancher/issues/25478) in which the Rancher UI doesn't show state of etcd and controlplane as drained, even though they are being drained.
|
||||
There is a [known issue](https://github.com/rancher/rancher/issues/25478) in which the Rancher UI doesn't show the state of etcd and controlplane as drained, even though they are being drained.
|
||||
|
||||
:::
|
||||
|
||||
|
||||
@@ -33,7 +33,7 @@ This Quick Start Guide is divided into different tasks for easier consumption.
|
||||
|
||||
### 1. Provision a Equinix Metal Host
|
||||
|
||||
Begin deploying an Equinix Metal Host. Equinix Metal Servers can be provisioned from either the Equinix Metal console, CLI, or API. You can find instructions for each deployment type on the [Equinix Metal deployment documentation](https://metal.equinix.com/developers/docs/deploy/on-demand/). You can find additional documentation on Equinix Metal server types and prices below:
|
||||
Begin deploying an Equinix Metal Host. Equinix Metal Servers can be provisioned from either the Equinix Metal console, API, or CLI. You can find instructions for each deployment type on the [Equinix Metal deployment documentation](https://metal.equinix.com/developers/docs/deploy/on-demand/). You can find additional documentation on Equinix Metal server types and prices below:
|
||||
- [Equinix Metal Server Types](https://metal.equinix.com/developers/docs/servers/about/)
|
||||
- [Equinix Metal Pricing](https://metal.equinix.com/developers/docs/servers/server-specs/)
|
||||
|
||||
|
||||
@@ -108,7 +108,7 @@ helm repo add rancher-latest https://releases.rancher.com/server-charts/latest
|
||||
|
||||
kubectl create namespace cattle-system
|
||||
|
||||
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.11.0/cert-manager.crds.yaml
|
||||
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/<VERSION>/cert-manager.crds.yaml
|
||||
|
||||
helm repo add jetstack https://charts.jetstack.io
|
||||
|
||||
@@ -116,14 +116,12 @@ helm repo update
|
||||
|
||||
helm install cert-manager jetstack/cert-manager \
|
||||
--namespace cert-manager \
|
||||
--create-namespace \
|
||||
--version v1.11.0
|
||||
--create-namespace
|
||||
|
||||
# Windows Powershell
|
||||
helm install cert-manager jetstack/cert-manager `
|
||||
--namespace cert-manager `
|
||||
--create-namespace `
|
||||
--version v1.11.0
|
||||
--create-namespace
|
||||
```
|
||||
|
||||
The final command to install Rancher is below. The command requires a domain name that forwards traffic to the Linux machine. For the sake of simplicity in this tutorial, you can use a fake domain name to create your proof-of-concept. An example of a fake domain name would be `<IP_OF_LINUX_NODE>.sslip.io`.
|
||||
|
||||
+115
-8
@@ -14,20 +14,46 @@ For configuration details, refer to the [official Kubernetes documentation](http
|
||||
|
||||
|
||||
<Tabs groupId="k8s-distro">
|
||||
<TabItem value="RKE2/K3s" default>
|
||||
<TabItem value="RKE2" default>
|
||||
|
||||
### Method 1 (Recommended): Set `audit-policy-file` in `machineGlobalConfig`
|
||||
|
||||
You can set `audit-policy-file` in the configuration file. Rancher delivers the file to the path `/var/lib/rancher/rke2/etc/config-files/audit-policy-file` in control plane nodes, and sets the proper options in the RKE2 server.
|
||||
|
||||
Example:
|
||||
```yaml
|
||||
apiVersion: provisioning.cattle.io/v1
|
||||
kind: Cluster
|
||||
spec:
|
||||
rkeConfig:
|
||||
machineGlobalConfig:
|
||||
audit-policy-file: |
|
||||
apiVersion: audit.k8s.io/v1
|
||||
kind: Policy
|
||||
rules:
|
||||
- level: RequestResponse
|
||||
resources:
|
||||
- group: ""
|
||||
resources:
|
||||
- pods
|
||||
```
|
||||
|
||||
### Method 2: Use the Directives, `machineSelectorFiles` and `machineGlobalConfig`
|
||||
|
||||
:::note
|
||||
|
||||
This feature is available in Rancher v2.7.2 and above.
|
||||
This feature is available in Rancher v2.7.2 and later.
|
||||
|
||||
:::
|
||||
|
||||
As a prerequisite, you need to create a secret or configmap which will be the source of the audit policy.
|
||||
You can use `machineSelectorFiles` to deliver the audit policy file to the control plane nodes, and `machineGlobalConfig` to set the options on kube-apiserver.
|
||||
|
||||
The secret or configmap must meet the following two requirements:
|
||||
As a prerequisite, you must create a [secret](../new-user-guides/kubernetes-resources-setup/secrets.md) or [configmap](../new-user-guides/kubernetes-resources-setup/configmaps.md) to be the source of the audit policy.
|
||||
|
||||
The secret or configmap must meet the following requirements:
|
||||
|
||||
1. It must be in the `fleet-default` namespace where the Cluster object exists.
|
||||
2. It must have the annotation `rke.cattle.io/object-authorized-for-clusters: cluster-name1,cluster-name2` which permits the target clusters to use it.
|
||||
2. It must have the annotation `rke.cattle.io/object-authorized-for-clusters: <cluster-name1>,<cluster-name2>` which permits the target clusters to use it.
|
||||
|
||||
:::tip
|
||||
|
||||
@@ -46,11 +72,11 @@ kind: Secret
|
||||
metadata:
|
||||
annotations:
|
||||
rke.cattle.io/object-authorized-for-clusters: cluster1
|
||||
name: name1
|
||||
name: <name1>
|
||||
namespace: fleet-default
|
||||
```
|
||||
|
||||
The audit log can be enabled and configured by editing the cluster in YAML and utilizing the `machineSelectorFiles` and `machineGlobalConfig` directives.
|
||||
Enable and configure the audit log by editing the cluster in YAML, and utilizing the `machineSelectorFiles` and `machineGlobalConfig` directives.
|
||||
|
||||
Example:
|
||||
|
||||
@@ -77,7 +103,88 @@ spec:
|
||||
rke.cattle.io/control-plane-role: 'true'
|
||||
```
|
||||
|
||||
For more information about cluster configuration, refer to the REK2 or K3s cluster configuration reference pages.
|
||||
:::tip
|
||||
|
||||
You can also use the directive `machineSelectorConfig` with proper machineLabelSelectors to achieve the same effect.
|
||||
|
||||
:::
|
||||
|
||||
For more information about cluster configuration, refer to the [RKE2 cluster configuration reference](../../reference-guides/cluster-configuration/rancher-server-configuration/rke2-cluster-configuration.md) pages.
|
||||
|
||||
</TabItem>
|
||||
|
||||
<TabItem value="K3s">
|
||||
|
||||
:::note
|
||||
|
||||
This feature is available in Rancher v2.7.2 and later.
|
||||
|
||||
:::
|
||||
|
||||
You can use `machineSelectorFiles` to deliver the audit policy file to the control plane nodes, and `machineGlobalConfig` to set the options on kube-apiserver.
|
||||
|
||||
As a prerequisite, you must create a [secret](../new-user-guides/kubernetes-resources-setup/secrets.md) or [configmap](../new-user-guides/kubernetes-resources-setup/configmaps.md) to be the source of the audit policy.
|
||||
|
||||
The secret or configmap must meet the following requirements:
|
||||
|
||||
1. It must be in the `fleet-default` namespace where the Cluster object exists.
|
||||
2. It must have the annotation `rke.cattle.io/object-authorized-for-clusters: <cluster-name1>,<cluster-name2>` which permits the target clusters to use it.
|
||||
|
||||
:::tip
|
||||
|
||||
Rancher Dashboard provides an easy-to-use form for creating the [secret](../new-user-guides/kubernetes-resources-setup/secrets.md) or [configmap](../new-user-guides/kubernetes-resources-setup/configmaps.md).
|
||||
|
||||
:::
|
||||
|
||||
Example:
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
data:
|
||||
audit-policy: >-
|
||||
IyBMb2cgYWxsIHJlcXVlc3RzIGF0IHRoZSBNZXRhZGF0YSBsZXZlbC4KYXBpVmVyc2lvbjogYXVkaXQuazhzLmlvL3YxCmtpbmQ6IFBvbGljeQpydWxlczoKLSBsZXZlbDogTWV0YWRhdGE=
|
||||
kind: Secret
|
||||
metadata:
|
||||
annotations:
|
||||
rke.cattle.io/object-authorized-for-clusters: cluster1
|
||||
name: <name1>
|
||||
namespace: fleet-default
|
||||
```
|
||||
|
||||
Enable and configure the audit log by editing the cluster in YAML, and utilizing the `machineSelectorFiles` and `machineGlobalConfig` directives.
|
||||
|
||||
Example:
|
||||
|
||||
```yaml
|
||||
apiVersion: provisioning.cattle.io/v1
|
||||
kind: Cluster
|
||||
spec:
|
||||
rkeConfig:
|
||||
machineGlobalConfig:
|
||||
kube-apiserver-arg:
|
||||
- audit-policy-file=<customized-path>/dev-audit-policy.yaml
|
||||
- audit-log-path=<customized-path>/dev-audit.logs
|
||||
machineSelectorFiles:
|
||||
- fileSources:
|
||||
- configMap:
|
||||
name: ''
|
||||
secret:
|
||||
items:
|
||||
- key: audit-policy
|
||||
path: <customized-path>/dev-audit-policy.yaml
|
||||
name: dev-audit-policy
|
||||
machineLabelSelector:
|
||||
matchLabels:
|
||||
rke.cattle.io/control-plane-role: 'true'
|
||||
```
|
||||
|
||||
:::tip
|
||||
|
||||
You can also use the directive `machineSelectorConfig` with proper machineLabelSelectors to achieve the same effect.
|
||||
|
||||
:::
|
||||
|
||||
For more information about cluster configuration, refer to the [K3s cluster configuration reference](../../reference-guides/cluster-configuration/rancher-server-configuration/k3s-cluster-configuration.md) pages.
|
||||
|
||||
</TabItem>
|
||||
|
||||
|
||||
-21
@@ -45,27 +45,6 @@ If the requested resources exceed the remaining capacity in the project for thos
|
||||
However, this is not the case with namespaces created **_outside_** of Rancher's UI. For namespaces created via `kubectl`, Rancher
|
||||
will assign a resource quota that has a **zero** amount for any resource that requested more capacity than what remains in the project.
|
||||
|
||||
To create a project with a resource quota via `kubectl`, use the `spec.resourceQuota` and `spec.namespaceDefaultResourceQuota` fields:
|
||||
|
||||
```
|
||||
apiVersion: management.cattle.io/v3
|
||||
kind: Project
|
||||
metadata:
|
||||
generateName: p-
|
||||
namespace: c-m-abcde
|
||||
spec:
|
||||
clusterName: c-m-abcde
|
||||
displayName: myproject
|
||||
resourceQuota:
|
||||
limit:
|
||||
limitsCpu: 1000m
|
||||
namespaceDefaultResourceQuota:
|
||||
limit:
|
||||
limitsCpu: 50m
|
||||
```
|
||||
|
||||
Use `metadata.generateName` to ensure a unique project ID. Set `metadata.namespace` and `spec.clusterName` to the ID for the cluster the project belongs to.
|
||||
|
||||
To create a namespace in an existing project via `kubectl`, use the `field.cattle.io/projectId` annotation. To override the default
|
||||
requested quota limit, use the `field.cattle.io/resourceQuota` annotation.
|
||||
|
||||
|
||||
+1
-1
@@ -12,7 +12,7 @@ The steps for setting up monitoring for workloads depend on whether you want bas
|
||||
|
||||
If you only need CPU and memory time series for the workload, you don't need to deploy a ServiceMonitor or PodMonitor because the monitoring application already collects metrics data on resource usage by default. The resource usage time series data is in Prometheus's local time series database.
|
||||
|
||||
Grafana shows the data in aggregate, but you can see the data for the individual workload by using a PromQL query that extracts the data for that workload. Once you have the PromQL query, you can execute the query individually in the Prometheus UI and see the time series visualized there, or you can use the query to customize a Grafana dashboard to display the workload metrics. For examples of PromQL queries for workload metrics, see [this section.](../../../integrations-in-rancher/monitoring-and-alerting/promql-expressions.md#workload-metrics)
|
||||
Grafana shows the data in aggregate, but you can see the data for the individual workload by using a PromQL query that extracts the data for that workload. Once you have the PromQL query, you can execute the query individually in the Prometheus UI and see the time series visualized there, or you can use the query to customize a Grafana dashboard to display the workload metrics. For examples of PromQL queries for workload metrics, see [this section](../../../integrations-in-rancher/monitoring-and-alerting/promql-expressions.md#workload-metrics).
|
||||
|
||||
To set up custom metrics for your workload, you will need to set up an exporter and create a new ServiceMonitor custom resource to configure Prometheus to scrape metrics from your exporter.
|
||||
|
||||
|
||||
@@ -6,7 +6,7 @@ title: Tuning etcd for Large Installations
|
||||
<link rel="canonical" href="https://ranchermanager.docs.rancher.com/how-to-guides/advanced-user-guides/tune-etcd-for-large-installs"/>
|
||||
</head>
|
||||
|
||||
When running larger Rancher installations with 15 or more clusters it is recommended to increase the default keyspace for etcd from the default 2GB. The maximum setting is 8GB and the host should have enough RAM to keep the entire dataset in memory. When increasing this value you should also increase the size of the host. The keyspace size can also be adjusted in smaller installations if you anticipate a high rate of change of pods during the garbage collection interval.
|
||||
When Rancher is used to manage [a large infrastructure](../../pages-for-subheaders/installation-requirements.md) it is recommended to increase the default keyspace for etcd from the default 2 GB. The maximum setting is 8 GB and the host should have enough RAM to keep the entire dataset in memory. When increasing this value you should also increase the size of the host. The keyspace size can also be adjusted in smaller installations if you anticipate a high rate of change of pods during the garbage collection interval.
|
||||
|
||||
The etcd data set is automatically cleaned up on a five minute interval by Kubernetes. There are situations, e.g. deployment thrashing, where enough events could be written to etcd and deleted before garbage collection occurs and cleans things up causing the keyspace to fill up. If you see `mvcc: database space exceeded` errors, in the etcd logs or Kubernetes API server logs, you should consider increasing the keyspace size. This can be accomplished by setting the [quota-backend-bytes](https://etcd.io/docs/v3.4.0/op-guide/maintenance/#space-quota) setting on the etcd servers.
|
||||
|
||||
|
||||
+1
-1
@@ -108,4 +108,4 @@ The OpenLDAP service account is used for all searches. Rancher users will see us
|
||||
1. Click **Okta** or, if SAML is already configured, **Edit Config**
|
||||
1. Under **User and Group Search**, check **Configure an OpenLDAP server**
|
||||
|
||||
If you experience issues when you test the connection to the OpenLDAP server, ensure that you entered the credentials for the service account and configured the search base correctly. Inspecting the Rancher logs can help pinpoint the root cause. Debug logs may contain more detailed information about the error. Please refer to [How can I enable debug logging](../../../../faq/technical-items.md#how-can-i-enable-debug-logging) for more information.
|
||||
If you experience issues when you test the connection to the OpenLDAP server, ensure that you entered the credentials for the service account and configured the search base correctly. Inspecting the Rancher logs can help pinpoint the root cause. Debug logs may contain more detailed information about the error. Please refer to [How can I enable debug logging](../../../../faq/technical-items.md#how-can-i-enable-debug-logging) for more information.
|
||||
+4
-1
@@ -85,12 +85,13 @@ Any user that is bound to the above permission will be able to change the restri
|
||||
|
||||
## Exempting Required Rancher Namespaces
|
||||
|
||||
When you run Rancher on a Kubernetes cluster that enforces a restrictive security policy by default, you'll need to [exempt the following namespaces](#exempting-namespaces), otherwise the policy might prevent Rancher system pods from running properly.
|
||||
When you run Rancher on a Kubernetes cluster that enforces a restrictive security policy by default, you'll need to [exempt the following namespaces](#exempting-namespaces), otherwise the policy might prevent Rancher system pods from running properly.
|
||||
|
||||
- `calico-apiserver`
|
||||
- `calico-system`
|
||||
- `cattle-alerting`
|
||||
- `cattle-csp-adapter-system`
|
||||
- `cattle-elemental-system`
|
||||
- `cattle-epinio-system`
|
||||
- `cattle-externalip-system`
|
||||
- `cattle-fleet-local-system`
|
||||
@@ -106,6 +107,8 @@ When you run Rancher on a Kubernetes cluster that enforces a restrictive securit
|
||||
- `cattle-monitoring-system`
|
||||
- `cattle-neuvector-system`
|
||||
- `cattle-prometheus`
|
||||
- `cattle-provisioning-capi-system`
|
||||
- `cattle-resources-system`
|
||||
- `cattle-sriov-system`
|
||||
- `cattle-system`
|
||||
- `cattle-ui-plugin-system`
|
||||
|
||||
+13
-28
@@ -8,39 +8,24 @@ title: Restoring Rancher
|
||||
|
||||
This page outlines how to perform a restore with Rancher.
|
||||
|
||||
:::note Important:
|
||||
Follow the instructions from this page when:
|
||||
- The running Rancher instance is at the same version as when the backup was made.
|
||||
- The upstream (local) cluster is the same as where the backup was made.
|
||||
|
||||
* Follow the instructions from this page for restoring rancher on the same cluster where it was backed up from. In order to migrate rancher to a new cluster, follow the steps to [migrate rancher.](migrate-rancher-to-new-cluster.md)
|
||||
* While restoring rancher on the same setup, the operator will scale down the rancher deployment when restore starts, and it will scale back up the deployment once restore completes. So Rancher will be unavailable during the restore.
|
||||
* If you need to restore Rancher to a previous version after an upgrade, see the [rollback documentation.](../../../getting-started/installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster/rollbacks.md)
|
||||
:::note Important
|
||||
|
||||
While restoring Rancher on the same setup, the operator will scale down the Rancher deployment when restore starts, and it will scale back up the deployment once restore completes. Rancher will be unavailable during the restore.
|
||||
|
||||
:::
|
||||
|
||||
## Additional Steps for Rollbacks with Rancher v2.6.4+
|
||||
:::tip
|
||||
|
||||
Rancher v2.6.4 upgrades the cluster-api module from v0.4.4 to v1.0.2. Version v1.0.2 of the cluster-api, in turn, upgrades the Cluster API's Custom Resource Definitions (CRDs) from `cluster.x-k8s.io/v1alpha4` to `cluster.x-k8s.io/v1beta1`. The CRDs upgrade to v1beta1 causes rollbacks to fail when you attempt to move from Rancher v2.6.4 to any previous version of Rancher v2.6.x. This is because CRDs that use the older apiVersion (v1alpha4) are incompatible with v1beta1.
|
||||
|
||||
To avoid rollback failure, the following Rancher scripts should be run **before** you attempt a restore operation or rollback:
|
||||
|
||||
* `verify.sh`: Checks for any Rancher-related resources in the cluster.
|
||||
* `cleanup.sh`: Cleans up the cluster.
|
||||
|
||||
See the [rancher/rancher-cleanup repo](https://github.com/rancher/rancher-cleanup) for more details and source code.
|
||||
|
||||
:::caution
|
||||
|
||||
There will be downtime while `cleanup.sh` runs, since the script deletes resources created by Rancher.
|
||||
* Follow those steps to [migrate Rancher](migrate-rancher-to-new-cluster.md).
|
||||
* If you need to restore Rancher to a previous version after an upgrade, see the [rollback documentation](../../../getting-started/installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster/rollbacks.md).
|
||||
|
||||
:::
|
||||
|
||||
### Rolling back from v2.6.4+ to lower versions of v2.6.x
|
||||
|
||||
1. Follow these [instructions](https://github.com/rancher/rancher-cleanup/blob/main/README.md) to run the scripts.
|
||||
1. Follow these [instructions](../backup-restore-and-disaster-recovery/migrate-rancher-to-new-cluster.md) to install the rancher-backup Helm chart on the existing cluster and restore the previous state.
|
||||
1. Omit Step 3.
|
||||
1. When you reach Step 4, install the Rancher v2.6.x version on the local cluster you intend to roll back to.
|
||||
|
||||
### Create the Restore Custom Resource
|
||||
## Create the Restore Custom Resource
|
||||
|
||||
A restore is performed by creating a Restore custom resource.
|
||||
|
||||
@@ -79,7 +64,7 @@ A restore is performed by creating a Restore custom resource.
|
||||
2. Cluster-scoped resources
|
||||
3. Namespaced resources
|
||||
|
||||
### Logs
|
||||
## Logs
|
||||
|
||||
To check how the restore is progressing, you can check the logs of the operator. Run this command to follow the logs:
|
||||
|
||||
@@ -87,11 +72,11 @@ To check how the restore is progressing, you can check the logs of the operator.
|
||||
kubectl logs -n cattle-resources-system -l app.kubernetes.io/name=rancher-backup -f
|
||||
```
|
||||
|
||||
### Cleanup
|
||||
## Cleanup
|
||||
|
||||
If you created the restore resource with kubectl, remove the resource to prevent a naming conflict with future restores.
|
||||
|
||||
### Known Issues
|
||||
## Known Issues
|
||||
In some cases, after restoring the backup, Rancher logs will show errors similar to the following:
|
||||
```
|
||||
2021/10/05 21:30:45 [ERROR] error syncing 'c-89d82/m-4067aa68dd78': handler rke-worker-upgrader: clusters.management.cattle.io "c-89d82" not found, requeuing
|
||||
|
||||
+1
-1
@@ -111,4 +111,4 @@ After creating your cluster, you can access it through the Rancher UI. As a best
|
||||
|
||||
- **Access your cluster with the kubectl CLI:** Follow [these steps](../../../../new-user-guides/manage-clusters/access-clusters/use-kubectl-and-kubeconfig.md#accessing-clusters-with-kubectl-from-your-workstation) to access clusters with kubectl on your workstation. In this case, you will be authenticated through the Rancher server’s authentication proxy, then Rancher will connect you to the downstream cluster. This method lets you manage the cluster without the Rancher UI.
|
||||
- **Access your cluster with the kubectl CLI, using the authorized cluster endpoint:** Follow [these steps](../../../../new-user-guides/manage-clusters/access-clusters/use-kubectl-and-kubeconfig.md#authenticating-directly-with-a-downstream-cluster) to access your cluster with kubectl directly, without authenticating through Rancher. We recommend setting up this alternative method to access your cluster so that in case you can’t connect to Rancher, you can still access the cluster.
|
||||
- **Provision Storage:** For an example of how to provision storage in vSphere using Rancher, refer to [this section.](../../../../../pages-for-subheaders/provisioning-storage-examples.md) In order to dynamically provision storage in vSphere, the vSphere provider must be enabled. For details, refer to [in-tree vSphere cloud provider docs](../../../../../how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/set-up-cloud-providers/configure-in-tree-vsphere.md) and [out-of-tree vSphere cloud provider docs](../../../../../how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/set-up-cloud-providers/configure-out-of-tree-vsphere.md).
|
||||
- **Provision Storage:** For an example of how to provision storage in vSphere using Rancher, refer to [this section.](../../../../../pages-for-subheaders/provisioning-storage-examples.md) In order to dynamically provision storage in vSphere, the vSphere provider must be enabled. For details, refer to [in-tree vSphere cloud provider docs](../../../../../how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/set-up-cloud-providers/configure-in-tree-vsphere.md) and [out-of-tree vSphere cloud provider docs](../../../../../how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/set-up-cloud-providers/configure-out-of-tree-vsphere.md).
|
||||
+1
-1
@@ -156,4 +156,4 @@ Finally, restart the rancher-csp-adapter deployment to ensure that the updated v
|
||||
kubectl rollout restart deploy rancher-csp-adapter -n cattle-csp-adapter-system
|
||||
```
|
||||
|
||||
> **Note:** There are methods such as cert-manager's [trust operator](https://cert-manager.io/docs/projects/trust/) which can help reduce the number of manual rotation tasks over time. While these options are not officially supported, they may be useful to users wishing to automate some of these tasks.
|
||||
> **Note:** There are methods such as cert-manager's [trust operator](https://cert-manager.io/docs/projects/trust/) which can help reduce the number of manual rotation tasks over time. While these options are not officially supported, they may be useful to users wishing to automate some of these tasks.
|
||||
@@ -103,4 +103,4 @@ Configure the settings for the OpenLDAP server, groups and users. For help filli
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
If you are experiencing issues while testing the connection to the OpenLDAP server, first double-check the credentials entered for the service account as well as the search base configuration. You may also inspect the Rancher logs to help pinpointing the problem cause. Debug logs may contain more detailed information about the error. Please refer to [How can I enable debug logging](../faq/technical-items.md#how-can-i-enable-debug-logging) in this documentation.
|
||||
If you are experiencing issues while testing the connection to the OpenLDAP server, first double-check the credentials entered for the service account as well as the search base configuration. You may also inspect the Rancher logs to help pinpointing the problem cause. Debug logs may contain more detailed information about the error. Please refer to [How can I enable debug logging](../faq/technical-items.md#how-can-i-enable-debug-logging) in this documentation.
|
||||
@@ -149,7 +149,7 @@ To see options on how to customize the cert-manager install (including for cases
|
||||
|
||||
```
|
||||
# If you have installed the CRDs manually instead of with the `--set installCRDs=true` option added to your Helm install command, you should upgrade your CRD resources before upgrading the Helm chart:
|
||||
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.11.0/cert-manager.crds.yaml
|
||||
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/<VERSION>/cert-manager.crds.yaml
|
||||
|
||||
# Add the Jetstack Helm repository
|
||||
helm repo add jetstack https://charts.jetstack.io
|
||||
@@ -160,8 +160,7 @@ helm repo update
|
||||
# Install the cert-manager Helm chart
|
||||
helm install cert-manager jetstack/cert-manager \
|
||||
--namespace cert-manager \
|
||||
--create-namespace \
|
||||
--version v1.11.0
|
||||
--create-namespace
|
||||
```
|
||||
|
||||
Once you’ve installed cert-manager, you can verify it is deployed correctly by checking the cert-manager namespace for running pods:
|
||||
|
||||
@@ -23,6 +23,10 @@ See our page on [best practices](../reference-guides/best-practices/rancher-serv
|
||||
|
||||
Rancher needs to be installed on a supported Kubernetes version. Consult the [Rancher support matrix](https://www.suse.com/suse-rancher/support-matrix/all-supported-versions) to ensure that your intended version of Kubernetes is supported.
|
||||
|
||||
### Install Rancher on a Hardened Kubernetes cluster
|
||||
|
||||
If you install Rancher on a hardened Kubernetes cluster, check the [Exempting Required Rancher Namespaces](../../../docs/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/psa-config-templates.md#exempting-required-rancher-namespaces) section for detailed requirements.
|
||||
|
||||
## Operating Systems and Container Runtime Requirements
|
||||
|
||||
All supported operating systems are 64-bit x86. Rancher should work with any modern Linux distribution.
|
||||
@@ -39,11 +43,11 @@ If you don't feel comfortable doing so, you might check suggestions in the [resp
|
||||
|
||||
If you plan to run Rancher on ARM64, see [Running on ARM64 (Experimental).](../how-to-guides/advanced-user-guides/enable-experimental-features/rancher-on-arm64.md)
|
||||
|
||||
### RKE Specific Requirements
|
||||
### RKE2 Specific Requirements
|
||||
|
||||
For the container runtime, RKE should work with any modern Docker version.
|
||||
RKE2 bundles its own container runtime, containerd. Docker is not required for RKE2 installs.
|
||||
|
||||
For more information see [Installing Docker,](../getting-started/installation-and-upgrade/installation-requirements/install-docker.md)
|
||||
For details on which OS versions were tested with RKE2, refer to the [Rancher support matrix](https://www.suse.com/suse-rancher/support-matrix/all-supported-versions).
|
||||
|
||||
### K3s Specific Requirements
|
||||
|
||||
@@ -55,68 +59,126 @@ If you are installing Rancher on a K3s cluster with **Raspbian Buster**, follow
|
||||
|
||||
If you are installing Rancher on a K3s cluster with Alpine Linux, follow [these steps](https://rancher.com/docs/k3s/latest/en/advanced/#additional-preparation-for-alpine-linux-setup) for additional setup.
|
||||
|
||||
### RKE2 Specific Requirements
|
||||
### RKE Specific Requirements
|
||||
|
||||
For the container runtime, RKE2 bundles its own containerd. Docker is not required for RKE2 installs.
|
||||
RKE requires a Docker container runtime. Supported Docker versions are specified in the [Support Matrix](https://www.suse.com/suse-rancher/support-matrix/all-supported-versions/) page.
|
||||
|
||||
For details on which OS versions were tested with RKE2, refer to the [Rancher support matrix](https://www.suse.com/suse-rancher/support-matrix/all-supported-versions).
|
||||
For more information, see [Installing Docker](../getting-started/installation-and-upgrade/installation-requirements/install-docker.md).
|
||||
|
||||
## Hardware Requirements
|
||||
|
||||
The following sections describe the CPU, memory, and disk requirements for the nodes where the Rancher server is installed.
|
||||
The following sections describe the CPU, memory, and I/O requirements for nodes where Rancher is installed. Requirements vary based on the size of the infrastructure.
|
||||
|
||||
## CPU and Memory
|
||||
### Practical Considerations
|
||||
|
||||
Hardware requirements scale based on the size of your Rancher deployment. Provision each individual node according to the requirements. The requirements are different depending on if you are installing Rancher in a single container with Docker, or if you are installing Rancher on a Kubernetes cluster.
|
||||
Rancher's hardware footprint depends on a number of factors, including:
|
||||
|
||||
### RKE and Hosted Kubernetes
|
||||
- Size of the managed infrastructure (e.g., node count, cluster count).
|
||||
- Complexity of the desired access control rules (e.g., `RoleBinding` object count).
|
||||
- Number of workloads (e.g., Kubernetes deployments, Fleet deployments).
|
||||
- Usage patterns (e.g., subset of functionality actively used, frequency of use, number of concurrent users).
|
||||
|
||||
These CPU and memory requirements apply to each host in the Kubernetes cluster where the Rancher server is installed.
|
||||
Since there are a high number of influencing factors that may vary over time, the requirements listed here should be understood as reasonable starting points that work well for most use cases. Nevertheless, your use case may have different requirements. For inquiries about a specific scenario please [contact Rancher](https://rancher.com/contact/) for further guidance.
|
||||
|
||||
These requirements apply to RKE Kubernetes clusters, as well as to hosted Kubernetes clusters such as EKS.
|
||||
In particular, requirements on this page are subject to typical use assumptions, which include:
|
||||
- Under 60,000 total Kubernetes resources, per type.
|
||||
- Up to 120 pods per node.
|
||||
- Up to 200 CRDs in the upstream (local) cluster.
|
||||
- Up to 100 CRDs in downstream clusters.
|
||||
- Up to 50 Fleet deployments.
|
||||
|
||||
| Deployment Size | Clusters | Nodes | vCPUs | RAM |
|
||||
| --------------- | ---------- | ------------ | -------| ------- |
|
||||
| Small | Up to 150 | Up to 1500 | 2 | 8 GB |
|
||||
| Medium | Up to 300 | Up to 3000 | 4 | 16 GB |
|
||||
| Large | Up to 500 | Up to 5000 | 8 | 32 GB |
|
||||
| X-Large | Up to 1000 | Up to 10,000 | 16 | 64 GB |
|
||||
| XX-Large | Up to 2000 | Up to 20,000 | 32 | 128 GB |
|
||||
Higher numbers are possible but requirements might be higher. If you have more than 20,000 resources of the same type, loading time of the whole list through the Rancher UI might take several seconds.
|
||||
|
||||
Every use case and environment is different. Please [contact Rancher](https://rancher.com/contact/) to review yours.
|
||||
:::note Evolution:
|
||||
|
||||
### K3s Kubernetes
|
||||
Rancher's codebase evolves, use cases change, and the body of accumulated Rancher experience grows every day.
|
||||
|
||||
These CPU and memory requirements apply to each host in a [K3s Kubernetes cluster where the Rancher server is installed.](install-upgrade-on-a-kubernetes-cluster.md)
|
||||
Hardware requirement recommendations are subject to change over time, as guidelines improve in accuracy and become more concrete.
|
||||
|
||||
| Deployment Size | Clusters | Nodes | vCPUs | RAM | Database Size |
|
||||
| --------------- | ---------- | ------------ | -------| ---------| ------------------------- |
|
||||
| Small | Up to 150 | Up to 1500 | 2 | 8 GB | 2 cores, 4 GB + 1000 IOPS |
|
||||
| Medium | Up to 300 | Up to 3000 | 4 | 16 GB | 2 cores, 4 GB + 1000 IOPS |
|
||||
| Large | Up to 500 | Up to 5000 | 8 | 32 GB | 2 cores, 4 GB + 1000 IOPS |
|
||||
| X-Large | Up to 1000 | Up to 10,000 | 16 | 64 GB | 2 cores, 4 GB + 1000 IOPS |
|
||||
| XX-Large | Up to 2000 | Up to 20,000 | 32 | 128 GB | 2 cores, 4 GB + 1000 IOPS |
|
||||
If you find that your Rancher deployment no longer complies with the listed recommendations, [contact Rancher](https://rancher.com/contact/) for a re-evaluation.
|
||||
|
||||
Every use case and environment is different. Please [contact Rancher](https://rancher.com/contact/) to review yours.
|
||||
:::
|
||||
|
||||
|
||||
### RKE2 Kubernetes
|
||||
|
||||
These CPU and memory requirements apply to each instance with RKE2 installed. Minimum recommendations are outlined here.
|
||||
The following table lists minimum CPU and memory requirements for each node in the [upstream cluster](install-upgrade-on-a-kubernetes-cluster.md).
|
||||
|
||||
| Deployment Size | Clusters | Nodes | vCPUs | RAM |
|
||||
| --------------- | -------- | --------- | ----- | ---- |
|
||||
| Small | Up to 5 | Up to 50 | 2 | 5 GB |
|
||||
| Medium | Up to 15 | Up to 200 | 3 | 9 GB |
|
||||
Please note that a highly available setup with at least three nodes is required for production.
|
||||
|
||||
| Managed Infrastructure Size | Maximum Number of Clusters | Maximum Number of Nodes | vCPUs | RAM |
|
||||
|-----------------------------|----------------------------|-------------------------|-------|-------|
|
||||
| Small | 150 | 1500 | 4 | 16 GB |
|
||||
| Medium | 300 | 3000 | 8 | 32 GB |
|
||||
| Large (*) | 500 | 5000 | 16 | 64 GB |
|
||||
| Larger (†) | (†) | (†) | (†) | (†) |
|
||||
|
||||
(*): Large deployments require that you [follow best practices](../reference-guides/best-practices/rancher-server/tuning-and-best-practices-for-rancher-at-scale.md) for adequate performance.
|
||||
|
||||
(†): Larger deployment sizes are generally possible with ad-hoc hardware recommendations and tuning. You can [contact Rancher](https://rancher.com/contact/) for a custom evaluation.
|
||||
|
||||
Refer to RKE2 documentation for more detailed information on [RKE2 general requirements](https://docs.rke2.io/install/requirements).
|
||||
|
||||
### K3s Kubernetes
|
||||
|
||||
The following table lists minimum CPU and memory requirements for each node in the [upstream cluster](install-upgrade-on-a-kubernetes-cluster.md).
|
||||
|
||||
Please note that a highly available setup with at least three nodes is required for production.
|
||||
|
||||
| Managed Infrastructure Size | Maximum Number of Clusters | Maximum Number of Nodes | vCPUs | RAM | External Database Host (*) |
|
||||
|-----------------------------|----------------------------|-------------------------|-------|-------|----------------------------|
|
||||
| Small | 150 | 1500 | 4 | 16 GB | 2 vCPUs, 8 GB + 1000 IOPS |
|
||||
| Medium | 300 | 3000 | 8 | 32 GB | 4 vCPUs, 16 GB + 2000 IOPS |
|
||||
| Large (†) | 500 | 5000 | 16 | 64 GB | 8 vCPUs, 32 GB + 4000 IOPS |
|
||||
|
||||
(*): External Database Host refers to hosting the K3s cluster data store on an [dedicated external host](https://docs.k3s.io/datastore). This is optional. Exact requirements depend on the external data store.
|
||||
|
||||
(†): Large deployments require that you [follow best practices](../reference-guides/best-practices/rancher-server/tuning-and-best-practices-for-rancher-at-scale.md) for adequate performance.
|
||||
|
||||
Refer to the K3s documentation for more detailed information on [general requirements](https://docs.k3s.io/installation/requirements).
|
||||
|
||||
### Hosted Kubernetes
|
||||
|
||||
The following table lists minimum CPU and memory requirements for each node in the [upstream cluster](install-upgrade-on-a-kubernetes-cluster.md).
|
||||
|
||||
Please note that a highly available setup with at least three nodes is required for production.
|
||||
|
||||
These requirements apply to hosted Kubernetes clusters such as Amazon Elastic Kubernetes Service (EKS), Azure Kubernetes Service (AKS), or Google Kubernetes Engine (GKE). They don't apply to Rancher SaaS solutions such as [Rancher Prime Hosted](https://www.rancher.com/products/rancher).
|
||||
|
||||
| Managed Infrastructure Size | Maximum Number of Clusters | Maximum Number of Nodes | vCPUs | RAM |
|
||||
|-----------------------------|----------------------------|-------------------------|-------|-------|
|
||||
| Small | 150 | 1500 | 4 | 16 GB |
|
||||
| Medium | 300 | 3000 | 8 | 32 GB |
|
||||
| Large (*) | 500 | 5000 | 16 | 64 GB |
|
||||
|
||||
(*): Large deployments require that you [follow best practices](../reference-guides/best-practices/rancher-server/tuning-and-best-practices-for-rancher-at-scale.md) for adequate performance.
|
||||
|
||||
### RKE
|
||||
|
||||
The following table lists minimum CPU and memory requirements for each node in the [upstream cluster](install-upgrade-on-a-kubernetes-cluster.md).
|
||||
|
||||
Please note that a highly available setup with at least three nodes is required for production.
|
||||
|
||||
| Managed Infrastructure Size | Maximum Number of Clusters | Maximum Number of Nodes | vCPUs | RAM |
|
||||
|-----------------------------|----------------------------|-------------------------|-------|-------|
|
||||
| Small | 150 | 1500 | 4 | 16 GB |
|
||||
| Medium | 300 | 3000 | 8 | 32 GB |
|
||||
| Large (*) | 500 | 5000 | 16 | 64 GB |
|
||||
|
||||
(*): Large deployments require that you [follow best practices](../reference-guides/best-practices/rancher-server/tuning-and-best-practices-for-rancher-at-scale.md) for adequate performance.
|
||||
|
||||
Refer to the RKE documentation for more detailed information on [general requirements](https://rke.docs.rancher.com/os).
|
||||
|
||||
### Docker
|
||||
|
||||
These CPU and memory requirements apply to a host with a [single-node](rancher-on-a-single-node-with-docker.md) installation of Rancher.
|
||||
The following table lists minimum CPU and memory requirements for a [single Docker node installation of Rancher](rancher-on-a-single-node-with-docker.md).
|
||||
|
||||
| Deployment Size | Clusters | Nodes | vCPUs | RAM |
|
||||
| --------------- | -------- | --------- | ----- | ---- |
|
||||
| Small | Up to 5 | Up to 50 | 1 | 4 GB |
|
||||
| Medium | Up to 15 | Up to 200 | 2 | 8 GB |
|
||||
Please note that a Docker installation is only suitable for development or testing purposes and is not meant to be used in production environments.
|
||||
|
||||
| Managed Infrastructure Size | Maximum Number of Clusters | Maximum Number of Nodes | vCPUs | RAM |
|
||||
|-----------------------------|----------------------------|-------------------------|-------|------|
|
||||
| Small | 5 | 50 | 1 | 4 GB |
|
||||
| Medium | 15 | 200 | 2 | 8 GB |
|
||||
|
||||
## Ingress
|
||||
|
||||
|
||||
@@ -88,7 +88,6 @@ Rancher is committed to informing the community of security issues in our produc
|
||||
|
||||
For recommendations on securing your Kubernetes cluster, refer to the [Kubernetes Security Best Practices](../reference-guides/rancher-security/kubernetes-security-best-practices.md) guide.
|
||||
|
||||
|
||||
### Rancher Security Best Practices
|
||||
|
||||
For recommendations on securing your Rancher Manager deployments, refer to the [Rancher Security Best Practices](../reference-guides/rancher-security/rancher-security-best-practices.md) guide.
|
||||
|
||||
@@ -209,41 +209,44 @@ services:
|
||||
exemptions:
|
||||
usernames: []
|
||||
runtimeClasses: []
|
||||
namespaces: [ calico-apiserver,
|
||||
calico-system,
|
||||
cattle-alerting,
|
||||
cattle-csp-adapter-system,
|
||||
cattle-epinio-system,
|
||||
cattle-externalip-system,
|
||||
cattle-fleet-local-system,
|
||||
cattle-fleet-system,
|
||||
cattle-gatekeeper-system,
|
||||
cattle-global-data,
|
||||
cattle-global-nt,
|
||||
cattle-impersonation-system,
|
||||
cattle-istio,
|
||||
cattle-istio-system,
|
||||
cattle-logging,
|
||||
cattle-logging-system,
|
||||
cattle-monitoring-system,
|
||||
cattle-neuvector-system,
|
||||
cattle-prometheus,
|
||||
cattle-sriov-system,
|
||||
cattle-system,
|
||||
cattle-ui-plugin-system,
|
||||
cattle-windows-gmsa-system,
|
||||
cert-manager,
|
||||
cis-operator-system,
|
||||
fleet-default,
|
||||
ingress-nginx,
|
||||
istio-system,
|
||||
kube-node-lease,
|
||||
kube-public,
|
||||
kube-system,
|
||||
longhorn-system,
|
||||
rancher-alerting-drivers,
|
||||
security-scan,
|
||||
tigera-operator ]
|
||||
namespaces: [calico-apiserver,
|
||||
calico-system,
|
||||
cattle-alerting,
|
||||
cattle-csp-adapter-system,
|
||||
cattle-elemental-system,
|
||||
cattle-epinio-system,
|
||||
cattle-externalip-system,
|
||||
cattle-fleet-local-system,
|
||||
cattle-fleet-system,
|
||||
cattle-gatekeeper-system,
|
||||
cattle-global-data,
|
||||
cattle-global-nt,
|
||||
cattle-impersonation-system,
|
||||
cattle-istio,
|
||||
cattle-istio-system,
|
||||
cattle-logging,
|
||||
cattle-logging-system,
|
||||
cattle-monitoring-system,
|
||||
cattle-neuvector-system,
|
||||
cattle-prometheus,
|
||||
cattle-provisioning-capi-system,
|
||||
cattle-resources-system,
|
||||
cattle-sriov-system,
|
||||
cattle-system,
|
||||
cattle-ui-plugin-system,
|
||||
cattle-windows-gmsa-system,
|
||||
cert-manager,
|
||||
cis-operator-system,
|
||||
fleet-default,
|
||||
ingress-nginx,
|
||||
istio-system,
|
||||
kube-node-lease,
|
||||
kube-public,
|
||||
kube-system,
|
||||
longhorn-system,
|
||||
rancher-alerting-drivers,
|
||||
security-scan,
|
||||
tigera-operator]
|
||||
kube-controller:
|
||||
extra_args:
|
||||
feature-gates: RotateKubeletServerCertificate=true
|
||||
|
||||
@@ -8,7 +8,7 @@ description: "Rancher adds significant value on top of Kubernetes: managing hund
|
||||
<head>
|
||||
<link rel="canonical" href="https://ranchermanager.docs.rancher.com"/>
|
||||
</head>
|
||||
|
||||
|
||||
Rancher is a Kubernetes management tool to deploy and run clusters anywhere and on any provider.
|
||||
|
||||
Rancher can provision Kubernetes from a hosted provider, provision compute nodes and then install Kubernetes onto them, or import existing Kubernetes clusters running anywhere.
|
||||
|
||||
@@ -1,65 +0,0 @@
|
||||
---
|
||||
title: Tips for Scaling Rancher
|
||||
---
|
||||
|
||||
<head>
|
||||
<link rel="canonical" href="https://ranchermanager.docs.rancher.com/reference-guides/best-practices/rancher-server/tips-for-scaling-rancher"/>
|
||||
</head>
|
||||
|
||||
This guide aims to introduce the approaches that should be considered to scale Rancher setups, and associated challenges with doing so. As systems grow performance will naturally reduce, but there are steps we can take to minimize the load put on Rancher, as well as optimize Rancher's ability to handle these larger setups.
|
||||
|
||||
## General Tips on Optimizing Rancher's Performance
|
||||
* It is advisable to keep Rancher up to date with patch releases. Performance improvements and bug fixes are made throughout the life of a minor release. You can review the release notes to help inform your own decisions on whether an upgrade is necessary but we recommend keeping yourself up to date in most cases.
|
||||
|
||||
* Performance will be negatively impacted by increased latency between Rancher's infrastructure and a downstream cluster's infrastructure (eg. geographic distance). If a user or organization requires clusters/nodes all over the world or spread across many regions, it is best to use multiple Rancher installations.
|
||||
|
||||
* Please always try to scale up gradually, monitoring and observing any change in behavior while doing do. It is usually easier to resolve performance problems as soon as they surface, and before other problems confuse symptoms.
|
||||
|
||||
## Minimizing Load on the local cluster
|
||||
The largest bottleneck when scaling Rancher is resource growth in the local Kubernetes cluster. The local cluster contains information for all downstream clusters. Many operations that apply to downstream clusters will create new objects in the local cluster and require computation from handlers running in the local cluster.
|
||||
|
||||
### Managing Your Object Counts
|
||||
ETCD eventually encounters limitations to the number of a single Kubernetes resource type it can store. These exact numbers are not well documented. From internal observations we usually see performance issues once a single resource type's object count exceeds 60k, and often that type is Rolebindings.
|
||||
|
||||
Rolebindings are created in the local cluster as a side effect of many operations.
|
||||
|
||||
Considerations when attempting reduce rolebindings in the local cluster:
|
||||
* Only add users to clusters and projects when necessary
|
||||
* Remove clusters and projects when they are no longer needed
|
||||
* Only use custom roles if necessary
|
||||
* Use as few rules as possible in custom roles
|
||||
* Consider whether adding a role to a user is redundant
|
||||
* Consider that using less, but more powerful, clusters may be more efficient
|
||||
* Experiment to see if creating new projects or creating new clusters manifests in fewer rolebindings for your specific use case.
|
||||
|
||||
### Using New Apps Over Legacy Apps
|
||||
There are two app kubernetes resources that Rancher uses: apps.projects.cattle.io and apps.cattle.cattle.io. The legacy apps, apps.projects.cattle.io, were introduced first in the Cluster Manager and are now outdated. The new apps, apps.catalog.cattle.io, are found in the Cluster Explorer for their respective cluster. The new apps are preferrable because they live in the downstream cluster while the legacy apps live in the local cluster.
|
||||
|
||||
We recommend removing apps that appear in the Cluster Manager, replacing them with apps in the Cluster Explorer for their target cluster if necessary and creating any future apps in the cluster's Cluster Explorer only.
|
||||
|
||||
### Using the Authorized Cluster Endpoint (ACE)
|
||||
There is an _Authorized Cluster Endpoint_ option for Rancher provisioned RKE1, RKE2, and K3s clusters. When enabled this adds a context to kubeconfigs generated for the cluster that uses a direct endpoint to the cluster and bypasses Rancher. However, it is not enough to only enable this option. The user of the Kubeconfig needs to use `kubectl use-context <ace context name>` in order to start using it.
|
||||
|
||||
Without using ACE, all kubeconfig requests first route through Rancher.
|
||||
|
||||
### Experimental: Option to Reduce Event Handler Executions
|
||||
The bulk of Rancher's logic occurs on event handlers. These event handlers run on an object whenever the object is updated, and when Rancher is started. Additionally, they run every 15 hours when caches are synced. In scaled setups these scheduled runs come with huge performance costs because every handler is being run on every applicable object. However, this scheduled execution of handlers can be disabled using the CATTLE_SYNC_ONLY_CHANGED_OBJECTS environment variable. If resource allocation spikes are seen on an interval of about 15 hours it is possible this setting can help.
|
||||
|
||||
The value for the environment variable can be a comma separated list of the following options. The values refer to types of controllers (the structures that contain and run handlers) and their handlers. Adding the controller types to the variable will disable that set of controllers from running their handlers as part of cache resyncing.
|
||||
|
||||
* `mgmt` refers to management controllers which only run on one Rancher node.
|
||||
* `user` refers to user controllers which run for every cluster. Some of these are ran on the same node as management controllers, while other run in the downstream cluster. This will option targets the former.
|
||||
* `scaled` refers to scaled controllers which run on every Rancher node. This is not recommended to be set due to the critical functionality the scaled handlers are responsible for.
|
||||
|
||||
In short, if you notice CPU usage peaks every 15 hours, add the CATTLE_SYNC_ONLY_CHANGED_OBJECTS environment variable to your rancher deployment with the value `mgmt,user`.
|
||||
|
||||
## Optimizations Outside of Rancher
|
||||
A large component of performance is the local cluster and how it was configured. This cluster can introduce a bottleneck before Rancher software ever runs. When Rancher nodes experience high resource usage, you can use the command "top" to identify whether it is Rancher or a Kubernetes component that is consuming the resource in excess.
|
||||
|
||||
### Keeping Kubernetes Versions Up to Date
|
||||
Similar to Rancher versions, it is advisable to keep your kubernetes cluster up to date. This will ensure that your cluster contains any available performance enhancements or bug fixes.
|
||||
|
||||
### Optimizing ETCD
|
||||
The two main bottlenecks to [ETCD performance](https://etcd.io/docs/v3.4/op-guide/performance/) are disk speed and network speed. Optimization to either should improve performance. For information regarding ETCD performance see [Slow etcd performance (performance testing and optimization)](https://www.suse.com/support/kb/doc/?id=000020100) and [Tuning etcd for Large Installations](https://docs.ranchermanager.rancher.io/how-to-guides/advanced-user-guides/tune-etcd-for-large-installs). Information on disks can also be found [in our docs](https://docs.Ranchermanager.Rancher.io/v2.5/pages-for-subheaders/installation-requirements#disks).
|
||||
|
||||
Theoretically, the more nodes in an ETCD cluster the slower it will be due to replication requirements [source](https://etcd.io/docs/v3.3/faq). This may be counter-intuitive to common scaling approaches. It can also be inferred that ETCD performance will be inversely affected by distance between nodes as that will slow down network communication.
|
||||
+100
@@ -0,0 +1,100 @@
|
||||
---
|
||||
title: Tuning and Best Practices for Rancher at Scale
|
||||
---
|
||||
|
||||
<head>
|
||||
<link rel="canonical" href="https://ranchermanager.docs.rancher.com/reference-guides/best-practices/rancher-server/tuning-and-best-practices-for-rancher-at-scale"/>
|
||||
</head>
|
||||
|
||||
|
||||
This guide describes the best practices and tuning approaches to scale Rancher setups and the associated challenges with doing so. As systems grow, performance will naturally reduce, but there are steps that can minimize the load put on Rancher and optimize Rancher's ability to manage larger infrastructures.
|
||||
|
||||
## Optimizing Rancher Performance
|
||||
|
||||
* Keep Rancher up to date with patch releases. We are continuously improving Rancher with performance enhancements and bug fixes. The latest Rancher release contains all accumulated improvements to performance and stability, plus updates based on developer experience and user feedback.
|
||||
|
||||
* Always scale up gradually, and monitor and observe any changes in behavior while doing do. It is usually easier to resolve performance problems as soon as they surface, before other problems obscure the root cause.
|
||||
|
||||
* Reduce network latency between the upstream Rancher cluster and downstream clusters to the extent possible. Note that latency is, among other factors, a function of geographic distance - if you require clusters or nodes spread across the world, consider multiple Rancher installations.
|
||||
|
||||
## Minimizing Load on the Upstream Cluster
|
||||
|
||||
When scaling up Rancher, one typical bottleneck is resource growth in the upstream (local) Kubernetes cluster. The upstream cluster contains information for all downstream clusters. Many operations that apply to downstream clusters create new objects in the upstream cluster and require computation from handlers running in the upstream cluster.
|
||||
|
||||
### Managing Your Object Counts
|
||||
|
||||
Etcd is the backing database for Kubernetes and for Rancher. The database may eventually encounter limitations to the number of a single Kubernetes resource type it can store. Exact limits vary and depend on a number of factors. However, experience indicates that performance issues frequently arise once a single resource type's object count exceeds 60,000. Often that type is `RoleBinding`.
|
||||
|
||||
This is typical in Rancher, as many operations create new `RoleBinding` objects in the upstream cluster as a side effect.
|
||||
|
||||
You can reduce the number of `RoleBindings` in the upstream cluster in the following ways:
|
||||
* Limit the use of the [Restricted Admin](../../../how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/manage-role-based-access-control-rbac/global-permissions#restricted-admin) role. Apply other roles wherever possible.
|
||||
* If you use [external authentication](../../../pages-for-subheaders/authentication-config), use groups to assign roles.
|
||||
* Only add users to clusters and projects when necessary.
|
||||
* Remove clusters and projects when they are no longer needed.
|
||||
* Only use custom roles if necessary.
|
||||
* Use as few rules as possible in custom roles.
|
||||
* Consider whether adding a role to a user is redundant.
|
||||
* Consider using less, but more powerful, clusters.
|
||||
* Kubernetes permissions are always "additive" (allow-list) rather than "subtractive" (deny-list). Try to minimize configurations that gives access to all but one aspect of a cluster, project, or namespace, as that will result in the creation of a high number of `RoleBinding` objects.
|
||||
* Experiment to see if creating new projects or clusters manifests in fewer `RoleBindings` for your specific use case.
|
||||
|
||||
### RoleBinding Count Estimation
|
||||
|
||||
Predicting how many `RoleBinding` objects a given configuration will create is complicated. However, the following considerations can offer a rough estimate:
|
||||
* For a minimum estimate, use the formula `32C + U + 2UaC + 8P + 5Pa`.
|
||||
* `C` is the total number of clusters.
|
||||
* `U` is the total number of users.
|
||||
* `Ua` is the average number of users with a membership on a cluster.
|
||||
* `P` is the total number of projects.
|
||||
* `Pa` is the average number of users with a membership on a project.
|
||||
* The Restricted Admin role follows a different formula, as every user with this role results in at least `7C + 2P + 2` additional `RoleBinding` objects.
|
||||
* The number of `RoleBindings` increases linearly with the number of clusters, projects, and users.
|
||||
|
||||
### Using New Apps Over Legacy Apps
|
||||
|
||||
Rancher uses two Kubernetes app resources: `apps.projects.cattle.io` and `apps.cattle.cattle.io`. Legacy apps, represented by `apps.projects.cattle.io`, were introduced with the former Cluster Manager UI and are now outdated. Current apps, represented by `apps.catalog.cattle.io`, are found in the Cluster Explorer UI for their respective cluster. `Apps.cattle.cattle.io` apps are preferable because their data resides in downstream clusters, which frees up resources in the upstream cluster.
|
||||
|
||||
You should remove any remaining legacy apps that appear in the Cluster Manager UI, and replace them with apps in the Cluster Explorer UI. Create any new apps only in the Cluster Explorer UI.
|
||||
|
||||
### Using the Authorized Cluster Endpoint (ACE)
|
||||
|
||||
An [Authorized Cluster Endpoint](../../../reference-guides/rancher-manager-architecture/communicating-with-downstream-user-clusters#4-authorized-cluster-endpoint) (ACE) provides access to the Kubernetes API of Rancher-provisioned RKE, RKE2, and K3s clusters. When enabled, the ACE adds a context to kubeconfig files generated for the cluster. The context uses a direct endpoint to the cluster, thereby bypassing Rancher. This reduces load on Rancher for cases where unmediated API access is acceptable or preferable. See [Authorized Cluster Endpoint](../../../reference-guides/rancher-manager-architecture/communicating-with-downstream-user-clusters#4-authorized-cluster-endpoint) for more information and configuration instructions.
|
||||
|
||||
### Reducing Event Handler Executions
|
||||
|
||||
The bulk of Rancher's logic occurs on event handlers. These event handlers run on an object whenever the object is updated, and when Rancher is started. Additionally, they run every 15 hours when Rancher syncs caches. In scaled setups these scheduled runs come with huge performance costs because every handler is being run on every applicable object. However, the scheduled handler execution can be disabled with the `CATTLE_SYNC_ONLY_CHANGED_OBJECTS` environment variable. If resource allocation spikes are seen every 15 hours, this setting can help.
|
||||
|
||||
The value for `CATTLE_SYNC_ONLY_CHANGED_OBJECTS` can be a comma separated list of the following options. The values refer to types of handlers and controllers (the structures that contain and run handlers). Adding the controller types to the variable disables that set of controllers from running their handlers as part of cache resyncing.
|
||||
|
||||
* `mgmt` refers to management controllers which only run on one Rancher node.
|
||||
* `user` refers to user controllers which run for every cluster. Some of these run on the same node as management controllers, while others run in the downstream cluster. This option targets the former.
|
||||
* `scaled` refers to scaled controllers which run on every Rancher node. You should avoid setting this value, as the scaled handlers are responsible for critical functions and changes may disrupt cluster stability.
|
||||
|
||||
In short, if you notice CPU usage peaks every 15 hours, add the `CATTLE_SYNC_ONLY_CHANGED_OBJECTS` environment variable to your Rancher deployment (in the `spec.containers.env` list) with the value `mgmt,user`
|
||||
|
||||
## Optimizations Outside of Rancher
|
||||
|
||||
Important influencing factors are the underlying cluster's own performance and configuration. The upstream cluster, if misconfigured, can introduce a bottleneck Rancher software has no chance to resolve.
|
||||
|
||||
### Manage Upstream Cluster Nodes Directly with RKE2
|
||||
|
||||
As Rancher can be very demanding on the upstream cluster, especially at scale, you should have full administrative control of the cluster's configuration and nodes. To identify the root cause of excess resource consumption, use standard Linux troubleshooting techniques and tools. This can aid in distinguishing between whether Rancher, Kubernetes, or operating system components are causing issues.
|
||||
|
||||
Although managed Kubernetes services make it easier to deploy and run Kubernetes clusters, they are discouraged for the upstream cluster in high scale scenarios. Managed Kubernetes services typically limit access to configuration and insights on individual nodes and services.
|
||||
|
||||
Use RKE2 for large scale use cases.
|
||||
|
||||
### Keeping Kubernetes Versions Up to Date
|
||||
|
||||
You should keep the local Kubernetes cluster up to date. This will ensure that your cluster has all available performance enhancements and bug fixes.
|
||||
|
||||
### Optimizing etcd
|
||||
|
||||
Etcd is the backend database for Kubernetes and for Rancher. It plays a very important role in Rancher performance.
|
||||
|
||||
The two main bottlenecks to [etcd performance](https://etcd.io/docs/v3.4/op-guide/performance/) are disk and network speed. Etcd should run on dedicated nodes with a fast network setup and with SSDs that have high input/output operations per second (IOPS). For more information regarding etcd performance, see [Slow etcd performance (performance testing and optimization)](https://www.suse.com/support/kb/doc/?id=000020100) and [Tuning etcd for Large Installations](../../../how-to-guides/advanced-user-guides/tune-etcd-for-large-installs). Information on disks can also be found in the [Installation Requirements](../../../pages-for-subheaders/installation-requirements#disks).
|
||||
|
||||
It's best to run etcd on exactly three nodes, as adding more nodes will reduce operation speed. This may be counter-intuitive to common scaling approaches, but it's due to etcd's [replication mechanisms](https://etcd.io/docs/v3.5/faq/#what-is-maximum-cluster-size).
|
||||
|
||||
Etcd performance will also be negatively affected by network latency between nodes as that will slow down network communication. Etcd nodes should be located together with Rancher nodes.
|
||||
+2
-2
@@ -17,7 +17,7 @@ Your cloud credential has these fields:
|
||||
| Credential Field | Description |
|
||||
|-----------------|--------------|
|
||||
| vCenter or ESXi Server | Enter the vCenter or ESXi hostname/IP. ESXi is the virtualization platform where you create and run virtual machines and virtual appliances. vCenter Server is the service through which you manage multiple hosts connected in a network and pool host resources. |
|
||||
| Port | Optional: configure configure the port of the vCenter or ESXi server. |
|
||||
| Port | Configure the port of the vCenter or ESXi server. |
|
||||
| Username and password | Enter your vSphere login username and password. |
|
||||
|
||||
## Scheduling
|
||||
@@ -96,4 +96,4 @@ In the **Engine Options** section of the node template, you can configure the co
|
||||
If you're provisioning Red Hat Enterprise Linux (RHEL) or CentOS nodes, leave the **Docker Install URL** field as the default value, or select **none**. This will bypass a check for Docker installation, as Docker is already installed on these node types.
|
||||
|
||||
If you set **Docker Install URL** to a value other than the default or **none**, you might see an error message such as the following: `Error creating machine: RHEL ssh command error: command: sudo -E yum install -y curl err: exit status 1 output: Updating Subscription Management repositories.`
|
||||
:::
|
||||
:::
|
||||
|
||||
-1
@@ -152,4 +152,3 @@ The syncing interval can be changed by running `kubectl edit setting eks-refresh
|
||||
If the `eks-refresh-cron` setting was previously set, the migration will happen automatically.
|
||||
|
||||
The shorter the refresh window, the less likely any race conditions will occur, but it does increase the likelihood of encountering request limits that may be in place for AWS APIs.
|
||||
|
||||
|
||||
+27
@@ -318,6 +318,7 @@ chartValues:
|
||||
chart-name:
|
||||
key: value
|
||||
```
|
||||
|
||||
### machineGlobalConfig
|
||||
|
||||
Specify K3s configurations. Any configuration change made here will apply to every node. The configuration options available in the [standalone version of k3s](https://docs.k3s.io/cli/server) can be applied here.
|
||||
@@ -331,6 +332,31 @@ machineGlobalConfig:
|
||||
- key2=value2
|
||||
```
|
||||
|
||||
To make it easier to put files on nodes beforehand, Rancher expects the following values to be included in the configuration, while K3s expects the values to be entered as file paths:
|
||||
- private-registry
|
||||
- flannel-conf
|
||||
|
||||
Rancher delivers the files to the path `/var/lib/rancher/k3s/etc/config-files/<option>` in target nodes, and sets the proper options in the K3s server.
|
||||
|
||||
Example:
|
||||
```yaml
|
||||
apiVersion: provisioning.cattle.io/v1
|
||||
kind: Cluster
|
||||
spec:
|
||||
rkeConfig:
|
||||
machineGlobalConfig:
|
||||
private-registry: |
|
||||
mirrors:
|
||||
docker.io:
|
||||
endpoint:
|
||||
- "http://mycustomreg.com:5000"
|
||||
configs:
|
||||
"mycustomreg:5000":
|
||||
auth:
|
||||
username: xxxxxx # this is the registry username
|
||||
password: xxxxxx # this is the registry password
|
||||
```
|
||||
|
||||
### machineSelectorConfig
|
||||
|
||||
`machineSelectorConfig` is the same as [`machineGlobalConfig`](#machineglobalconfig) except that a [label](#kubernetes-node-labels) selector can be specified with the configuration. The configuration will only be applied to nodes that match the provided label selector.
|
||||
@@ -354,6 +380,7 @@ machineSelectorConfig
|
||||
key1: value1
|
||||
key2: value2
|
||||
```
|
||||
|
||||
### machineSelectorFiles
|
||||
|
||||
:::note
|
||||
|
||||
+28
-1
@@ -348,6 +348,7 @@ chartValues:
|
||||
chart-name:
|
||||
key: value
|
||||
```
|
||||
|
||||
### machineGlobalConfig
|
||||
|
||||
Specify RKE2 configurations. Any configuration change made here will apply to every node. The configuration options available in the [standalone version of RKE2](https://docs.rke2.io/reference/server_config) can be applied here.
|
||||
@@ -361,6 +362,31 @@ machineGlobalConfig:
|
||||
- key2=value2
|
||||
```
|
||||
|
||||
To make it easier to put files on nodes beforehand, Rancher expects the following values to be included in the configuration, while RKE2 expects the values to be entered as file paths:
|
||||
- audit-policy-file
|
||||
- cloud-provider-config
|
||||
- private-registry
|
||||
|
||||
Rancher delivers the files to the path `/var/lib/rancher/rke2/etc/config-files/<option>` in target nodes, and sets the proper options in the RKE2 server.
|
||||
|
||||
Example:
|
||||
```yaml
|
||||
apiVersion: provisioning.cattle.io/v1
|
||||
kind: Cluster
|
||||
spec:
|
||||
rkeConfig:
|
||||
machineGlobalConfig:
|
||||
audit-policy-file:
|
||||
apiVersion: audit.k8s.io/v1
|
||||
kind: Policy
|
||||
rules:
|
||||
- level: RequestResponse
|
||||
resources:
|
||||
- group: ""
|
||||
resources:
|
||||
- pods
|
||||
```
|
||||
|
||||
### machineSelectorConfig
|
||||
|
||||
`machineSelectorConfig` is the same as [`machineGlobalConfig`](#machineglobalconfig) except that a [label](#kubernetes-node-labels) selector can be specified with the configuration. The configuration will only be applied to nodes that match the provided label selector.
|
||||
@@ -384,6 +410,7 @@ machineSelectorConfig
|
||||
key1: value1
|
||||
key2: value2
|
||||
```
|
||||
|
||||
### machineSelectorFiles
|
||||
|
||||
:::note
|
||||
@@ -440,7 +467,7 @@ machineSelectorFiles:
|
||||
The secret or configmap must meet the following requirements:
|
||||
|
||||
1. It must be in the `fleet-default` namespace where the Cluster object exists.
|
||||
2. It must have the annotation `rke.cattle.io/object-authorized-for-clusters: cluster-name1,cluster-name2`, which permits the target clusters to use it.
|
||||
2. It must have the annotation `rke.cattle.io/object-authorized-for-clusters: cluster-name1,cluster-name2`, which permits the target clusters to use it.
|
||||
|
||||
:::tip
|
||||
|
||||
|
||||
@@ -46,7 +46,7 @@ The route needs to refer to a [receiver](./receivers.md) that has already been c
|
||||
|
||||
| Field | Default | Description |
|
||||
|-------|--------------|---------|
|
||||
| Group By | N/a | List of labels to group by. All labels must be unique. Special label "..." (aggregate by all possible labels), if provided, must be the only element in the list. Accepts a list of strings. See the [upstream documentation](https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/api.md#route) for details. |
|
||||
| Group By | N/a | List of labels to group by. Labels must be unique. Special label "..." (aggregate by all possible labels), if provided, must be the only element in the list. Accepts a list of strings. See the [upstream documentation](https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/api.md#route) for details. |
|
||||
| Group Wait | 30s | How long to wait to buffer alerts of the same group before sending initially. |
|
||||
| Group Interval | 5m | How long to wait before sending an alert that has been added to a group of alerts for which an initial notification has already been sent. |
|
||||
| Repeat Interval | 4h | How long to wait before re-sending a given alert that has already been sent. |
|
||||
|
||||
+9
-2
@@ -64,7 +64,6 @@ An authorized cluster endpoint (ACE) allows users to connect to the Kubernetes A
|
||||
|
||||
> ACE is available on RKE, RKE2, and K3s clusters that are provisioned or registered with Rancher. It's not available on clusters in a hosted Kubernetes provider, such as Amazon's EKS.
|
||||
|
||||
|
||||
There are two main reasons why a user might need the authorized cluster endpoint:
|
||||
|
||||
- To access a downstream user cluster while Rancher is down
|
||||
@@ -78,7 +77,15 @@ Like the authorized cluster endpoint, the `kube-api-auth` authentication service
|
||||
|
||||
With this endpoint enabled for the downstream cluster, Rancher generates an extra Kubernetes context in the kubeconfig file in order to connect directly to the cluster. This file has the credentials for `kubectl` and `helm`.
|
||||
|
||||
You will need to use a context defined in this kubeconfig file to access the cluster if Rancher goes down. Therefore, we recommend exporting the kubeconfig file so that if Rancher goes down, you can still use the credentials in the file to access your cluster. For more information, refer to the section on accessing your cluster with [kubectl and the kubeconfig file.](../../how-to-guides/new-user-guides/manage-clusters/access-clusters/use-kubectl-and-kubeconfig.md)
|
||||
:::note
|
||||
|
||||
To use the ACE context in your kubeconfig, run `kubectl use-context <ace context name>` after enabling it.
|
||||
|
||||
:::
|
||||
|
||||
For more information, refer to the section on accessing your cluster with [kubectl and the kubeconfig file](../../how-to-guides/new-user-guides/manage-clusters/access-clusters/use-kubectl-and-kubeconfig.md).
|
||||
|
||||
We recommend exporting the kubeconfig file so that if Rancher goes down, you can still use the credentials in the file to access your cluster.
|
||||
|
||||
## Impersonation
|
||||
|
||||
|
||||
@@ -46,6 +46,7 @@ plugins:
|
||||
cattle-monitoring-system,
|
||||
cattle-neuvector-system,
|
||||
cattle-prometheus,
|
||||
cattle-provisioning-capi-system,
|
||||
cattle-resources-system,
|
||||
cattle-sriov-system,
|
||||
cattle-system,
|
||||
|
||||
@@ -1,133 +0,0 @@
|
||||
---
|
||||
title: Hardening the Rancher Webhook
|
||||
---
|
||||
|
||||
Rancher Webhook is an important component within Rancher, playing a role in enforcing security requirements for Rancher and its workloads. To decrease its attack surface, access to it should be limited to the only valid caller it has: the Kubernetes API server. This can be done by using network policies and authentication independently or in conjunction with each other to harden the webhook against attacks.
|
||||
|
||||
## Block External Traffic Using Network Policies
|
||||
|
||||
The webhook is only expected to accept requests from the Kubernetes API server. By default, however, the webhook can accept traffic from any source. If you are using a CNI that supports Network Policies, you can create a policy that blocks traffic that doesn't originate from the API server.
|
||||
|
||||
The built-in NetworkPolicy resource in Kubernetes can't block or admit traffic from the cluster hosts, and the `kube-apiserver` process is always running on the host network. Therefore, you must use the advanced network policy resources from the CNI in use. Examples for Calico and Cilium follow. Consult the documentation for your CNI for more details.
|
||||
|
||||
### Calico
|
||||
|
||||
Use the NetworkPolicy resource in the `crd.projectcalico.org/v1` API group. Use the selector `app == 'rancher-webhook'` to create a rule for the webhook, and set the CIDR of the control plane hosts as the ingress source:
|
||||
|
||||
```yaml
|
||||
apiVersion: crd.projectcalico.org/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: allow-k8s
|
||||
namespace: cattle-system
|
||||
spec:
|
||||
selector: app == 'rancher-webhook'
|
||||
types:
|
||||
- Ingress
|
||||
ingress:
|
||||
- action: Allow
|
||||
protocol: TCP
|
||||
source:
|
||||
nets:
|
||||
- 192.168.42.0/24 # CIDR of the control plane host. May list more than 1 if the hosts are in different subnets.
|
||||
destination:
|
||||
selector:
|
||||
app == 'rancher-webhook'
|
||||
```
|
||||
|
||||
### Cilium
|
||||
|
||||
Use the CiliumNetworkPolicy resource in the `cilium.io/v2` API group. Add the `host` and `remote-node` keys to the `fromEntities` ingress rule. This blocks in-cluster and external traffic while allowing traffic from the hosts.
|
||||
|
||||
```yaml
|
||||
apiVersion: "cilium.io/v2"
|
||||
kind: CiliumNetworkPolicy
|
||||
metadata:
|
||||
name: allow-k8s
|
||||
namespace: cattle-system
|
||||
spec:
|
||||
endpointSelector:
|
||||
matchLabels:
|
||||
app: rancher-webhook
|
||||
ingress:
|
||||
- fromEntities:
|
||||
- host
|
||||
- remote-node
|
||||
```
|
||||
|
||||
## Require the Kubernetes API Server to Authenticate to the Webhook
|
||||
|
||||
The webhook should only accept requests from the Kubernetes API server. By default, the webhook doesn't require clients to authenticate to it. It will accept any request. You can configure the webhook to require credentials so that only the API server can access it. More information can be found in the [Kubernetes documentation](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#authenticate-apiservers).
|
||||
|
||||
1. Configure the API server to present a client certificate to the webhook, pointing to an AdmissionConfiguration file to configure the ValidatingAdmissionWebhook and MutatingAdmissionWebhook plugins:
|
||||
|
||||
```yaml
|
||||
# /etc/rancher/admission/admission.yaml
|
||||
apiVersion: apiserver.config.k8s.io/v1
|
||||
kind: AdmissionConfiguration
|
||||
plugins:
|
||||
- name: ValidatingAdmissionWebhook
|
||||
configuration:
|
||||
apiVersion: apiserver.config.k8s.io/v1
|
||||
kind: WebhookAdmissionConfiguration
|
||||
kubeConfigFile: "/etc/rancher/admission/kubeconfig"
|
||||
- name: MutatingAdmissionWebhook
|
||||
configuration:
|
||||
apiVersion: apiserver.config.k8s.io/v1
|
||||
kind: WebhookAdmissionConfiguration
|
||||
kubeConfigFile: "/etc/rancher/admission/kubeconfig"
|
||||
```
|
||||
|
||||
This is also the same config file where other admission plugins are configured, such as PodSecurity. If your distro or your setup uses additional admission plugins, configure those as well. For example, add [RKE2's PodSecurity configuration](https://docs.rke2.io/security/pod_security_standards) to this file.
|
||||
|
||||
2. Create the kubeconfig file that the admission plugins refer to. Rancher Webhook only supports client certificate authentication, so generate a TLS key pair, and set the kubeconfig to use either `client-certificate` and `client-key` or `client-certificate-data` and `client-key-data`. For example:
|
||||
|
||||
```yaml
|
||||
# /etc/rancher/admission/kubeconfig
|
||||
apiVersion: v1
|
||||
kind: Config
|
||||
users:
|
||||
- name: 'rancher-webhook.cattle-system.svc'
|
||||
user:
|
||||
client-certificate: /path/to/client/cert
|
||||
client-key: /path/to/client/key
|
||||
```
|
||||
|
||||
3. Start the kube-apiserver binary with the flag `--admission-control-config-file` pointing to your AdmissionConfiguration file. The way to do this varies by distro, and it isn't supported universally, such as in hosted Kubernetes providers. Consult the documentation for your Kubernetes distribution.
|
||||
|
||||
For RKE2, `rke2-server` can be started with a config file like so:
|
||||
|
||||
```yaml
|
||||
# /etc/rancher/rke2/config.yaml
|
||||
kube-apiserver-arg:
|
||||
- admission-control-config-file=/etc/rancher/admission/admission.yaml
|
||||
kube-apiserver-extra-mount:
|
||||
- /etc/rancher/admission:/etc/rancher/admission:ro
|
||||
```
|
||||
|
||||
:::danger
|
||||
Some distros set this flag by default. If your distro provisions its own AdmissionConfiguration, you must include it in your custom admission control config file. For example, RKE2 installs an AdmissionConfiguration file at `/etc/rancher/rke2/rke2-pss.yaml`, which configures the PodSecurity admission plugin. Setting `admission-control-config-file` in config.yaml will override this essential security setting. To include both plugins, consult [the Default Pod Security Standards documentation](https://docs.rke2.io/security/pod_security_standards) and copy the appropriate plugin configuration to your admission.yaml.
|
||||
:::
|
||||
|
||||
4. If you're using Rancher to provision your cluster using existing nodes, create these files on the node before you provision them.
|
||||
|
||||
If you're using Rancher to provision your cluster on new nodes, allow the provisioning to complete, then use the provided SSH key and IP address to connect to the nodes, and place the RKE2 config file in the `/etc/rancher/rke2/config.yaml.d/` directory.
|
||||
|
||||
5. After the cluster is configured with these credentials, configure the Rancher cluster agent to enable authentication in the webhook. Create a file containing these chart values:
|
||||
|
||||
```yaml
|
||||
# values.yaml
|
||||
auth:
|
||||
clientCA: <base64-encoded certificate authority that signed the kube-apiserver's client certificates>
|
||||
allowedCNs:
|
||||
- <list of Common Names identifying the kube-apiserver's client certificates.>
|
||||
- <if not provided, any cert signed by the given CA will be accepted.>
|
||||
```
|
||||
|
||||
6. Create a configmap in the `cattle-system` namespace on the provisioned cluster with these values:
|
||||
|
||||
```
|
||||
kubectl --namespace cattle-system create configmap --from-file=rancher-webhook=values.yaml
|
||||
```
|
||||
|
||||
The webhook will restart with these values.
|
||||
Reference in New Issue
Block a user