diff --git a/content/rancher/v2.x/en/installation/options/etcd/_index.md b/content/rancher/v2.x/en/installation/options/etcd/_index.md index 3298755f3e1..8cc87fc3954 100644 --- a/content/rancher/v2.x/en/installation/options/etcd/_index.md +++ b/content/rancher/v2.x/en/installation/options/etcd/_index.md @@ -6,9 +6,9 @@ aliases: ## Tuning etcd for large installations ## -When running larger Rancher installations with 15 or more clusters it is recommended to increase the default keyspace for etcd from the default 2GB. The maximum setting is 8GB and the host should have enough RAM to keep the entire dataset in memory. When increasing this value you should also increase the size of the host. This value can also be adjusted in smaller installations if you have a large number of deployments across clusters every 5 minutes. +When running larger Rancher installations with 15 or more clusters it is recommended to increase the default keyspace for etcd from the default 2GB. The maximum setting is 8GB and the host should have enough RAM to keep the entire dataset in memory. When increasing this value you should also increase the size of the host. The keyspace size can also be adjusted in smaller installations if you anticipate a high rate of change of pods during the garbage collection interval. -The etcd data set is automatically cleaned up on a five minute interval by Kubernetes. There are situations, e.g. deployment thrasing, where enough events could be written to etcd and deleted before garbage collection occurs and cleans things up causing the keyspace to fill up. If you see `mvcc: database space exceeded` errors, in the etcd logs or Kubernetes API server logs, you should consider increasing the keyspace size. This can be accomplished by setting the [quota-backend-bytes](https://etcd.io/docs/v3.3.12/op-guide/maintenance/#space-quota) setting on the etcd servers. +The etcd data set is automatically cleaned up on a five minute interval by Kubernetes. There are situations, e.g. deployment thrashing, where enough events could be written to etcd and deleted before garbage collection occurs and cleans things up causing the keyspace to fill up. If you see `mvcc: database space exceeded` errors, in the etcd logs or Kubernetes API server logs, you should consider increasing the keyspace size. This can be accomplished by setting the [quota-backend-bytes](https://etcd.io/docs/v3.3.12/op-guide/maintenance/#space-quota) setting on the etcd servers. ### Example: This snippet of the RKE cluster.yml file increases the keyspace size to 5GB ### @@ -28,7 +28,7 @@ You can follow the recommendations from [the etcd docs](https://etcd.io/docs/v3. Additionally, to reduce IO contention on the disks for etcd, you can use a dedicated device for the data and wal directory. Based on etcd best practices, mirroring RAID configurations are unnecessary because etcd replicates data between the nodes in the cluster. You can use stripping RAID configurations to increase available IOPS. -To implement this solution in an RKE cluster, the `/var/lib/etcd/data` and `/var/lib/etc/wal` directories will need to have disks mounted and formated on the underlying host. Notice the additional `wal_dir` directory added to the extra_arg, without this the etcd process will try to manipulate the underlying `wal` mount without sufficient permissions. +To implement this solution in an RKE cluster, the `/var/lib/etcd/data` and `/var/lib/etc/wal` directories will need to have disks mounted and formmated on the underlying host. In the `extra_args` directive of the `etcd` service, you must include the `wal_dir` directory. Without specifying the `wal_dir`, etcd process will try to manipulate the underlying `wal` mount with insufficient permissions. ```yaml # RKE cluster.yml @@ -37,7 +37,7 @@ services: etcd: extra_args: data-dir: "/var/lib/rancher/etcd/data/" - wal-dir: "/var/lib/rancher/etcd/wal/wall_dir" + wal-dir: "/var/lib/rancher/etcd/wal/wal_dir" extra_binds: - "/var/lib/etcd/data:/var/lib/rancher/etcd/data" - "/var/lib/etcd/wal:/var/lib/rancher/etcd/wal" diff --git a/content/rancher/v2.x/en/installation/requirements/_index.md b/content/rancher/v2.x/en/installation/requirements/_index.md index d1366a75b75..e0a8724cdc1 100644 --- a/content/rancher/v2.x/en/installation/requirements/_index.md +++ b/content/rancher/v2.x/en/installation/requirements/_index.md @@ -67,7 +67,7 @@ Medium | Up to 15 | Up to 200 | 2 | 8 GB | **Disks** -Rancher performance depends on etcd in the cluster performance. To ensure optimal speed, we recommend always using SSD disks to back your Rancher management Kubernetes cluster. On cloud providers, you will also want to use the size minimum size that allows the maximum IOPs. Consider in larger clusters using dedicated dedicated storage devices for etcd data and wal directories. +Rancher performance depends on etcd in the cluster performance. To ensure optimal speed, we recommend always using SSD disks to back your Rancher management Kubernetes cluster. On cloud providers, you will also want to use the minimum size that allows the maximum IOPs. In larger clusters consider using dedicated dedicated storage devices for etcd data and wal directories.