diff --git a/content/rancher/v2.x/en/installation/options/etcd/_index.md b/content/rancher/v2.x/en/installation/options/etcd/_index.md index 33357564133..ac03044ac75 100644 --- a/content/rancher/v2.x/en/installation/options/etcd/_index.md +++ b/content/rancher/v2.x/en/installation/options/etcd/_index.md @@ -6,13 +6,12 @@ aliases: ## Tuning etcd for large installations ## -When running larger Rancher installations with 15 or more clusters it is recommended to increase the default key space from the default 2GB. The maximum setting is 8GB and the host should have enough RAM to keep the entire dataset in memory. When increasing this value you should also increase the size of the host. This value can also be adjusted in smaller installations if you have a large number of deployments across clusters every 5 minutes. If you run into +When running larger Rancher installations with 15 or more clusters it is recommended to increase the default keyspace for etcd from the default 2GB. The maximum setting is 8GB and the host should have enough RAM to keep the entire dataset in memory. When increasing this value you should also increase the size of the host. This value can also be adjusted in smaller installations if you have a large number of deployments across clusters every 5 minutes. -The etcd data set is automatically cleaned up on a five minute interval by Kubernetes. In situations where a lot of events, like deployment thrashing, enough events could be written to etcd and deleted before garbage collection occurs and cleans things up. The risk of this happening can be minimized by increasing the quota-backend-bytes. +The etcd data set is automatically cleaned up on a five minute interval by Kubernetes. There are situations, e.g. deployment thrasing, where enough events could be written to etcd and deleted before garbage collection occurs and cleans things up causing the keyspace to fill up. If you see `mvcc: database space exceeded` errors, in the etcd logs or Kubernetes API server logs, you should consider increasing the keyspace size. This can be accomplished by setting the quota-backend-bytes setting on the etcd servers. -If you ever run into `mvcc: database space exceeded` errors, once you compact, defrag and disarm to recover you should consider increasing the keyspace to mitigate this issue. +### Example: This snippet of the RKE cluster.yml file increases the keyspace size to 5GB ### -**Example: This snippet of the RKE cluster.yml file increases the key space size to 5GB** ```yaml # RKE cluster.yml ... @@ -27,6 +26,8 @@ services: To reduce IO contention on the disks for etcd, you can split the data and WAL directories onto separate devices. To do so, you need to provision nodes with devices and format them before running etcd on them. Based on etcd best practices, mirroring array configurations are unnecessary because etcd replicates the data between the nodes. However, you can use stripping RAID configurations to increase the IOPS available to etcd. To configure dedicated arrays, you can configure this in the RKE cluster.yml file with the following snippet. +For the solution above to work, this requires `/var/lib/etcd/data` and `/var/lib/etcd/wal` to be mounted and formated on the underlying host. Note the addition of the 'wall_dir' directory in the extra_args section. Without this directory being defined, etcd tries to modify the root directory and without sufficient permissions. + ```yaml # RKE cluster.yml ... @@ -40,5 +41,3 @@ services: - "/var/lib/etcd/wal:/var/lib/rancher/etcd/wal" ... ``` - -For the solution above to work, this requires `/var/lib/etcd/data` and `/var/lib/etcd/wal` to be mounted and formated on the underlying host. Note the addition of the 'wall_dir' directory in the extra_args section. Without this directory being defined, etcd tries to modify the root directory and without sufficient permissions. \ No newline at end of file diff --git a/content/rancher/v2.x/en/installation/requirements/_index.md b/content/rancher/v2.x/en/installation/requirements/_index.md index 3f7a5de3140..d1366a75b75 100644 --- a/content/rancher/v2.x/en/installation/requirements/_index.md +++ b/content/rancher/v2.x/en/installation/requirements/_index.md @@ -67,7 +67,7 @@ Medium | Up to 15 | Up to 200 | 2 | 8 GB | **Disks** -Rancher performance depends on etcd in the cluster performance. To ensure optimal speed, we recommend always using SSD disks to back your Rancher management plane. On cloud providers, you will also want to use the size minimum size that allows the maximum IOPs. Consider in larger clusters using dedicated dedicated storage devices for etcd data and wal directories. +Rancher performance depends on etcd in the cluster performance. To ensure optimal speed, we recommend always using SSD disks to back your Rancher management Kubernetes cluster. On cloud providers, you will also want to use the size minimum size that allows the maximum IOPs. Consider in larger clusters using dedicated dedicated storage devices for etcd data and wal directories.