From a7c7ef12646c23c4a91e7aa383f1c8afa39220ca Mon Sep 17 00:00:00 2001
From: Bill Maxwell <bill@rancher.com>
Date: Wed, 29 May 2019 17:02:02 -0700
Subject: [PATCH] add advanced etcd configuration

---
 .../v2.x/en/installation/options/_index.md    |  1 +
 .../en/installation/options/etcd/_index.md    | 38 +++++++++++++++++++
 .../en/installation/requirements/_index.md    |  6 +++
 3 files changed, 45 insertions(+)
 create mode 100644 content/rancher/v2.x/en/installation/options/etcd/_index.md

diff --git a/content/rancher/v2.x/en/installation/options/_index.md b/content/rancher/v2.x/en/installation/options/_index.md
index 410c5f6bc95..15fb2960c37 100644
--- a/content/rancher/v2.x/en/installation/options/_index.md
+++ b/content/rancher/v2.x/en/installation/options/_index.md
@@ -10,3 +10,4 @@ When installing Rancher, there are several advanced options that can be enabled
 | [Custom CA Certificate]({{< baseurl >}}/rancher/v2.x/en/installation/options/custom-ca-root-certificate/) | v2.0.0 |
 | [API Audit Log]({{< baseurl >}}/rancher/v2.x/en/installation/options/api-audit-log/) | v2.0.0 |
 | [TLS Settings]({{< baseurl >}}/rancher/v2.x/en/installation/options/tls-settings/) | v2.1.7 |
+| [etcd configuration]({{< baseurl >}}/rancher/v2.x/en/installation/options/etcd/) | v2.2.0 |
diff --git a/content/rancher/v2.x/en/installation/options/etcd/_index.md b/content/rancher/v2.x/en/installation/options/etcd/_index.md
new file mode 100644
index 00000000000..8209b3fef47
--- /dev/null
+++ b/content/rancher/v2.x/en/installation/options/etcd/_index.md
@@ -0,0 +1,38 @@
+---
+title: etcd Advanced Configurations
+weight: 1
+aliases:
+---
+
+## Tuning etcd for large installations ##
+
+When running larger Rancher installations with 15 or more clusters it is recommended to increase the default key space from the default 2GB. The maximum setting is 8GB and the host should have enough RAM to keep the entire dataset in memory. When increasing this value you should also increase the size of the host. This value can also be adjusted in smaller installations if you have a large number of deployments across clusters every 5 minutes. If you run into 
+
+The etcd data set is automatically cleaned up on a five minute interval by Kubernetes. In situations where a lot of events, like deployment thrashing, enough events could be written to etcd and deleted before garbage collection occurs and cleans things up. The risk of this happening can be minimized by increasing the quota-backend-bytes.
+
+If you ever run into `mvcc: database space exceeded` errors, once you compact, defrag and disarm to recover you should consider increasing the keyspace to mitigate this issue.
+
+**Example: This snippet increases the key space size to 5GB**
+```yaml
+services:
+  etcd:
+    extra_args:
+      quota-backend-bytes: 5368709120
+```
+
+## Scaling etcd Disk Performance ##
+
+To reduce IO contention on the disks for etcd, you can split the data and WAL directories onto separate devices. To do so, you need to provision nodes with devices and format them before running etcd on them. Based on etcd best practices, mirroring array configurations are unnecessary because etcd replicates the data between the nodes. However, you can use stripping RAID configurations to increase the IOPS available to etcd. To configure dedicated arrays, you can configure this in the RKE cluster.yml file with the following snippet.
+
+```yaml
+services:
+  etcd:
+    extra_args:
+      data-dir: "/var/lib/rancher/etcd/data/"
+      wal-dir: "/var/lib/rancher/etcd/wal/wall_dir"
+    extra_binds:
+    - "/var/lib/etcd/data:/var/lib/rancher/etcd/data"
+    - "/var/lib/etcd/wal:/var/lib/rancher/etcd/wal"
+```
+
+For the solution above to work, this requires `/var/lib/etcd/data` and `/var/lib/etcd/wal` to be mounted and formated on the underlying host. Note the addition of the 'wall_dir' directory in the extra_args section. Without this directory being defined, etcd tries to modify the root directory and without sufficient permissions.
\ No newline at end of file
diff --git a/content/rancher/v2.x/en/installation/requirements/_index.md b/content/rancher/v2.x/en/installation/requirements/_index.md
index 9b046959f1f..3f7a5de3140 100644
--- a/content/rancher/v2.x/en/installation/requirements/_index.md
+++ b/content/rancher/v2.x/en/installation/requirements/_index.md
@@ -65,6 +65,12 @@ Medium | Up to 15 | Up to 200 | 2 | 8 GB |
 
 <br/>
 
+**Disks**
+
+Rancher performance depends on etcd in the cluster performance. To ensure optimal speed, we recommend always using SSD disks to back your Rancher management plane. On cloud providers, you will also want to use the size minimum size that allows the maximum IOPs. Consider in larger clusters using dedicated dedicated storage devices for etcd data and wal directories.
+
+<br/>
+
 {{% /tab %}}
 {{% tab  "Networking" %}}
 <br>