public/rancher-docs
1
0
Fork 0
mirror of https://github.com/rancher/rancher-docs.git synced 2026-09-24 20:18:18 +00:00
Code Issues Packages Projects Releases Wiki Activity

added some etcd io doc refs

Browse Source
This commit is contained in:
Bill Maxwell
2019-06-11 15:04:16 -07:00
parent b2d6630d01
commit 34603daaf1
1 changed files with 6 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files
content/rancher/v2.x/en/installation/options/etcd/_index.md
+6 -4
View File
@@ -8,7 +8,7 @@ aliases:
When running larger Rancher installations with 15 or more clusters it is recommended to increase the default keyspace for etcd from the default 2GB. The maximum setting is 8GB and the host should have enough RAM to keep the entire dataset in memory. When increasing this value you should also increase the size of the host. This value can also be adjusted in smaller installations if you have a large number of deployments across clusters every 5 minutes.
The etcd data set is automatically cleaned up on a five minute interval by Kubernetes. There are situations, e.g. deployment thrasing, where enough events could be written to etcd and deleted before garbage collection occurs and cleans things up causing the keyspace to fill up. If you see `mvcc: database space exceeded` errors, in the etcd logs or Kubernetes API server logs, you should consider increasing the keyspace size. This can be accomplished by setting the quota-backend-bytes setting on the etcd servers.
The etcd data set is automatically cleaned up on a five minute interval by Kubernetes. There are situations, e.g. deployment thrasing, where enough events could be written to etcd and deleted before garbage collection occurs and cleans things up causing the keyspace to fill up. If you see `mvcc: database space exceeded` errors, in the etcd logs or Kubernetes API server logs, you should consider increasing the keyspace size. This can be accomplished by setting the [quota-backend-bytes](https://etcd.io/docs/v3.3.12/op-guide/maintenance/#space-quota) setting on the etcd servers.
### Example: This snippet of the RKE cluster.yml file increases the keyspace size to 5GB ###
@@ -24,9 +24,11 @@ services:
## Scaling etcd disk performance ##
To reduce IO contention on the disks for etcd, you can split the data and WAL directories onto separate devices. To do so, you need to provision nodes with devices and format them before running etcd on them. Based on etcd best practices, mirroring array configurations are unnecessary because etcd replicates the data between the nodes. However, you can use stripping RAID configurations to increase the IOPS available to etcd. To configure dedicated arrays, you can configure this in the RKE cluster.yml file with the following snippet.
You can follow the recommendations from [the etcd docs](https://etcd.io/docs/v3.3.12/tuning/#disk) on how to tune the disk priority on the host.
For the solution above to work, this requires `/var/lib/etcd/data` and `/var/lib/etcd/wal` to be mounted and formated on the underlying host. Note the addition of the 'wall_dir' directory in the extra_args section. Without this directory being defined, etcd tries to modify the root directory and without sufficient permissions.
Additionally, to reduce IO contention on the disks for etcd, you can use a dedicated device for the data and wal directory. Based on etcd best practices, mirroring RAID configurations are unnecessary because etcd replicates data between the nodes in the cluster. You can use stripping RAID configurations to increase available IOPS.
To implement this solution in an RKE cluster, the `/var/lib/etcd/data` and `/var/lib/etc/wal` directories will need to have disks mounted and formated on the underlying host. Notice the additional `wal_dir` directory added to the extra_arg, without this the etcd process will try to manipulate the underlying `wal` mount without sufficient permissions.
```yaml
# RKE cluster.yml
@@ -40,4 +42,4 @@ services:
- "/var/lib/etcd/data:/var/lib/rancher/etcd/data"
- "/var/lib/etcd/wal:/var/lib/rancher/etcd/wal"
...
```
```