mirror of
https://github.com/rancher/rancher-docs.git
synced 2026-09-25 20:48:11 +00:00
#1358 Link to 'Settings for etcd tuning' no longer valid, plus other fixes to links to etcd.io/docs (#1362)
* 1358 Link to 'Settings for etcd tuning' no longer valid updates dead link to new URL * updated etcd version links 2.0-2.4 was tricky, I opted for etcd v3.3 as it's identical to the text on v3.4 except for a single clarifying heading * update link in v2.5 to etcd v3.3 to cover downstream * other 3.4 files in /docs * other 3.4 files in /v2.9 * other 3.4 files in /v2.8 * other 3.4 files in /v2.7 * other 3.4 files in /v2.6 * other 3.4 files in /v2.5 * other 3.4 files in /v2.0-2.4 * rm'ing '.0' from URLs * update v2.6 links to account for differing etcd versions * chinese links
This commit is contained in:
+1
-1
@@ -285,7 +285,7 @@ Input or select an **Expression**, the dropdown shows the original metrics from
|
||||
|
||||
- [**Node**](https://github.com/prometheus/node_exporter)
|
||||
- [**Container**](https://github.com/google/cadvisor)
|
||||
- [**ETCD**](https://etcd.io/docs/v3.4.0/op-guide/monitoring/)
|
||||
- [**ETCD**](https://etcd.io/docs/v3.3/op-guide/monitoring/)
|
||||
- [**Kubernetes Components**](https://github.com/kubernetes/metrics)
|
||||
- [**Kubernetes Resources**](https://github.com/kubernetes/kube-state-metrics)
|
||||
- [**Fluentd**](https://docs.fluentd.org/v1.0/articles/monitoring-prometheus) (supported by [Logging](../cluster-logging/cluster-logging.md))
|
||||
|
||||
+2
-2
@@ -4,7 +4,7 @@ title: Tuning etcd for Large Installations
|
||||
|
||||
When running larger Rancher installations with 15 or more clusters it is recommended to increase the default keyspace for etcd from the default 2GB. The maximum setting is 8GB and the host should have enough RAM to keep the entire dataset in memory. When increasing this value you should also increase the size of the host. The keyspace size can also be adjusted in smaller installations if you anticipate a high rate of change of pods during the garbage collection interval.
|
||||
|
||||
The etcd data set is automatically cleaned up on a five minute interval by Kubernetes. There are situations, e.g. deployment thrashing, where enough events could be written to etcd and deleted before garbage collection occurs and cleans things up causing the keyspace to fill up. If you see `mvcc: database space exceeded` errors, in the etcd logs or Kubernetes API server logs, you should consider increasing the keyspace size. This can be accomplished by setting the [quota-backend-bytes](https://etcd.io/docs/v3.4.0/op-guide/maintenance/#space-quota) setting on the etcd servers.
|
||||
The etcd data set is automatically cleaned up on a five minute interval by Kubernetes. There are situations, e.g. deployment thrashing, where enough events could be written to etcd and deleted before garbage collection occurs and cleans things up causing the keyspace to fill up. If you see `mvcc: database space exceeded` errors, in the etcd logs or Kubernetes API server logs, you should consider increasing the keyspace size. This can be accomplished by setting the [quota-backend-bytes](https://etcd.io/docs/v3.3/op-guide/maintenance/#space-quota) setting on the etcd servers.
|
||||
|
||||
### Example: This snippet of the RKE cluster.yml file increases the keyspace size to 5GB
|
||||
|
||||
@@ -19,7 +19,7 @@ services:
|
||||
|
||||
## Scaling etcd disk performance
|
||||
|
||||
You can follow the recommendations from [the etcd docs](https://etcd.io/docs/v3.4.0/tuning/#disk) on how to tune the disk priority on the host.
|
||||
You can follow the recommendations from [the etcd docs](https://etcd.io/docs/v3.3/tuning/#disk) on how to tune the disk priority on the host.
|
||||
|
||||
Additionally, to reduce IO contention on the disks for etcd, you can use a dedicated device for the data and wal directory. Based on etcd best practices, mirroring RAID configurations are unnecessary because etcd replicates data between the nodes in the cluster. You can use striping RAID configurations to increase available IOPS.
|
||||
|
||||
|
||||
+1
-1
@@ -22,4 +22,4 @@ For details on taking database snapshots and restoring your database from them,
|
||||
|
||||
- [Official MySQL documentation](https://dev.mysql.com/doc/refman/8.0/en/replication-snapshot-method.html)
|
||||
- [Official PostgreSQL documentation](https://www.postgresql.org/docs/8.3/backup-dump.html)
|
||||
- [Official etcd documentation](https://etcd.io/docs/v3.4/op-guide/recovery/)
|
||||
- [Official etcd documentation](https://etcd.io/docs/v3.3/op-guide/recovery/)
|
||||
+1
-1
@@ -18,4 +18,4 @@ For details on taking database snapshots and restoring your database from them,
|
||||
|
||||
- [Official MySQL documentation](https://dev.mysql.com/doc/refman/8.0/en/replication-snapshot-method.html)
|
||||
- [Official PostgreSQL documentation](https://www.postgresql.org/docs/8.3/backup-dump.html)
|
||||
- [Official etcd documentation](https://etcd.io/docs/v3.4/op-guide/recovery/)
|
||||
- [Official etcd documentation](https://etcd.io/docs/v3.3/op-guide/recovery/)
|
||||
+1
-1
@@ -49,5 +49,5 @@ number of nodes for each Kubernetes role, refer to the section on [recommended a
|
||||
|
||||
### Networking
|
||||
|
||||
* Minimize network latency. Rancher recommends minimizing latency between the etcd nodes. The default setting for `heartbeat-interval` is `500`, and the default setting for `election-timeout` is `5000`. These [settings for etcd tuning](https://coreos.com/etcd/docs/latest/tuning.html) allow etcd to run in most networks (except really high latency networks).
|
||||
* Minimize network latency. Rancher recommends minimizing latency between the etcd nodes. The default setting for `heartbeat-interval` is `500`, and the default setting for `election-timeout` is `5000`. These [settings for etcd tuning](https://etcd.io/docs/v3.3/tuning/) allow etcd to run in most networks (except really high latency networks).
|
||||
* Cluster nodes should be located within a single region. Most cloud providers provide multiple availability zones within a region, which can be used to create higher availability for your cluster. Using multiple availability zones is fine for nodes with any role. If you are using [Kubernetes Cloud Provider](../launch-kubernetes-with-rancher/set-up-cloud-providers/set-up-cloud-providers.md) resources, consult the documentation for any restrictions (i.e. zone storage restrictions).
|
||||
|
||||
+1
-1
@@ -57,7 +57,7 @@ The number of nodes that you can lose at once while maintaining cluster availabi
|
||||
|
||||
References:
|
||||
|
||||
* [Official etcd documentation on optimal etcd cluster size](https://etcd.io/docs/v3.4.0/faq/#what-is-failure-tolerance)
|
||||
* [Official etcd documentation on optimal etcd cluster size](https://etcd.io/docs/v3.3/faq/#what-is-failure-tolerance)
|
||||
* [Official Kubernetes documentation on operating etcd clusters for Kubernetes](https://kubernetes.io/docs/tasks/administer-cluster/configure-upgrade-etcd/)
|
||||
|
||||
### Number of Worker Nodes
|
||||
|
||||
+1
-1
@@ -95,7 +95,7 @@ Regarding CPU and memory, it is recommended that the different planes of Kuberne
|
||||
|
||||
For hardware recommendations for large Kubernetes clusters, refer to the official Kubernetes documentation on [building large clusters.](https://kubernetes.io/docs/setup/best-practices/cluster-large/)
|
||||
|
||||
For hardware recommendations for etcd clusters in production, refer to the official [etcd documentation.](https://etcd.io/docs/v3.4.0/op-guide/hardware/)
|
||||
For hardware recommendations for etcd clusters in production, refer to the official [etcd documentation.](https://etcd.io/docs/v3.3/op-guide/hardware/)
|
||||
|
||||
## Networking Requirements
|
||||
|
||||
|
||||
@@ -15,7 +15,7 @@ When the Rancher server is installed on a Kubernetes cluster, it should not be r
|
||||
It is strongly recommended to use hosted infrastructure such as Amazon's EC2 or Google's GCE instead. When you create a cluster using RKE on an infrastructure provider, you can configure the cluster to create etcd snapshots as a backup. You can then [use RKE](https://rancher.com/docs/rke/latest/en/etcd-snapshots/) or [Rancher](../../how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/backup-restore-and-disaster-recovery.md) to restore your cluster from one of these snapshots. In a hosted Kubernetes environment, this backup and restore functionality is not supported.
|
||||
|
||||
### Make sure nodes are configured correctly for Kubernetes
|
||||
It's important to follow K8s and etcd best practices when deploying your nodes, including disabling swap, double checking you have full network connectivity between all machines in the cluster, using unique hostnames, MAC addresses, and product_uuids for every node, checking that all correct ports are opened, and deploying with ssd backed etcd. More details can be found in the [kubernetes docs](https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/#before-you-begin) and [etcd's performance op guide](https://etcd.io/docs/v3.4/op-guide/performance/)
|
||||
It's important to follow K8s and etcd best practices when deploying your nodes, including disabling swap, double checking you have full network connectivity between all machines in the cluster, using unique hostnames, MAC addresses, and product_uuids for every node, checking that all correct ports are opened, and deploying with ssd backed etcd. More details can be found in the [kubernetes docs](https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/#before-you-begin) and [etcd's performance op guide](https://etcd.io/docs/v3.3/op-guide/performance/)
|
||||
|
||||
### When using RKE: Backup the Statefile
|
||||
RKE keeps record of the cluster state in a file called `cluster.rkestate`. This file is important for the recovery of a cluster and/or the continued maintenance of the cluster through RKE. Because this file contains certificate material, we strongly recommend encrypting this file before backing up. After each run of `rke up` you should backup the state file.
|
||||
|
||||
Reference in New Issue
Block a user