mirror of
https://github.com/rancher/rancher-docs.git
synced 2026-09-25 04:28:15 +00:00
#1358 Link to 'Settings for etcd tuning' no longer valid, plus other fixes to links to etcd.io/docs (#1362)
* 1358 Link to 'Settings for etcd tuning' no longer valid updates dead link to new URL * updated etcd version links 2.0-2.4 was tricky, I opted for etcd v3.3 as it's identical to the text on v3.4 except for a single clarifying heading * update link in v2.5 to etcd v3.3 to cover downstream * other 3.4 files in /docs * other 3.4 files in /v2.9 * other 3.4 files in /v2.8 * other 3.4 files in /v2.7 * other 3.4 files in /v2.6 * other 3.4 files in /v2.5 * other 3.4 files in /v2.0-2.4 * rm'ing '.0' from URLs * update v2.6 links to account for differing etcd versions * chinese links
This commit is contained in:
@@ -8,7 +8,7 @@ title: Tuning etcd for Large Installations
|
||||
|
||||
When Rancher is used to manage [a large infrastructure](../../getting-started/installation-and-upgrade/installation-requirements/installation-requirements.md) it is recommended to increase the default keyspace for etcd from the default 2 GB. The maximum setting is 8 GB and the host should have enough RAM to keep the entire dataset in memory. When increasing this value you should also increase the size of the host. The keyspace size can also be adjusted in smaller installations if you anticipate a high rate of change of pods during the garbage collection interval.
|
||||
|
||||
The etcd data set is automatically cleaned up on a five minute interval by Kubernetes. There are situations, e.g. deployment thrashing, where enough events could be written to etcd and deleted before garbage collection occurs and cleans things up causing the keyspace to fill up. If you see `mvcc: database space exceeded` errors, in the etcd logs or Kubernetes API server logs, you should consider increasing the keyspace size. This can be accomplished by setting the [quota-backend-bytes](https://etcd.io/docs/v3.4.0/op-guide/maintenance/#space-quota) setting on the etcd servers.
|
||||
The etcd data set is automatically cleaned up on a five minute interval by Kubernetes. There are situations, e.g. deployment thrashing, where enough events could be written to etcd and deleted before garbage collection occurs and cleans things up causing the keyspace to fill up. If you see `mvcc: database space exceeded` errors, in the etcd logs or Kubernetes API server logs, you should consider increasing the keyspace size. This can be accomplished by setting the [quota-backend-bytes](https://etcd.io/docs/v3.5/op-guide/maintenance/#space-quota) setting on the etcd servers.
|
||||
|
||||
### Example: This snippet of the RKE cluster.yml file increases the keyspace size to 5GB
|
||||
|
||||
@@ -23,7 +23,7 @@ services:
|
||||
|
||||
## Scaling etcd disk performance
|
||||
|
||||
You can follow the recommendations from [the etcd docs](https://etcd.io/docs/v3.4.0/tuning/#disk) on how to tune the disk priority on the host.
|
||||
You can follow the recommendations from [the etcd docs](https://etcd.io/docs/v3.5/tuning/#disk) on how to tune the disk priority on the host.
|
||||
|
||||
Additionally, to reduce IO contention on the disks for etcd, you can use a dedicated device for the data and wal directory. Based on etcd best practices, mirroring RAID configurations are unnecessary because etcd replicates data between the nodes in the cluster. You can use striping RAID configurations to increase available IOPS.
|
||||
|
||||
|
||||
+1
-1
@@ -49,5 +49,5 @@ number of nodes for each Kubernetes role, refer to the section on [recommended a
|
||||
|
||||
### Networking
|
||||
|
||||
* Minimize network latency. Rancher recommends minimizing latency between the etcd nodes. The default setting for `heartbeat-interval` is `500`, and the default setting for `election-timeout` is `5000`. These [settings for etcd tuning](https://coreos.com/etcd/docs/latest/tuning.html) allow etcd to run in most networks (except really high latency networks).
|
||||
* Minimize network latency. Rancher recommends minimizing latency between the etcd nodes. The default setting for `heartbeat-interval` is `500`, and the default setting for `election-timeout` is `5000`. These [settings for etcd tuning](https://etcd.io/docs/v3.5/tuning/) allow etcd to run in most networks (except really high latency networks).
|
||||
* Cluster nodes should be located within a single region. Most cloud providers provide multiple availability zones within a region, which can be used to create higher availability for your cluster. Using multiple availability zones is fine for nodes with any role. If you are using [Kubernetes Cloud Provider](../set-up-cloud-providers/set-up-cloud-providers.md) resources, consult the documentation for any restrictions (i.e. zone storage restrictions).
|
||||
|
||||
+1
-1
@@ -57,7 +57,7 @@ The number of nodes that you can lose at once while maintaining cluster availabi
|
||||
|
||||
References:
|
||||
|
||||
* [Official etcd documentation on optimal etcd cluster size](https://etcd.io/docs/v3.4.0/faq/#what-is-failure-tolerance)
|
||||
* [Official etcd documentation on optimal etcd cluster size](https://etcd.io/docs/v3.5/faq/#what-is-failure-tolerance)
|
||||
* [Official Kubernetes documentation on operating etcd clusters for Kubernetes](https://kubernetes.io/docs/tasks/administer-cluster/configure-upgrade-etcd/)
|
||||
|
||||
### Number of Worker Nodes
|
||||
|
||||
+1
-1
@@ -108,7 +108,7 @@ Regarding CPU and memory, it is recommended that the different planes of Kuberne
|
||||
|
||||
For hardware recommendations for large Kubernetes clusters, refer to the official Kubernetes documentation on [building large clusters.](https://kubernetes.io/docs/setup/best-practices/cluster-large/)
|
||||
|
||||
For hardware recommendations for etcd clusters in production, refer to the official [etcd documentation.](https://etcd.io/docs/v3.4.0/op-guide/hardware/)
|
||||
For hardware recommendations for etcd clusters in production, refer to the official [etcd documentation.](https://etcd.io/docs/v3.5/op-guide/hardware/)
|
||||
|
||||
## Networking Requirements
|
||||
|
||||
|
||||
@@ -18,7 +18,7 @@ When you set up your high-availability Rancher installation, consider the follow
|
||||
Don't run other workloads or microservices in the Kubernetes cluster that Rancher is installed on.
|
||||
|
||||
### Make sure nodes are configured correctly for Kubernetes
|
||||
It's important to follow K8s and etcd best practices when deploying your nodes, including disabling swap, double checking you have full network connectivity between all machines in the cluster, using unique hostnames, MAC addresses, and product_uuids for every node, checking that all correct ports are opened, and deploying with ssd backed etcd. More details can be found in the [kubernetes docs](https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/#before-you-begin) and [etcd's performance op guide](https://etcd.io/docs/v3.4/op-guide/performance/).
|
||||
It's important to follow K8s and etcd best practices when deploying your nodes, including disabling swap, double checking you have full network connectivity between all machines in the cluster, using unique hostnames, MAC addresses, and product_uuids for every node, checking that all correct ports are opened, and deploying with ssd backed etcd. More details can be found in the [kubernetes docs](https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/#before-you-begin) and [etcd's performance op guide](https://etcd.io/docs/v3.5/op-guide/performance/).
|
||||
|
||||
### When using RKE: Back up the Statefile
|
||||
RKE keeps record of the cluster state in a file called `cluster.rkestate`. This file is important for the recovery of a cluster and/or the continued maintenance of the cluster through RKE. Because this file contains certificate material, we strongly recommend encrypting this file before backing up. After each run of `rke up` you should backup the state file.
|
||||
|
||||
+1
-1
@@ -126,7 +126,7 @@ You should keep the local Kubernetes cluster up to date. This will ensure that y
|
||||
|
||||
Etcd is the backend database for Kubernetes and for Rancher. It plays a very important role in Rancher performance.
|
||||
|
||||
The two main bottlenecks to [etcd performance](https://etcd.io/docs/v3.4/op-guide/performance/) are disk and network speed. Etcd should run on dedicated nodes with a fast network setup and with SSDs that have high input/output operations per second (IOPS). For more information regarding etcd performance, see [Slow etcd performance (performance testing and optimization)](https://www.suse.com/support/kb/doc/?id=000020100) and [Tuning etcd for Large Installations](../../../how-to-guides/advanced-user-guides/tune-etcd-for-large-installs.md). Information on disks can also be found in the [Installation Requirements](../../../getting-started/installation-and-upgrade/installation-requirements/installation-requirements.md#disks).
|
||||
The two main bottlenecks to [etcd performance](https://etcd.io/docs/v3.5/op-guide/performance/) are disk and network speed. Etcd should run on dedicated nodes with a fast network setup and with SSDs that have high input/output operations per second (IOPS). For more information regarding etcd performance, see [Slow etcd performance (performance testing and optimization)](https://www.suse.com/support/kb/doc/?id=000020100) and [Tuning etcd for Large Installations](../../../how-to-guides/advanced-user-guides/tune-etcd-for-large-installs.md). Information on disks can also be found in the [Installation Requirements](../../../getting-started/installation-and-upgrade/installation-requirements/installation-requirements.md#disks).
|
||||
|
||||
It's best to run etcd on exactly three nodes, as adding more nodes will reduce operation speed. This may be counter-intuitive to common scaling approaches, but it's due to etcd's [replication mechanisms](https://etcd.io/docs/v3.5/faq/#what-is-maximum-cluster-size).
|
||||
|
||||
|
||||
+2
-2
@@ -4,7 +4,7 @@ title: 为大型安装进行 etcd 调优
|
||||
|
||||
当你运行具有 15 个或更多集群的大型 Rancher 安装时,我们建议你扩大 etcd 的默认 keyspace(默认为 2GB)。你最大可以将它设置为 8GB。此外,请确保主机有足够的 RAM 来保存整个数据集。如果需要增加这个值,你还需要同步增加主机的大小。如果你预计在垃圾回收间隔期间 Pod 的变化率很高,你也可以在较小的安装中调整 Keyspace 大小。
|
||||
|
||||
Kubernetes 每隔五分钟会自动清理 etcd 数据集。在某些情况下(例如发生部署抖动),在垃圾回收发生并进行清理之前会有大量事件写入 etcd 并删除,从而导致 Keyspace 填满。如果你在 etcd 日志或 Kubernetes API Server 日志中看到 `mvcc: database space exceeded` 错误,你可以在 etcd 服务器上设置 [quota-backend-bytes](https://etcd.io/docs/v3.4.0/op-guide/maintenance/#space-quota) 来增加 Keyspace 的大小。
|
||||
Kubernetes 每隔五分钟会自动清理 etcd 数据集。在某些情况下(例如发生部署抖动),在垃圾回收发生并进行清理之前会有大量事件写入 etcd 并删除,从而导致 Keyspace 填满。如果你在 etcd 日志或 Kubernetes API Server 日志中看到 `mvcc: database space exceeded` 错误,你可以在 etcd 服务器上设置 [quota-backend-bytes](https://etcd.io/docs/v3.5/op-guide/maintenance/#space-quota) 来增加 Keyspace 的大小。
|
||||
|
||||
### 示例:此 RKE cluster.yml 文件的代码片段将 Keyspace 的大小增加到 5GB
|
||||
|
||||
@@ -19,7 +19,7 @@ services:
|
||||
|
||||
## 扩展 etcd 磁盘性能
|
||||
|
||||
你可以参见 [etcd 文档](https://etcd.io/docs/v3.4.0/tuning/#disk)中的建议,了解如何调整主机上的磁盘优先级。
|
||||
你可以参见 [etcd 文档](https://etcd.io/docs/v3.5/tuning/#disk)中的建议,了解如何调整主机上的磁盘优先级。
|
||||
|
||||
此外,为了减少 etcd 磁盘上的 IO 争用,你可以为 data 和 wal 目录使用专用设备。etcd 最佳实践不建议配置 Mirror RAID(因为 etcd 在集群中的节点之间复制数据)。你可以使用 striping RAID 配置来增加可用的 IOPS。
|
||||
|
||||
|
||||
+1
-1
@@ -48,5 +48,5 @@ title: 生产就绪集群检查清单
|
||||
|
||||
### 网络
|
||||
|
||||
* 最小化网络延迟。Rancher 建议尽量减少 etcd 节点之间的延迟。`heartbeat-interval` 的默认设置是 `500`,`election-timeout` 的默认设置是 `5000`。这些 [etcd 调优设置](https://coreos.com/etcd/docs/latest/tuning.html) 允许 etcd 在大多数网络(网络延迟特别高的情况下除外)中运行。
|
||||
* 最小化网络延迟。Rancher 建议尽量减少 etcd 节点之间的延迟。`heartbeat-interval` 的默认设置是 `500`,`election-timeout` 的默认设置是 `5000`。这些 [etcd 调优设置](https://etcd.io/docs/v3.5/tuning/) 允许 etcd 在大多数网络(网络延迟特别高的情况下除外)中运行。
|
||||
* 集群节点应位于单个区域内。大多数云厂商在一个区域内提供多个可用区,这可以提高你集群的可用性。任何角色的节点都可以使用多个可用区。如果你使用 [Kubernetes Cloud Provider](../set-up-cloud-providers/set-up-cloud-providers.md) 资源,请查阅文档以了解限制(即区域存储限制)。
|
||||
|
||||
+1
-1
@@ -53,7 +53,7 @@ title: 推荐的集群架构
|
||||
|
||||
参考:
|
||||
|
||||
* [最佳 etcd 集群大小的官方 etcd 文档](https://etcd.io/docs/v3.4.0/faq/#what-is-failure-tolerance)
|
||||
* [最佳 etcd 集群大小的官方 etcd 文档](https://etcd.io/docs/v3.5/faq/#what-is-failure-tolerance)
|
||||
* [为 Kubernetes 操作 etcd 集群的官方 Kubernetes 文档](https://kubernetes.io/docs/tasks/administer-cluster/configure-upgrade-etcd/)
|
||||
|
||||
### Worker 节点数
|
||||
|
||||
+1
-1
@@ -104,7 +104,7 @@ Windows 节点只能用于 Worker 节点。请参阅[配置 Windows 自定义集
|
||||
|
||||
有关大型 Kubernetes 集群的硬件建议,请参阅[构建大型集群](https://kubernetes.io/docs/setup/best-practices/cluster-large/)的官方 Kubernetes 文档。
|
||||
|
||||
有关生产环境中 etcd 集群的硬件建议,请参阅官方 [etcd 文档](https://etcd.io/docs/v3.4.0/op-guide/hardware/)。
|
||||
有关生产环境中 etcd 集群的硬件建议,请参阅官方 [etcd 文档](https://etcd.io/docs/v3.5/op-guide/hardware/)。
|
||||
|
||||
## 网络要求
|
||||
|
||||
|
||||
+1
-1
@@ -44,5 +44,5 @@ title: 生产就绪集群检查清单
|
||||
|
||||
### 网络
|
||||
|
||||
* 最小化网络延迟。Rancher 建议尽量减少 etcd 节点之间的延迟。`heartbeat-interval` 的默认设置是 `500`,`election-timeout` 的默认设置是 `5000`。这些 [etcd 调优设置](https://coreos.com/etcd/docs/latest/tuning.html) 允许 etcd 在大多数网络(网络延迟特别高的情况下除外)中运行。
|
||||
* 最小化网络延迟。Rancher 建议尽量减少 etcd 节点之间的延迟。`heartbeat-interval` 的默认设置是 `500`,`election-timeout` 的默认设置是 `5000`。这些 [etcd 调优设置](https://etcd.io/docs/v3.5/tuning/) 允许 etcd 在大多数网络(网络延迟特别高的情况下除外)中运行。
|
||||
* 集群节点应位于单个区域内。大多数云厂商在一个区域内提供多个可用区,这可以提高你集群的可用性。任何角色的节点都可以使用多个可用区。如果你使用 [Kubernetes Cloud Provider](set-up-cloud-providers.md) 资源,请查阅文档以了解限制(即区域存储限制)。
|
||||
|
||||
+1
-1
@@ -14,7 +14,7 @@ title: Rancher 运行技巧
|
||||
不要在安装了 Rancher 的 Kubernetes 集群上运行其他工作负载或微服务。
|
||||
|
||||
### 确保 Kubernetes 节点配置正确
|
||||
在部署节点时,请遵循 K8s 和 etcd 的最佳实践,其中包括禁用 swap,检查集群中的所有主机之间是否有良好的网络连接,为每个节点使用唯一的主机名、MAC 地址和 `product_uuids`,检查所需端口是否已经打开,并使用配置 SSD 的 etcd 进行部署。详情请参见 [kubernetes 官方文档](https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/#before-you-begin)和 [etcd 性能操作指南](https://etcd.io/docs/v3.4/op-guide/performance/)。
|
||||
在部署节点时,请遵循 K8s 和 etcd 的最佳实践,其中包括禁用 swap,检查集群中的所有主机之间是否有良好的网络连接,为每个节点使用唯一的主机名、MAC 地址和 `product_uuids`,检查所需端口是否已经打开,并使用配置 SSD 的 etcd 进行部署。详情请参见 [kubernetes 官方文档](https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/#before-you-begin)和 [etcd 性能操作指南](https://etcd.io/docs/v3.5/op-guide/performance/)。
|
||||
|
||||
### 使用 RKE 时:备份状态文件(Statefile)
|
||||
RKE 将集群状态记录在一个名为 `cluster.rkestate` 的文件中,该文件对集群的恢复和/或通过 RKE 维护集群非常重要。由于这个文件包含证书材料,我们强烈建议在备份前对该文件进行加密。请在每次运行 `rke up` 后备份状态文件。
|
||||
|
||||
+1
-1
@@ -56,6 +56,6 @@ Rancher 的大部分逻辑都发生在事件处理程序上。每当更新对象
|
||||
与 Rancher 版本类似,我们建议让你的 kubernetes 集群保持使用最新版本。这将确保你的集群能包含可用的性能增强或错误修复。
|
||||
|
||||
### 优化 ETCD
|
||||
[ETCD 性能](https://etcd.io/docs/v3.4/op-guide/performance/)的两个主要瓶颈是磁盘速度和网络速度。对任何一个进行优化都应该能提高性能。有关 ETCD 性能的信息,请参阅 [etcd 性能慢(性能测试和优化)](https://www.suse.com/support/kb/doc/?id=000020100)和[为大型安装调优 etcd](https://docs.ranchermanager.rancher.io/how-to-guides/advanced-user-guides/tune-etcd-for-large-installs)。有关磁盘的信息,你也可以参阅[我们的文档](https://docs.Ranchermanager.Rancher.io/v2.5/pages-for-subheaders/installation-requirements#disks)。
|
||||
[ETCD 性能](https://etcd.io/docs/v3.5/op-guide/performance/)的两个主要瓶颈是磁盘速度和网络速度。对任何一个进行优化都应该能提高性能。有关 ETCD 性能的信息,请参阅 [etcd 性能慢(性能测试和优化)](https://www.suse.com/support/kb/doc/?id=000020100)和[为大型安装调优 etcd](https://docs.ranchermanager.rancher.io/how-to-guides/advanced-user-guides/tune-etcd-for-large-installs)。有关磁盘的信息,你也可以参阅[我们的文档](https://docs.Ranchermanager.Rancher.io/v2.5/pages-for-subheaders/installation-requirements#disks)。
|
||||
|
||||
理论上,ETCD 集群中的节点越多,由于复制要求 [source](https://etcd.io/docs/v3.3/faq),它就会越慢。这可能与常见的缩放方法相悖。我们还可以推断,ETCD 的性能将受到节点间距离的反面影响,因为这将减慢网络通信。
|
||||
|
||||
+1
-1
@@ -110,7 +110,7 @@ Rancher 的大部分逻辑发生在 Event Handler 上。每当资源对象产生
|
||||
|
||||
Etcd 是 Kubernetes 和 Rancher 的后端数据库,在 Rancher 性能中扮演重要的角色。
|
||||
|
||||
[Etcd 性能](https://etcd.io/docs/v3.4/op-guide/performance/)的两个主要瓶颈是磁盘和网络速度。Etcd 应当在具有高速网络和高读写速度 (IOPS) SSD 硬盘的专用节点上运行。有关 etcd 性能的更多信息,请参阅 [etcd 性能缓慢(性能测试和优化)](https://www.suse.com/support/kb/doc/?id=000020100)和[为大型安装进行 etcd 调优](../../../how-to-guides/advanced-user-guides/tune-etcd-for-large-installs.md)。有关磁盘的信息可以在[安装要求](../../../getting-started/installation-and-upgrade/installation-requirements/installation-requirements.md#磁盘)中找到。
|
||||
[Etcd 性能](https://etcd.io/docs/v3.5/op-guide/performance/)的两个主要瓶颈是磁盘和网络速度。Etcd 应当在具有高速网络和高读写速度 (IOPS) SSD 硬盘的专用节点上运行。有关 etcd 性能的更多信息,请参阅 [etcd 性能缓慢(性能测试和优化)](https://www.suse.com/support/kb/doc/?id=000020100)和[为大型安装进行 etcd 调优](../../../how-to-guides/advanced-user-guides/tune-etcd-for-large-installs.md)。有关磁盘的信息可以在[安装要求](../../../getting-started/installation-and-upgrade/installation-requirements/installation-requirements.md#磁盘)中找到。
|
||||
|
||||
根据 etcd 的[复制机制](https://etcd.io/docs/v3.5/faq/#what-is-maximum-cluster-size),建议在三个节点上运行 etcd,运行在更多的节点上反而会降低速度。
|
||||
|
||||
|
||||
+2
-2
@@ -4,7 +4,7 @@ title: 为大型安装进行 etcd 调优
|
||||
|
||||
当你运行具有 15 个或更多集群的大型 Rancher 安装时,我们建议你扩大 etcd 的默认 keyspace(默认为 2GB)。你最大可以将它设置为 8GB。此外,请确保主机有足够的 RAM 来保存整个数据集。如果需要增加这个值,你还需要同步增加主机的大小。如果你预计在垃圾回收间隔期间 Pod 的变化率很高,你也可以在较小的安装中调整 Keyspace 大小。
|
||||
|
||||
Kubernetes 每隔五分钟会自动清理 etcd 数据集。在某些情况下(例如发生部署抖动),在垃圾回收发生并进行清理之前会有大量事件写入 etcd 并删除,从而导致 Keyspace 填满。如果你在 etcd 日志或 Kubernetes API Server 日志中看到 `mvcc: database space exceeded` 错误,你可以在 etcd 服务器上设置 [quota-backend-bytes](https://etcd.io/docs/v3.4.0/op-guide/maintenance/#space-quota) 来增加 Keyspace 的大小。
|
||||
Kubernetes 每隔五分钟会自动清理 etcd 数据集。在某些情况下(例如发生部署抖动),在垃圾回收发生并进行清理之前会有大量事件写入 etcd 并删除,从而导致 Keyspace 填满。如果你在 etcd 日志或 Kubernetes API Server 日志中看到 `mvcc: database space exceeded` 错误,你可以在 etcd 服务器上设置 [quota-backend-bytes](https://etcd.io/docs/v3.4/op-guide/maintenance/#space-quota) 来增加 Keyspace 的大小。
|
||||
|
||||
### 示例:此 RKE cluster.yml 文件的代码片段将 Keyspace 的大小增加到 5GB
|
||||
|
||||
@@ -19,7 +19,7 @@ services:
|
||||
|
||||
## 扩展 etcd 磁盘性能
|
||||
|
||||
你可以参见 [etcd 文档](https://etcd.io/docs/v3.4.0/tuning/#disk)中的建议,了解如何调整主机上的磁盘优先级。
|
||||
你可以参见 [etcd 文档](https://etcd.io/docs/v3.4/tuning/#disk)中的建议,了解如何调整主机上的磁盘优先级。
|
||||
|
||||
此外,为了减少 etcd 磁盘上的 IO 争用,你可以为 data 和 wal 目录使用专用设备。etcd 最佳实践不建议配置 Mirror RAID(因为 etcd 在集群中的节点之间复制数据)。你可以使用 striping RAID 配置来增加可用的 IOPS。
|
||||
|
||||
|
||||
+1
-1
@@ -48,5 +48,5 @@ title: 生产就绪集群检查清单
|
||||
|
||||
### 网络
|
||||
|
||||
* 最小化网络延迟。Rancher 建议尽量减少 etcd 节点之间的延迟。`heartbeat-interval` 的默认设置是 `500`,`election-timeout` 的默认设置是 `5000`。这些 [etcd 调优设置](https://coreos.com/etcd/docs/latest/tuning.html) 允许 etcd 在大多数网络(网络延迟特别高的情况下除外)中运行。
|
||||
* 最小化网络延迟。Rancher 建议尽量减少 etcd 节点之间的延迟。`heartbeat-interval` 的默认设置是 `500`,`election-timeout` 的默认设置是 `5000`。这些 [etcd 调优设置](https://etcd.io/docs/v3.5/tuning/) 允许 etcd 在大多数网络(网络延迟特别高的情况下除外)中运行。
|
||||
* 集群节点应位于单个区域内。大多数云厂商在一个区域内提供多个可用区,这可以提高你集群的可用性。任何角色的节点都可以使用多个可用区。如果你使用 [Kubernetes Cloud Provider](../set-up-cloud-providers/set-up-cloud-providers.md) 资源,请查阅文档以了解限制(即区域存储限制)。
|
||||
|
||||
+1
-1
@@ -53,7 +53,7 @@ title: 推荐的集群架构
|
||||
|
||||
参考:
|
||||
|
||||
* [最佳 etcd 集群大小的官方 etcd 文档](https://etcd.io/docs/v3.4.0/faq/#what-is-failure-tolerance)
|
||||
* [最佳 etcd 集群大小的官方 etcd 文档](https://etcd.io/docs/v3.4/faq/#what-is-failure-tolerance)
|
||||
* [为 Kubernetes 操作 etcd 集群的官方 Kubernetes 文档](https://kubernetes.io/docs/tasks/administer-cluster/configure-upgrade-etcd/)
|
||||
|
||||
### Worker 节点数
|
||||
|
||||
+1
-1
@@ -104,7 +104,7 @@ Windows 节点只能用于 Worker 节点。请参阅[配置 Windows 自定义集
|
||||
|
||||
有关大型 Kubernetes 集群的硬件建议,请参阅[构建大型集群](https://kubernetes.io/docs/setup/best-practices/cluster-large/)的官方 Kubernetes 文档。
|
||||
|
||||
有关生产环境中 etcd 集群的硬件建议,请参阅官方 [etcd 文档](https://etcd.io/docs/v3.4.0/op-guide/hardware/)。
|
||||
有关生产环境中 etcd 集群的硬件建议,请参阅官方 [etcd 文档](https://etcd.io/docs/v3.4/op-guide/hardware/)。
|
||||
|
||||
## 网络要求
|
||||
|
||||
|
||||
+1
-1
@@ -44,5 +44,5 @@ title: 生产就绪集群检查清单
|
||||
|
||||
### 网络
|
||||
|
||||
* 最小化网络延迟。Rancher 建议尽量减少 etcd 节点之间的延迟。`heartbeat-interval` 的默认设置是 `500`,`election-timeout` 的默认设置是 `5000`。这些 [etcd 调优设置](https://coreos.com/etcd/docs/latest/tuning.html) 允许 etcd 在大多数网络(网络延迟特别高的情况下除外)中运行。
|
||||
* 最小化网络延迟。Rancher 建议尽量减少 etcd 节点之间的延迟。`heartbeat-interval` 的默认设置是 `500`,`election-timeout` 的默认设置是 `5000`。这些 [etcd 调优设置](https://etcd.io/docs/v3.5/tuning/) 允许 etcd 在大多数网络(网络延迟特别高的情况下除外)中运行。
|
||||
* 集群节点应位于单个区域内。大多数云厂商在一个区域内提供多个可用区,这可以提高你集群的可用性。任何角色的节点都可以使用多个可用区。如果你使用 [Kubernetes Cloud Provider](set-up-cloud-providers.md) 资源,请查阅文档以了解限制(即区域存储限制)。
|
||||
|
||||
+2
-2
@@ -4,7 +4,7 @@ title: 为大型安装进行 etcd 调优
|
||||
|
||||
当你运行具有 15 个或更多集群的大型 Rancher 安装时,我们建议你扩大 etcd 的默认 keyspace(默认为 2GB)。你最大可以将它设置为 8GB。此外,请确保主机有足够的 RAM 来保存整个数据集。如果需要增加这个值,你还需要同步增加主机的大小。如果你预计在垃圾回收间隔期间 Pod 的变化率很高,你也可以在较小的安装中调整 Keyspace 大小。
|
||||
|
||||
Kubernetes 每隔五分钟会自动清理 etcd 数据集。在某些情况下(例如发生部署抖动),在垃圾回收发生并进行清理之前会有大量事件写入 etcd 并删除,从而导致 Keyspace 填满。如果你在 etcd 日志或 Kubernetes API Server 日志中看到 `mvcc: database space exceeded` 错误,你可以在 etcd 服务器上设置 [quota-backend-bytes](https://etcd.io/docs/v3.4.0/op-guide/maintenance/#space-quota) 来增加 Keyspace 的大小。
|
||||
Kubernetes 每隔五分钟会自动清理 etcd 数据集。在某些情况下(例如发生部署抖动),在垃圾回收发生并进行清理之前会有大量事件写入 etcd 并删除,从而导致 Keyspace 填满。如果你在 etcd 日志或 Kubernetes API Server 日志中看到 `mvcc: database space exceeded` 错误,你可以在 etcd 服务器上设置 [quota-backend-bytes](https://etcd.io/docs/v3.5/op-guide/maintenance/#space-quota) 来增加 Keyspace 的大小。
|
||||
|
||||
### 示例:此 RKE cluster.yml 文件的代码片段将 Keyspace 的大小增加到 5GB
|
||||
|
||||
@@ -19,7 +19,7 @@ services:
|
||||
|
||||
## 扩展 etcd 磁盘性能
|
||||
|
||||
你可以参见 [etcd 文档](https://etcd.io/docs/v3.4.0/tuning/#disk)中的建议,了解如何调整主机上的磁盘优先级。
|
||||
你可以参见 [etcd 文档](https://etcd.io/docs/v3.5/tuning/#disk)中的建议,了解如何调整主机上的磁盘优先级。
|
||||
|
||||
此外,为了减少 etcd 磁盘上的 IO 争用,你可以为 data 和 wal 目录使用专用设备。etcd 最佳实践不建议配置 Mirror RAID(因为 etcd 在集群中的节点之间复制数据)。你可以使用 striping RAID 配置来增加可用的 IOPS。
|
||||
|
||||
|
||||
+1
-1
@@ -48,5 +48,5 @@ title: 生产就绪集群检查清单
|
||||
|
||||
### 网络
|
||||
|
||||
* 最小化网络延迟。Rancher 建议尽量减少 etcd 节点之间的延迟。`heartbeat-interval` 的默认设置是 `500`,`election-timeout` 的默认设置是 `5000`。这些 [etcd 调优设置](https://coreos.com/etcd/docs/latest/tuning.html) 允许 etcd 在大多数网络(网络延迟特别高的情况下除外)中运行。
|
||||
* 最小化网络延迟。Rancher 建议尽量减少 etcd 节点之间的延迟。`heartbeat-interval` 的默认设置是 `500`,`election-timeout` 的默认设置是 `5000`。这些 [etcd 调优设置](https://etcd.io/docs/v3.5/tuning/) 允许 etcd 在大多数网络(网络延迟特别高的情况下除外)中运行。
|
||||
* 集群节点应位于单个区域内。大多数云厂商在一个区域内提供多个可用区,这可以提高你集群的可用性。任何角色的节点都可以使用多个可用区。如果你使用 (../set-up-cloud-providers/set-up-cloud-providers.md) 资源,请查阅文档以了解限制(即区域存储限制)。
|
||||
|
||||
+1
-1
@@ -53,7 +53,7 @@ title: 推荐的集群架构
|
||||
|
||||
参考:
|
||||
|
||||
* [最佳 etcd 集群大小的官方 etcd 文档](https://etcd.io/docs/v3.4.0/faq/#what-is-failure-tolerance)
|
||||
* [最佳 etcd 集群大小的官方 etcd 文档](https://etcd.io/docs/v3.5/faq/#what-is-failure-tolerance)
|
||||
* [为 Kubernetes 操作 etcd 集群的官方 Kubernetes 文档](https://kubernetes.io/docs/tasks/administer-cluster/configure-upgrade-etcd/)
|
||||
|
||||
### Worker 节点数
|
||||
|
||||
+1
-1
@@ -104,7 +104,7 @@ Windows 节点只能用于 Worker 节点。请参阅[配置 Windows 自定义集
|
||||
|
||||
有关大型 Kubernetes 集群的硬件建议,请参阅[构建大型集群](https://kubernetes.io/docs/setup/best-practices/cluster-large/)的官方 Kubernetes 文档。
|
||||
|
||||
有关生产环境中 etcd 集群的硬件建议,请参阅官方 [etcd 文档](https://etcd.io/docs/v3.4.0/op-guide/hardware/)。
|
||||
有关生产环境中 etcd 集群的硬件建议,请参阅官方 [etcd 文档](https://etcd.io/docs/v3.5/op-guide/hardware/)。
|
||||
|
||||
## 网络要求
|
||||
|
||||
|
||||
+1
-1
@@ -44,5 +44,5 @@ title: 生产就绪集群检查清单
|
||||
|
||||
### 网络
|
||||
|
||||
* 最小化网络延迟。Rancher 建议尽量减少 etcd 节点之间的延迟。`heartbeat-interval` 的默认设置是 `500`,`election-timeout` 的默认设置是 `5000`。这些 [etcd 调优设置](https://coreos.com/etcd/docs/latest/tuning.html) 允许 etcd 在大多数网络(网络延迟特别高的情况下除外)中运行。
|
||||
* 最小化网络延迟。Rancher 建议尽量减少 etcd 节点之间的延迟。`heartbeat-interval` 的默认设置是 `500`,`election-timeout` 的默认设置是 `5000`。这些 [etcd 调优设置](https://etcd.io/docs/v3.5/tuning/) 允许 etcd 在大多数网络(网络延迟特别高的情况下除外)中运行。
|
||||
* 集群节点应位于单个区域内。大多数云厂商在一个区域内提供多个可用区,这可以提高你集群的可用性。任何角色的节点都可以使用多个可用区。如果你使用 [Kubernetes Cloud Provider](set-up-cloud-providers.md) 资源,请查阅文档以了解限制(即区域存储限制)。
|
||||
|
||||
+1
-1
@@ -14,7 +14,7 @@ title: Rancher 运行技巧
|
||||
不要在安装了 Rancher 的 Kubernetes 集群上运行其他工作负载或微服务。
|
||||
|
||||
### 确保 Kubernetes 节点配置正确
|
||||
在部署节点时,请遵循 K8s 和 etcd 的最佳实践,其中包括禁用 swap,检查集群中的所有主机之间是否有良好的网络连接,为每个节点使用唯一的主机名、MAC 地址和 `product_uuids`,检查所需端口是否已经打开,并使用配置 SSD 的 etcd 进行部署。详情请参见 [kubernetes 官方文档](https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/#before-you-begin)和 [etcd 性能操作指南](https://etcd.io/docs/v3.4/op-guide/performance/)。
|
||||
在部署节点时,请遵循 K8s 和 etcd 的最佳实践,其中包括禁用 swap,检查集群中的所有主机之间是否有良好的网络连接,为每个节点使用唯一的主机名、MAC 地址和 `product_uuids`,检查所需端口是否已经打开,并使用配置 SSD 的 etcd 进行部署。详情请参见 [kubernetes 官方文档](https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/#before-you-begin)和 [etcd 性能操作指南](https://etcd.io/docs/v3.5/op-guide/performance/)。
|
||||
|
||||
### 使用 RKE 时:备份状态文件(Statefile)
|
||||
RKE 将集群状态记录在一个名为 `cluster.rkestate` 的文件中,该文件对集群的恢复和/或通过 RKE 维护集群非常重要。由于这个文件包含证书材料,我们强烈建议在备份前对该文件进行加密。请在每次运行 `rke up` 后备份状态文件。
|
||||
|
||||
+1
-1
@@ -56,6 +56,6 @@ Rancher 的大部分逻辑都发生在事件处理程序上。每当更新对象
|
||||
与 Rancher 版本类似,我们建议让你的 kubernetes 集群保持使用最新版本。这将确保你的集群能包含可用的性能增强或错误修复。
|
||||
|
||||
### 优化 ETCD
|
||||
[ETCD 性能](https://etcd.io/docs/v3.4/op-guide/performance/)的两个主要瓶颈是磁盘速度和网络速度。对任何一个进行优化都应该能提高性能。有关 ETCD 性能的信息,请参阅 [etcd 性能慢(性能测试和优化)](https://www.suse.com/support/kb/doc/?id=000020100)和[为大型安装调优 etcd](https://docs.ranchermanager.rancher.io/how-to-guides/advanced-user-guides/tune-etcd-for-large-installs)。有关磁盘的信息,你也可以参阅[我们的文档](https://docs.Ranchermanager.Rancher.io/v2.5/pages-for-subheaders/installation-requirements#disks)。
|
||||
[ETCD 性能](https://etcd.io/docs/v3.5/op-guide/performance/)的两个主要瓶颈是磁盘速度和网络速度。对任何一个进行优化都应该能提高性能。有关 ETCD 性能的信息,请参阅 [etcd 性能慢(性能测试和优化)](https://www.suse.com/support/kb/doc/?id=000020100)和[为大型安装调优 etcd](https://docs.ranchermanager.rancher.io/how-to-guides/advanced-user-guides/tune-etcd-for-large-installs)。有关磁盘的信息,你也可以参阅[我们的文档](https://docs.Ranchermanager.Rancher.io/v2.5/pages-for-subheaders/installation-requirements#disks)。
|
||||
|
||||
理论上,ETCD 集群中的节点越多,由于复制要求 [source](https://etcd.io/docs/v3.3/faq),它就会越慢。这可能与常见的缩放方法相悖。我们还可以推断,ETCD 的性能将受到节点间距离的反面影响,因为这将减慢网络通信。
|
||||
|
||||
+1
-1
@@ -110,7 +110,7 @@ Rancher 的大部分逻辑发生在 Event Handler 上。每当资源对象产生
|
||||
|
||||
Etcd 是 Kubernetes 和 Rancher 的后端数据库,在 Rancher 性能中扮演重要的角色。
|
||||
|
||||
[Etcd 性能](https://etcd.io/docs/v3.4/op-guide/performance/)的两个主要瓶颈是磁盘和网络速度。Etcd 应当在具有高速网络和高读写速度 (IOPS) SSD 硬盘的专用节点上运行。有关 etcd 性能的更多信息,请参阅 [etcd 性能缓慢(性能测试和优化)](https://www.suse.com/support/kb/doc/?id=000020100)和[为大型安装进行 etcd 调优](../../../how-to-guides/advanced-user-guides/tune-etcd-for-large-installs.md)。有关磁盘的信息可以在[安装要求](../../../getting-started/installation-and-upgrade/installation-requirements/installation-requirements.md#磁盘)中找到。
|
||||
[Etcd 性能](https://etcd.io/docs/v3.5/op-guide/performance/)的两个主要瓶颈是磁盘和网络速度。Etcd 应当在具有高速网络和高读写速度 (IOPS) SSD 硬盘的专用节点上运行。有关 etcd 性能的更多信息,请参阅 [etcd 性能缓慢(性能测试和优化)](https://www.suse.com/support/kb/doc/?id=000020100)和[为大型安装进行 etcd 调优](../../../how-to-guides/advanced-user-guides/tune-etcd-for-large-installs.md)。有关磁盘的信息可以在[安装要求](../../../getting-started/installation-and-upgrade/installation-requirements/installation-requirements.md#磁盘)中找到。
|
||||
|
||||
根据 etcd 的[复制机制](https://etcd.io/docs/v3.5/faq/#what-is-maximum-cluster-size),建议在三个节点上运行 etcd,运行在更多的节点上反而会降低速度。
|
||||
|
||||
|
||||
+2
-2
@@ -4,7 +4,7 @@ title: 为大型安装进行 etcd 调优
|
||||
|
||||
当你运行具有 15 个或更多集群的大型 Rancher 安装时,我们建议你扩大 etcd 的默认 keyspace(默认为 2GB)。你最大可以将它设置为 8GB。此外,请确保主机有足够的 RAM 来保存整个数据集。如果需要增加这个值,你还需要同步增加主机的大小。如果你预计在垃圾回收间隔期间 Pod 的变化率很高,你也可以在较小的安装中调整 Keyspace 大小。
|
||||
|
||||
Kubernetes 每隔五分钟会自动清理 etcd 数据集。在某些情况下(例如发生部署抖动),在垃圾回收发生并进行清理之前会有大量事件写入 etcd 并删除,从而导致 Keyspace 填满。如果你在 etcd 日志或 Kubernetes API Server 日志中看到 `mvcc: database space exceeded` 错误,你可以在 etcd 服务器上设置 [quota-backend-bytes](https://etcd.io/docs/v3.4.0/op-guide/maintenance/#space-quota) 来增加 Keyspace 的大小。
|
||||
Kubernetes 每隔五分钟会自动清理 etcd 数据集。在某些情况下(例如发生部署抖动),在垃圾回收发生并进行清理之前会有大量事件写入 etcd 并删除,从而导致 Keyspace 填满。如果你在 etcd 日志或 Kubernetes API Server 日志中看到 `mvcc: database space exceeded` 错误,你可以在 etcd 服务器上设置 [quota-backend-bytes](https://etcd.io/docs/v3.5/op-guide/maintenance/#space-quota) 来增加 Keyspace 的大小。
|
||||
|
||||
### 示例:此 RKE cluster.yml 文件的代码片段将 Keyspace 的大小增加到 5GB
|
||||
|
||||
@@ -19,7 +19,7 @@ services:
|
||||
|
||||
## 扩展 etcd 磁盘性能
|
||||
|
||||
你可以参见 [etcd 文档](https://etcd.io/docs/v3.4.0/tuning/#disk)中的建议,了解如何调整主机上的磁盘优先级。
|
||||
你可以参见 [etcd 文档](https://etcd.io/docs/v3.5/tuning/#disk)中的建议,了解如何调整主机上的磁盘优先级。
|
||||
|
||||
此外,为了减少 etcd 磁盘上的 IO 争用,你可以为 data 和 wal 目录使用专用设备。etcd 最佳实践不建议配置 Mirror RAID(因为 etcd 在集群中的节点之间复制数据)。你可以使用 striping RAID 配置来增加可用的 IOPS。
|
||||
|
||||
|
||||
+1
-1
@@ -48,5 +48,5 @@ title: 生产就绪集群检查清单
|
||||
|
||||
### 网络
|
||||
|
||||
* 最小化网络延迟。Rancher 建议尽量减少 etcd 节点之间的延迟。`heartbeat-interval` 的默认设置是 `500`,`election-timeout` 的默认设置是 `5000`。这些 [etcd 调优设置](https://coreos.com/etcd/docs/latest/tuning.html) 允许 etcd 在大多数网络(网络延迟特别高的情况下除外)中运行。
|
||||
* 最小化网络延迟。Rancher 建议尽量减少 etcd 节点之间的延迟。`heartbeat-interval` 的默认设置是 `500`,`election-timeout` 的默认设置是 `5000`。这些 [etcd 调优设置](https://etcd.io/docs/v3.5/tuning/) 允许 etcd 在大多数网络(网络延迟特别高的情况下除外)中运行。
|
||||
* 集群节点应位于单个区域内。大多数云厂商在一个区域内提供多个可用区,这可以提高你集群的可用性。任何角色的节点都可以使用多个可用区。如果你使用 (../set-up-cloud-providers/set-up-cloud-providers.md) 资源,请查阅文档以了解限制(即区域存储限制)。
|
||||
|
||||
+1
-1
@@ -53,7 +53,7 @@ title: 推荐的集群架构
|
||||
|
||||
参考:
|
||||
|
||||
* [最佳 etcd 集群大小的官方 etcd 文档](https://etcd.io/docs/v3.4.0/faq/#what-is-failure-tolerance)
|
||||
* [最佳 etcd 集群大小的官方 etcd 文档](https://etcd.io/docs/v3.5/faq/#what-is-failure-tolerance)
|
||||
* [为 Kubernetes 操作 etcd 集群的官方 Kubernetes 文档](https://kubernetes.io/docs/tasks/administer-cluster/configure-upgrade-etcd/)
|
||||
|
||||
### Worker 节点数
|
||||
|
||||
+1
-1
@@ -104,7 +104,7 @@ Windows 节点只能用于 Worker 节点。请参阅[配置 Windows 自定义集
|
||||
|
||||
有关大型 Kubernetes 集群的硬件建议,请参阅[构建大型集群](https://kubernetes.io/docs/setup/best-practices/cluster-large/)的官方 Kubernetes 文档。
|
||||
|
||||
有关生产环境中 etcd 集群的硬件建议,请参阅官方 [etcd 文档](https://etcd.io/docs/v3.4.0/op-guide/hardware/)。
|
||||
有关生产环境中 etcd 集群的硬件建议,请参阅官方 [etcd 文档](https://etcd.io/docs/v3.5/op-guide/hardware/)。
|
||||
|
||||
## 网络要求
|
||||
|
||||
|
||||
+1
-1
@@ -44,5 +44,5 @@ title: 生产就绪集群检查清单
|
||||
|
||||
### 网络
|
||||
|
||||
* 最小化网络延迟。Rancher 建议尽量减少 etcd 节点之间的延迟。`heartbeat-interval` 的默认设置是 `500`,`election-timeout` 的默认设置是 `5000`。这些 [etcd 调优设置](https://coreos.com/etcd/docs/latest/tuning.html) 允许 etcd 在大多数网络(网络延迟特别高的情况下除外)中运行。
|
||||
* 最小化网络延迟。Rancher 建议尽量减少 etcd 节点之间的延迟。`heartbeat-interval` 的默认设置是 `500`,`election-timeout` 的默认设置是 `5000`。这些 [etcd 调优设置](https://etcd.io/docs/v3.5/tuning/) 允许 etcd 在大多数网络(网络延迟特别高的情况下除外)中运行。
|
||||
* 集群节点应位于单个区域内。大多数云厂商在一个区域内提供多个可用区,这可以提高你集群的可用性。任何角色的节点都可以使用多个可用区。如果你使用 [Kubernetes Cloud Provider](set-up-cloud-providers.md) 资源,请查阅文档以了解限制(即区域存储限制)。
|
||||
|
||||
+1
-1
@@ -14,7 +14,7 @@ title: Rancher 运行技巧
|
||||
不要在安装了 Rancher 的 Kubernetes 集群上运行其他工作负载或微服务。
|
||||
|
||||
### 确保 Kubernetes 节点配置正确
|
||||
在部署节点时,请遵循 K8s 和 etcd 的最佳实践,其中包括禁用 swap,检查集群中的所有主机之间是否有良好的网络连接,为每个节点使用唯一的主机名、MAC 地址和 `product_uuids`,检查所需端口是否已经打开,并使用配置 SSD 的 etcd 进行部署。详情请参见 [kubernetes 官方文档](https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/#before-you-begin)和 [etcd 性能操作指南](https://etcd.io/docs/v3.4/op-guide/performance/)。
|
||||
在部署节点时,请遵循 K8s 和 etcd 的最佳实践,其中包括禁用 swap,检查集群中的所有主机之间是否有良好的网络连接,为每个节点使用唯一的主机名、MAC 地址和 `product_uuids`,检查所需端口是否已经打开,并使用配置 SSD 的 etcd 进行部署。详情请参见 [kubernetes 官方文档](https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/#before-you-begin)和 [etcd 性能操作指南](https://etcd.io/docs/v3.5/op-guide/performance/)。
|
||||
|
||||
### 使用 RKE 时:备份状态文件(Statefile)
|
||||
RKE 将集群状态记录在一个名为 `cluster.rkestate` 的文件中,该文件对集群的恢复和/或通过 RKE 维护集群非常重要。由于这个文件包含证书材料,我们强烈建议在备份前对该文件进行加密。请在每次运行 `rke up` 后备份状态文件。
|
||||
|
||||
+1
-1
@@ -56,6 +56,6 @@ Rancher 的大部分逻辑都发生在事件处理程序上。每当更新对象
|
||||
与 Rancher 版本类似,我们建议让你的 kubernetes 集群保持使用最新版本。这将确保你的集群能包含可用的性能增强或错误修复。
|
||||
|
||||
### 优化 ETCD
|
||||
[ETCD 性能](https://etcd.io/docs/v3.4/op-guide/performance/)的两个主要瓶颈是磁盘速度和网络速度。对任何一个进行优化都应该能提高性能。有关 ETCD 性能的信息,请参阅 [etcd 性能慢(性能测试和优化)](https://www.suse.com/support/kb/doc/?id=000020100)和[为大型安装调优 etcd](https://docs.ranchermanager.rancher.io/how-to-guides/advanced-user-guides/tune-etcd-for-large-installs)。有关磁盘的信息,你也可以参阅[我们的文档](https://docs.Ranchermanager.Rancher.io/v2.5/pages-for-subheaders/installation-requirements#disks)。
|
||||
[ETCD 性能](https://etcd.io/docs/v3.5/op-guide/performance/)的两个主要瓶颈是磁盘速度和网络速度。对任何一个进行优化都应该能提高性能。有关 ETCD 性能的信息,请参阅 [etcd 性能慢(性能测试和优化)](https://www.suse.com/support/kb/doc/?id=000020100)和[为大型安装调优 etcd](https://docs.ranchermanager.rancher.io/how-to-guides/advanced-user-guides/tune-etcd-for-large-installs)。有关磁盘的信息,你也可以参阅[我们的文档](https://docs.Ranchermanager.Rancher.io/v2.5/pages-for-subheaders/installation-requirements#disks)。
|
||||
|
||||
理论上,ETCD 集群中的节点越多,由于复制要求 [source](https://etcd.io/docs/v3.3/faq),它就会越慢。这可能与常见的缩放方法相悖。我们还可以推断,ETCD 的性能将受到节点间距离的反面影响,因为这将减慢网络通信。
|
||||
|
||||
+1
-1
@@ -110,7 +110,7 @@ Rancher 的大部分逻辑发生在 Event Handler 上。每当资源对象产生
|
||||
|
||||
Etcd 是 Kubernetes 和 Rancher 的后端数据库,在 Rancher 性能中扮演重要的角色。
|
||||
|
||||
[Etcd 性能](https://etcd.io/docs/v3.4/op-guide/performance/)的两个主要瓶颈是磁盘和网络速度。Etcd 应当在具有高速网络和高读写速度 (IOPS) SSD 硬盘的专用节点上运行。有关 etcd 性能的更多信息,请参阅 [etcd 性能缓慢(性能测试和优化)](https://www.suse.com/support/kb/doc/?id=000020100)和[为大型安装进行 etcd 调优](../../../how-to-guides/advanced-user-guides/tune-etcd-for-large-installs.md)。有关磁盘的信息可以在[安装要求](../../../getting-started/installation-and-upgrade/installation-requirements/installation-requirements.md#磁盘)中找到。
|
||||
[Etcd 性能](https://etcd.io/docs/v3.5/op-guide/performance/)的两个主要瓶颈是磁盘和网络速度。Etcd 应当在具有高速网络和高读写速度 (IOPS) SSD 硬盘的专用节点上运行。有关 etcd 性能的更多信息,请参阅 [etcd 性能缓慢(性能测试和优化)](https://www.suse.com/support/kb/doc/?id=000020100)和[为大型安装进行 etcd 调优](../../../how-to-guides/advanced-user-guides/tune-etcd-for-large-installs.md)。有关磁盘的信息可以在[安装要求](../../../getting-started/installation-and-upgrade/installation-requirements/installation-requirements.md#磁盘)中找到。
|
||||
|
||||
根据 etcd 的[复制机制](https://etcd.io/docs/v3.5/faq/#what-is-maximum-cluster-size),建议在三个节点上运行 etcd,运行在更多的节点上反而会降低速度。
|
||||
|
||||
|
||||
+2
-2
@@ -4,7 +4,7 @@ title: 为大型安装进行 etcd 调优
|
||||
|
||||
当你运行具有 15 个或更多集群的大型 Rancher 安装时,我们建议你扩大 etcd 的默认 keyspace(默认为 2GB)。你最大可以将它设置为 8GB。此外,请确保主机有足够的 RAM 来保存整个数据集。如果需要增加这个值,你还需要同步增加主机的大小。如果你预计在垃圾回收间隔期间 Pod 的变化率很高,你也可以在较小的安装中调整 Keyspace 大小。
|
||||
|
||||
Kubernetes 每隔五分钟会自动清理 etcd 数据集。在某些情况下(例如发生部署抖动),在垃圾回收发生并进行清理之前会有大量事件写入 etcd 并删除,从而导致 Keyspace 填满。如果你在 etcd 日志或 Kubernetes API Server 日志中看到 `mvcc: database space exceeded` 错误,你可以在 etcd 服务器上设置 [quota-backend-bytes](https://etcd.io/docs/v3.4.0/op-guide/maintenance/#space-quota) 来增加 Keyspace 的大小。
|
||||
Kubernetes 每隔五分钟会自动清理 etcd 数据集。在某些情况下(例如发生部署抖动),在垃圾回收发生并进行清理之前会有大量事件写入 etcd 并删除,从而导致 Keyspace 填满。如果你在 etcd 日志或 Kubernetes API Server 日志中看到 `mvcc: database space exceeded` 错误,你可以在 etcd 服务器上设置 [quota-backend-bytes](https://etcd.io/docs/v3.5/op-guide/maintenance/#space-quota) 来增加 Keyspace 的大小。
|
||||
|
||||
### 示例:此 RKE cluster.yml 文件的代码片段将 Keyspace 的大小增加到 5GB
|
||||
|
||||
@@ -19,7 +19,7 @@ services:
|
||||
|
||||
## 扩展 etcd 磁盘性能
|
||||
|
||||
你可以参见 [etcd 文档](https://etcd.io/docs/v3.4.0/tuning/#disk)中的建议,了解如何调整主机上的磁盘优先级。
|
||||
你可以参见 [etcd 文档](https://etcd.io/docs/v3.5/tuning/#disk)中的建议,了解如何调整主机上的磁盘优先级。
|
||||
|
||||
此外,为了减少 etcd 磁盘上的 IO 争用,你可以为 data 和 wal 目录使用专用设备。etcd 最佳实践不建议配置 Mirror RAID(因为 etcd 在集群中的节点之间复制数据)。你可以使用 striping RAID 配置来增加可用的 IOPS。
|
||||
|
||||
|
||||
+1
-1
@@ -48,5 +48,5 @@ title: 生产就绪集群检查清单
|
||||
|
||||
### 网络
|
||||
|
||||
* 最小化网络延迟。Rancher 建议尽量减少 etcd 节点之间的延迟。`heartbeat-interval` 的默认设置是 `500`,`election-timeout` 的默认设置是 `5000`。这些 [etcd 调优设置](https://coreos.com/etcd/docs/latest/tuning.html) 允许 etcd 在大多数网络(网络延迟特别高的情况下除外)中运行。
|
||||
* 最小化网络延迟。Rancher 建议尽量减少 etcd 节点之间的延迟。`heartbeat-interval` 的默认设置是 `500`,`election-timeout` 的默认设置是 `5000`。这些 [etcd 调优设置](https://etcd.io/docs/v3.5/tuning/) 允许 etcd 在大多数网络(网络延迟特别高的情况下除外)中运行。
|
||||
* 集群节点应位于单个区域内。大多数云厂商在一个区域内提供多个可用区,这可以提高你集群的可用性。任何角色的节点都可以使用多个可用区。如果你使用 [Kubernetes Cloud Provider](../set-up-cloud-providers/set-up-cloud-providers.md) 资源,请查阅文档以了解限制(即区域存储限制)。
|
||||
|
||||
+1
-1
@@ -53,7 +53,7 @@ title: 推荐的集群架构
|
||||
|
||||
参考:
|
||||
|
||||
* [最佳 etcd 集群大小的官方 etcd 文档](https://etcd.io/docs/v3.4.0/faq/#what-is-failure-tolerance)
|
||||
* [最佳 etcd 集群大小的官方 etcd 文档](https://etcd.io/docs/v3.5/faq/#what-is-failure-tolerance)
|
||||
* [为 Kubernetes 操作 etcd 集群的官方 Kubernetes 文档](https://kubernetes.io/docs/tasks/administer-cluster/configure-upgrade-etcd/)
|
||||
|
||||
### Worker 节点数
|
||||
|
||||
+1
-1
@@ -104,7 +104,7 @@ Windows 节点只能用于 Worker 节点。请参阅[配置 Windows 自定义集
|
||||
|
||||
有关大型 Kubernetes 集群的硬件建议,请参阅[构建大型集群](https://kubernetes.io/docs/setup/best-practices/cluster-large/)的官方 Kubernetes 文档。
|
||||
|
||||
有关生产环境中 etcd 集群的硬件建议,请参阅官方 [etcd 文档](https://etcd.io/docs/v3.4.0/op-guide/hardware/)。
|
||||
有关生产环境中 etcd 集群的硬件建议,请参阅官方 [etcd 文档](https://etcd.io/docs/v3.5/op-guide/hardware/)。
|
||||
|
||||
## 网络要求
|
||||
|
||||
|
||||
+1
-1
@@ -44,5 +44,5 @@ title: 生产就绪集群检查清单
|
||||
|
||||
### 网络
|
||||
|
||||
* 最小化网络延迟。Rancher 建议尽量减少 etcd 节点之间的延迟。`heartbeat-interval` 的默认设置是 `500`,`election-timeout` 的默认设置是 `5000`。这些 [etcd 调优设置](https://coreos.com/etcd/docs/latest/tuning.html) 允许 etcd 在大多数网络(网络延迟特别高的情况下除外)中运行。
|
||||
* 最小化网络延迟。Rancher 建议尽量减少 etcd 节点之间的延迟。`heartbeat-interval` 的默认设置是 `500`,`election-timeout` 的默认设置是 `5000`。这些 [etcd 调优设置](https://etcd.io/docs/v3.5/tuning/) 允许 etcd 在大多数网络(网络延迟特别高的情况下除外)中运行。
|
||||
* 集群节点应位于单个区域内。大多数云厂商在一个区域内提供多个可用区,这可以提高你集群的可用性。任何角色的节点都可以使用多个可用区。如果你使用 [Kubernetes Cloud Provider](set-up-cloud-providers.md) 资源,请查阅文档以了解限制(即区域存储限制)。
|
||||
|
||||
+1
-1
@@ -14,7 +14,7 @@ title: Rancher 运行技巧
|
||||
不要在安装了 Rancher 的 Kubernetes 集群上运行其他工作负载或微服务。
|
||||
|
||||
### 确保 Kubernetes 节点配置正确
|
||||
在部署节点时,请遵循 K8s 和 etcd 的最佳实践,其中包括禁用 swap,检查集群中的所有主机之间是否有良好的网络连接,为每个节点使用唯一的主机名、MAC 地址和 `product_uuids`,检查所需端口是否已经打开,并使用配置 SSD 的 etcd 进行部署。详情请参见 [kubernetes 官方文档](https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/#before-you-begin)和 [etcd 性能操作指南](https://etcd.io/docs/v3.4/op-guide/performance/)。
|
||||
在部署节点时,请遵循 K8s 和 etcd 的最佳实践,其中包括禁用 swap,检查集群中的所有主机之间是否有良好的网络连接,为每个节点使用唯一的主机名、MAC 地址和 `product_uuids`,检查所需端口是否已经打开,并使用配置 SSD 的 etcd 进行部署。详情请参见 [kubernetes 官方文档](https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/#before-you-begin)和 [etcd 性能操作指南](https://etcd.io/docs/v3.5/op-guide/performance/)。
|
||||
|
||||
### 使用 RKE 时:备份状态文件(Statefile)
|
||||
RKE 将集群状态记录在一个名为 `cluster.rkestate` 的文件中,该文件对集群的恢复和/或通过 RKE 维护集群非常重要。由于这个文件包含证书材料,我们强烈建议在备份前对该文件进行加密。请在每次运行 `rke up` 后备份状态文件。
|
||||
|
||||
+1
-1
@@ -56,6 +56,6 @@ Rancher 的大部分逻辑都发生在事件处理程序上。每当更新对象
|
||||
与 Rancher 版本类似,我们建议让你的 kubernetes 集群保持使用最新版本。这将确保你的集群能包含可用的性能增强或错误修复。
|
||||
|
||||
### 优化 ETCD
|
||||
[ETCD 性能](https://etcd.io/docs/v3.4/op-guide/performance/)的两个主要瓶颈是磁盘速度和网络速度。对任何一个进行优化都应该能提高性能。有关 ETCD 性能的信息,请参阅 [etcd 性能慢(性能测试和优化)](https://www.suse.com/support/kb/doc/?id=000020100)和[为大型安装调优 etcd](https://docs.ranchermanager.rancher.io/how-to-guides/advanced-user-guides/tune-etcd-for-large-installs)。有关磁盘的信息,你也可以参阅[我们的文档](https://docs.Ranchermanager.Rancher.io/v2.5/pages-for-subheaders/installation-requirements#disks)。
|
||||
[ETCD 性能](https://etcd.io/docs/v3.5/op-guide/performance/)的两个主要瓶颈是磁盘速度和网络速度。对任何一个进行优化都应该能提高性能。有关 ETCD 性能的信息,请参阅 [etcd 性能慢(性能测试和优化)](https://www.suse.com/support/kb/doc/?id=000020100)和[为大型安装调优 etcd](https://docs.ranchermanager.rancher.io/how-to-guides/advanced-user-guides/tune-etcd-for-large-installs)。有关磁盘的信息,你也可以参阅[我们的文档](https://docs.Ranchermanager.Rancher.io/v2.5/pages-for-subheaders/installation-requirements#disks)。
|
||||
|
||||
理论上,ETCD 集群中的节点越多,由于复制要求 [source](https://etcd.io/docs/v3.3/faq),它就会越慢。这可能与常见的缩放方法相悖。我们还可以推断,ETCD 的性能将受到节点间距离的反面影响,因为这将减慢网络通信。
|
||||
|
||||
+1
-1
@@ -110,7 +110,7 @@ Rancher 的大部分逻辑发生在 Event Handler 上。每当资源对象产生
|
||||
|
||||
Etcd 是 Kubernetes 和 Rancher 的后端数据库,在 Rancher 性能中扮演重要的角色。
|
||||
|
||||
[Etcd 性能](https://etcd.io/docs/v3.4/op-guide/performance/)的两个主要瓶颈是磁盘和网络速度。Etcd 应当在具有高速网络和高读写速度 (IOPS) SSD 硬盘的专用节点上运行。有关 etcd 性能的更多信息,请参阅 [etcd 性能缓慢(性能测试和优化)](https://www.suse.com/support/kb/doc/?id=000020100)和[为大型安装进行 etcd 调优](../../../how-to-guides/advanced-user-guides/tune-etcd-for-large-installs.md)。有关磁盘的信息可以在[安装要求](../../../getting-started/installation-and-upgrade/installation-requirements/installation-requirements.md#磁盘)中找到。
|
||||
[Etcd 性能](https://etcd.io/docs/v3.5/op-guide/performance/)的两个主要瓶颈是磁盘和网络速度。Etcd 应当在具有高速网络和高读写速度 (IOPS) SSD 硬盘的专用节点上运行。有关 etcd 性能的更多信息,请参阅 [etcd 性能缓慢(性能测试和优化)](https://www.suse.com/support/kb/doc/?id=000020100)和[为大型安装进行 etcd 调优](../../../how-to-guides/advanced-user-guides/tune-etcd-for-large-installs.md)。有关磁盘的信息可以在[安装要求](../../../getting-started/installation-and-upgrade/installation-requirements/installation-requirements.md#磁盘)中找到。
|
||||
|
||||
根据 etcd 的[复制机制](https://etcd.io/docs/v3.5/faq/#what-is-maximum-cluster-size),建议在三个节点上运行 etcd,运行在更多的节点上反而会降低速度。
|
||||
|
||||
|
||||
+1
-1
@@ -285,7 +285,7 @@ Input or select an **Expression**, the dropdown shows the original metrics from
|
||||
|
||||
- [**Node**](https://github.com/prometheus/node_exporter)
|
||||
- [**Container**](https://github.com/google/cadvisor)
|
||||
- [**ETCD**](https://etcd.io/docs/v3.4.0/op-guide/monitoring/)
|
||||
- [**ETCD**](https://etcd.io/docs/v3.3/op-guide/monitoring/)
|
||||
- [**Kubernetes Components**](https://github.com/kubernetes/metrics)
|
||||
- [**Kubernetes Resources**](https://github.com/kubernetes/kube-state-metrics)
|
||||
- [**Fluentd**](https://docs.fluentd.org/v1.0/articles/monitoring-prometheus) (supported by [Logging](../cluster-logging/cluster-logging.md))
|
||||
|
||||
+2
-2
@@ -4,7 +4,7 @@ title: Tuning etcd for Large Installations
|
||||
|
||||
When running larger Rancher installations with 15 or more clusters it is recommended to increase the default keyspace for etcd from the default 2GB. The maximum setting is 8GB and the host should have enough RAM to keep the entire dataset in memory. When increasing this value you should also increase the size of the host. The keyspace size can also be adjusted in smaller installations if you anticipate a high rate of change of pods during the garbage collection interval.
|
||||
|
||||
The etcd data set is automatically cleaned up on a five minute interval by Kubernetes. There are situations, e.g. deployment thrashing, where enough events could be written to etcd and deleted before garbage collection occurs and cleans things up causing the keyspace to fill up. If you see `mvcc: database space exceeded` errors, in the etcd logs or Kubernetes API server logs, you should consider increasing the keyspace size. This can be accomplished by setting the [quota-backend-bytes](https://etcd.io/docs/v3.4.0/op-guide/maintenance/#space-quota) setting on the etcd servers.
|
||||
The etcd data set is automatically cleaned up on a five minute interval by Kubernetes. There are situations, e.g. deployment thrashing, where enough events could be written to etcd and deleted before garbage collection occurs and cleans things up causing the keyspace to fill up. If you see `mvcc: database space exceeded` errors, in the etcd logs or Kubernetes API server logs, you should consider increasing the keyspace size. This can be accomplished by setting the [quota-backend-bytes](https://etcd.io/docs/v3.3/op-guide/maintenance/#space-quota) setting on the etcd servers.
|
||||
|
||||
### Example: This snippet of the RKE cluster.yml file increases the keyspace size to 5GB
|
||||
|
||||
@@ -19,7 +19,7 @@ services:
|
||||
|
||||
## Scaling etcd disk performance
|
||||
|
||||
You can follow the recommendations from [the etcd docs](https://etcd.io/docs/v3.4.0/tuning/#disk) on how to tune the disk priority on the host.
|
||||
You can follow the recommendations from [the etcd docs](https://etcd.io/docs/v3.3/tuning/#disk) on how to tune the disk priority on the host.
|
||||
|
||||
Additionally, to reduce IO contention on the disks for etcd, you can use a dedicated device for the data and wal directory. Based on etcd best practices, mirroring RAID configurations are unnecessary because etcd replicates data between the nodes in the cluster. You can use striping RAID configurations to increase available IOPS.
|
||||
|
||||
|
||||
+1
-1
@@ -22,4 +22,4 @@ For details on taking database snapshots and restoring your database from them,
|
||||
|
||||
- [Official MySQL documentation](https://dev.mysql.com/doc/refman/8.0/en/replication-snapshot-method.html)
|
||||
- [Official PostgreSQL documentation](https://www.postgresql.org/docs/8.3/backup-dump.html)
|
||||
- [Official etcd documentation](https://etcd.io/docs/v3.4/op-guide/recovery/)
|
||||
- [Official etcd documentation](https://etcd.io/docs/v3.3/op-guide/recovery/)
|
||||
+1
-1
@@ -18,4 +18,4 @@ For details on taking database snapshots and restoring your database from them,
|
||||
|
||||
- [Official MySQL documentation](https://dev.mysql.com/doc/refman/8.0/en/replication-snapshot-method.html)
|
||||
- [Official PostgreSQL documentation](https://www.postgresql.org/docs/8.3/backup-dump.html)
|
||||
- [Official etcd documentation](https://etcd.io/docs/v3.4/op-guide/recovery/)
|
||||
- [Official etcd documentation](https://etcd.io/docs/v3.3/op-guide/recovery/)
|
||||
+1
-1
@@ -49,5 +49,5 @@ number of nodes for each Kubernetes role, refer to the section on [recommended a
|
||||
|
||||
### Networking
|
||||
|
||||
* Minimize network latency. Rancher recommends minimizing latency between the etcd nodes. The default setting for `heartbeat-interval` is `500`, and the default setting for `election-timeout` is `5000`. These [settings for etcd tuning](https://coreos.com/etcd/docs/latest/tuning.html) allow etcd to run in most networks (except really high latency networks).
|
||||
* Minimize network latency. Rancher recommends minimizing latency between the etcd nodes. The default setting for `heartbeat-interval` is `500`, and the default setting for `election-timeout` is `5000`. These [settings for etcd tuning](https://etcd.io/docs/v3.3/tuning/) allow etcd to run in most networks (except really high latency networks).
|
||||
* Cluster nodes should be located within a single region. Most cloud providers provide multiple availability zones within a region, which can be used to create higher availability for your cluster. Using multiple availability zones is fine for nodes with any role. If you are using [Kubernetes Cloud Provider](../launch-kubernetes-with-rancher/set-up-cloud-providers/set-up-cloud-providers.md) resources, consult the documentation for any restrictions (i.e. zone storage restrictions).
|
||||
|
||||
+1
-1
@@ -57,7 +57,7 @@ The number of nodes that you can lose at once while maintaining cluster availabi
|
||||
|
||||
References:
|
||||
|
||||
* [Official etcd documentation on optimal etcd cluster size](https://etcd.io/docs/v3.4.0/faq/#what-is-failure-tolerance)
|
||||
* [Official etcd documentation on optimal etcd cluster size](https://etcd.io/docs/v3.3/faq/#what-is-failure-tolerance)
|
||||
* [Official Kubernetes documentation on operating etcd clusters for Kubernetes](https://kubernetes.io/docs/tasks/administer-cluster/configure-upgrade-etcd/)
|
||||
|
||||
### Number of Worker Nodes
|
||||
|
||||
+1
-1
@@ -95,7 +95,7 @@ Regarding CPU and memory, it is recommended that the different planes of Kuberne
|
||||
|
||||
For hardware recommendations for large Kubernetes clusters, refer to the official Kubernetes documentation on [building large clusters.](https://kubernetes.io/docs/setup/best-practices/cluster-large/)
|
||||
|
||||
For hardware recommendations for etcd clusters in production, refer to the official [etcd documentation.](https://etcd.io/docs/v3.4.0/op-guide/hardware/)
|
||||
For hardware recommendations for etcd clusters in production, refer to the official [etcd documentation.](https://etcd.io/docs/v3.3/op-guide/hardware/)
|
||||
|
||||
## Networking Requirements
|
||||
|
||||
|
||||
@@ -15,7 +15,7 @@ When the Rancher server is installed on a Kubernetes cluster, it should not be r
|
||||
It is strongly recommended to use hosted infrastructure such as Amazon's EC2 or Google's GCE instead. When you create a cluster using RKE on an infrastructure provider, you can configure the cluster to create etcd snapshots as a backup. You can then [use RKE](https://rancher.com/docs/rke/latest/en/etcd-snapshots/) or [Rancher](../../how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/backup-restore-and-disaster-recovery.md) to restore your cluster from one of these snapshots. In a hosted Kubernetes environment, this backup and restore functionality is not supported.
|
||||
|
||||
### Make sure nodes are configured correctly for Kubernetes
|
||||
It's important to follow K8s and etcd best practices when deploying your nodes, including disabling swap, double checking you have full network connectivity between all machines in the cluster, using unique hostnames, MAC addresses, and product_uuids for every node, checking that all correct ports are opened, and deploying with ssd backed etcd. More details can be found in the [kubernetes docs](https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/#before-you-begin) and [etcd's performance op guide](https://etcd.io/docs/v3.4/op-guide/performance/)
|
||||
It's important to follow K8s and etcd best practices when deploying your nodes, including disabling swap, double checking you have full network connectivity between all machines in the cluster, using unique hostnames, MAC addresses, and product_uuids for every node, checking that all correct ports are opened, and deploying with ssd backed etcd. More details can be found in the [kubernetes docs](https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/#before-you-begin) and [etcd's performance op guide](https://etcd.io/docs/v3.3/op-guide/performance/)
|
||||
|
||||
### When using RKE: Backup the Statefile
|
||||
RKE keeps record of the cluster state in a file called `cluster.rkestate`. This file is important for the recovery of a cluster and/or the continued maintenance of the cluster through RKE. Because this file contains certificate material, we strongly recommend encrypting this file before backing up. After each run of `rke up` you should backup the state file.
|
||||
|
||||
+2
-2
@@ -4,7 +4,7 @@ title: Tuning etcd for Large Installations
|
||||
|
||||
When running larger Rancher installations with 15 or more clusters it is recommended to increase the default keyspace for etcd from the default 2GB. The maximum setting is 8GB and the host should have enough RAM to keep the entire dataset in memory. When increasing this value you should also increase the size of the host. The keyspace size can also be adjusted in smaller installations if you anticipate a high rate of change of pods during the garbage collection interval.
|
||||
|
||||
The etcd data set is automatically cleaned up on a five minute interval by Kubernetes. There are situations, e.g. deployment thrashing, where enough events could be written to etcd and deleted before garbage collection occurs and cleans things up causing the keyspace to fill up. If you see `mvcc: database space exceeded` errors, in the etcd logs or Kubernetes API server logs, you should consider increasing the keyspace size. This can be accomplished by setting the [quota-backend-bytes](https://etcd.io/docs/v3.4.0/op-guide/maintenance/#space-quota) setting on the etcd servers.
|
||||
The etcd data set is automatically cleaned up on a five minute interval by Kubernetes. There are situations, e.g. deployment thrashing, where enough events could be written to etcd and deleted before garbage collection occurs and cleans things up causing the keyspace to fill up. If you see `mvcc: database space exceeded` errors, in the etcd logs or Kubernetes API server logs, you should consider increasing the keyspace size. This can be accomplished by setting the [quota-backend-bytes](https://etcd.io/docs/v3.3/op-guide/maintenance/#space-quota) setting on the etcd servers.
|
||||
|
||||
### Example: This snippet of the RKE cluster.yml file increases the keyspace size to 5GB
|
||||
|
||||
@@ -19,7 +19,7 @@ services:
|
||||
|
||||
## Scaling etcd disk performance
|
||||
|
||||
You can follow the recommendations from [the etcd docs](https://etcd.io/docs/v3.4.0/tuning/#disk) on how to tune the disk priority on the host.
|
||||
You can follow the recommendations from [the etcd docs](https://etcd.io/docs/v3.3/tuning/#disk) on how to tune the disk priority on the host.
|
||||
|
||||
Additionally, to reduce IO contention on the disks for etcd, you can use a dedicated device for the data and wal directory. Based on etcd best practices, mirroring RAID configurations are unnecessary because etcd replicates data between the nodes in the cluster. You can use striping RAID configurations to increase available IOPS.
|
||||
|
||||
|
||||
+1
-1
@@ -49,5 +49,5 @@ number of nodes for each Kubernetes role, refer to the section on [recommended a
|
||||
|
||||
### Networking
|
||||
|
||||
* Minimize network latency. Rancher recommends minimizing latency between the etcd nodes. The default setting for `heartbeat-interval` is `500`, and the default setting for `election-timeout` is `5000`. These [settings for etcd tuning](https://coreos.com/etcd/docs/latest/tuning.html) allow etcd to run in most networks (except really high latency networks).
|
||||
* Minimize network latency. Rancher recommends minimizing latency between the etcd nodes. The default setting for `heartbeat-interval` is `500`, and the default setting for `election-timeout` is `5000`. These [settings for etcd tuning](https://etcd.io/docs/v3.3/tuning/) allow etcd to run in most networks (except really high latency networks).
|
||||
* Cluster nodes should be located within a single region. Most cloud providers provide multiple availability zones within a region, which can be used to create higher availability for your cluster. Using multiple availability zones is fine for nodes with any role. If you are using [Kubernetes Cloud Provider](../launch-kubernetes-with-rancher/set-up-cloud-providers/set-up-cloud-providers.md) resources, consult the documentation for any restrictions (i.e. zone storage restrictions).
|
||||
|
||||
+1
-1
@@ -57,7 +57,7 @@ The number of nodes that you can lose at once while maintaining cluster availabi
|
||||
|
||||
References:
|
||||
|
||||
* [Official etcd documentation on optimal etcd cluster size](https://etcd.io/docs/v3.4.0/faq/#what-is-failure-tolerance)
|
||||
* [Official etcd documentation on optimal etcd cluster size](https://etcd.io/docs/v3.3/faq/#what-is-failure-tolerance)
|
||||
* [Official Kubernetes documentation on operating etcd clusters for Kubernetes](https://kubernetes.io/docs/tasks/administer-cluster/configure-upgrade-etcd/)
|
||||
|
||||
### Number of Worker Nodes
|
||||
|
||||
+1
-1
@@ -101,7 +101,7 @@ Regarding CPU and memory, it is recommended that the different planes of Kuberne
|
||||
|
||||
For hardware recommendations for large Kubernetes clusters, refer to the official Kubernetes documentation on [building large clusters.](https://kubernetes.io/docs/setup/best-practices/cluster-large/)
|
||||
|
||||
For hardware recommendations for etcd clusters in production, refer to the official [etcd documentation.](https://etcd.io/docs/v3.4.0/op-guide/hardware/)
|
||||
For hardware recommendations for etcd clusters in production, refer to the official [etcd documentation.](https://etcd.io/docs/v3.3/op-guide/hardware/)
|
||||
|
||||
## Networking Requirements
|
||||
|
||||
|
||||
+1
-1
@@ -18,7 +18,7 @@ When you set up your high-availability Rancher installation, consider the follow
|
||||
Don't run other workloads or microservices in the Kubernetes cluster that Rancher is installed on.
|
||||
|
||||
### Make sure nodes are configured correctly for Kubernetes
|
||||
It's important to follow K8s and etcd best practices when deploying your nodes, including disabling swap, double checking you have full network connectivity between all machines in the cluster, using unique hostnames, MAC addresses, and product_uuids for every node, checking that all correct ports are opened, and deploying with ssd backed etcd. More details can be found in the [kubernetes docs](https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/#before-you-begin) and [etcd's performance op guide](https://etcd.io/docs/v3.4/op-guide/performance/).
|
||||
It's important to follow K8s and etcd best practices when deploying your nodes, including disabling swap, double checking you have full network connectivity between all machines in the cluster, using unique hostnames, MAC addresses, and product_uuids for every node, checking that all correct ports are opened, and deploying with ssd backed etcd. More details can be found in the [kubernetes docs](https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/#before-you-begin) and [etcd's performance op guide](https://etcd.io/docs/v3.3/op-guide/performance/).
|
||||
|
||||
### When using RKE: Back up the Statefile
|
||||
RKE keeps record of the cluster state in a file called `cluster.rkestate`. This file is important for the recovery of a cluster and/or the continued maintenance of the cluster through RKE. Because this file contains certificate material, we strongly recommend encrypting this file before backing up. After each run of `rke up` you should backup the state file.
|
||||
|
||||
+4
-2
@@ -8,7 +8,9 @@ title: Tuning etcd for Large Installations
|
||||
|
||||
When Rancher is used to manage [a large infrastructure](../../getting-started/installation-and-upgrade/installation-requirements/installation-requirements.md) it is recommended to increase the default keyspace for etcd from the default 2 GB. The maximum setting is 8 GB and the host should have enough RAM to keep the entire dataset in memory. When increasing this value you should also increase the size of the host. The keyspace size can also be adjusted in smaller installations if you anticipate a high rate of change of pods during the garbage collection interval.
|
||||
|
||||
The etcd data set is automatically cleaned up on a five minute interval by Kubernetes. There are situations, e.g. deployment thrashing, where enough events could be written to etcd and deleted before garbage collection occurs and cleans things up causing the keyspace to fill up. If you see `mvcc: database space exceeded` errors, in the etcd logs or Kubernetes API server logs, you should consider increasing the keyspace size. This can be accomplished by setting the [quota-backend-bytes](https://etcd.io/docs/v3.4.0/op-guide/maintenance/#space-quota) setting on the etcd servers.
|
||||
The etcd data set is automatically cleaned up on a five minute interval by Kubernetes. There are situations, e.g. deployment thrashing, where enough events could be written to etcd and deleted before garbage collection occurs and cleans things up causing the keyspace to fill up. If you see `mvcc: database space exceeded` errors, in the etcd logs or Kubernetes API server logs, you should consider increasing the keyspace size. This can be accomplished by setting the `quota-backend-bytes` value on the etcd servers.
|
||||
|
||||
Details about `quota-backend-bytes` differs by etcd version. For more information, see the [official etcd v3.5 documentation](https://etcd.io/docs/v3.5/op-guide/maintenance/#space-quota) if you're running Kubernetes v1.22 and later. Otherwise, see the [official etcd v3.4 documentation](https://etcd.io/docs/v3.4/op-guide/maintenance/#space-quota).
|
||||
|
||||
### Example: This snippet of the RKE cluster.yml file increases the keyspace size to 5GB
|
||||
|
||||
@@ -23,7 +25,7 @@ services:
|
||||
|
||||
## Scaling etcd disk performance
|
||||
|
||||
You can follow the recommendations from [the etcd docs](https://etcd.io/docs/v3.4.0/tuning/#disk) on how to tune the disk priority on the host.
|
||||
You can follow the recommendations from [the etcd docs](https://etcd.io/docs/v3.5/tuning/#disk) on how to tune the disk priority on the host.
|
||||
|
||||
Additionally, to reduce IO contention on the disks for etcd, you can use a dedicated device for the data and wal directory. Based on etcd best practices, mirroring RAID configurations are unnecessary because etcd replicates data between the nodes in the cluster. You can use striping RAID configurations to increase available IOPS.
|
||||
|
||||
|
||||
+2
-1
@@ -49,5 +49,6 @@ number of nodes for each Kubernetes role, refer to the section on [recommended a
|
||||
|
||||
### Networking
|
||||
|
||||
* Minimize network latency. Rancher recommends minimizing latency between the etcd nodes. The default setting for `heartbeat-interval` is `500`, and the default setting for `election-timeout` is `5000`. These [settings for etcd tuning](https://coreos.com/etcd/docs/latest/tuning.html) allow etcd to run in most networks (except really high latency networks).
|
||||
* Minimize network latency. Rancher recommends minimizing latency between the etcd nodes. The default setting for `heartbeat-interval` is `500`, and the default setting for `election-timeout` is `5000`. These settings for etcd tuning allow etcd to run in most networks (except really high latency networks). Details about tuning etcd differ based on etcd version. For more information, see the [official etcd v3.5 documentation](https://etcd.io/docs/v3.5/tuning/) if you're running Kubernetes v1.22 and later. Otherwise, see the [official etcd v3.4 documentation](https://etcd.io/docs/v3.4/tuning/).
|
||||
|
||||
* Cluster nodes should be located within a single region. Most cloud providers provide multiple availability zones within a region, which can be used to create higher availability for your cluster. Using multiple availability zones is fine for nodes with any role. If you are using [Kubernetes Cloud Provider](../set-up-cloud-providers/set-up-cloud-providers.md) resources, consult the documentation for any restrictions (i.e. zone storage restrictions).
|
||||
|
||||
+3
-1
@@ -57,7 +57,9 @@ The number of nodes that you can lose at once while maintaining cluster availabi
|
||||
|
||||
References:
|
||||
|
||||
* [Official etcd documentation on optimal etcd cluster size](https://etcd.io/docs/v3.4.0/faq/#what-is-failure-tolerance)
|
||||
* Official etcd documentation on optimal etcd cluster size(https://etcd.io/docs/v3.5/faq/#what-is-failure-tolerance)
|
||||
* Docs on [etcd v3.5](https://etcd.io/docs/v3.5/faq/#what-is-failure-tolerance), for clusters running Kubernetes v1.22 and later.
|
||||
* Docs on [etcd v3.4](https://etcd.io/docs/v3.4/faq/#what-is-failure-tolerance) for clusters running Kubernetes v1.18 to Kubernetes v1.21.
|
||||
* [Official Kubernetes documentation on operating etcd clusters for Kubernetes](https://kubernetes.io/docs/tasks/administer-cluster/configure-upgrade-etcd/)
|
||||
|
||||
### Number of Worker Nodes
|
||||
|
||||
+4
-1
@@ -108,7 +108,10 @@ Regarding CPU and memory, it is recommended that the different planes of Kuberne
|
||||
|
||||
For hardware recommendations for large Kubernetes clusters, refer to the official Kubernetes documentation on [building large clusters.](https://kubernetes.io/docs/setup/best-practices/cluster-large/)
|
||||
|
||||
For hardware recommendations for etcd clusters in production, refer to the official [etcd documentation.](https://etcd.io/docs/v3.4.0/op-guide/hardware/)
|
||||
For hardware recommendations for etcd clusters in production, refer to the official [etcd documentation.](https://etcd.io/docs/v3.5/op-guide/hardware/)
|
||||
|
||||
|
||||
Hardware recommendations for etcd clusters in production differ by etcd version. For more information, see the [official etcd v3.5 documentation](https://etcd.io/docs/v3.5/op-guide/hardware/) if you're running Kubernetes v1.22 and later. Otherwise, see the [official etcd v3.4 documentation](https://etcd.io/docs/v3.4/op-guide/hardware/).
|
||||
|
||||
## Networking Requirements
|
||||
|
||||
|
||||
+4
-1
@@ -18,7 +18,10 @@ When you set up your high-availability Rancher installation, consider the follow
|
||||
Don't run other workloads or microservices in the Kubernetes cluster that Rancher is installed on.
|
||||
|
||||
### Make sure nodes are configured correctly for Kubernetes
|
||||
It's important to follow K8s and etcd best practices when deploying your nodes, including disabling swap, double checking you have full network connectivity between all machines in the cluster, using unique hostnames, MAC addresses, and product_uuids for every node, checking that all correct ports are opened, and deploying with ssd backed etcd. More details can be found in the [kubernetes docs](https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/#before-you-begin) and [etcd's performance op guide](https://etcd.io/docs/v3.4/op-guide/performance/).
|
||||
It's important to follow K8s and etcd best practices when deploying your nodes, including disabling swap, double checking you have full network connectivity between all machines in the cluster, using unique hostnames, MAC addresses, and product_uuids for every node, checking that all correct ports are opened, and deploying with ssd backed etcd. More details can be found in the [kubernetes docs](https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/#before-you-begin) and etcd's performance op guide. Since details about etcd differ by version, see the [official etcd v3.5 documentation](https://etcd.io/docs/v3.5/op-guide/performance/) if you're running Kubernetes v1.22 and later, or see the [official etcd v3.4 documentation](https://etcd.io/docs/v3.5/op-guide/performance/) if you are running an earlier Kubernetes version.
|
||||
|
||||
|
||||
(https://etcd.io/docs/v3.5/op-guide/performance/).
|
||||
|
||||
### When using RKE: Back up the Statefile
|
||||
RKE keeps record of the cluster state in a file called `cluster.rkestate`. This file is important for the recovery of a cluster and/or the continued maintenance of the cluster through RKE. Because this file contains certificate material, we strongly recommend encrypting this file before backing up. After each run of `rke up` you should backup the state file.
|
||||
|
||||
+1
-1
@@ -126,7 +126,7 @@ You should keep the local Kubernetes cluster up to date. This will ensure that y
|
||||
|
||||
Etcd is the backend database for Kubernetes and for Rancher. It plays a very important role in Rancher performance.
|
||||
|
||||
The two main bottlenecks to [etcd performance](https://etcd.io/docs/v3.4/op-guide/performance/) are disk and network speed. Etcd should run on dedicated nodes with a fast network setup and with SSDs that have high input/output operations per second (IOPS). For more information regarding etcd performance, see [Slow etcd performance (performance testing and optimization)](https://www.suse.com/support/kb/doc/?id=000020100) and [Tuning etcd for Large Installations](../../../how-to-guides/advanced-user-guides/tune-etcd-for-large-installs.md). Information on disks can also be found in the [Installation Requirements](../../../getting-started/installation-and-upgrade/installation-requirements/installation-requirements.md#disks).
|
||||
The two main bottlenecks to etcd performance are disk and network speed. Etcd should run on dedicated nodes with a fast network setup and with SSDs that have high input/output operations per second (IOPS). For more information regarding etcd performance, see the [official etcd v3.5 documentation](https://etcd.io/docs/v3.5/op-guide/performance/) if you are running Kubernetes v1.22 and later, or the [official etcd v3.4 documentation](https://etcd.io/docs/v3.4/op-guide/performance/) if you are running an earlier version of Kubernetes. Also see [Slow etcd performance (performance testing and optimization)](https://www.suse.com/support/kb/doc/?id=000020100) and [Tuning etcd for Large Installations](../../../how-to-guides/advanced-user-guides/tune-etcd-for-large-installs.md). Information on disks can also be found in the [Installation Requirements](../../../getting-started/installation-and-upgrade/installation-requirements/installation-requirements.md#disks).
|
||||
|
||||
It's best to run etcd on exactly three nodes, as adding more nodes will reduce operation speed. This may be counter-intuitive to common scaling approaches, but it's due to etcd's [replication mechanisms](https://etcd.io/docs/v3.5/faq/#what-is-maximum-cluster-size).
|
||||
|
||||
|
||||
+2
-2
@@ -8,7 +8,7 @@ title: Tuning etcd for Large Installations
|
||||
|
||||
When Rancher is used to manage [a large infrastructure](../../getting-started/installation-and-upgrade/installation-requirements/installation-requirements.md) it is recommended to increase the default keyspace for etcd from the default 2 GB. The maximum setting is 8 GB and the host should have enough RAM to keep the entire dataset in memory. When increasing this value you should also increase the size of the host. The keyspace size can also be adjusted in smaller installations if you anticipate a high rate of change of pods during the garbage collection interval.
|
||||
|
||||
The etcd data set is automatically cleaned up on a five minute interval by Kubernetes. There are situations, e.g. deployment thrashing, where enough events could be written to etcd and deleted before garbage collection occurs and cleans things up causing the keyspace to fill up. If you see `mvcc: database space exceeded` errors, in the etcd logs or Kubernetes API server logs, you should consider increasing the keyspace size. This can be accomplished by setting the [quota-backend-bytes](https://etcd.io/docs/v3.4.0/op-guide/maintenance/#space-quota) setting on the etcd servers.
|
||||
The etcd data set is automatically cleaned up on a five minute interval by Kubernetes. There are situations, e.g. deployment thrashing, where enough events could be written to etcd and deleted before garbage collection occurs and cleans things up causing the keyspace to fill up. If you see `mvcc: database space exceeded` errors, in the etcd logs or Kubernetes API server logs, you should consider increasing the keyspace size. This can be accomplished by setting the [quota-backend-bytes](https://etcd.io/docs/v3.5/op-guide/maintenance/#space-quota) setting on the etcd servers.
|
||||
|
||||
### Example: This snippet of the RKE cluster.yml file increases the keyspace size to 5GB
|
||||
|
||||
@@ -23,7 +23,7 @@ services:
|
||||
|
||||
## Scaling etcd disk performance
|
||||
|
||||
You can follow the recommendations from [the etcd docs](https://etcd.io/docs/v3.4.0/tuning/#disk) on how to tune the disk priority on the host.
|
||||
You can follow the recommendations from [the etcd docs](https://etcd.io/docs/v3.5/tuning/#disk) on how to tune the disk priority on the host.
|
||||
|
||||
Additionally, to reduce IO contention on the disks for etcd, you can use a dedicated device for the data and wal directory. Based on etcd best practices, mirroring RAID configurations are unnecessary because etcd replicates data between the nodes in the cluster. You can use striping RAID configurations to increase available IOPS.
|
||||
|
||||
|
||||
+1
-1
@@ -49,5 +49,5 @@ number of nodes for each Kubernetes role, refer to the section on [recommended a
|
||||
|
||||
### Networking
|
||||
|
||||
* Minimize network latency. Rancher recommends minimizing latency between the etcd nodes. The default setting for `heartbeat-interval` is `500`, and the default setting for `election-timeout` is `5000`. These [settings for etcd tuning](https://coreos.com/etcd/docs/latest/tuning.html) allow etcd to run in most networks (except really high latency networks).
|
||||
* Minimize network latency. Rancher recommends minimizing latency between the etcd nodes. The default setting for `heartbeat-interval` is `500`, and the default setting for `election-timeout` is `5000`. These [settings for etcd tuning](https://etcd.io/docs/v3.5/tuning/) allow etcd to run in most networks (except really high latency networks).
|
||||
* Cluster nodes should be located within a single region. Most cloud providers provide multiple availability zones within a region, which can be used to create higher availability for your cluster. Using multiple availability zones is fine for nodes with any role. If you are using [Kubernetes Cloud Provider](../set-up-cloud-providers/set-up-cloud-providers.md) resources, consult the documentation for any restrictions (i.e. zone storage restrictions).
|
||||
|
||||
+1
-1
@@ -57,7 +57,7 @@ The number of nodes that you can lose at once while maintaining cluster availabi
|
||||
|
||||
References:
|
||||
|
||||
* [Official etcd documentation on optimal etcd cluster size](https://etcd.io/docs/v3.4.0/faq/#what-is-failure-tolerance)
|
||||
* [Official etcd documentation on optimal etcd cluster size](https://etcd.io/docs/v3.5/faq/#what-is-failure-tolerance)
|
||||
* [Official Kubernetes documentation on operating etcd clusters for Kubernetes](https://kubernetes.io/docs/tasks/administer-cluster/configure-upgrade-etcd/)
|
||||
|
||||
### Number of Worker Nodes
|
||||
|
||||
+1
-1
@@ -108,7 +108,7 @@ Regarding CPU and memory, it is recommended that the different planes of Kuberne
|
||||
|
||||
For hardware recommendations for large Kubernetes clusters, refer to the official Kubernetes documentation on [building large clusters.](https://kubernetes.io/docs/setup/best-practices/cluster-large/)
|
||||
|
||||
For hardware recommendations for etcd clusters in production, refer to the official [etcd documentation.](https://etcd.io/docs/v3.4.0/op-guide/hardware/)
|
||||
For hardware recommendations for etcd clusters in production, refer to the official [etcd documentation.](https://etcd.io/docs/v3.5/op-guide/hardware/)
|
||||
|
||||
## Networking Requirements
|
||||
|
||||
|
||||
+1
-1
@@ -18,7 +18,7 @@ When you set up your high-availability Rancher installation, consider the follow
|
||||
Don't run other workloads or microservices in the Kubernetes cluster that Rancher is installed on.
|
||||
|
||||
### Make sure nodes are configured correctly for Kubernetes
|
||||
It's important to follow K8s and etcd best practices when deploying your nodes, including disabling swap, double checking you have full network connectivity between all machines in the cluster, using unique hostnames, MAC addresses, and product_uuids for every node, checking that all correct ports are opened, and deploying with ssd backed etcd. More details can be found in the [kubernetes docs](https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/#before-you-begin) and [etcd's performance op guide](https://etcd.io/docs/v3.4/op-guide/performance/).
|
||||
It's important to follow K8s and etcd best practices when deploying your nodes, including disabling swap, double checking you have full network connectivity between all machines in the cluster, using unique hostnames, MAC addresses, and product_uuids for every node, checking that all correct ports are opened, and deploying with ssd backed etcd. More details can be found in the [kubernetes docs](https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/#before-you-begin) and [etcd's performance op guide](https://etcd.io/docs/v3.5/op-guide/performance/).
|
||||
|
||||
### When using RKE: Back up the Statefile
|
||||
RKE keeps record of the cluster state in a file called `cluster.rkestate`. This file is important for the recovery of a cluster and/or the continued maintenance of the cluster through RKE. Because this file contains certificate material, we strongly recommend encrypting this file before backing up. After each run of `rke up` you should backup the state file.
|
||||
|
||||
+1
-1
@@ -126,7 +126,7 @@ You should keep the local Kubernetes cluster up to date. This will ensure that y
|
||||
|
||||
Etcd is the backend database for Kubernetes and for Rancher. It plays a very important role in Rancher performance.
|
||||
|
||||
The two main bottlenecks to [etcd performance](https://etcd.io/docs/v3.4/op-guide/performance/) are disk and network speed. Etcd should run on dedicated nodes with a fast network setup and with SSDs that have high input/output operations per second (IOPS). For more information regarding etcd performance, see [Slow etcd performance (performance testing and optimization)](https://www.suse.com/support/kb/doc/?id=000020100) and [Tuning etcd for Large Installations](../../../how-to-guides/advanced-user-guides/tune-etcd-for-large-installs.md). Information on disks can also be found in the [Installation Requirements](../../../getting-started/installation-and-upgrade/installation-requirements/installation-requirements.md#disks).
|
||||
The two main bottlenecks to [etcd performance](https://etcd.io/docs/v3.5/op-guide/performance/) are disk and network speed. Etcd should run on dedicated nodes with a fast network setup and with SSDs that have high input/output operations per second (IOPS). For more information regarding etcd performance, see [Slow etcd performance (performance testing and optimization)](https://www.suse.com/support/kb/doc/?id=000020100) and [Tuning etcd for Large Installations](../../../how-to-guides/advanced-user-guides/tune-etcd-for-large-installs.md). Information on disks can also be found in the [Installation Requirements](../../../getting-started/installation-and-upgrade/installation-requirements/installation-requirements.md#disks).
|
||||
|
||||
It's best to run etcd on exactly three nodes, as adding more nodes will reduce operation speed. This may be counter-intuitive to common scaling approaches, but it's due to etcd's [replication mechanisms](https://etcd.io/docs/v3.5/faq/#what-is-maximum-cluster-size).
|
||||
|
||||
|
||||
+2
-2
@@ -8,7 +8,7 @@ title: Tuning etcd for Large Installations
|
||||
|
||||
When Rancher is used to manage [a large infrastructure](../../getting-started/installation-and-upgrade/installation-requirements/installation-requirements.md) it is recommended to increase the default keyspace for etcd from the default 2 GB. The maximum setting is 8 GB and the host should have enough RAM to keep the entire dataset in memory. When increasing this value you should also increase the size of the host. The keyspace size can also be adjusted in smaller installations if you anticipate a high rate of change of pods during the garbage collection interval.
|
||||
|
||||
The etcd data set is automatically cleaned up on a five minute interval by Kubernetes. There are situations, e.g. deployment thrashing, where enough events could be written to etcd and deleted before garbage collection occurs and cleans things up causing the keyspace to fill up. If you see `mvcc: database space exceeded` errors, in the etcd logs or Kubernetes API server logs, you should consider increasing the keyspace size. This can be accomplished by setting the [quota-backend-bytes](https://etcd.io/docs/v3.4.0/op-guide/maintenance/#space-quota) setting on the etcd servers.
|
||||
The etcd data set is automatically cleaned up on a five minute interval by Kubernetes. There are situations, e.g. deployment thrashing, where enough events could be written to etcd and deleted before garbage collection occurs and cleans things up causing the keyspace to fill up. If you see `mvcc: database space exceeded` errors, in the etcd logs or Kubernetes API server logs, you should consider increasing the keyspace size. This can be accomplished by setting the [quota-backend-bytes](https://etcd.io/docs/v3.5/op-guide/maintenance/#space-quota) setting on the etcd servers.
|
||||
|
||||
### Example: This snippet of the RKE cluster.yml file increases the keyspace size to 5GB
|
||||
|
||||
@@ -23,7 +23,7 @@ services:
|
||||
|
||||
## Scaling etcd disk performance
|
||||
|
||||
You can follow the recommendations from [the etcd docs](https://etcd.io/docs/v3.4.0/tuning/#disk) on how to tune the disk priority on the host.
|
||||
You can follow the recommendations from [the etcd docs](https://etcd.io/docs/v3.5/tuning/#disk) on how to tune the disk priority on the host.
|
||||
|
||||
Additionally, to reduce IO contention on the disks for etcd, you can use a dedicated device for the data and wal directory. Based on etcd best practices, mirroring RAID configurations are unnecessary because etcd replicates data between the nodes in the cluster. You can use striping RAID configurations to increase available IOPS.
|
||||
|
||||
|
||||
+1
-1
@@ -49,5 +49,5 @@ number of nodes for each Kubernetes role, refer to the section on [recommended a
|
||||
|
||||
### Networking
|
||||
|
||||
* Minimize network latency. Rancher recommends minimizing latency between the etcd nodes. The default setting for `heartbeat-interval` is `500`, and the default setting for `election-timeout` is `5000`. These [settings for etcd tuning](https://coreos.com/etcd/docs/latest/tuning.html) allow etcd to run in most networks (except really high latency networks).
|
||||
* Minimize network latency. Rancher recommends minimizing latency between the etcd nodes. The default setting for `heartbeat-interval` is `500`, and the default setting for `election-timeout` is `5000`. These [settings for etcd tuning](https://etcd.io/docs/v3.5/tuning/) allow etcd to run in most networks (except really high latency networks).
|
||||
* Cluster nodes should be located within a single region. Most cloud providers provide multiple availability zones within a region, which can be used to create higher availability for your cluster. Using multiple availability zones is fine for nodes with any role. If you are using [Kubernetes Cloud Provider](../set-up-cloud-providers/set-up-cloud-providers.md) resources, consult the documentation for any restrictions (i.e. zone storage restrictions).
|
||||
|
||||
+1
-1
@@ -57,7 +57,7 @@ The number of nodes that you can lose at once while maintaining cluster availabi
|
||||
|
||||
References:
|
||||
|
||||
* [Official etcd documentation on optimal etcd cluster size](https://etcd.io/docs/v3.4.0/faq/#what-is-failure-tolerance)
|
||||
* [Official etcd documentation on optimal etcd cluster size](https://etcd.io/docs/v3.5/faq/#what-is-failure-tolerance)
|
||||
* [Official Kubernetes documentation on operating etcd clusters for Kubernetes](https://kubernetes.io/docs/tasks/administer-cluster/configure-upgrade-etcd/)
|
||||
|
||||
### Number of Worker Nodes
|
||||
|
||||
+1
-1
@@ -108,7 +108,7 @@ Regarding CPU and memory, it is recommended that the different planes of Kuberne
|
||||
|
||||
For hardware recommendations for large Kubernetes clusters, refer to the official Kubernetes documentation on [building large clusters.](https://kubernetes.io/docs/setup/best-practices/cluster-large/)
|
||||
|
||||
For hardware recommendations for etcd clusters in production, refer to the official [etcd documentation.](https://etcd.io/docs/v3.4.0/op-guide/hardware/)
|
||||
For hardware recommendations for etcd clusters in production, refer to the official [etcd documentation.](https://etcd.io/docs/v3.5/op-guide/hardware/)
|
||||
|
||||
## Networking Requirements
|
||||
|
||||
|
||||
+1
-1
@@ -18,7 +18,7 @@ When you set up your high-availability Rancher installation, consider the follow
|
||||
Don't run other workloads or microservices in the Kubernetes cluster that Rancher is installed on.
|
||||
|
||||
### Make sure nodes are configured correctly for Kubernetes
|
||||
It's important to follow K8s and etcd best practices when deploying your nodes, including disabling swap, double checking you have full network connectivity between all machines in the cluster, using unique hostnames, MAC addresses, and product_uuids for every node, checking that all correct ports are opened, and deploying with ssd backed etcd. More details can be found in the [kubernetes docs](https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/#before-you-begin) and [etcd's performance op guide](https://etcd.io/docs/v3.4/op-guide/performance/).
|
||||
It's important to follow K8s and etcd best practices when deploying your nodes, including disabling swap, double checking you have full network connectivity between all machines in the cluster, using unique hostnames, MAC addresses, and product_uuids for every node, checking that all correct ports are opened, and deploying with ssd backed etcd. More details can be found in the [kubernetes docs](https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/#before-you-begin) and [etcd's performance op guide](https://etcd.io/docs/v3.5/op-guide/performance/).
|
||||
|
||||
### When using RKE: Back up the Statefile
|
||||
RKE keeps record of the cluster state in a file called `cluster.rkestate`. This file is important for the recovery of a cluster and/or the continued maintenance of the cluster through RKE. Because this file contains certificate material, we strongly recommend encrypting this file before backing up. After each run of `rke up` you should backup the state file.
|
||||
|
||||
+1
-1
@@ -126,7 +126,7 @@ You should keep the local Kubernetes cluster up to date. This will ensure that y
|
||||
|
||||
Etcd is the backend database for Kubernetes and for Rancher. It plays a very important role in Rancher performance.
|
||||
|
||||
The two main bottlenecks to [etcd performance](https://etcd.io/docs/v3.4/op-guide/performance/) are disk and network speed. Etcd should run on dedicated nodes with a fast network setup and with SSDs that have high input/output operations per second (IOPS). For more information regarding etcd performance, see [Slow etcd performance (performance testing and optimization)](https://www.suse.com/support/kb/doc/?id=000020100) and [Tuning etcd for Large Installations](../../../how-to-guides/advanced-user-guides/tune-etcd-for-large-installs.md). Information on disks can also be found in the [Installation Requirements](../../../getting-started/installation-and-upgrade/installation-requirements/installation-requirements.md#disks).
|
||||
The two main bottlenecks to [etcd performance](https://etcd.io/docs/v3.5/op-guide/performance/) are disk and network speed. Etcd should run on dedicated nodes with a fast network setup and with SSDs that have high input/output operations per second (IOPS). For more information regarding etcd performance, see [Slow etcd performance (performance testing and optimization)](https://www.suse.com/support/kb/doc/?id=000020100) and [Tuning etcd for Large Installations](../../../how-to-guides/advanced-user-guides/tune-etcd-for-large-installs.md). Information on disks can also be found in the [Installation Requirements](../../../getting-started/installation-and-upgrade/installation-requirements/installation-requirements.md#disks).
|
||||
|
||||
It's best to run etcd on exactly three nodes, as adding more nodes will reduce operation speed. This may be counter-intuitive to common scaling approaches, but it's due to etcd's [replication mechanisms](https://etcd.io/docs/v3.5/faq/#what-is-maximum-cluster-size).
|
||||
|
||||
|
||||
+2
-2
@@ -8,7 +8,7 @@ title: Tuning etcd for Large Installations
|
||||
|
||||
When Rancher is used to manage [a large infrastructure](../../getting-started/installation-and-upgrade/installation-requirements/installation-requirements.md) it is recommended to increase the default keyspace for etcd from the default 2 GB. The maximum setting is 8 GB and the host should have enough RAM to keep the entire dataset in memory. When increasing this value you should also increase the size of the host. The keyspace size can also be adjusted in smaller installations if you anticipate a high rate of change of pods during the garbage collection interval.
|
||||
|
||||
The etcd data set is automatically cleaned up on a five minute interval by Kubernetes. There are situations, e.g. deployment thrashing, where enough events could be written to etcd and deleted before garbage collection occurs and cleans things up causing the keyspace to fill up. If you see `mvcc: database space exceeded` errors, in the etcd logs or Kubernetes API server logs, you should consider increasing the keyspace size. This can be accomplished by setting the [quota-backend-bytes](https://etcd.io/docs/v3.4.0/op-guide/maintenance/#space-quota) setting on the etcd servers.
|
||||
The etcd data set is automatically cleaned up on a five minute interval by Kubernetes. There are situations, e.g. deployment thrashing, where enough events could be written to etcd and deleted before garbage collection occurs and cleans things up causing the keyspace to fill up. If you see `mvcc: database space exceeded` errors, in the etcd logs or Kubernetes API server logs, you should consider increasing the keyspace size. This can be accomplished by setting the [quota-backend-bytes](https://etcd.io/docs/v3.5/op-guide/maintenance/#space-quota) setting on the etcd servers.
|
||||
|
||||
### Example: This snippet of the RKE cluster.yml file increases the keyspace size to 5GB
|
||||
|
||||
@@ -23,7 +23,7 @@ services:
|
||||
|
||||
## Scaling etcd disk performance
|
||||
|
||||
You can follow the recommendations from [the etcd docs](https://etcd.io/docs/v3.4.0/tuning/#disk) on how to tune the disk priority on the host.
|
||||
You can follow the recommendations from [the etcd docs](https://etcd.io/docs/v3.5/tuning/#disk) on how to tune the disk priority on the host.
|
||||
|
||||
Additionally, to reduce IO contention on the disks for etcd, you can use a dedicated device for the data and wal directory. Based on etcd best practices, mirroring RAID configurations are unnecessary because etcd replicates data between the nodes in the cluster. You can use striping RAID configurations to increase available IOPS.
|
||||
|
||||
|
||||
+1
-1
@@ -49,5 +49,5 @@ number of nodes for each Kubernetes role, refer to the section on [recommended a
|
||||
|
||||
### Networking
|
||||
|
||||
* Minimize network latency. Rancher recommends minimizing latency between the etcd nodes. The default setting for `heartbeat-interval` is `500`, and the default setting for `election-timeout` is `5000`. These [settings for etcd tuning](https://coreos.com/etcd/docs/latest/tuning.html) allow etcd to run in most networks (except really high latency networks).
|
||||
* Minimize network latency. Rancher recommends minimizing latency between the etcd nodes. The default setting for `heartbeat-interval` is `500`, and the default setting for `election-timeout` is `5000`. These [settings for etcd tuning](https://etcd.io/docs/v3.5/tuning/) allow etcd to run in most networks (except really high latency networks).
|
||||
* Cluster nodes should be located within a single region. Most cloud providers provide multiple availability zones within a region, which can be used to create higher availability for your cluster. Using multiple availability zones is fine for nodes with any role. If you are using [Kubernetes Cloud Provider](../set-up-cloud-providers/set-up-cloud-providers.md) resources, consult the documentation for any restrictions (i.e. zone storage restrictions).
|
||||
|
||||
+1
-1
@@ -57,7 +57,7 @@ The number of nodes that you can lose at once while maintaining cluster availabi
|
||||
|
||||
References:
|
||||
|
||||
* [Official etcd documentation on optimal etcd cluster size](https://etcd.io/docs/v3.4.0/faq/#what-is-failure-tolerance)
|
||||
* [Official etcd documentation on optimal etcd cluster size](https://etcd.io/docs/v3.5/faq/#what-is-failure-tolerance)
|
||||
* [Official Kubernetes documentation on operating etcd clusters for Kubernetes](https://kubernetes.io/docs/tasks/administer-cluster/configure-upgrade-etcd/)
|
||||
|
||||
### Number of Worker Nodes
|
||||
|
||||
+1
-1
@@ -108,7 +108,7 @@ Regarding CPU and memory, it is recommended that the different planes of Kuberne
|
||||
|
||||
For hardware recommendations for large Kubernetes clusters, refer to the official Kubernetes documentation on [building large clusters.](https://kubernetes.io/docs/setup/best-practices/cluster-large/)
|
||||
|
||||
For hardware recommendations for etcd clusters in production, refer to the official [etcd documentation.](https://etcd.io/docs/v3.4.0/op-guide/hardware/)
|
||||
For hardware recommendations for etcd clusters in production, refer to the official [etcd documentation.](https://etcd.io/docs/v3.5/op-guide/hardware/)
|
||||
|
||||
## Networking Requirements
|
||||
|
||||
|
||||
+1
-1
@@ -18,7 +18,7 @@ When you set up your high-availability Rancher installation, consider the follow
|
||||
Don't run other workloads or microservices in the Kubernetes cluster that Rancher is installed on.
|
||||
|
||||
### Make sure nodes are configured correctly for Kubernetes
|
||||
It's important to follow K8s and etcd best practices when deploying your nodes, including disabling swap, double checking you have full network connectivity between all machines in the cluster, using unique hostnames, MAC addresses, and product_uuids for every node, checking that all correct ports are opened, and deploying with ssd backed etcd. More details can be found in the [kubernetes docs](https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/#before-you-begin) and [etcd's performance op guide](https://etcd.io/docs/v3.4/op-guide/performance/).
|
||||
It's important to follow K8s and etcd best practices when deploying your nodes, including disabling swap, double checking you have full network connectivity between all machines in the cluster, using unique hostnames, MAC addresses, and product_uuids for every node, checking that all correct ports are opened, and deploying with ssd backed etcd. More details can be found in the [kubernetes docs](https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/#before-you-begin) and [etcd's performance op guide](https://etcd.io/docs/v3.5/op-guide/performance/).
|
||||
|
||||
### When using RKE: Back up the Statefile
|
||||
RKE keeps record of the cluster state in a file called `cluster.rkestate`. This file is important for the recovery of a cluster and/or the continued maintenance of the cluster through RKE. Because this file contains certificate material, we strongly recommend encrypting this file before backing up. After each run of `rke up` you should backup the state file.
|
||||
|
||||
+1
-1
@@ -126,7 +126,7 @@ You should keep the local Kubernetes cluster up to date. This will ensure that y
|
||||
|
||||
Etcd is the backend database for Kubernetes and for Rancher. It plays a very important role in Rancher performance.
|
||||
|
||||
The two main bottlenecks to [etcd performance](https://etcd.io/docs/v3.4/op-guide/performance/) are disk and network speed. Etcd should run on dedicated nodes with a fast network setup and with SSDs that have high input/output operations per second (IOPS). For more information regarding etcd performance, see [Slow etcd performance (performance testing and optimization)](https://www.suse.com/support/kb/doc/?id=000020100) and [Tuning etcd for Large Installations](../../../how-to-guides/advanced-user-guides/tune-etcd-for-large-installs.md). Information on disks can also be found in the [Installation Requirements](../../../getting-started/installation-and-upgrade/installation-requirements/installation-requirements.md#disks).
|
||||
The two main bottlenecks to [etcd performance](https://etcd.io/docs/v3.5/op-guide/performance/) are disk and network speed. Etcd should run on dedicated nodes with a fast network setup and with SSDs that have high input/output operations per second (IOPS). For more information regarding etcd performance, see [Slow etcd performance (performance testing and optimization)](https://www.suse.com/support/kb/doc/?id=000020100) and [Tuning etcd for Large Installations](../../../how-to-guides/advanced-user-guides/tune-etcd-for-large-installs.md). Information on disks can also be found in the [Installation Requirements](../../../getting-started/installation-and-upgrade/installation-requirements/installation-requirements.md#disks).
|
||||
|
||||
It's best to run etcd on exactly three nodes, as adding more nodes will reduce operation speed. This may be counter-intuitive to common scaling approaches, but it's due to etcd's [replication mechanisms](https://etcd.io/docs/v3.5/faq/#what-is-maximum-cluster-size).
|
||||
|
||||
|
||||
Reference in New Issue
Block a user