From 586c109b17fe40c39447353667adf728d9f421b5 Mon Sep 17 00:00:00 2001 From: Silvio Moioli Date: Fri, 6 Oct 2023 09:53:58 +0200 Subject: [PATCH] Apply suggestions from code review Co-authored-by: Billy Tat --- .../tune-etcd-for-large-installs.md | 2 +- .../installation-requirements.md | 14 +++++++------- ...ning-and-best-practices-for-rancher-at-scale.md | 10 ++++++++-- 3 files changed, 16 insertions(+), 10 deletions(-) diff --git a/docs/how-to-guides/advanced-user-guides/tune-etcd-for-large-installs.md b/docs/how-to-guides/advanced-user-guides/tune-etcd-for-large-installs.md index fcf9566142e..7d803ff697e 100644 --- a/docs/how-to-guides/advanced-user-guides/tune-etcd-for-large-installs.md +++ b/docs/how-to-guides/advanced-user-guides/tune-etcd-for-large-installs.md @@ -6,7 +6,7 @@ title: Tuning etcd for Large Installations -When Rancher is used to manage [a large infrastructure](../../pages-for-subheaders/installation-requirements.md) it is recommended to increase the default keyspace for etcd from the default 2GB. The maximum setting is 8GB and the host should have enough RAM to keep the entire dataset in memory. When increasing this value you should also increase the size of the host. The keyspace size can also be adjusted in smaller installations if you anticipate a high rate of change of pods during the garbage collection interval. +When Rancher is used to manage [a large infrastructure](../../pages-for-subheaders/installation-requirements.md) it is recommended to increase the default keyspace for etcd from the default 2 GB. The maximum setting is 8 GB and the host should have enough RAM to keep the entire dataset in memory. When increasing this value you should also increase the size of the host. The keyspace size can also be adjusted in smaller installations if you anticipate a high rate of change of pods during the garbage collection interval. The etcd data set is automatically cleaned up on a five minute interval by Kubernetes. There are situations, e.g. deployment thrashing, where enough events could be written to etcd and deleted before garbage collection occurs and cleans things up causing the keyspace to fill up. If you see `mvcc: database space exceeded` errors, in the etcd logs or Kubernetes API server logs, you should consider increasing the keyspace size. This can be accomplished by setting the [quota-backend-bytes](https://etcd.io/docs/v3.4.0/op-guide/maintenance/#space-quota) setting on the etcd servers. diff --git a/docs/pages-for-subheaders/installation-requirements.md b/docs/pages-for-subheaders/installation-requirements.md index c45b748c1b2..3f6b50d66e7 100644 --- a/docs/pages-for-subheaders/installation-requirements.md +++ b/docs/pages-for-subheaders/installation-requirements.md @@ -69,12 +69,12 @@ The following sections describe the CPU, memory, and I/O requirements for nodes Rancher's hardware footprint depends on a number of factors, including: - - Size of the managed infrastructure (eg. node count, cluster count). - - Complexity of the desired access control rules (eg. `RoleBinding` object count). - - Number of workloads (eg. Kubernetes deployments, Fleet deployments). - - Usage patterns (eg. subset of functionality actively used, frequency of use, number of concurrent users). + - Size of the managed infrastructure (e.g., node count, cluster count). + - Complexity of the desired access control rules (e.g., `RoleBinding` object count). + - Number of workloads (e.g., Kubernetes deployments, Fleet deployments). + - Usage patterns (e.g., subset of functionality actively used, frequency of use, number of concurrent users). -Since there are a high number of influencing factors that may vary over time, the requirements listed here should be understood as reasonable starting points that work well for most use cases. Nevertheless, your use case may have higher or lower requirements. For enquiries about a specific scenario please [contact Rancher](https://rancher.com/contact/) for further guidance. +Since there are a high number of influencing factors that may vary over time, the requirements listed here should be understood as reasonable starting points that work well for most use cases. Nevertheless, your use case may have different requirements. For inquiries about a specific scenario please [contact Rancher](https://rancher.com/contact/) for further guidance. In particular, requirements on this page are subject to typical use assumptions, which include: - Under 60 thousand total Kubernetes resources, per type. @@ -102,7 +102,7 @@ The following table lists minimum CPU and memory requirements for each node in t Please note that a highly available setup with at least three nodes is required for production. -| Managed Infrastructure Size | Maximum number of Clusters | Maximum number of Nodes | vCPUs | RAM | +| Managed Infrastructure Size | Maximum Number of Clusters | Maximum Number of Nodes | vCPUs | RAM | |-----------------------------|----------------------------|-------------------------|-------|-------| | Small | 150 | 1500 | 4 | 16 GB | | Medium | 300 | 3000 | 8 | 32 GB | @@ -139,7 +139,7 @@ The following table lists minimum CPU and memory requirements for each node in t Please note that a highly available setup with at least three nodes is required for production. -These requirements apply to hosted Kubernetes clusters such as Amazon Elastic Kubernetes Service (EKS), Azure Kubernetes Service (AKS), or Google Kubernetes Engine (GKE). They don't apply to Rancher SaaS solutions such as [Rancher Prime Hosted](https://www.rancher.com/products/rancher). +These requirements apply to hosted Kubernetes clusters such as Amazon Elastic Kubernetes Service (EKS), Azure Kubernetes Service (AKS), or Google Kubernetes Engine (GKE). They don't apply to Rancher SaaS solutions such as [Rancher Prime Hosted](https://www.rancher.com/products/rancher). | Managed Infrastructure Size | Maximum number of Clusters | Maximum number of Nodes | vCPUs | RAM | |-----------------------------|----------------------------|-------------------------|-------|-------| diff --git a/docs/reference-guides/best-practices/rancher-server/tuning-and-best-practices-for-rancher-at-scale.md b/docs/reference-guides/best-practices/rancher-server/tuning-and-best-practices-for-rancher-at-scale.md index cf6aa839ce6..e8fdf28d045 100644 --- a/docs/reference-guides/best-practices/rancher-server/tuning-and-best-practices-for-rancher-at-scale.md +++ b/docs/reference-guides/best-practices/rancher-server/tuning-and-best-practices-for-rancher-at-scale.md @@ -15,7 +15,7 @@ This guide describes the best practices and tuning approaches to scale Rancher s * Always scale up gradually, and monitor and observe any changes in behavior while doing do. It is usually easier to resolve performance problems as soon as they surface, before other problems obscure the root cause. -* Reduce network latency between the upstream Rancher cluster and downstream clusters to the extent possible. Note that latency is, among other factors, a function of geographic distance - if a you require clusters or nodes spread across the world, consider multiple Rancher installations. +* Reduce network latency between the upstream Rancher cluster and downstream clusters to the extent possible. Note that latency is, among other factors, a function of geographic distance - if you require clusters or nodes spread across the world, consider multiple Rancher installations. ## Minimizing Load on the Upstream Cluster @@ -42,7 +42,12 @@ You can reduce the number of `RoleBindings` in the upstream cluster in the follo ### RoleBinding Count Estimation Predicting how many `RoleBinding` objects a given configuration will create is complicated. However, the following considerations can offer a rough estimate: -* For a minimum estimate, use the formula `32C + U + 2UaC + 8P + 5Pa`, where `C` is the total number of clusters, `U` is the total number of users, `Ua` is the average number of users with a membership on a cluster, `P` is the total number of projects, and `Pa` is the average number of users with a membership on a project. +* For a minimum estimate, use the formula `32C + U + 2UaC + 8P + 5Pa`. + * `C` is the total number of clusters. + * `U` is the total number of users. + * `Ua` is the average number of users with a membership on a cluster. + * `P` is the total number of projects. + * `Pa` is the average number of users with a membership on a project. * The Restricted Admin role follows a different formula, as every user with this role results in at least `7C + 2P + 2` additional `RoleBinding` objects. * The number of `RoleBindings` increases linearly with the number of clusters, projects, and users. @@ -57,6 +62,7 @@ You should remove any remaining legacy apps that appear in the Cluster Manager U An [Authorized Cluster Endpoint](../../../reference-guides/rancher-manager-architecture/communicating-with-downstream-user-clusters#4-authorized-cluster-endpoint) (ACE) provides access to the Kubernetes API of Rancher-provisioned RKE, RKE2, and K3s clusters. When enabled, the ACE adds a context to kubeconfig files generated for the cluster. The context uses a direct endpoint to the cluster, thereby bypassing Rancher. This reduces load on Rancher for cases where unmediated API access is acceptable or preferable. See [Authorized Cluster Endpoint](../../../reference-guides/rancher-manager-architecture/communicating-with-downstream-user-clusters#4-authorized-cluster-endpoint) for more information and configuration instructions. ### Reducing Event Handler Executions + The bulk of Rancher's logic occurs on event handlers. These event handlers run on an object whenever the object is updated, and when Rancher is started. Additionally, they run every 15 hours when Rancher syncs caches. In scaled setups these scheduled runs come with huge performance costs because every handler is being run on every applicable object. However, the scheduled handler execution can be disabled with the `CATTLE_SYNC_ONLY_CHANGED_OBJECTS` environment variable. If resource allocation spikes are seen every 15 hours, this setting can help. The value for `CATTLE_SYNC_ONLY_CHANGED_OBJECTS` can be a comma separated list of the following options. The values refer to types of handlers and controllers (the structures that contain and run handlers). Adding the controller types to the variable disables that set of controllers from running their handlers as part of cache resyncing.