mirror of
https://github.com/rancher/rancher-docs.git
synced 2026-09-26 13:08:11 +00:00
Add version-2.7 docs
This commit is contained in:
Billy Tat
+84
@@ -0,0 +1,84 @@
|
||||
---
|
||||
title: Installing Rancher in a vSphere Environment
|
||||
---
|
||||
|
||||
This guide outlines a reference architecture for installing Rancher on an RKE Kubernetes cluster in a vSphere environment, in addition to standard vSphere best practices as documented by VMware.
|
||||
|
||||
|
||||
<figcaption>Solution Overview</figcaption>
|
||||
|
||||
|
||||
|
||||
## 1. Load Balancer Considerations
|
||||
|
||||
A load balancer is required to direct traffic to the Rancher workloads residing on the RKE nodes.
|
||||
|
||||
### Leverage Fault Tolerance and High Availability
|
||||
|
||||
Leverage the use of an external (hardware or software) load balancer that has inherit high-availability functionality (F5, NSX-T, Keepalived, etc).
|
||||
|
||||
### Back Up Load Balancer Configuration
|
||||
|
||||
In the event of a Disaster Recovery activity, availability of the Load balancer configuration will expedite the recovery process.
|
||||
|
||||
### Configure Health Checks
|
||||
|
||||
Configure the Load balancer to automatically mark nodes as unavailable if a health check is failed. For example, NGINX can facilitate this with:
|
||||
|
||||
`max_fails=3 fail_timeout=5s`
|
||||
|
||||
### Leverage an External Load Balancer
|
||||
|
||||
Avoid implementing a software load balancer within the management cluster.
|
||||
|
||||
### Secure Access to Rancher
|
||||
|
||||
Configure appropriate Firewall / ACL rules to only expose access to Rancher
|
||||
|
||||
## 2. VM Considerations
|
||||
|
||||
### Size the VM's According to Rancher Documentation
|
||||
|
||||
See [Installation Requirements](../../../pages-for-subheaders/installation-requirements.md).
|
||||
|
||||
### Leverage VM Templates to Construct the Environment
|
||||
|
||||
To facilitate the consistency of Virtual Machines deployed across the environment, consider the use of "Golden Images" in the form of VM templates. Packer can be used to accomplish this, adding greater customization options.
|
||||
|
||||
### Leverage DRS Anti-Affinity Rules (Where Possible) to Separate Rancher Cluster Nodes Across ESXi Hosts
|
||||
|
||||
Doing so will ensure node VM's are spread across multiple ESXi hosts - preventing a single point of failure at the host level.
|
||||
|
||||
### Leverage DRS Anti-Affinity Rules (Where Possible) to Separate Rancher Cluster Nodes Across Datastores
|
||||
|
||||
Doing so will ensure node VM's are spread across multiple datastores - preventing a single point of failure at the datastore level.
|
||||
|
||||
### Configure VM's as Appropriate for Kubernetes
|
||||
|
||||
It’s important to follow K8s and etcd best practices when deploying your nodes, including disabling swap, double-checking you have full network connectivity between all machines in the cluster, using unique hostnames, MAC addresses, and product_uuids for every node.
|
||||
|
||||
## 3. Network Considerations
|
||||
|
||||
### Leverage Low Latency, High Bandwidth Connectivity Between ETCD Nodes
|
||||
|
||||
Deploy etcd members within a single data center where possible to avoid latency overheads and reduce the likelihood of network partitioning. For most setups, 1Gb connections will suffice. For large clusters, 10Gb connections can reduce the time taken to restore from backup.
|
||||
|
||||
### Consistent IP Addressing for VM's
|
||||
|
||||
Each node used should have a static IP configured. In the case of DHCP, each node should have a DHCP reservation to make sure the node gets the same IP allocated.
|
||||
|
||||
## 4. Storage Considerations
|
||||
|
||||
### Leverage SSD Drives for ETCD Nodes
|
||||
|
||||
ETCD is very sensitive to write latency. Therefore, leverage SSD disks where possible.
|
||||
|
||||
## 5. Backups and Disaster Recovery
|
||||
|
||||
### Perform Regular Management Cluster Backups
|
||||
|
||||
Rancher stores its data in the ETCD datastore of the Kubernetes cluster it resides on. Like with any Kubernetes cluster, perform frequent, tested backups of this cluster.
|
||||
|
||||
### Back up Rancher Cluster Node VMs
|
||||
|
||||
Incorporate the Rancher management node VM's within a standard VM backup policy.
|
||||
+39
@@ -0,0 +1,39 @@
|
||||
---
|
||||
title: Rancher Deployment Strategy
|
||||
---
|
||||
|
||||
There are two recommended deployment strategies for a Rancher instance that manages downstream Kubernetes clusters. Each one has its own pros and cons. Read more about which one would fit best for your use case:
|
||||
|
||||
* [Hub and Spoke](#hub-and-spoke-strategy)
|
||||
* [Regional](#regional-strategy)
|
||||
|
||||
## Hub & Spoke Strategy
|
||||
---
|
||||
|
||||
In this deployment scenario, there is a single Rancher instance managing Kubernetes clusters across the globe. The Rancher instance would be run on a high-availability Kubernetes cluster, and there would be impact due to latencies.
|
||||
|
||||
### Pros
|
||||
|
||||
* Single control plane interface to view/see all regions and environments.
|
||||
* Kubernetes does not require Rancher to operate and can tolerate losing connectivity to the Rancher instance.
|
||||
|
||||
### Cons
|
||||
|
||||
* Subject to network latencies.
|
||||
* If Rancher goes down, global provisioning of new services is unavailable until it is restored. However, each Kubernetes cluster can continue to be managed individually.
|
||||
|
||||
## Regional Strategy
|
||||
---
|
||||
In the regional deployment model a Rancher instance is deployed in close proximity to the downstream Kubernetes clusters.
|
||||
|
||||
### Pros
|
||||
|
||||
* Rancher functionality in regions stay operational if a Rancher instance in another region goes down.
|
||||
* Network latency between Rancher and downstream clusters is greatly reduced, improving the performance of functionality in Rancher.
|
||||
* Upgrades of Rancher can be done independently per region.
|
||||
|
||||
### Cons
|
||||
|
||||
* Overhead of managing multiple Rancher installations.
|
||||
* Visibility into Kubernetes clusters in different regions requires multiple interfaces/panes of glass.
|
||||
* Deploying multi-cluster apps in Rancher requires repeating the process for each Rancher server.
|
||||
+36
@@ -0,0 +1,36 @@
|
||||
---
|
||||
title: Tips for Running Rancher
|
||||
---
|
||||
|
||||
This guide is geared toward use cases where Rancher is used to manage downstream Kubernetes clusters. The high-availability setup is intended to prevent losing access to downstream clusters if the Rancher server is not available.
|
||||
|
||||
A high-availability Kubernetes installation, defined as an installation of Rancher on a Kubernetes cluster with at least three nodes, should be used in any production installation of Rancher, as well as any installation deemed "important." Multiple Rancher instances running on multiple nodes ensure high availability that cannot be accomplished with a single node environment.
|
||||
|
||||
If you are installing Rancher in a vSphere environment, refer to the best practices documented [here.](on-premises-rancher-in-vsphere.md)
|
||||
|
||||
When you set up your high-availability Rancher installation, consider the following:
|
||||
|
||||
### Run Rancher on a Separate Cluster
|
||||
Don't run other workloads or microservices in the Kubernetes cluster that Rancher is installed on.
|
||||
|
||||
### Make sure nodes are configured correctly for Kubernetes
|
||||
It's important to follow K8s and etcd best practices when deploying your nodes, including disabling swap, double checking you have full network connectivity between all machines in the cluster, using unique hostnames, MAC addresses, and product_uuids for every node, checking that all correct ports are opened, and deploying with ssd backed etcd. More details can be found in the [kubernetes docs](https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/#before-you-begin) and [etcd's performance op guide](https://etcd.io/docs/v3.4/op-guide/performance/).
|
||||
|
||||
### When using RKE: Back up the Statefile
|
||||
RKE keeps record of the cluster state in a file called `cluster.rkestate`. This file is important for the recovery of a cluster and/or the continued maintenance of the cluster through RKE. Because this file contains certificate material, we strongly recommend encrypting this file before backing up. After each run of `rke up` you should backup the state file.
|
||||
|
||||
### Run All Nodes in the Cluster in the Same Datacenter
|
||||
For best performance, run all three of your nodes in the same geographic datacenter. If you are running nodes in the cloud, such as AWS, run each node in a separate Availability Zone. For example, launch node 1 in us-west-2a, node 2 in us-west-2b, and node 3 in us-west-2c.
|
||||
|
||||
### Development and Production Environments Should be Similar
|
||||
It's strongly recommended to have a "staging" or "pre-production" environment of the Kubernetes cluster that Rancher runs on. This environment should mirror your production environment as closely as possible in terms of software and hardware configuration.
|
||||
|
||||
### Monitor Your Clusters to Plan Capacity
|
||||
The Rancher server's Kubernetes cluster should run within the [system and hardware requirements](../../../pages-for-subheaders/installation-requirements.md) as closely as possible. The more you deviate from the system and hardware requirements, the more risk you take.
|
||||
|
||||
However, metrics-driven capacity planning analysis should be the ultimate guidance for scaling Rancher, because the published requirements take into account a variety of workload types.
|
||||
|
||||
Using Rancher, you can monitor the state and processes of your cluster nodes, Kubernetes components, and software deployments through integration with Prometheus, a leading open-source monitoring solution, and Grafana, which lets you visualize the metrics from Prometheus.
|
||||
|
||||
After you [enable monitoring](../../../pages-for-subheaders/monitoring-and-alerting.md) in the cluster, you can set up alerts to let you know if your cluster is approaching its capacity. You can also use the Prometheus and Grafana monitoring framework to establish a baseline for key metrics as you scale.
|
||||
|
||||
+61
@@ -0,0 +1,61 @@
|
||||
---
|
||||
title: Tips for Scaling Rancher
|
||||
---
|
||||
|
||||
This guide aims to introduce the approaches that should be considered to scale Rancher setups, and associated challenges with doing so. As systems grow performance will naturally reduce, but there are steps we can take to minimize the load put on Rancher, as well as optimize Rancher's ability to handle these larger setups.
|
||||
|
||||
## General Tips on Optimizing Rancher's Performance
|
||||
* It is advisable to keep Rancher up to date with patch releases. Performance improvements and bug fixes are made throughout the life of a minor release. You can review the release notes to help inform your own decisions on whether an upgrade is necessary but we recommend keeping yourself up to date in most cases.
|
||||
|
||||
* Performance will be negatively impacted by increased latency between Rancher's infrastructure and a downstream cluster's infrastructure (eg. geographic distance). If a user or organization requires clusters/nodes all over the world or spread across many regions, it is best to use multiple Rancher installations.
|
||||
|
||||
* Please always try to scale up gradually, monitoring and observing any change in behavior while doing do. It is usually easier to resolve performance problems as soon as they surface, and before other problems confuse symptoms.
|
||||
|
||||
## Minimizing Load on the local cluster
|
||||
The largest bottleneck when scaling Rancher is resource growth in the local Kubernetes cluster. The local cluster contains information for all downstream clusters. Many operations that apply to downstream clusters will create new objects in the local cluster and require computation from handlers running in the local cluster.
|
||||
|
||||
### Managing Your Object Counts
|
||||
ETCD eventually encounters limitations to the number of a single Kubernetes resource type it can store. These exact numbers are not well documented. From internal observations we usually see performance issues once a single resource type's object count exceeds 60k, and often that type is Rolebindings.
|
||||
|
||||
Rolebindings are created in the local cluster as a side effect of many operations.
|
||||
|
||||
Considerations when attempting reduce rolebindings in the local cluster:
|
||||
* Only add users to clusters and projects when necessary
|
||||
* Remove clusters and projects when they are no longer needed
|
||||
* Only use custom roles if necessary
|
||||
* Use as few rules as possible in custom roles
|
||||
* Consider whether adding a role to a user is redundant
|
||||
* Consider that using less, but more powerful, clusters may be more efficient
|
||||
* Experiment to see if creating new projects or creating new clusters manifests in fewer rolebindings for your specific use case.
|
||||
|
||||
### Using New Apps Over Legacy Apps
|
||||
There are two app kubernetes resources that Rancher uses: apps.projects.cattle.io and apps.cattle.cattle.io. The legacy apps, apps.projects.cattle.io, were introduced first in the Cluster Manager and are now outdated. The new apps, apps.catalog.cattle.io, are found in the Cluster Explorer for their respective cluster. The new apps are preferrable because they live in the downstream cluster while the legacy apps live in the local cluster.
|
||||
|
||||
We recommend removing apps that appear in the Cluster Manager, replacing them with apps in the Cluster Explorer for their target cluster if necessary and creating any future apps in the cluster's Cluster Explorer only.
|
||||
|
||||
### Using the Authorized Cluster Endpoint (ACE)
|
||||
There is an _Authorized Cluster Endpoint_ option for Rancher provisioned RKE1, RKE2, and K3s clusters. When enabled this adds a context to kubeconfigs generated for the cluster that uses a direct endpoint to the cluster and bypasses Rancher. However, it is not enough to only enable this option. The user of the Kubeconfig needs to use `kubectl use-context <ace context name>` in order to start using it.
|
||||
|
||||
Without using ACE, all kubeconfig requests first route through Rancher.
|
||||
|
||||
### Experimental: Option to Reduce Event Handler Executions
|
||||
The bulk of Rancher's logic occurs on event handlers. These event handlers run on an object whenever the object is updated, and when Rancher is started. Additionally, they run every 15 hours when caches are synced. In scaled setups these scheduled runs come with huge performance costs because every handler is being run on every applicable object. However, this scheduled execution of handlers can be disabled using the CATTLE_SYNC_ONLY_CHANGED_OBJECTS environment variable. If resource allocation spikes are seen on an interval of about 15 hours it is possible this setting can help.
|
||||
|
||||
The value for the environment variable can be a comma separated list of the following options. The values refer to types of controllers (the structures that contain and run handlers) and their handlers. Adding the controller types to the variable will disable that set of controllers from running their handlers as part of cache resyncing.
|
||||
|
||||
* `mgmt` refers to management controllers which only run on one Rancher node.
|
||||
* `user` refers to user controllers which run for every cluster. Some of these are ran on the same node as management controllers, while other run in the downstream cluster. This will option targets the former.
|
||||
* `scaled` refers to scaled controllers which run on every Rancher node. This is not recommended to be set due to the critical functionality the scaled handlers are responsible for.
|
||||
|
||||
In short, if you notice CPU usage peaks every 15 hours, add the CATTLE_SYNC_ONLY_CHANGED_OBJECTS environment variable to your rancher deployment with the value `mgmt,user`.
|
||||
|
||||
## Optimizations Outside of Rancher
|
||||
A large component of performance is the local cluster and how it was configured. This cluster can introduce a bottleneck before Rancher software ever runs. When Rancher nodes experience high resource usage, you can use the command "top" to identify whether it is Rancher or a Kubernetes component that is consuming the resource in excess.
|
||||
|
||||
### Keeping Kubernetes Versions Up to Date
|
||||
Similar to Rancher versions, it is advisable to keep your kubernetes cluster up to date. This will ensure that your cluster contains any available performance enhancements or bug fixes.
|
||||
|
||||
### Optimizing ETCD
|
||||
The two main bottlenecks to [ETCD performance](https://etcd.io/docs/v3.4/op-guide/performance/) are disk speed and network speed. Optimization to either should improve performance. For information regarding ETCD performance see [Slow etcd performance (performance testing and optimization)](https://www.suse.com/support/kb/doc/?id=000020100) and [Tuning etcd for Large Installations](https://docs.ranchermanager.rancher.io/how-to-guides/advanced-user-guides/tune-etcd-for-large-installs). Information on disks can also be found [in our docs](https://docs.Ranchermanager.Rancher.io/v2.5/pages-for-subheaders/installation-requirements#disks).
|
||||
|
||||
Theoretically, the more nodes in an ETCD cluster the slower it will be due to replication requirements [source](https://etcd.io/docs/v3.3/faq). This may be counter-intuitive to common scaling approaches. It can also be inferred that ETCD performance will be inversely affected by distance between nodes as that will slow down network communication.
|
||||
Reference in New Issue
Block a user