Clarify the configuration for external TLS termination with NGINX
v0.22, including:
- Instructions to enable the `use-forwarded-headers` option for
ingress in both RKE and RKE2 installations.
- A new section for RKE2 detailing how to create a custom
configuration file for this setting.
- Updates to the required headers section to include
`X-Forwarded-Proto` and `X-Forwarded-Port`.
The documentation has been revised to provide clearer guidance on
configuring Ingress for external TLS with NGINX v0.22. Key changes
include:
- Removed references to NGINX v0.25.
- Added instructions for enabling the `use-forwarded-headers`
option in the `cluster.yml` for RKE installations.
- Included steps for creating a custom
`rke2-ingress-nginx-config.yaml` for RKE2 installations.
- Provided a YAML snippet for HelmChartConfig to demonstrate
how to set the `use-forwarded-headers` option in Helm chart
values.
Updating external TLS termination required settings. Updating the documentation to be clear on RKE configuration vs RKE2 configuration.
Based on closed issue: https://github.com/rancher/rancher/issues/35088
When cleaning nodes for re-use which were deployed manually with RKE2, the process differs slightly. This is more important in reverse so I'm editing that doc as well. I think there's like a note callout or something that might be better?
This adds a few points which should shed light on how to search users
using a drop-down list in Rancher, whether to add users to clusters or
to projects.
Migrating rancher across different Kubernetes distributions causes the local
cluster object not to specify the correct distribution and driver.
This commit adds a workaround to be applied after restoration for a migration
and before bringing Rancher up, so that the local cluster will have the
correct data for the new distribution it runs on.
* [2.9.4] versions table entry
* [2.9.4] webhook table entry
* [2.9.4] csp adapter table entry
* [2.9.4] deprecated features table entry
* Apply suggestions from code review
Co-authored-by: Billy Tat <btat@suse.com>
* [2.9.4] Update link to prime quick start guide for v2.9
---------
Co-authored-by: Billy Tat <btat@suse.com>
* Use crds.enabled to install cert-manager
Signed-off-by: Dharmit Shah <dharmit.shah@suse.com>
* versioned
* reverted changes to earliest versions and slightly modified comment
* spacing issue
---------
Signed-off-by: Dharmit Shah <dharmit.shah@suse.com>
Co-authored-by: Dharmit Shah <dharmit.shah@suse.com>
Co-authored-by: Billy Tat <btat@suse.com>
# This action gets all changed markdown files in /docs and /versioned_docs using tj-actions/changed-files@v42
# It compares new commits in the PR with the base commit (github.event.pull_request.base.sha)
# It checks if no markdown files are changed
# It shows a count of markdown files and lists all changed markdown files
# It uses Vale (https://vale.sh/docs/vale-cli/installation/) to provide feedback base off the SUSE Style Guide / OpenSUSE style rules (https://github.com/openSUSE/suse-vale-styleguide)
name:Style check
on:
pull_request:
paths-ignore:
- '**/README.md'
jobs:
vale-lint:
name:runner / vale
runs-on:ubuntu-latest
steps:
- uses:actions/checkout@v4
continue-on-error:true
with:
fetch-depth:0# OR "2" -> To retrieve the preceding commit.
submodules:true
- name:Get all changed markdown files
continue-on-error:true
id:changed-markdown-files
uses:tj-actions/changed-files@v42
with:
# Avoid using single or double quotes for multiline patterns
> Monitoring V1 requires a Kubernetes verison less than or equal to v1.20.x. To install monitoring on Kubernetes v1.21+, you will need to [migrate to Monitoring V2.](/versioned_docs/version-2.5/how-to-guides/advanced-user-guides/monitoring-alerting-guides/migrate-to-rancher-v2.5%2B-monitoring.md)
> Monitoring V1 requires a Kubernetes verison less than or equal to v1.20.x. To install monitoring on Kubernetes v1.21+, you will need to [migrate to Monitoring V2.](https://github.com/rancher/rancher-docs/tree/main/archived_docs/en/version-2.5/how-to-guides/advanced-user-guides/monitoring-alerting-guides/migrate-to-rancher-v2.5%2B-monitoring.md)
1. From the **Global** view, navigate to the cluster that you want to configure cluster monitoring.
While configuring monitoring at either the [cluster level](cluster-monitoring.md) or [project level](./project-monitoring.md), there are multiple options that can be configured.
- [Basic Configuration](#basic-configuration)
- [Advanced Options](#advanced-options)
- [Node Exporter](#node-exporter)
- [Persistent Storage](#persistent-storage)
- [Remote Storage](#remote-storage)
## Basic Configuration
Option | Description
-------|-------------
Data Retention | How long your Prometheus instance retains monitoring data scraped from Rancher objects before it's purged.
[Enable Node Exporter](#node-exporter) | Whether or not to deploy the node exporter.
Node Exporter Host Port | The host port on which data is exposed, i.e. data that Prometheus collects from your node hardware. Required if you have enabled the node exporter.
[Enable Persistent Storage](#persistent-storage) for Prometheus | Whether or not to configure storage for Prometheus so that metrics can be retained even if the Prometheus pod fails.
[Enable Persistent Storage](#persistent-storage) for Grafana | Whether or not to configure storage for Grafana so that the Grafana dashboards and configuration can be retained even if the Grafana pod fails.
Prometheus [CPU Limit](https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/#meaning-of-cpu) | CPU resource limit for the Prometheus pod.
Prometheus [CPU Reservation](https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/#meaning-of-cpu) | CPU reservation for the Prometheus pod.
Prometheus [Memory Limit](https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/#meaning-of-memory) | Memory resource limit for the Prometheus pod.
Prometheus [Memory Reservation](https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/#meaning-of-memory) | Memory resource requests for the Prometheus pod.
Selector | Ability to select the nodes in which Prometheus and Grafana pods are deployed to. To use this option, the nodes must have labels.
## Advanced Options
Since monitoring is an [application](https://github.com/rancher/system-charts/tree/dev/charts/rancher-monitoring) from the [Rancher catalog](../../../how-to-guides/new-user-guides/helm-charts-in-rancher/helm-charts-in-rancher.md), it can be configured like any other catalog application, by passing in values to Helm.
> **Warning:** Any modification to the application without understanding the entire application can lead to catastrophic errors.
### Prometheus RemoteRead and RemoteWrite
_Available as of v2.4.0_
Prometheus RemoteRead and RemoteWrite can be configured as custom answers in the **Advanced Options** section.
For more information on remote endpoints and storage, refer to the [Prometheus documentation.](https://prometheus.io/docs/operating/integrations/#remote-endpoints-and-storage)
The Prometheus operator documentation contains the full [RemoteReadSpec](https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/api-reference/api.md#remotereadspec) and [RemoteWriteSpec.](https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/api-reference/api.md#remotewritespec)
The [node exporter](https://github.com/prometheus/node_exporter/blob/master/README.md) is a popular open source exporter, which exposes the metrics for hardware and \*NIX kernels OS. It is designed to monitor the host system. However, there are still issues with namespaces when running it in a container, mostly around filesystem mount spaces. In order to monitor actual network metrics for the container network, the node exporter must be deployed with the `hostNetwork` mode.
When configuring Prometheus and enabling the node exporter, enter a host port in the **Node Exporter Host Port** that will not produce port conflicts with existing applications. The host port chosen must be open to allow internal traffic between Prometheus and the Node Exporter.
>**Warning:** In order for Prometheus to collect the metrics of the node exporter, after enabling cluster monitoring, you must open the <b>Node Exporter Host Port</b> in the host firewall rules to allow intranet access. By default, `9796` is used as that host port.
## Persistent Storage
>**Prerequisite:** Configure one or more StorageClasses to use as [persistent storage](../../../how-to-guides/advanced-user-guides/manage-clusters/create-kubernetes-persistent-storage/create-kubernetes-persistent-storage.md) for your Prometheus or Grafana pod.
By default, when you enable Prometheus for either a cluster or project, all monitoring data that Prometheus collects is stored on its own pod. With local storage, if the Prometheus or Grafana pods fail, all the data is lost. Rancher recommends configuring an external persistent storage to the cluster. With the external persistent storage, if the Prometheus or Grafana pods fail, the new pods can recover using data from the persistent storage.
When enabling persistent storage for Prometheus or Grafana, specify the size of the persistent volume and select the StorageClass.
## Remote Storage
>**Prerequisite:** Need a remote storage endpoint to be available. The possible list of integrations is available [here](https://prometheus.io/docs/operating/integrations/)
Using advanced options, remote storage integration for the Prometheus installation can be configured as follows:
Additional fields can be set up based on the [ReadSpec](https://github.com/coreos/prometheus-operator/blob/master/Documentation/api.md#remotereadspec) and [RemoteWriteSpec](https://github.com/coreos/prometheus-operator/blob/master/Documentation/api.md#remotewritespec)
`kubectl` is a CLI utility for running commands against Kubernetes clusters. It's required for many maintenance and administrative tasks in Rancher 2.x.
## Installation
See [kubectl Installation](https://kubernetes.io/docs/tasks/tools/install-kubectl/) for installation on your operating system.
## Configuration
When you create a Kubernetes cluster with RKE, RKE creates a `kube_config_rancher-cluster.yml` in the local directory that contains credentials to connect to your new cluster with tools like `kubectl` or `helm`.
You can copy this file to `$HOME/.kube/config` or if you are working with multiple Kubernetes clusters, set the `KUBECONFIG` environmental variable to the path of `kube_config_rancher-cluster.yml`.
This page is intended to answer questions about what happens if you don't want Rancher anymore, if you don't want a cluster to be managed by Rancher anymore, or if the Rancher server is deleted.
## If the Rancher server is deleted, what happens to the workloads in my downstream clusters?
If Rancher is ever deleted or unrecoverable, all workloads in the downstream Kubernetes clusters managed by Rancher will continue to function as normal.
## If the Rancher server is deleted, how do I access my downstream clusters?
The capability to access a downstream cluster without Rancher depends on the type of cluster and the way that the cluster was created. To summarize:
- **Imported clusters:** The cluster will be unaffected and you can access the cluster using the same methods that you did before the cluster was imported into Rancher.
- **Hosted Kubernetes clusters:** If you created the cluster in a cloud-hosted Kubernetes provider such as EKS, GKE, or AKS, you can continue to manage the cluster using your provider's cloud credentials.
- **RKE clusters:** To access an [RKE cluster,](../how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/launch-kubernetes-with-rancher/launch-kubernetes-with-rancher.md) the cluster must have the [authorized cluster endpoint](../reference-guides/rancher-manager-architecture/communicating-with-downstream-user-clusters.md#4-authorized-cluster-endpoint) enabled, and you must have already downloaded the cluster's kubeconfig file from the Rancher UI. (The authorized cluster endpoint is enabled by default for RKE clusters.) With this endpoint, you can access your cluster with kubectl directly instead of communicating through the Rancher server's [authentication proxy.](../reference-guides/rancher-manager-architecture/communicating-with-downstream-user-clusters.md#1-the-authentication-proxy) For instructions on how to configure kubectl to use the authorized cluster endpoint, refer to the section about directly accessing clusters with [kubectl and the kubeconfig file.](../how-to-guides/advanced-user-guides/manage-clusters/access-clusters/use-kubectl-and-kubeconfig.md#authenticating-directly-with-a-downstream-cluster) These clusters will use a snapshot of the authentication as it was configured when Rancher was removed.
## What if I don't want Rancher anymore?
If you [installed Rancher on a Kubernetes cluster,](../getting-started/installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster/install-upgrade-on-a-kubernetes-cluster.md) remove Rancher by using the [System Tools](../reference-guides/system-tools.md) with the `remove` subcommand.
If you installed Rancher with Docker, you can uninstall Rancher by removing the single Docker container that it runs in.
Imported clusters will not be affected by Rancher being removed. For other types of clusters, refer to the section on [accessing downstream clusters when Rancher is removed.](#if-the-rancher-server-is-deleted-how-do-i-access-my-downstream-clusters)
## What if I don't want my imported cluster managed by Rancher?
If an imported cluster is deleted from the Rancher UI, the cluster is detached from Rancher, leaving it intact and accessible by the same methods that were used to access it before it was imported into Rancher.
To detach the cluster,
1. From the **Global** view in Rancher, go to the **Clusters** tab.
2. Go to the imported cluster that should be detached from Rancher and click **⋮ > Delete.**
3. Click **Delete.**
**Result:** The imported cluster is detached from Rancher and functions normally outside of Rancher.
## What if I don't want my RKE cluster or hosted Kubernetes cluster managed by Rancher?
At this time, there is no functionality to detach these clusters from Rancher. In this context, "detach" is defined as the ability to remove Rancher components from the cluster and manage access to the cluster independently of Rancher.
The capability to manage these clusters without Rancher is being tracked in this [issue.](https://github.com/rancher/rancher/issues/25234)
For information about how to access clusters if the Rancher server is deleted, refer to [this section.](#if-the-rancher-server-is-deleted-how-do-i-access-my-downstream-clusters)
The Hardening Guide is now located in the main [Security](../reference-guides/rancher-security/rancher-security.md) section.
## What are the results of Rancher's Kubernetes cluster when it is CIS benchmarked?
We have run the CIS Kubernetes benchmark against a hardened Rancher Kubernetes cluster. The results of that assessment can be found in the main [Security](../reference-guides/rancher-security/rancher-security.md) section.
New password for default administrator (user-xxxxx):
<new_password>
```
### RKE add-on install (only supported up to Rancher v2.0.8)
:::tip
> If you are currently using the RKE add-on install method, see [Migrating from a Kubernetes Install with an RKE Add-on](../getting-started/installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster/upgrades/migrating-from-rke-add-on.md) for details on how to start using the Helm chart.
New password for default administrator (user-xxxxx):
<new_password>
```
## How can I enable debug logging?
See [Troubleshooting: Logging](../troubleshooting/other-troubleshooting-tips/logging.md)
## My ClusterIP does not respond to ping
ClusterIP is a virtual IP, which will not respond to ping. Best way to test if the ClusterIP is configured correctly, is by using `curl` to access the IP and port to see if it responds.
## Where can I manage Node Templates?
Node Templates can be accessed by opening your account menu (top right) and selecting `Node Templates`.
## Why is my Layer-4 Load Balancer in `Pending` state?
The Layer-4 Load Balancer is created as `type: LoadBalancer`. In Kubernetes, this needs a cloud provider or controller that can satisfy these requests, otherwise these will be in `Pending` state forever. More information can be found on [Cloud Providers](../how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/launch-kubernetes-with-rancher/set-up-cloud-providers/set-up-cloud-providers.md) or [Create External Load Balancer](https://kubernetes.io/docs/tasks/access-application-cluster/create-external-load-balancer/)
## Where is the state of Rancher stored?
- Docker Install: in the embedded etcd of the `rancher/rancher` container, located at `/var/lib/rancher`.
- Kubernetes install: in the etcd of the RKE cluster created to run Rancher.
## How are the supported Docker versions determined?
We follow the validated Docker versions for upstream Kubernetes releases. The validated versions can be found under [External Dependencies](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.10.md#external-dependencies) in the Kubernetes release CHANGELOG.md.
## How can I access nodes created by Rancher?
SSH keys to access the nodes created by Rancher can be downloaded via the **Nodes** view. Choose the node which you want to access and click on the vertical ⋮ button at the end of the row, and choose **Download Keys** as shown in the picture below.
Unzip the downloaded zip file, and use the file `id_rsa` to connect to you host. Be sure to use the correct username (`rancher` or `docker` for RancherOS, `ubuntu` for Ubuntu, `ec2-user` for Amazon Linux)
```
$ ssh -i id_rsa user@ip_of_node
```
## How can I automate task X in Rancher?
The UI consists of static files, and works based on responses of the API. That means every action/task that you can execute in the UI, can be automated via the API. There are 2 ways to do this:
* Visit `https://your_rancher_ip/v3` and browse the API options.
* Capture the API calls when using the UI (Most commonly used for this is [Chrome Developer Tools](https://developers.google.com/web/tools/chrome-devtools/#network) but you can use anything you like)
## The IP address of a node changed, how can I recover?
A node is required to have a static IP configured (or a reserved IP via DHCP). If the IP of a node has changed, you will have to remove it from the cluster and readd it. After it is removed, Rancher will update the cluster to the correct state. If the cluster is no longer in `Provisioning` state, the node is removed from the cluster.
When the IP address of the node changed, Rancher lost connection to the node, so it will be unable to clean the node properly. See [Cleaning cluster nodes](../how-to-guides/advanced-user-guides/manage-clusters/clean-cluster-nodes.md) to clean the node.
When the node is removed from the cluster, and the node is cleaned, you can readd the node to the cluster.
## How can I add more arguments/binds/environment variables to Kubernetes components in a Rancher Launched Kubernetes cluster?
You can add more arguments/binds/environment variables via the [Config File](../reference-guides/cluster-configuration/rancher-server-configuration/rke1-cluster-configuration.md#cluster-config-file) option in Cluster Options. For more information, see the [Extra Args, Extra Binds, and Extra Environment Variables](https://rancher.com/docs/rke/latest/en/config-options/services/services-extras/) in the RKE documentation or browse the [Example Cluster.ymls](https://rancher.com/docs/rke/latest/en/example-yamls/).
## How do I check if my certificate chain is valid?
Use the `openssl verify` command to validate your certificate chain:
:::tip
Configure `SSL_CERT_DIR` and `SSL_CERT_FILE` to a dummy location to make sure the OS installed certificates are not used when verifying manually.
If you receive the error `unable to get local issuer certificate`, the chain is incomplete. This usually means that there is an intermediate CA certificate that issued your server certificate. If you already have this certificate, you can use it in the verification of the certificate like shown below:
If you have successfully verified your certificate chain, you should include needed intermediate CA certificates in the server certificate to complete the certificate chain for any connection made to Rancher (for example, by the Rancher agent). The order of the certificates in the server certificate file should be first the server certificate itself (contents of `rancher.yourdomain.com.pem`), followed by intermediate CA certificate(s) (contents of `intermediate.pem`).
```
-----BEGIN CERTIFICATE-----
%YOUR_CERTIFICATE%
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
%YOUR_INTERMEDIATE_CERTIFICATE%
-----END CERTIFICATE-----
```
If you still get errors during verification, you can retrieve the subject and the issuer of the server certificate using the following command:
issuer= /C=GB/ST=England/O=Alice Ltd/CN=Alice Intermediate CA
```
## How do I check `Common Name` and `Subject Alternative Names` in my server certificate?
Although technically an entry in `Subject Alternative Names` is required, having the hostname in both `Common Name` and as entry in `Subject Alternative Names` gives you maximum compatibility with older browser/applications.
Check `Common Name`:
```
openssl x509 -noout -subject -in cert.pem
subject= /CN=rancher.my.org
```
Check `Subject Alternative Names`:
```
openssl x509 -noout -in cert.pem -text | grep DNS
DNS:rancher.my.org
```
## Why does it take 5+ minutes for a pod to be rescheduled when a node has failed?
This is due to a combination of the following default Kubernetes settings:
* kubelet
*`node-status-update-frequency`: Specifies how often kubelet posts node status to master (default 10s)
* kube-controller-manager
*`node-monitor-period`: The period for syncing NodeStatus in NodeController (default 5s)
*`node-monitor-grace-period`: Amount of time which we allow running Node to be unresponsive before marking it unhealthy (default 40s)
*`pod-eviction-timeout`: The grace period for deleting pods on failed nodes (default 5m0s)
See [Kubernetes: kubelet](https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/) and [Kubernetes: kube-controller-manager](https://kubernetes.io/docs/reference/command-line-tools-reference/kube-controller-manager/) for more information on these settings.
In Kubernetes v1.13, the `TaintBasedEvictions` feature is enabled by default. See [Kubernetes: Taint based Evictions](https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/#taint-based-evictions) for more information.
* kube-apiserver (Kubernetes v1.13 and up)
*`default-not-ready-toleration-seconds`: Indicates the tolerationSeconds of the toleration for notReady:NoExecute that is added by default to every pod that does not already have such a toleration.
*`default-unreachable-toleration-seconds`: Indicates the tolerationSeconds of the toleration for unreachable:NoExecute that is added by default to every pod that does not already have such a toleration.
## Can I use keyboard shortcuts in the UI?
Yes, most parts of the UI can be reached using keyboard shortcuts. For an overview of the available shortcuts, press `?` anywhere in the UI.
Telemetry collects aggregate information about the size of Rancher installations, versions of components used, and which features are used. This information is used by Rancher Labs to help make the product better and is not shared with third-parties.
## What information is collected?
No specific identifying information like usernames, passwords, or the names or addresses of user resources will ever be collected.
The primary things collected include:
- Aggregate counts (smallest, average, largest, total) of nodes per-cluster and their size (e.g. CPU cores & RAM).
- Aggregate counts of logical resources like Clusters, Projects, Namespaces, and Pods.
- Counts of what driver was used to deploy clusters and nodes (e.g. GKE vs EC2 vs Imported vs Custom).
- Versions of Kubernetes components, Operating Systems and Docker that are deployed on nodes.
- Whether some optional components are enabled or not (e.g. which auth providers are used).
- The image name & version of Rancher that is running.
- A unique randomly-generated identifier for this installation.
## Can I see the information that is being sent?
If Telemetry is enabled, you can go to `https://<your rancher server>/v1-telemetry` in your installation to see the current data.
If Telemetry is not enabled, the process that collects the data is not running, so there is nothing being collected to look at.
## How do I turn it on or off?
After initial setup, an administrator can go to the `Settings` page in the `Global` section of the UI and click Edit to change the `telemetry-opt` setting to either `in` or `out`.
@@ -155,15 +155,32 @@ You may terminate the SSL/TLS on a L7 load balancer external to the Rancher clus
Your load balancer must support long lived websocket connections and will need to insert proxy headers so Rancher can route links correctly.
#### Configuring Ingress for External TLS when Using NGINX v0.25
### Configuring Ingress for External TLS when Using NGINX v0.22
In NGINX v0.25, the behavior of NGINX has [changed](https://github.com/kubernetes/ingress-nginx/blob/main/Changelog.md#0220) regarding forwarding headers and external TLS termination. Therefore, in the scenario that you are using external TLS termination configuration with NGINX v0.25, you must edit the `cluster.yml` to enable the `use-forwarded-headers` option for ingress:
In NGINX v0.22, the behavior of NGINX has [changed](https://github.com/kubernetes/ingress-nginx/blob/06efac9f0b6f8f84b553f58ccecf79dc42c75cc6/Changelog.md) regarding forwarding headers and external TLS termination. Therefore, in the scenario that you are using external TLS termination configuration with NGINX v0.22, you must enable the `use-forwarded-headers` option for ingress:
For RKE installations, edit the `cluster.yml` to add the following settings.
```yaml
ingress:
provider: nginx
options:
use-forwarded-headers: "true"
use-forwarded-headers: 'true'
```
For RKE2 installations, you can create a custom `rke2-ingress-nginx-config.yaml` file at `/var/lib/rancher/rke2/server/manifests/rke2-ingress-nginx-config.yaml` containing this required setting to enable using forwarded headers with external TLS termination. Without this required setting applied, the external LB will continuously respond with redirect loops it receives from the ingress controller. (This can be created before or after rancher is installed, rke2 server agent will notice this addition and automatically apply it.)
When running larger Rancher installations with 15 or more clusters it is recommended to increase the default keyspace for etcd from the default 2GB. The maximum setting is 8GB and the host should have enough RAM to keep the entire dataset in memory. When increasing this value you should also increase the size of the host. The keyspace size can also be adjusted in smaller installations if you anticipate a high rate of change of pods during the garbage collection interval.
The etcd data set is automatically cleaned up on a five minute interval by Kubernetes. There are situations, e.g. deployment thrashing, where enough events could be written to etcd and deleted before garbage collection occurs and cleans things up causing the keyspace to fill up. If you see `mvcc: database space exceeded` errors, in the etcd logs or Kubernetes API server logs, you should consider increasing the keyspace size. This can be accomplished by setting the [quota-backend-bytes](https://etcd.io/docs/v3.3/op-guide/maintenance/#space-quota) setting on the etcd servers.
## Example: This Snippet of the RKE Cluster.yml file Increases the Keyspace Size to 5GB
```yaml
# RKE cluster.yml
---
services:
etcd:
extra_args:
quota-backend-bytes:5368709120
```
## Scaling etcd Disk Performance
You can follow the recommendations from [the etcd docs](https://etcd.io/docs/v3.3/tuning/#disk) on how to tune the disk priority on the host.
Additionally, to reduce IO contention on the disks for etcd, you can use a dedicated device for the data and wal directory. Based on etcd best practices, mirroring RAID configurations are unnecessary because etcd replicates data between the nodes in the cluster. You can use striping RAID configurations to increase available IOPS.
To implement this solution in an RKE cluster, the `/var/lib/etcd/data` and `/var/lib/etcd/wal` directories will need to have disks mounted and formatted on the underlying host. In the `extra_args` directive of the `etcd` service, you must include the `wal_dir` directory. Without specifying the `wal_dir`, etcd process will try to manipulate the underlying `wal` mount with insufficient permissions.
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Sunil Singh
Diogo Souza
Billy Tat
Tom Lebreux
Jonathan Crowther
John Carew
Petr Kovar
Alessio GreggiandLucas Saintarbor
Angelo Cicero
Marty Hernandez Avedon
Tejeev
Corentin Néau
Alex Seymour
SjuulJanssen
hridyesh bisht
hridyesh bisht
Mikhail Krutov
burnedoutman
Tejeev
Pedro Franco de Carvalho
Harrison Affel
Jiaqi Luo
Venkata Krishna Rohit Sakala
dependabot[bot]
Andreas Jaeger
Andreas Jaeger
Jake Hyde
Pratik Jagrut
Julia Bier
yiannistri
Nick Hecker
aseques
Pietro Dell'Amore
Michal Jura
Adam Pickering
Gabriel Bueno
Kinara Shah
Peter Matseykanets
Silvio Moioli
Silvio MoioliandBilly Tat
eliaoggian
Sharuzzaman Ahmat Raslan
Ian
Tony Hansen
Dharmit ShahandMarty Hernandez Avedon
Artem Sadovskyi
Kourosh Maneshni
gunamata
Dan Pock
DanandMarty Hernandez Avedon