public/rancher-docs
1
0
Fork 0
mirror of https://github.com/rancher/rancher-docs.git synced 2026-09-25 12:38:05 +00:00
Code Issues Packages Projects Releases Wiki Activity

Edit zero-downtime upgrade docs for Rancher

Browse Source
This commit is contained in:
Catherine Luse
2020-03-30 13:33:04 -07:00
parent 95802e6603
commit 302b3ee918
8 changed files with 143 additions and 390 deletions
Download Patch File Download Diff File Expand all files Collapse all files
content/rancher/v2.x/en/_index.md
+6 -5
View File
@@ -8,13 +8,14 @@ insertOneSix: true
weight: 1
ctaBanner: intro-k8s-rancher-online-training
---
Rancher was originally built to work with multiple orchestrators, and it included its own orchestrator called Cattle. With the rise of Kubernetes in the marketplace, Rancher 2.x exclusively deploys and manages Kubernetes clusters running anywhere, on any provider.
# What's New?
Rancher can provision Kubernetes from a hosted provider, provision compute nodes and then install Kubernetes onto them, or import existing Kubernetes clusters running anywhere.
Rancher was originally built to work with multiple orchestrators, and it included its own orchestrator called Cattle. With the rise of Kubernetes in the marketplace, Rancher now exclusively deploys and manages multiple Kubernetes clusters running anywhere, on any provider. It can provision Kubernetes from a hosted provider, provision compute nodes and then install Kubernetes onto them, or inherit existing Kubernetes clusters running anywhere.
One Rancher server installation can manage up to 2,000 Kubernetes clusters and 100,000 nodes from the same user interface.
One Rancher server installation can manage hundreds of Kubernetes clusters from the same interface.
Rancher adds significant value on top of Kubernetes, first by centralizing authentication and role-based access control (RBAC) for all of the clusters, giving global admins the ability to control cluster access from one location.
Rancher adds significant value on top of Kubernetes, first by centralizing role-based access control (RBAC) for all of the clusters and giving global admins the ability to control cluster access from one location. It then enables detailed monitoring and alerting for clusters and their resources, ships logs to external providers, and integrates directly with Helm via the Application Catalog. If you have an external CI/CD system, you can plug it into Rancher, but if you don't, Rancher even includes a pipeline engine to help you automatically deploy and upgrade workloads.
It then enables detailed monitoring and alerting for clusters and their resources, ships logs to external providers, and integrates directly with Helm via the Application Catalog. If you have an external CI/CD system, you can plug it into Rancher, but if you don't, Rancher even includes a pipeline engine to help you automatically deploy and upgrade workloads.
Rancher is a _complete_ container management platform for Kubernetes, giving you the tools to successfully run Kubernetes anywhere.
Rancher is a _complete_ container management platform for Kubernetes, giving you the tools to successfully run Kubernetes anywhere.
content/rancher/v2.x/en/cluster-admin/nodes/_index.md
+7 -4
View File
@@ -71,8 +71,10 @@ However, you can override the conditions draining when you initiate the drain. Y
The node draining options are different based on your version of Rancher.
### Aggressive and Safe Draining Options for Rancher v2.2.x+
### Aggressive and Safe Draining Options
{{% tabs %}}
{{% tab "Rancher v2.2.x+" %}}
There are two drain modes: aggressive and safe.
- **Aggressive Mode**
@@ -84,9 +86,8 @@ There are two drain modes: aggressive and safe.
- **Safe Mode**
If a node has standalone pods or ephemeral data it will be cordoned but not drained.
### Aggressive and Safe Draining Options for Rancher Prior to v2.2.x
{{% /tab %}}
{{% tab "Rancher Prior to v2.2.x" %}}
The following list describes each drain option:
- **Even if there are pods not managed by a ReplicationController, ReplicaSet, Job, DaemonSet or StatefulSet**
@@ -101,6 +102,8 @@ The following list describes each drain option:
If a pod uses emptyDir to store local data, you might not be able to safely delete it, since the data in the emptyDir will be deleted once the pod is removed from the node. Similar to the first option, Kubernetes expects the implementation to decide what to do with these pods. Choosing this option will delete these pods.
{{% /tab %}}
{{% /tabs %}}
### Grace Period
content/rancher/v2.x/en/cluster-admin/upgrading-kubernetes/_index.md
+105 -17
View File
@@ -3,25 +3,46 @@ title: Upgrading and Rolling Back Kubernetes
weight: 70
---
Following an upgrade to the latest version of Rancher, you can update your existing clusters to use the latest supported version of Kubernetes.
Following an upgrade to the latest version of Rancher, downstream Kubernetes clusters can be upgraded to use the latest supported version of Kubernetes.
Before a new version of Rancher is released, it's tested with the latest minor versions of Kubernetes to ensure compatibility. For example, Rancher v2.3.0 is was tested with Kubernetes v1.15.4, v1.14.7, and v1.13.11. For details on which versions of Kubernetes were tested on each Rancher version, refer to the [support maintenance terms.](https://rancher.com/support-maintenance-terms/all-supported-versions/rancher-v2.3.0/)
Rancher calls RKE (Rancher Kubernetes Engine) as a library when provisioning and editing RKE clusters. For more information on configuring the upgrade strategy for RKE clusters, refer to the [RKE documentation]({{<baseurl>}}/rke/latest/en/).
This section covers the following topics:
- [New Features](#new-features)
- [Tested Kubernetes Versions](#tested-kubernetes-versions)
- [How Upgrades Work](#how-upgrades-work)
- [Recommended Best Practice for Upgrades](#recommended-best-practice-for-upgrades)
- [Upgrading the Kubernetes Version](#upgrading-the-kubernetes-version)
- [Rolling Back the Kubernetes Version](#rolling-back-the-kubernetes-version)
- [Configuring the Upgrade Strategy](#configuring-the-upgrade-strategy)
- [Configuring the Maximum Unavailable Worker Nodes in the Rancher UI](#configuring-the-maximum-unavailable-worker-nodes-in-the-rancher-ui)
- [Enabling Draining Nodes During Upgrades from the Rancher UI](#enabling-draining-nodes-during-upgrades-from-the-rancher-ui)
- [Maintaining Availability for Applications During Upgrades](#maintaining-availability-for-applications-during-upgrades)
- [Configuring the Upgrade Strategy in the cluster.yml](#configuring-the-upgrade-strategy-in-the-cluster-yml)
- [Troubleshooting](#troubleshooting)
# New Features
As of Rancher v2.3.0, the Kubernetes metadata feature was added, which allows Rancher to ship Kubernetes patch versions without upgrading Rancher. For details, refer to the [section on Kubernetes metadata.]({{<baseurl>}}/rancher/v2.x/en/admin-settings/k8s-metadata)
As of Rancher v2.4.0, the ability to import K3s Kubernetes clusters into Rancher was added, along with the ability to upgrade Kubernetes when editing those clusters.
As of Rancher v2.4.0,
### Zero-downtime Upgrades
- The ability to import K3s Kubernetes clusters into Rancher was added, along with the ability to upgrade Kubernetes when editing those clusters. For details, refer to the [section on imported clusters.]({{<baseurl>}}/rancher/v2.x/en/cluster-provisioning/imported-clusters)
- New advanced options are exposed in the Rancher UI for configuring the upgrade strategy of an RKE cluster: **Maximum Worker Nodes Unavailable** and **Drain nodes.** These options leverage the new cluster upgrade process of RKE v1.1.0, in which worker nodes are upgraded in batches, so that applications can remain available during cluster upgrades, under [certain conditions.](#maintaining-availability-for-applications-during-upgrades)
_Available as of v2.4_
# Tested Kubernetes Versions
It is now possible to upgrade or edit an RKE Kubernetes cluster without downtime for your applications.
Before a new version of Rancher is released, it's tested with the latest minor versions of Kubernetes to ensure compatibility. For example, Rancher v2.3.0 is was tested with Kubernetes v1.15.4, v1.14.7, and v1.13.11. For details on which versions of Kubernetes were tested on each Rancher version, refer to the [support maintenance terms.](https://rancher.com/support-maintenance-terms/all-supported-versions/rancher-v2.3.0/)
A zero-downtime upgrade is one in which your workloads are available on at least a single node, and all critical addon services, such as Ingress and DNS, are available during the upgrade.
# How Upgrades Work
For details, refer to [this section.]({{<baseurl>}}/rancher/v2.x/en/cluster-admin/zero-downtime)
RKE v1.1.0 changed the way that clusters are upgraded.
### Recommended Best Practice for Upgrades
In this section of the [RKE documentation,]({{<baseurl>}}/rke/latest/en/upgrades/how-upgrades-work) you'll learn what happens when you edit or upgrade your RKE Kubernetes cluster.
# Recommended Best Practice for Upgrades
{{% tabs %}}
{{% tab "Rancher v2.4+" %}}
@@ -29,7 +50,7 @@ When upgrading the Kubernetes version of a cluster, we recommend that you:
1. Take a snapshot.
1. Initiate a Kubernetes upgrade.
1. If the upgrade fails, revert the Kubernetes upgrade to the pre-upgrade version. Before restoring the cluster from the snapshot in the etcd datastore, the cluster should be running the pre-upgrade Kubernetes version.
1. If the upgrade fails, revert the cluster to the pre-upgrade Kubernetes version. Before restoring the cluster from the snapshot in the etcd datastore, the cluster should be running the pre-upgrade Kubernetes version.
1. Restore the cluster from the etcd snapshot.
The restore operation will work on a cluster that is not in a healthy or active state.
@@ -41,17 +62,16 @@ When upgrading the Kubernetes version of a cluster, we recommend that you:
1. Initiate a Kubernetes upgrade.
1. If the upgrade fails, restore the cluster from the etcd snapshot.
> The cluster cannot be downgraded to a previous Kubernetes version.
The cluster cannot be downgraded to a previous Kubernetes version.
{{% /tab %}}
{{% /tabs %}}
### Upgrading Kubernetes
# Upgrading the Kubernetes Version
> **Prerequisites:**
>
> The options below are available only for [Rancher-launched RKE Kubernetes clusters]({{<baseurl>}}/rancher/v2.x/en/cluster-provisioning/rke-clusters/) and [imported K3s Kubernetes clusters.]({{<baseurl>}}/rancher/v2.x/en/cluster-provisioning/imported-clusters/#additional-features-for-imported-k3s-clusters)
>
> Before upgrading Kubernetes, [back up your cluster]({{<baseurl>}}/rancher/v2.x/en/backups).
> - The options below are available only for [Rancher-launched RKE Kubernetes clusters]({{<baseurl>}}/rancher/v2.x/en/cluster-provisioning/rke-clusters/) and [imported K3s Kubernetes clusters.]({{<baseurl>}}/rancher/v2.x/en/cluster-provisioning/imported-clusters/#additional-features-for-imported-k3s-clusters)
> - Before upgrading Kubernetes, [back up your cluster.]({{<baseurl>}}/rancher/v2.x/en/backups)
1. From the **Global** view, find the cluster for which you want to upgrade Kubernetes. Select **Vertical Ellipsis (...) > Edit**.
@@ -63,8 +83,76 @@ When upgrading the Kubernetes version of a cluster, we recommend that you:
**Result:** Kubernetes begins upgrading for the cluster.
### Rolling Back Kubernetes
# Rolling Back the Kubernetes Version
_Available as of v2.4_
A cluster can be restored to a backup in which the previous Kubernetes version was used.
A cluster can be restored to a backup in which the previous Kubernetes version was used.
RKE v1.1.0 introduced the ability to restore a Kubernetes cluster to an older Kubernetes version by restoring it to a snapshot that includes the older Kubernetes version.
# Configuring the Upgrade Strategy
As of RKE v1.1.0, additional upgrade options became available to give you more granular control over the upgrade process. These options can be used to maintain availability of your applications during a cluster upgrade if certain [conditions and requirements]({{<baseurl>}}/rke/latest/en/upgrades/maintaining-availability) are met.
The upgrade strategy can be configured in the Rancher UI, or by editing the `cluster.yml`. More advanced options are available by editing the `cluster.yml`.
### Configuring the Maximum Unavailable Worker Nodes in the Rancher UI
From the Rancher UI, the maximum number of unavailable worker nodes can be configured. During a cluster upgrade, worker nodes will be upgraded in batches of this size.
By default, the maximum number of unavailable worker is defined as 10 percent of all worker nodes. This number can be configured as a percentage or as an integer. When defined as a percentage, the batch size is rounded down to the nearest node, with a minimum of one node.
To change the default number or percentage of worker nodes,
1. Go to the cluster view in the Rancher UI.
1. Click **Ellipsis (...) > Edit.**
1. In the **Advanced Options** section, go to the **Maxiumum Worker Nodes Unavailable** field. Enter the percentage of worker nodes that can be upgraded in a batch. Optionally, select **Count** from the drop-down menu and enter the maximum unavailable worker nodes as an integer.
1. Click **Save.**
**Result:** The cluster is updated to use the new upgrade strategy.
### Enabling Draining Nodes During Upgrades from the Rancher UI
By default, RKE [cordons](https://kubernetes.io/docs/concepts/architecture/nodes/#manual-node-administration) each node before upgrading it. [Draining](https://kubernetes.io/docs/tasks/administer-cluster/safely-drain-node/) is disabled during upgrades by default. If draining is enabled in the cluster configuration, RKE will both cordon and drain the node before it is upgraded.
To enable draining each node during a cluster upgrade,
1. Go to the cluster view in the Rancher UI.
1. Click **Ellipsis (...) > Edit.**
1. In the **Advanced Options** section, go to the **Drain nodes** field and click **Yes.**
1. Choose a safe or aggressive drain option. For more information about each option, refer to [this section.]({{<baseurl>}}/rancher/v2.x/en/cluster-admin/nodes/#aggressive-and-safe-draining-options)
1. Optionally, configure a grace period. The grace period is the timeout given to each pod for cleaning things up, so they will have chance to exit gracefully. Pods might need to finish any outstanding requests, roll back transactions or save state to some external storage. If this value is negative, the default value specified in the pod will be used.
1. Optionally, configure a timeout, which is the amount of time the drain should continue to wait before giving up.
1. Click **Save.**
**Result:** The cluster is updated to use the new upgrade strategy.
### Maintaining Availability for Applications During Upgrades
_Available as of RKE v1.1.0_
In [this section of the RKE documentation,]({{<baseurl>}}/rke/latest/en/upgrades/maintaining-availability/) you'll learn the requirements to prevent downtime for your applications when upgrading the cluster.
### Configuring the Upgrade Strategy in the cluster.yml
More advanced upgrade strategy configuration options are available by editing the `cluster.yml`.
For details, refer to [Configuring the Upgrade Strategy]({{<baseurl>}}/rke/latest/en/upgrades/configuring-strategy) in the RKE documentation. The section also includes an example `cluster.yml` for configuring the upgrade strategy.
# Troubleshooting
If a node doesn't come up after an upgrade, the `rke up` command errors out.
No upgrade will proceed if the number of unavailable nodes exceeds the configured maximum.
If an upgrade stops, you may need to fix an unavailable node or remove it from the cluster before the upgrade can continue.
A failed node could be in many different states:
- Powered off
- Unavailable
- User drains a node while upgrade is in process, so there are no kubelets on the node
- The upgrade itself failed
If the max unavailable number of nodes is reached during an upgrade, Rancher user clusters will be stuck in updating state and not move forward with upgrading any other control plane nodes. It will continue to evaluate the set of unavailable nodes in case one of the nodes becomes available. If the node cannot be fixed, you must remove the node in order to continue the upgrade.
content/rancher/v2.x/en/cluster-admin/upgrading-kubernetes/how-upgrades-work/_index.md
-95
View File
@@ -1,95 +0,0 @@
---
title: How Upgrades Work
weight: 1
---
In this section, you'll learn how each type of node is upgraded by default when a cluster is upgraded.
Rancher calls [RKE]({{<baseurl>}}/rke/latest/en/) (Rancher Kubernetes Engine) as a library when upgrading Kubernetes clusters. RKE v1.1.0 changed the way that clusters are upgraded to make it possible to maintain availability of your applications during the upgrade.
{{% tabs %}}
{{% tab "RKE v1.1.0+" %}}
The following features are new in RKE v1.1.0:
- The ability to upgrade or edit a cluster without downtime for your applications.
- The ability to manually upgrade nodes of a certain role without upgrading others.
- The ability to restore a Kubernetes cluster to an older Kubernetes version by restoring it to a snapshot that includes the older Kubernetes version. This capability allows you to safely upgrade one type of node at a time, because if an upgrade cannot be completed by all nodes in the cluster, you can downgrade the Kubernetes version of the nodes that were already upgraded.
When a cluster is upgraded with `rke up`, using the default options, the following process is used:
1. etcd nodes get updated first, one at a time.
1. Controlplane nodes get updated second, one at a time.
1. Worker nodes get updated third, one by one, because the default configuration for the maximum number of unavailable nodes is one.
1. [Addons]({{<baseurl>}}/rke/latest/en/config-options/add-ons/) get upgraded one by one.
The following sections break down in more detail what happens when etcd nodes, controlplane nodes, worker nodes, and addons are upgraded. This information is intended to be used to help you understand the update strategy for the cluster, and may be useful when troubleshooting problems with upgrading the cluster.
- [Upgrades of etcd nodes](#upgrades-of-etcd-nodes)
- [Upgrades of controlplane nodes](#upgrades-of-controlplane-nodes)
- [Upgrades of worker nodes](#upgrades-of-worker-nodes)
- [Upgrades of addons](#upgrades-of-addons)
### Upgrades of etcd Nodes
A cluster upgrade begins by upgrading the etcd nodes one at a time.
If an etcd node fails at any time, the upgrade will fail and no more nodes will be upgraded. The cluster will be stuck in an updating state and not move forward to upgrading controlplane or worker nodes.
### Upgrades of Controlplane Nodes
Controlplane nodes are upgraded one at a time by default. The maximum number of unavailable controlplane nodes can also be configured, so that they can be upgraded in batches.
As long as the maximum unavailable number or percentage of controlplane nodes has not been reached, Rancher will continue to upgrade other controlplane nodes, then the worker nodes.
If any controlplane nodes were unable to be upgraded, the upgrade will not proceed to the worker nodes.
### Upgrades of Worker Nodes
By default, worker nodes are upgraded in batches. The size of the batch is determined by the maximum number of unavailable worker nodes, configured as the `max_unavailable_worker` directive in the `cluster.yml`.
By default, the `max_unavailable_worker` nodes is defined as 10 percent of all worker nodes. This number can be configured as a percentage or as an integer. When defined as a percentage, the batch size is rounded down to the nearest node, with a minimum of one node.
For example, if you have 11 worker nodes and `max_unavailable_worker` is 25%, two nodes will be upgraded at once because 25% of 11 is 2.75. If you have two worker nodes and `max_unavailable_worker` is 1%, the worker nodes will be upgraded one at a time because the minimum batch size is one.
When each node in a batch returns to a Ready state, the next batch of nodes begins to upgrade. If `kubelet` and `kube-proxy` have started, the node is Ready. As long as the `max_unavailable_worker` number of nodes have not failed, Rancher will continue to upgrade other worker nodes.
RKE scans the cluster before starting the upgrade to find the powered down or unreachable hosts. The upgrade will stop if that number matches or exceeds the maximum number of unavailable nodes.
RKE will cordon each node before upgrading it, and uncordon the node afterward. RKE can also be configured to [drain](https://kubernetes.io/docs/tasks/administer-cluster/safely-drain-node/) nodes before upgrading them.
RKE will handle all worker node upgrades before upgrading any add-ons. As long as the maximum number of unavailable worker nodes is not reached, RKE will attempt to upgrade the [addons.](#upgrades-of-addons) For example, if a cluster has two worker nodes and one worker node fails, but the maximum unavailable worker nodes is greater than one, the addons will still be upgraded.
### Upgrades of Addons
The availability of your applications partly depends on the availability of [RKE addons.]({{<baseurl>}}/rke/latest/en/config-options/add-ons/) Addons are used to deploy several cluster components, including network plug-ins, the Ingress controller, DNS provider, and metrics server.
Because RKE addons are necessary for allowing traffic into the cluster, a zero-downtime upgrade requires them to be updated in batches. You will need to configure the maximum number of unavailable replicas for each addon in the `cluster.yml` to ensure that your cluster will retain enough available replicas during an upgrade.
For more information on configuring the number of replicas for each addon, refer to [this section.](#replicas-for-rke-addons)
For an example showing how to configure the addons for zero downtime, refer to the [example cluster.yml.](#example-cluster-yml)
{{% /tab %}}
{{% tab "RKE prior to v1.1.0" %}}
When a cluster is upgraded with `rke up`, using the default options, the following process is used:
- etcd nodes get updated first, one at a time.
- Controlplane nodes get updated second, one at a time.
- Worker nodes and addons get updated third, in batches of 50 or the total number of worker nodes, whichever is lower.
- Addons get upgraded one by one.
### Upgrades of Controlplane and etcd Nodes
Controlplane and etcd nodes would be upgraded in batches of 50 nodes or the total number of controlplane nodes, whichever is lower.
If a node fails at any time, the upgrade will stop upgrading any other nodes and fail.
### Upgrades of Worker Nodes
Worker nodes are upgraded simultaneously, in batches of either 50 or the total number of worker nodes, whichever is lower. If a worker node fails at any time, the upgrade stops.
When a worker node is upgraded, it restarts several Docker processes, including the `kubelet` and `kube-proxy`. When `kube-proxy` comes up, it flushes `iptables`. When this happens, pods on this node can’t be accessed, resulting in downtime for the applications.
{{% /tab %}}
{{% /tabs %}}
content/rancher/v2.x/en/cluster-admin/zero-downtime/_index.md
-246
View File
@@ -1,246 +0,0 @@
---
title: Zero-downtime Cluster Maintenance
weight: 71
---
_Available as of v2.4.0_
It is now possible to upgrade or edit an RKE Kubernetes cluster without downtime for your applications. In this section, you'll learn how to configure your cluster to prevent downtime when you upgrade the cluster using `rke up`.
A zero-downtime upgrade is one in which your workloads are available on at least a single node, and all critical addon services, such as Ingress and DNS, are available during the upgrade.
This availability is achieved by upgrading worker nodes in batches of a configurable size, and ensuring that your workloads run on a number of nodes that exceeds that maximum number of unavailable worker nodes.
Several requirements must be met before zero-downtime cluster upgrades can succeed. For details, refer to [this section.](#requirements)
- [How Upgrades Work](#how-upgrades-work)
- [Requirements for Zero-downtime Upgrades](#requirements-for-zero-downtime-upgrades)
- [Kubernetes Version Requirement](#1-kubernetes-version-requirement)
- [Cluster Requirements](#2-cluster-requirements)
- [Workload Requirements](#3-workload-requirements)
- [Configuring the Upgrade Strategy](#configuring-the-upgrade-strategy)
- [Maximum Unavailable Nodes](#maximum-unavailable-nodes)
- [Draining Nodes](#draining-nodes)
- [Replicas for Ingress and Networking Addons](#replicas-for-ingress-and-networking-addons)
- [Replicas for DNS and Monitoring Addons](#replicas-for-dns-and-monitoring-addons)
- [Example cluster.yml](#example-cluster-yml)
- [Troubleshooting](#troubleshooting)
# Requirements for Zero-downtime Upgrades
To achieve a cluster upgrade without downtime for your applications, you will need to configure your workloads to continue running despite the rolling upgrade of worker nodes. There are also requirements for the cluster architecture and Kubernetes target version.
1. [Kubernetes version requirement](#2-kubernetes-version-requirement)
2. [Cluster requirements](#1-cluster-requirements)
3. [Workload requirements](#3-workload-requirements)
### 1. Kubernetes Version Requirement
When upgrading to a newer Kubernetes version, the upgrade must be from a minor release to the next minor version, or to within the same patch release series.
### 2. Cluster Requirements
The following must be true of the cluster that will be upgraded:
1. The cluster has three or more etcd nodes.
1. The cluster has two or more controlplane nodes.
1. The cluster has two or more worker nodes.
1. The Ingress, DNS, and other addons are schedulable to a number of nodes that exceeds the maximum number of unavailable worker nodes. By default, the minimum number of unavailable worker nodes is one.
### 3. Workload Requirements
The following must be true of the cluster's applications:
1. The application and Ingress are deployed across a number of nodes exceeding the maximum number of unavailable worker nodes.
1. The applications must make use of liveness and readiness probes.
For information on how to use node selectors to assign pods to nodes, refer to the [official Kubernetes documentation.](https://kubernetes.io/docs/concepts/configuration/assign-pod-node/)
For information on configuring the number of replicas for each addon, refer to [this section.](#replicas-for-rke-addons)
# Configuring the Upgrade Strategy
In this section, you'll learn how to configure the maximum number of unavailable controlplane and worker nodes, how to drain nodes before upgrading them, and how to configure the replicas for addons such as Ingress.
### Maximum Unavailable Nodes
Optionally, configure the following upgrade directives in the `cluster.yml` before upgrading the cluster:
- **max_unavailable_controlplane:** The maximum number of controlplane nodes that can fail without causing the cluster upgrade to fail. By default, `max_unavailable_controlplane` is defined as 2 nodes.
- **max_unavailable_worker:** The maximum number of worker nodes that can fail without causing the cluster upgrade to fail. By default, `max_unavailable_worker` is defined as 10 percent of all worker nodes.*
/* This number can be configured as a percentage or as an integer. When defined as a percentage, the batch size is rounded down to the nearest node, with a minimum of one node per batch.
An example configuration of the cluster upgrade strategy is shown below:
```yaml
upgrade_strategy:
max_unavailable_worker: 10%
max_unavailable_controlplane: 2
```
### Draining Nodes
By default, nodes are cordoned first before upgrading. Each node should always be cordoned before starting its upgrade so that new pods will not be scheduled to it, and traffic will not reach the node. In addition to cordoning each node, RKE can also be configured to drain each node before starting its upgrade. Draining a node will evict all the pods running on the computing resource.
For information on draining and how to safely drain a node, refer to the [Kubernetes documentation.](https://kubernetes.io/docs/tasks/administer-cluster/safely-drain-node/)
If the `drain` directive is set to `true` in the `cluster.yml`, worker nodes will be drained before they are upgraded. The default value is false:
```yaml
upgrade_strategy:
max_unavailable_worker: 10%
max_unavailable_controlplane: 2
drain: false
node_drain_input:
force: false
ignore_daemonsets: true
delete_local_data: false
grace_period: -1 // grace period specified for each pod spec will be used
timeout: 60
```
### Replicas for Ingress and Networking Addons
The Ingress and network addons are launched as Kubernetes [daemonsets.](https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/) If no value is given for the [update strategy,](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#strategy) Kubernetes sets the update strategy to `rollingUpdate` by default, with `maxUnavailable` set to 1.
An example configuration of the Ingress and network addons is shown below:
```yaml
ingress:
provider: nginx
update_strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 5
network:
plugin: canal
update_strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 6
```
### Replicas for DNS and Monitoring Addons
The DNS and monitoring addons are launched as Kubernetes [deployments.](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/) These addons include `coredns`, `kubedns`, and `metrics-server`, the monitoring deployment.
If no value is configured for their [update strategy](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#strategy) in the `cluster.yml`, Kubernetes sets the update strategy to `rollingUpdate` by default, with `maxUnavailable` set to 25% and `maxSurge` set to 25%.
The DNS addons use `cluster-proportional-autoscaler`, which is an [open-source container image](https://github.com/kubernetes-incubator/cluster-proportional-autoscaler) that watches over the number of schedulable nodes and cores of the cluster and resizes the number of replicas for the required resource. This functionality is useful for applications that need to be autoscaled with the number of nodes in the cluster. For the DNS addon, the fields needed for the `cluster-proportional-autoscaler` are made configurable.
The following table shows the default values for these fields:
Field Name | Default Value
-----------|--------------
coresPerReplica | 128
nodesPerReplica | 4
min | 1
preventSinglePointFailure | true
The `cluster-proportional-autoscaler` uses this formula to calculate the number of replicas:
```plain
replicas = max( ceil( cores * 1/coresPerReplica ) , ceil( nodes * 1/nodesPerReplica ) )
replicas = min(replicas, max)
replicas = max(replicas, min)
```
An example configuration of the DNS and monitoring addons is shown below:
```yaml
dns:
provider: coredns
update_strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 20%
maxSurge: 15%
linear_autoscaler_params:
cores_per_replica: 0.34
nodes_per_replica: 4
prevent_single_point_failure: true
min: 2
max: 3
monitoring:
provider: metrics-server
update_strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 8
```
### Example cluster.yml
```yaml
# If you intened to deploy Kubernetes in an air-gapped environment,
# please consult the documentation on how to configure custom RKE images.
nodes:
# At least three etcd nodes, two controlplane nodes, and two worker nodes,
# nodes skipped for brevity
upgrade_strategy:
max_unavailable_worker: 10%
max_unavailable_controlplane: 2
drain: false
node_drain_input:
force: false
ignore_daemonsets: true
delete_local_data: false
grace_period: -1 // grace period specified for each pod spec will be used
timeout: 60
ingress:
provider: nginx
update_strategy: # Available in v2.4
type: RollingUpdate
rollingUpdate:
maxUnavailable: 5
network:
plugin: canal
update_strategy: # Available in v2.4
type: RollingUpdate
rollingUpdate:
maxUnavailable: 6
dns:
provider: coredns
update_strategy: # Available in v2.4
type: RollingUpdate
rollingUpdate:
maxUnavailable: 20%
maxSurge: 15%
linear_autoscaler_params:
cores_per_replica: 0.34
nodes_per_replica: 4
prevent_single_point_failure: true
min: 2
max: 3
monitoring:
provider: metrics-server
update_strategy: # Available in v2.4
type: RollingUpdate
rollingUpdate:
maxUnavailable: 8
```
# Upgrading Nodes Manually
You can manually update each type of node separately. As a best practice, upgrade the etcd nodes first, followed by controlplane and then worker nodes.
A cluster can be rolled back to a snapshot that uses a previous Kubernetes version.
# Troubleshooting
If a node doesn't come up after an upgrade, the `rke up` command errors out.
No upgrade will proceed if the number of unavailable nodes exceeds the configured maximum.
If an upgrade stops, you may need to fix an unavailable node or remove it from the cluster before the upgrade can continue.
A failed node could be in many different states:
- Powered off
- Unavailable
- User drains a node while upgrade is in process, so there are no kubelets on the node
- The upgrade itself failed
If the max unavailable number of nodes is reached during an upgrade, Rancher user clusters will be stuck in updating state and not move forward with upgrading any other control plane nodes. It will continue to evaluate the set of unavailable nodes in case one of the nodes becomes available. If the node cannot be fixed, you must remove the node in order to continue the upgrade.
content/rke/latest/en/upgrades/_index.md
+19 -11
View File
@@ -15,18 +15,24 @@ In case the Kubernetes version is defined in the `kubernetes_version` directive
This page covers the following topics:
- [How upgrades work](#how-upgrades-work)
- [Prerequisites](#prerequisites)
- [Upgrading Kubernetes](#upgrading-kubernetes)
- [Configuring the upgrade strategy](#configuring-the-upgrade-strategy)
- [Maintaining availability for applications during upgrades](#maintaining-availability-for-applications-during-upgrades)
- [Listing supported Kubernetes versions](#listing-supported-kubernetes-versions)
- [Kubernetes version precedence](#kubernetes-version-precedence)
- [Using an unsupported Kubernetes version](#using-an-unsupported-kubernetes-version)
- [Mapping the Kubernetes version to services](#mapping-the-kubernetes-version-to-services)
- [Service upgrades](#service-upgrades)
- [Configuring the upgrade strategy](#configuring-the-upgrade-strategy)
- [Upgrading Nodes Manually](#upgrading-nodes-manually)
- [Rolling Back the Kubernetes Version](#rolling-back-the-kubernetes-version)
- [Troubleshooting](#troubleshooting)
### How Upgrades Work
In [this section,]({{<baseurl>}}/rke/latest/en/upgrades/how-upgrades-work) you'll learn what happens when you edit or upgrade your RKE Kubernetes cluster.
### Prerequisites
- Ensure that any `system_images` configuration is absent from the `cluster.yml`. The Kubernetes version should only be listed under the `system_images` directive if an [unsupported version](#using-an-unsupported-kubernetes-version) is being used. Refer to [Kubernetes version precedence](#kubernetes-version-precedence) for more information.
@@ -63,6 +69,18 @@ Then invoke `rke up`:
$ rke up --config cluster.yml
```
### Configuring the Upgrade Strategy
As of v0.1.8, upgrades to add-ons are supported. [Add-ons]({{<baseurl>}}/rke/latest/en/config-options/add-ons/) can also be upgraded by changing any of the add-ons and running `rke up` again with the updated configuration file.
As of v1.1.0, additional upgrade options became available to give you more granular control over the upgrade process. These options can be used to maintain availability of your applications during a cluster upgrade.
For details on upgrade configuration options, refer to [Configuring the Upgrade Strategy.]({{<baseurl>}}/rke/latest/en/upgrades/configuring-strategy)
### Maintaining Availability for Applications During Upgrades
In [this section,]({{<baseurl>}}/rke/latest/en/upgrades/maintaining-availability/) you'll learn the requirements to prevent downtime for your applications when you upgrade the cluster using `rke up`.
### Listing Supported Kubernetes Versions
Please refer to the [release notes](https://github.com/rancher/rke/releases) of the RKE version that you are running, to find the list of supported Kubernetes versions as well as the default Kubernetes version.
@@ -105,16 +123,6 @@ For RKE prior to v0.3.0, the service defaults are located [here](https://github.
> **Note:** The following arguments, `service_cluster_ip_range` or `cluster_cidr`, cannot be changed as any changes to these arguments will result in a broken cluster. Currently, network pods are not automatically upgraded.
### Configuring the Upgrade Strategy
As of v0.1.8, upgrades to add-ons are supported. [Add-ons]({{<baseurl>}}/rke/latest/en/config-options/add-ons/) can also be upgraded by changing any of the add-ons and running `rke up` again with the updated configuration file.
As of v1.1.0, additional upgrade options became available to give you more granular control over the upgrade process. These options can be used to maintain availability of your applications during a cluster upgrade.
For details on upgrade configuration options, refer to [Configuring the Upgrade Strategy.]({{<baseurl>}}/rke/latest/en/upgrades/configuring-strategy)
For the requirements to maintain availability of applications during a cluster upgrade, refer to [this section.]({{<baseurl>}}/rke/latest/en/upgrades/maintaining-availability)
### Upgrading Nodes Manually
_Available as of v1.1.0_
content/rke/latest/en/upgrades/how-upgrades-work/_index.md
+3 -8
View File
@@ -3,7 +3,7 @@ title: How Upgrades Work
weight: 1
---
In this section, you'll learn how each type of node is upgraded by default when a cluster is upgraded using `rke up`.
In this section, you'll learn what happens when you edit or upgrade your RKE Kubernetes cluster. The below sections describe how each type of node is upgraded by default when a cluster is upgraded using `rke up`.
{{% tabs %}}
{{% tab "RKE v1.1.0+" %}}
@@ -23,11 +23,6 @@ When a cluster is upgraded with `rke up`, using the default options, the followi
The following sections break down in more detail what happens when etcd nodes, controlplane nodes, worker nodes, and addons are upgraded. This information is intended to be used to help you understand the update strategy for the cluster, and may be useful when troubleshooting problems with upgrading the cluster.
- [Upgrades of etcd nodes](#upgrades-of-etcd-nodes)
- [Upgrades of controlplane nodes](#upgrades-of-controlplane-nodes)
- [Upgrades of worker nodes](#upgrades-of-worker-nodes)
- [Upgrades of addons](#upgrades-of-addons)
### Upgrades of etcd Nodes
A cluster upgrade begins by upgrading the etcd nodes one at a time.
@@ -62,11 +57,11 @@ RKE will handle all worker node upgrades before upgrading any add-ons. As long a
The availability of your applications partly depends on the availability of [RKE addons.]({{<baseurl>}}/rke/latest/en/config-options/add-ons/) Addons are used to deploy several cluster components, including network plug-ins, the Ingress controller, DNS provider, and metrics server.
Because RKE addons are necessary for allowing traffic into the cluster, a zero-downtime upgrade requires them to be updated in batches. You will need to configure the maximum number of unavailable replicas for each addon in the `cluster.yml` to ensure that your cluster will retain enough available replicas during an upgrade.
Because RKE addons are necessary for allowing traffic into the cluster, they will need to be updated in batches to maintain availability. You will need to configure the maximum number of unavailable replicas for each addon in the `cluster.yml` to ensure that your cluster will retain enough available replicas during an upgrade.
For more information on configuring the number of replicas for each addon, refer to [this section.](#replicas-for-rke-addons)
For an example showing how to configure the addons for zero downtime, refer to the [example cluster.yml.](#example-cluster-yml)
For an example showing how to configure the addons, refer to the [example cluster.yml.](#example-cluster-yml)
{{% /tab %}}
{{% tab "RKE prior to v1.1.0" %}}
content/rke/latest/en/upgrades/maintaining-availability/_index.md
+3 -4
View File
@@ -29,16 +29,15 @@ The following must be true of the cluster that will be upgraded:
1. The cluster has three or more etcd nodes.
1. The cluster has two or more controlplane nodes.
1. The cluster has two or more worker nodes.
1. The Ingress, DNS, and other addons are schedulable to a number of nodes that exceeds the maximum number of unavailable worker nodes. By default, the minimum number of unavailable worker nodes is one.
1. The Ingress, DNS, and other addons are schedulable to a number of nodes that exceeds the maximum number of unavailable worker nodes, also called the batch size. By default, the minimum number of unavailable worker nodes is 10 percent of worker nodes, rounded down to the nearest node, with a minimum batch size of one node.
### 3. Workload Requirements
The following must be true of the cluster's applications:
1. The application and Ingress are deployed across a number of nodes exceeding the maximum number of unavailable worker nodes.
1. The application and Ingress are deployed across a number of nodes exceeding the maximum number of unavailable worker nodes, also called the batch size. By default, the minimum number of unavailable worker nodes is 10 percent of worker nodes, rounded down to the nearest node, with a minimum batch size of one node.
1. The applications must make use of liveness and readiness probes.
For information on how to use node selectors to assign pods to nodes, refer to the [official Kubernetes documentation.](https://kubernetes.io/docs/concepts/configuration/assign-pod-node/)
For information on configuring the number of replicas for each addon, refer to [this section.]({{<baseurl>}}/rke/latest/en/upgrades/configuring-strategy/)
For information on configuring the number of replicas for each addon, refer to [this section.]({{<baseurl>}}/rke/latest/en/upgrades/configuring-strategy/)