mirror of
https://github.com/rancher/rancher-docs.git
synced 2026-05-22 21:05:13 +00:00
Update docs for atomic rollback feature
This commit is contained in:
Catherine Luse
@@ -1,5 +1,5 @@
|
||||
---
|
||||
title: Backing up Cluster Data
|
||||
title: Backing up a Cluster
|
||||
weight: 2045
|
||||
---
|
||||
|
||||
@@ -14,34 +14,50 @@ Snapshots of the etcd database are taken and saved either [locally onto the etcd
|
||||
This section covers the following topics:
|
||||
|
||||
- [How snapshots work](#how-snapshots-work)
|
||||
- [Snapshot creation period and retention count](#snapshot-creation-period-and-retention-count)
|
||||
- [Configuring recurring snapshots for the cluster](#configuring-recurring-snapshots-for-the-cluster)
|
||||
- [Configuring recurring snapshots](#configuring-recurring-snapshots)
|
||||
- [One-time snapshots](#one-time-snapshots)
|
||||
- [Snapshot backup targets](#snapshot-backup-targets)
|
||||
- [Local backup target](#local-backup-target)
|
||||
- [S3 backup target](#s3-backup-target)
|
||||
- [Using a custom CA certificate for S3](#using-a-custom-ca-certificate-for-s3)
|
||||
- [IAM Support for storing snapshots in S3](#iam-support-for-storing-snapshots-in-s3)
|
||||
- [Viewing available snapshots](#viewing-available-snapshots)
|
||||
- [Safe timestamps](#safe-timestamps)
|
||||
- [Enabling snapshot features for clusters created before Rancher v2.2.0](#enabling-snapshot-features-for-clusters-created-before-Rancher-v2-2-0)
|
||||
- [Enabling snapshot features for clusters created before Rancher v2.2.0](#enabling-snapshot-features-for-clusters-created-before-rancher-v2-2-0)
|
||||
|
||||
# How Snapshots Work
|
||||
|
||||
{{% tabs %}}
|
||||
{{% tab "Rancher v2.4.0+" %}}
|
||||
When Rancher creates a snapshot, it includes three components:
|
||||
|
||||
- The cluster data in etcd
|
||||
- The Kubernetes version
|
||||
- The cluster configuration in the form of the `cluster.yml`
|
||||
|
||||
Because the Kubernetes version is now included in the snapshot, it is possible to restore a cluster to a prior Kubernetes version.
|
||||
|
||||
The multiple components of the snapshot allow you to select from the following options if you need to a cluster from a snapshot:
|
||||
|
||||
- **Restore just the etcd contents:** This restoration is similar to restoring to snapshots in Rancher prior to v2.4.0.
|
||||
- **Restore etcd and Kubernetes version:** This option should be used if a Kubernetes upgrade is the reason that your cluster is failing, and you haven't made any cluster configuration changes.
|
||||
- **Restore etcd, Kubernetes versions and cluster configuration:** This option should be used if you changed both the Kubernetes version and cluster configuration when upgrading.
|
||||
|
||||
It's always recommended to take a new snapshot before any upgrades.
|
||||
{{% /tab %}}
|
||||
{{% tab "Rancher prior to v2.4.0" %}}
|
||||
When Rancher creates a snapshot, only the etcd data is included in the snapshot.
|
||||
|
||||
Because the Kubernetes version is not included in the snapshot, there is no option to restore a cluster to a different Kubernetes version.
|
||||
|
||||
It's always recommended to take a new snapshot before any upgrades.
|
||||
{{% /tab %}}
|
||||
{{% /tabs %}}
|
||||
|
||||
# Snapshot Creation Period and Retention Count
|
||||
# Configuring Recurring Snapshots
|
||||
|
||||
Select how often you want recurring snapshots to be taken as well as how many snapshots to keep. The amount of time is measured in hours. With timestamped snapshots, the user has the ability to do a point-in-time recovery.
|
||||
|
||||
### Configuring Recurring Snapshots for the Cluster
|
||||
|
||||
By default, [Rancher launched Kubernetes clusters]({{<baseurl>}}/rancher/v2.x/en/cluster-provisioning/rke-clusters/) are configured to take recurring snapshots (saved to local disk). To protect against local disk failure, using the [S3 Target](#s3-backup-target) or replicating the path on disk is advised.
|
||||
|
||||
During cluster provisioning or editing the cluster, the configuration for snapshots can be found in the advanced section for **Cluster Options**. Click on **Show advanced options**.
|
||||
@@ -120,7 +136,7 @@ _Available as of v2.3.0_
|
||||
|
||||
As of v2.2.6, snapshot files are timestamped to simplify processing the files using external tools and scripts, but in some S3 compatible backends, these timestamps were unusable. As of Rancher v2.3.0, the option `safe_timestamp` is added to support compatible file names. When this flag is set to `true`, all special characters in the snapshot filename timestamp are replaced.
|
||||
|
||||
>>**Note:** This option is not available directly in the UI, and is only available through the `Edit as Yaml` interface.
|
||||
This option is not available directly in the UI, and is only available through the `Edit as Yaml` interface.
|
||||
|
||||
# Enabling Snapshot Features for Clusters Created Before Rancher v2.2.0
|
||||
|
||||
|
||||
@@ -7,9 +7,16 @@ _Available as of v2.2.0_
|
||||
|
||||
etcd backup and recovery for [Rancher launched Kubernetes clusters]({{<baseurl>}}/rancher/v2.x/en/cluster-provisioning/rke-clusters/) can be easily performed. Snapshots of the etcd database are taken and saved either locally onto the etcd nodes or to a S3 compatible target. The advantages of configuring S3 is that if all etcd nodes are lost, your snapshot is saved remotely and can be used to restore the cluster.
|
||||
|
||||
Rancher recommends enabling the [ability to set up recurring snapshots of etcd]({{<baseurl>}}/rancher/v2.x/en/cluster-admin/backing-up-etcd/#configuring-recurring-snapshots-for-the-cluster), but [one-time snapshots]({{<baseurl>}}/rancher/v2.x/en/cluster-admin/backing-up-etcd/#one-time-snapshots) can easily be taken as well. Rancher allows restore from [saved snapshots](#restoring-your-cluster-from-a-snapshot) or if you don't have any snapshots, you can still [restore etcd](#recovering-etcd-without-a-snapshot).
|
||||
Rancher recommends enabling the [ability to set up recurring snapshots of etcd]({{<baseurl>}}/rancher/v2.x/en/cluster-admin/backing-up-etcd/#configuring-recurring-snapshots), but [one-time snapshots]({{<baseurl>}}/rancher/v2.x/en/cluster-admin/backing-up-etcd/#one-time-snapshots) can easily be taken as well. Rancher allows restore from [saved snapshots](#restoring-a-cluster-from-a-snapshot) or if you don't have any snapshots, you can still [restore etcd](#recovering-etcd-without-a-snapshot).
|
||||
|
||||
>**Note:** If you have any Rancher launched Kubernetes clusters that were created prior to v2.2.0, after upgrading Rancher, you must [edit the cluster]({{<baseurl>}}/rancher/v2.x/en/cluster-admin/editing-clusters/) and _save_ it, in order to enable the [updated snapshot features]({{<baseurl>}}/rancher/v2.x/en/cluster-admin/backing-up-etcd/). Even if you were already creating snapshots prior to v2.2.0, you must do this step as the older snapshots will not be available to use to back up and restore etcd through the UI.
|
||||
As of Rancher v2.4.0, clusters can also be restored to a prior Kubernetes version and cluster configuration.
|
||||
|
||||
This section covers the following topics:
|
||||
|
||||
- [Viewing Available Snapshots](#viewing-available-snapshots)
|
||||
- [Restoring a Cluster from a Snapshot](#restoring-a-cluster-from-a-snapshot)
|
||||
- [Recovering etcd without a Snapshot](#recovering-etcd-without-a-snapshot)
|
||||
- [Enabling snapshot features for clusters created before Rancher v2.2.0](#enabling-snapshot-features-for-clusters-created-before-rancher-v2-2-0)
|
||||
|
||||
## Viewing Available Snapshots
|
||||
|
||||
@@ -19,21 +26,57 @@ The list of all available snapshots for the cluster is available.
|
||||
|
||||
2. Click **Tools > Snapshots** from the navigation bar to view the list of saved snapshots. These snapshots include a timestamp of when they were created.
|
||||
|
||||
## Restoring your Cluster from a Snapshot
|
||||
## Restoring a Cluster from a Snapshot
|
||||
|
||||
If your Kubernetes cluster is broken, you can restore the cluster from a snapshot.
|
||||
|
||||
1. In the **Global** view, navigate to the cluster that you want to view snapshots.
|
||||
Restorations changed in Rancher v2.4.0.
|
||||
|
||||
{{% tabs %}}
|
||||
{{% tab "Rancher v2.4.0+" %}}
|
||||
|
||||
Snapshots are composed of the cluster data in etcd, the Kubernetes version, and the cluster configuration in the `cluster.yml.` These components allow you can select from the following options when restoring a cluster from a snapshot:
|
||||
|
||||
- **Restore just the etcd contents:** This restoration is similar to restoring to snapshots in Rancher prior to v2.4.0.
|
||||
- **Restore etcd and Kubernetes version:** This option should be used if a Kubernetes upgrade is the reason that your cluster is failing, and you haven't made any cluster configuration changes.
|
||||
- **Restore etcd, Kubernetes versions and cluster configuration:** This option should be used if you changed both the Kubernetes version and cluster configuration when upgrading.
|
||||
|
||||
When rolling back to a prior Kubernetes version, the [upgrade strategy options]({{<baseurl>}}/rancher/v2.x/en/cluster-admin/upgrading-kubernetes/#configuring-the-upgrade-strategy) are ignored. Worker nodes are not cordoned or drained before being reverted to the older Kubernetes version, so that an unhealthy cluster can be more quickly restored to a healthy state.
|
||||
|
||||
> **Prerequisite:** To restore snapshots from S3, the cluster needs to be configured to [take recurring snapshots on S3.]({{<baseurl>}}/rancher/v2.x/en/cluster-admin/backing-up-etcd/#configuring-recurring-snapshots)
|
||||
|
||||
1. In the **Global** view, navigate to the cluster that you want to restore from a snapshots.
|
||||
|
||||
2. Click the **Vertical Ellipsis (...) > Restore Snapshot**.
|
||||
|
||||
3. Select the snapshot that you want to use for restoring your cluster from the dropdown of available snapshots. Click **Save**.
|
||||
3. Select the snapshot that you want to use for restoring your cluster from the dropdown of available snapshots.
|
||||
|
||||
> **Note:** Snapshots from S3 can only be restored from if the cluster is configured to take recurring snapshots on S3.
|
||||
4. In the **Restoration Type** field, choose one of the restoration options described above.
|
||||
|
||||
5. Click **Save**.
|
||||
|
||||
**Result:** The cluster will go into `updating` state and the process of restoring the `etcd` nodes from the snapshot will start. The cluster is restored when it returns to an `active` state.
|
||||
|
||||
> **Note:** If you are restoring a cluster with unavailable etcd nodes, it's recommended that all etcd nodes are removed from Rancher before attempting to restore. For clusters that were provisioned using [nodes hosted in an infrastructure provider]({{<baseurl>}}/rancher/v2.x/en/cluster-provisioning/rke-clusters/node-pools/), new etcd nodes will automatically be created. For [custom clusters]({{<baseurl>}}/rancher/v2.x/en/cluster-provisioning/rke-clusters/custom-nodes/), please ensure that you add new etcd nodes to the cluster.
|
||||
{{% /tab %}}
|
||||
{{% tab "Rancher prior to v2.4.0" %}}
|
||||
|
||||
> **Prerequisites:**
|
||||
>
|
||||
> - Make sure your etcd nodes are healthy. If you are restoring a cluster with unavailable etcd nodes, it's recommended that all etcd nodes are removed from Rancher before attempting to restore. For clusters in which Rancher used node pools to provision [nodes in an infrastructure provider]({{<baseurl>}}/rancher/v2.x/en/cluster-provisioning/rke-clusters/node-pools/), new etcd nodes will automatically be created. For [custom clusters]({{<baseurl>}}/rancher/v2.x/en/cluster-provisioning/rke-clusters/custom-nodes/), please ensure that you add new etcd nodes to the cluster.
|
||||
> - To restore snapshots from S3, the cluster needs to be configured to [take recurring snapshots on S3.]({{<baseurl>}}/rancher/v2.x/en/cluster-admin/backing-up-etcd/#configuring-recurring-snapshots)
|
||||
|
||||
1. In the **Global** view, navigate to the cluster that you want to restore from a snapshot.
|
||||
|
||||
2. Click the **Vertical Ellipsis (...) > Restore Snapshot**.
|
||||
|
||||
3. Select the snapshot that you want to use for restoring your cluster from the dropdown of available snapshots.
|
||||
|
||||
4. Click **Save**.
|
||||
|
||||
**Result:** The cluster will go into `updating` state and the process of restoring the `etcd` nodes from the snapshot will start. The cluster is restored when it returns to an `active` state.
|
||||
|
||||
{{% /tab %}}
|
||||
{{% /tabs %}}
|
||||
|
||||
## Recovering etcd without a Snapshot
|
||||
|
||||
@@ -64,3 +107,7 @@ If the group of etcd nodes loses quorum, the Kubernetes cluster will report a fa
|
||||
5. Run the revised command.
|
||||
|
||||
6. After the single nodes is up and running, Rancher recommends adding additional etcd nodes to your cluster. If you have a [custom cluster]({{<baseurl>}}/rancher/v2.x/en/cluster-provisioning/custom-clusters/) and you want to reuse an old node, you are required to [clean up the nodes]({{<baseurl>}}/rancher/v2.x/en/faq/cleaning-cluster-nodes/) before attempting to add them back into a cluster.
|
||||
|
||||
# Enabling Snapshot Features for Clusters Created Before Rancher v2.2.0
|
||||
|
||||
If you have any Rancher launched Kubernetes clusters that were created prior to v2.2.0, after upgrading Rancher, you must [edit the cluster]({{<baseurl>}}/rancher/v2.x/en/cluster-admin/editing-clusters/) and _save_ it, in order to enable the updated snapshot features. Even if you were already creating snapshots prior to v2.2.0, you must do this step as the older snapshots will not be available to use to [back up and restore etcd through the UI]({{<baseurl>}}/rancher/v2.x/en/cluster-admin/restoring-etcd/).
|
||||
@@ -87,9 +87,10 @@ The cluster cannot be downgraded to a previous Kubernetes version.
|
||||
|
||||
_Available as of v2.4_
|
||||
|
||||
A cluster can be restored to a backup in which the previous Kubernetes version was used.
|
||||
A cluster can be restored to a backup in which the previous Kubernetes version was used. For more information, refer to the following sections:
|
||||
|
||||
RKE v1.1.0 introduced the ability to restore a Kubernetes cluster to an older Kubernetes version by restoring it to a snapshot that includes the older Kubernetes version.
|
||||
- [Backing up a cluster]({{<baseurl>}}/rancher/v2.x/en/cluster-admin/backing-up-etcd/#how-snapshots-work)
|
||||
- [Restoring a cluster from backup]({{<baseurl>}}/rancher/v2.x/en/cluster-admin/restoring-etcd/restoring-a-cluster-from-a-snapshot)
|
||||
|
||||
# Configuring the Upgrade Strategy
|
||||
|
||||
@@ -128,6 +129,8 @@ To enable draining each node during a cluster upgrade,
|
||||
|
||||
**Result:** The cluster is updated to use the new upgrade strategy.
|
||||
|
||||
> **Note:** As of Rancher v2.4.0, there is a known issue in which the Rancher UI doesn't show state of nodes as drained, even though they are being drained.
|
||||
|
||||
### Maintaining Availability for Applications During Upgrades
|
||||
|
||||
_Available as of RKE v1.1.0_
|
||||
|
||||
Reference in New Issue
Block a user