public/rancher-docs
1
0
Fork 0
mirror of https://github.com/rancher/rancher-docs.git synced 2026-08-27 04:22:18 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge pull request #556 from MBishop17/rollbacks-merge

Browse Source 
Rollbacks merge
This commit is contained in:
Denise
2018-07-30 10:47:20 -07:00
committed by GitHub
parent 54e6b6bc0b 8c073124d9
commit d71f10aa6c
10 changed files with 174 additions and 176 deletions
Download Patch File Download Diff File Expand all files Collapse all files
content/rancher/v2.x/en/backups/_index.md
+3 -6
View File
@@ -5,14 +5,11 @@ weight: 1000
This section is devoted to protecting your Rancher Server data in a disaster scenario.
- [Backups](./backups/)
- [Backups]({{< baseurl >}}/rancher/v2.x/en/backups/backups)
To protect yourself from a disaster scenario, you should create Rancher backups on a regular basis.
- [Restorations](./restorations/)
In a disaster scenario, you can restore your `etcd` database by restoring a backup.
- [Rollbacks]({{< baseurl >}}/rancher/v2.x/en/backups/rollbacks)
- [Rollbacks](./rollbacks/)
If an upgrade does not complete successfully, you can revert you Rancher Server to its previous version along with its data prior to upgrade.
If you experience a disaster scenario, or an upgrade does not complete successfully, you can restore Rancher to its last healthy backup.
content/rancher/v2.x/en/backups/restorations/_index.md
-8
View File
@@ -1,8 +0,0 @@
---
title: Restorations
weight: 1010
---
If you lose the data on your Rancher Server, you can restore it if you have backups stored in a safe location.
- [Restoring Backups—Single Node Installs](./single-node-restoration/)
- [Restoring Backups—High Availablity Installs](./ha-restoration/)
content/rancher/v2.x/en/backups/restorations/ha-restoration/_index.md
-115
View File
@@ -1,115 +0,0 @@
---
title: Restoring Backups—High Availability Installs
weight: 370
aliases:
- /rancher/v2.x/en/installation/after-installation/ha-backup-and-restoration/
---
This procedure describes how to restore your a snapshot of `etcd` if you lose your Rancher data in a disaster scenario.
## Restoration Outline
Following a disaster scenario, restoration of your HA Rancher installation requires you to pull your snapshot from your chosen external location and then restore it.
1. [Create New Node and Pull Snapshot](#1-create-new-node-and-pull-snapshot)
If one of your `etcd` nodes goes down, create a new node, and then pull the most recent `etcd` snapshot to that node.
2. [Restore etcd Database](#2-restore-etcd-database)
After you pull the snapshot, run the RKE command to restore the `etcd` database.
<br/>
### 1. Create New Node and Pull Snapshot
If one of your `etcd` nodes go down, you need to replace it with a new node, and then pull the most recent working `etcd` snapshot to that node.
**To Create a New Node and Pull the Latest Snapshot:**
1. Create a new node of your choice—baremetal, on-prem virtual machine, cloud-based virtual machine, and so on. Provision it according to our [requirements]({{< baseurl >}}/rancher/v2.x/en/installation/ha-server-install/#host-requirements).
2. Log in to your new node using a remote Terminal connection.
3. Create a directory that mirrors your other nodes' snapshot directories:
```
root@newnode:~# mkdir -p /opt/rke/etcd-snapshots
```
4. Pull your most recent snapshot onto the node. Replace `<SNAPSHOT.db>` with the name of the snapshot you're restoring to.
```
root@newnode:~# s3cmd get s3://rke-etcd-snapshots/<SNAPSHOT.db> /opt/rke/etcd-snapshots/<SNAPSHOT.db>
```
>**Remember:** Our use of Amazon S3 is an example used for this documentation. The command for pulling your snapshot may vary.
After restoring the cluster you have to restart the kubernetes components on all nodes, otherwise there will be some conflicts with resource versions of objects stored in `etcd`, this will include restart to kubernetes components and the network components, for more information please refer to [kubernetes documentation](https://kubernetes.io/docs/tasks/administer-cluster/configure-upgrade-etcd/#etcd-upgrade-requirements), to do that you can run the following on each node:
```
docker restart kube-apiserver kubelet kube-controller-manager kube-scheduler kube-proxy
docker ps | grep flannel | cut -f 1 -d " " | xargs docker restart
docker ps | grep calico | cut -f 1 -d " " | xargs docker restart
```
### 2. Restore `etcd` Database
To restore the most recent `etcd` snapshot on your new node, run RKE the command `rke etcd snapshot-restore`. This command reverts to any snapshot stored in `/opt/rke/etcd-snapshots` that you explicitly define. When you run `rke etcd snapshot-restore`, RKE removes the old `etcd` container if it still exists. To restore operations, RKE creates a new `etcd` cluster using the snapshot you choose.
>**Warning:** Restoring an `etcd` snapshot deletes your current `etcd` cluster and replaces it with a new one. Before you run the `rke etcd snapshot-restore` command, backup any important data in your current cluster.
>
>**Prerequisites:** {{< requirements_rollback >}}
1. From your workstation, open `rancher-cluster.yml` in your favorite text editor.
2. Replace the unresponsive node (`3.3.3.3` in this example) with your new one (`4.4.4.4`). You IP addresses will be different obviously:
nodes:
- address: 1.1.1.1
user: root
role: [controlplane,etcd,worker]
ssh_key_path: ~/.ssh/id_rsa
- address: 2.2.2.2
user: root
role: [controlplane,etcd,worker]
ssh_key_path: ~/.ssh/id_rsa
# - address: 3.3.3.3 # UNRESPONSIVE NODE
# user: root
# role: [controlplane,etcd,worker]
# ssh_key_path: ~/.ssh/id_rsa
- address: 4.4.4.4 # NEW NODE
user: root
role: [controlplane,etcd,worker]
ssh_key_path: ~/.ssh/id_rsa
3. Save and close `rancher-cluster.yml`.
4. Open **Terminal** and change directory to the location of the RKE binary. Your `rancher-cluster.yml` file must reside in the same directory.
5. Run one of the following commands to restore the `etcd` database:
```
# MacOS
./rke_darwin-amd64 etcd snapshot-restore --name <SNAPSHOT.db> --config rancher-cluster.yml
# Linux
./rke_linux-amd64 etcd snapshot-restore --name <SNAPSHOT.db> --config rancher-cluster.yml
```
6. Run one of the following commands to bring your cluster back up:
```
# MacOS
./rke_darwin-amd64 up --config rancher-cluster.yml
# Linux
./rke_linux-amd64 up --config rancher-cluster.yml
```
7. Lastly, restart the Kubernetes components on all cluster nodes to prevent potential `etcd` conflicts. Run this command on each of your nodes.
```
docker restart kube-apiserver kubelet kube-controller-manager kube-scheduler kube-proxy
docker ps | grep flannel | cut -f 1 -d " " | xargs docker restart
docker ps | grep calico | cut -f 1 -d " " | xargs docker restart
```
content/rancher/v2.x/en/backups/restorations/single-node-restoration/_index.md
-31
View File
@@ -1,31 +0,0 @@
---
title: Restoring Backups—Single Node Installs
shortTitle: Singe Node Installs
weight: 365
aliases:
- /rancher/v2.x/en/installation/after-installation/single-node-backup-and-restoration/
---
Restoring to a backup for your Rancher install if you encounter issues in your Rancher setup.
1. Stop the container currently running Rancher Server. Replace `<RANCHER_CONTAINER_ID>` with the ID of your Rancher container.
```
docker stop <RANCHER_CONTAINER_ID>
```
2. Go to the location where you saved your [backup tar balls]({{< baseurl >}}/rancher/v2.x/en/backups/backups/single-node-backups/#backup). Run the following command to delete your current state data and start your backup data:
```
docker run --volumes-from <RANCHER_CONTAINER_ID> -v $PWD:/backup \
alpine sh -c "rm /var/lib/rancher/* -rf && \
tar zxvf /backup/<BACKUP_FILENAME>.tar.gz"
```
>**Warning!** Running this command will delete ALL current state data from your Rancher Server container. Any changes that happened after the backup point you are restoring will be lost.
3. Start you rancher server container back. The container will start with the data from the restored backup.
```
docker start <RANCHER_CONTAINER_ID>
```
content/rancher/v2.x/en/backups/rollbacks/_index.md
+2 -2
View File
@@ -4,7 +4,7 @@ shortTitle: Rollbacks
weight: 1010
---
If you upgrade your Rancher Server and the upgrade is unsuccessful, you can use the backup you created during the upgrade process to restore the previous version of Rancher, along with its data.
If you encounter a disaster scenario where you lose Rancher or its data, you can recover by restoring to your most recent backup. This process will return your cluster to a healthy state, with minimal data loss.
- [Rolling Back—Single Node Installs](./single-node-rollbacks/)
- [Rolling Back—High Availablity Installs](./ha-server-rollbacks/)
- [Rolling Back—High Availability Installs](./ha-server-rollbacks/)
content/rancher/v2.x/en/backups/rollbacks/ha-server-rollbacks/_index.md
+121 -6
View File
@@ -1,25 +1,37 @@
---
title: Rolling Back—High Availability Installs
title: Roll Backs—High Availability Installs
weight: 50
aliases:
- /rancher/v2.x/en/upgrades/rollbacks/ha-server-rollbacks/
- /rancher/v2.x/en/backups/restorations/ha-restoration
---
There are two scenarios where rolling back to a Rancher backup may be required.
<!-- TOC -->
- [Recovering from an Unsuccessful Upgrade](#recovering-from-an-unsuccessful-upgrade): In this scenario, you restore both the prior installation of Rancher and your backup data.
- [Recovering from a Disaster Scenario](#recovering-from-a-disaster-scenario): In this scenario, you replace a problematic node with a new one, and then restore your backup data.
<!-- /TOC -->
## Recovering from an Unsuccessful Upgrade
If you upgrade Rancher, but the upgrade does not complete successfully, you may need to roll back your Rancher Server to its last healthy state before upgrade. To restore Rancher:
- Restore the `etcd` snapshot taken before upgrade.
- Restore the etcd snapshot taken before upgrade.
- Run the command to revert to your prior version of Rancher.
>**Warning!**
>
> Rolling back to a previous version of Rancher destroys any changes made to Rancher following the upgrade. Unrecoverable data loss may occur.
>
>
>**Prerequisites:** {{< requirements_rollback >}}
1. Open **Terminal** and change directory to the location of the RKE binary that you used during upgrade.
1. Run one of the following commands to restore the `etcd` snapshot [that you took]({{< baseurl >}}/rancher/v2.x/en/upgrades/upgrade-scenarios/ha-server-upgrade/#snapshot) before your unsuccessful upgrade. Replace the `<SNAPSHOT.db>` placeholder with your snapshot.
1. Run one of the following commands to restore the etcd snapshot [that you took]({{< baseurl >}}/rancher/v2.x/en/upgrades/upgrade-scenarios/ha-server-upgrade/#snapshot) before your unsuccessful upgrade. Replace the `<SNAPSHOT.db>` placeholder with your snapshot.
```
# MacOS
@@ -37,7 +49,7 @@ If you upgrade Rancher, but the upgrade does not complete successfully, you may
./rke_linux-amd64 up --config rancher-cluster.yml
```
1. Lastly, restart the Kubernetes components on all cluster nodes to prevent potential `etcd` conflicts. Run this command on each of your nodes.
1. Restart the Kubernetes components on all cluster nodes to prevent potential etcd conflicts. Run this command on each of your nodes.
```
docker restart kube-apiserver kubelet kube-controller-manager kube-scheduler kube-proxy
@@ -53,4 +65,107 @@ kubectl --kubeconfig=kube_config-rancher-cluster.yml set image deployment/cattle
**Step Result:** The rollback begins. Rancher Server may be unavailable for a few minutes.
1. Log into Rancher. Confirm that the rollback succeeded by checking the version displayed in the bottom-left corner of the browser window.
1. Log into Rancher. Confirm that the rollback succeeded by checking the version displayed in the bottom-left corner of the browser window.
## Recovering from a Disaster Scenario
If you experience a disaster scenario where one of your cluster nodes fails, you'll need to replace the node and rollback to backup data.
- Create a new node and image it with a backup snapshot.
- Restore the etcd snapshot last taken before failure.
### 1. Create New Node and Pull Snapshot
If one of your etcd nodes go down, you need to replace it with a new node, and then pull the most recent working etcd snapshot to that node.
**To Create a New Node and Pull the Latest Snapshot:**
1. Create a new node of your choice—baremetal, on-prem virtual machine, cloud-based virtual machine, and so on. Provision it according to our [requirements]({{< baseurl >}}/rancher/v2.x/en/installation/ha-server-install/#host-requirements).
2. Log in to your new node using a remote Terminal connection.
3. Create a directory that mirrors your other nodes' snapshot directories:
```
root@newnode:~# mkdir -p /opt/rke/etcd-snapshots
```
4. Pull your most recent snapshot onto the node. Replace `<SNAPSHOT.db>` with the name of the snapshot you're restoring to.
```
root@newnode:~# s3cmd get s3://rke-etcd-snapshots/<SNAPSHOT.db> /opt/rke/etcd-snapshots/<SNAPSHOT.db>
```
>**Remember:** Our use of Amazon S3 is an example used for this documentation. The command for pulling your snapshot may vary.
After restoring the cluster, you have to restart the kubernetes components on all nodes. Otherwise there may be conflicts with resource versions of objects stored in etcd. This command restarts the kubernetes components and the network components. For more information please refer to the [Kubernetes documentation](https://kubernetes.io/docs/tasks/administer-cluster/configure-upgrade-etcd/#etcd-upgrade-requirements). To restart the components, run the following command on each node.
```
docker restart kube-apiserver kubelet kube-controller-manager kube-scheduler kube-proxy
docker ps | grep flannel | cut -f 1 -d " " | xargs docker restart
docker ps | grep calico | cut -f 1 -d " " | xargs docker restart
```
### 2. Restore `etcd` Database
To restore the most recent etcd snapshot on your new node, run the RKE command `rke etcd snapshot-restore`. This command reverts to any snapshot stored in `/opt/rke/etcd-snapshots` that you explicitly define. When you run `rke etcd snapshot-restore`, RKE removes the old etcd container if it still exists. To restore operations, RKE creates a new etcd cluster using the snapshot you choose.
>**Warning:** Restoring an etcd snapshot deletes your current etcd cluster and replaces it with a new one. Before you run the `rke etcd snapshot-restore` command, backup any important data in your current cluster.
>
>**Prerequisites:** {{< requirements_rollback >}}
1. From your workstation, open `rancher-cluster.yml` in your favorite text editor.
2. Replace the unresponsive node (`3.3.3.3` in this example) with your new one (`4.4.4.4`). You IP addresses will be different obviously:
nodes:
- address: 1.1.1.1
user: root
role: [controlplane,etcd,worker]
ssh_key_path: ~/.ssh/id_rsa
- address: 2.2.2.2
user: root
role: [controlplane,etcd,worker]
ssh_key_path: ~/.ssh/id_rsa
# - address: 3.3.3.3 # UNRESPONSIVE NODE
# user: root
# role: [controlplane,etcd,worker]
# ssh_key_path: ~/.ssh/id_rsa
- address: 4.4.4.4 # NEW NODE
user: root
role: [controlplane,etcd,worker]
ssh_key_path: ~/.ssh/id_rsa
3. Save and close `rancher-cluster.yml`.
4. Open **Terminal** and change directory to the location of the RKE binary. Your `rancher-cluster.yml` file must reside in the same directory.
5. Run one of the following commands to restore the etcd database:
```
# MacOS
./rke_darwin-amd64 etcd snapshot-restore --name <SNAPSHOT.db> --config rancher-cluster.yml
# Linux
./rke_linux-amd64 etcd snapshot-restore --name <SNAPSHOT.db> --config rancher-cluster.yml
```
6. Run one of the following commands to bring your cluster back up:
```
# MacOS
./rke_darwin-amd64 up --config rancher-cluster.yml
# Linux
./rke_linux-amd64 up --config rancher-cluster.yml
```
7. Restart the Kubernetes components on all cluster nodes to prevent potential etcd conflicts. Run this command on each of your nodes.
```
docker restart kube-apiserver kubelet kube-controller-manager kube-scheduler kube-proxy
docker ps | grep flannel | cut -f 1 -d " " | xargs docker restart
docker ps | grep calico | cut -f 1 -d " " | xargs docker restart
```
content/rancher/v2.x/en/backups/rollbacks/single-node-rollbacks/_index.md
+45 -5
View File
@@ -1,12 +1,25 @@
---
title: Rolling Back—Single Node Installs
title: Roll Backs—Single Node Installs
weight: 25
aliases:
- /rancher/v2.x/en/installation/after-installation/single-node-backup-and-restoration/
- /rancher/v2.x/en/backups/restorations/single-node-restoration/
---
Rolling back an unsuccessful Rancher upgrade requires you to
restore the backup you created while completing [Single Node Upgrade]({{< baseurl >}}/rancher/v2.x/en/upgrades/upgrade-scenarios/single-node-upgrade/).
There are two scenarios where rolling back to a recent backup may be required.
>**Warning!** Rolling back to a previous version of Rancher destroys any changes made to Rancher following the upgrade. Unrecoverable data loss may occur.
<!-- TOC -->
- [Recovering from an Unsuccessful Upgrade](#recovering-from-an-unsuccessful-upgrade): In this scenario, you restore the Rancher application and data to backups taken before upgrade.
- [Recovering from a Disaster Scenario](#recovering-from-a-disaster-scenario): In this scenario, you'll only need to restore your data.
<!-- /TOC -->
## Recovering from an Unsuccessful Upgrade
If you experience an unsuccessful upgrade, you can restore to backups you created during the [upgrade process]({{< baseurl >}}/rancher/v2.x/en/upgrades/upgrade-scenarios/single-node-upgrade/).
>**Warning!** Rolling back to a previous version destroys any changes made to Rancher following the upgrade. Unrecoverable data loss may occur.
1. Pull the version of Rancher that you were running prior to upgrade.
@@ -20,7 +33,7 @@ docker pull rancher/rancher:<PRIOR_VERSION>
docker stop <RANCHER_CONTAINER_ID>
```
3. Go to the location where you saved your [backup tar balls]({{< baseurl>}}/rancher/v2.x/en/upgrades/single-node-upgrade/#backup). Run the following command to create a data container from the backup tar ball.
3. Go to the location where you saved your [backup tarballs]({{< baseurl>}}/rancher/v2.x/en/upgrades/single-node-upgrade/#backup). Run the following command to create a data container from the backup tarball.
```
docker run --volumes-from rancher-data-<PRIOR_VERSION> \
@@ -34,3 +47,30 @@ docker run -d --volumes-from rancher-data-<PRIOR_VERSION> \
--restart=unless-stopped -p 80:80 -p 443:443 rancher/rancher:<PRIOR_VERSION>
```
>**Note:** _Do not_ stop the rollback after initiating it, even if the rollback process seems longer than expected. Stopping the rollback may result in database issues during future upgrades.
## Recovering from a Disaster Scenario
If you experience a disaster scenario, you can restore to your most recent backup with minimal data loss.
1. Stop the container currently running Rancher Server. Replace `<RANCHER_CONTAINER_ID>` with the ID of your Rancher container.
```
docker stop <RANCHER_CONTAINER_ID>
```
2. Go to the location where you saved your [backup tarballs]({{< baseurl >}}/rancher/v2.x/en/backups/backups/single-node-backups/#backup). Run the following command to delete your current state data and start your backup data:
```
docker run --volumes-from <RANCHER_CONTAINER_ID> -v $PWD:/backup \
alpine sh -c "rm /var/lib/rancher/* -rf && \
tar zxvf /backup/<BACKUP_FILENAME>.tar.gz"
```
>**Warning!** Running this command will delete ALL current state data from your Rancher Server container. Any changes that happened after the backup point you are restoring will be lost.
3. Start your Rancher Server container. The container will start with the data from the restored backup.
```
docker start <RANCHER_CONTAINER_ID>
```
content/rancher/v2.x/en/installation/ha-server-install-external-lb/_index.md
+1 -1
View File
@@ -331,7 +331,7 @@ During installation, RKE automatically generates a config file named `kube_confi
You have a couple of options:
- Create a backup of your Rancher Server in case of a disaster scenario: [High Availablility Back Up and Restoration]({{< baseurl >}}/rancher/v2.x/en/installation/backups-and-restoration/ha-backup-and-restoration).
- Create a backup of your Rancher Server in case of a disaster scenario: [High Availablility Back Up and Restoration]({{< baseurl >}}/rancher/v2.x/en/backups/backups/ha-backups).
- Create a Kubernetes cluster: [Creating a Cluster]({{< baseurl >}}/rancher/v2.x/en/tasks/clusters/creating-a-cluster/).
<br/>
content/rancher/v2.x/en/installation/single-node-install/_index.md
+1 -1
View File
@@ -164,7 +164,7 @@ Run the following commands from your Linux host.
You have a couple of options:
- Create a backup of your Rancher Server in case of a disaster scenario: [Single Node Backup and Restoration]({{< baseurl >}}/rancher/v2.x/en/installation/backups-and-restoration/single-node-backup-and-restoration/).
- Create a backup of your Rancher Server in case of a disaster scenario: [Single Node Backup and Restoration]({{< baseurl >}}/rancher/v2.x/en/backups/rollbacks/single-node-rollbacks).
- Create a Kubernetes cluster: [Provisioning Kubernetes Clusters]({{< baseurl >}}/rancher/v2.x/en/cluster-provisioning/).
<br/>
content/rancher/v2.x/en/upgrades/single-node-upgrade/_index.md
+1 -1
View File
@@ -66,4 +66,4 @@ docker run -d --volumes-from rancher-data --restart=unless-stopped \
**Result:** Rancher Server is upgraded to the latest version.
>**Note:** If your upgrade does not complete successfully, you can roll Rancher Server and its data back to its last healthy state. For more information, see [Restoring Backups—Single Node Installs]({{< baseurl >}}/rancher/v2.x/en/backups/restorations/single-node-restoration/).
>**Note:** If your upgrade does not complete successfully, you can roll Rancher Server and its data back to its last healthy state. For more information, see [Restoring Backups—Single Node Installs]({{< baseurl >}}/rancher/v2.x/en/backups/rollbacks/single-node-rollbacks).