public/rancher-docs
1
0
Fork 0
mirror of https://github.com/rancher/rancher-docs.git synced 2026-09-25 12:38:05 +00:00
Code Issues Packages Projects Releases Wiki Activity

merged rollback procedures into single document

Browse Source
This commit is contained in:
Mark Bishop
2018-07-29 14:18:40 -07:00
parent 54e6b6bc0b
commit 622ef48f3b
6 changed files with 158 additions and 163 deletions
Download Patch File Download Diff File Expand all files Collapse all files
content/rancher/v2.x/en/backups/_index.md
+3 -6
View File
@@ -5,14 +5,11 @@ weight: 1000
This section is devoted to protecting your Rancher Server data in a disaster scenario.
- [Backups](./backups/)
- [Backups]({{< baseurl >}}/rancher/v2.x/en/backups/backups)
To protect yourself from a disaster scenario, you should create Rancher backups on a regular basis.
- [Restorations](./restorations/)
In a disaster scenario, you can restore your `etcd` database by restoring a backup.
- [Rollbacks]({{< baseurl >}}/rancher/v2.x/en/backups/rollbacks)
- [Rollbacks](./rollbacks/)
If an upgrade does not complete successfully, you can revert you Rancher Server to its previous version along with its data prior to upgrade.
If you experience a disaster scenario, or an upgrade does not complete successfully, you can restore Rancher to its last healthy backup.
content/rancher/v2.x/en/backups/restorations/_index.md
-8
View File
@@ -1,8 +0,0 @@
---
title: Restorations
weight: 1010
---
If you lose the data on your Rancher Server, you can restore it if you have backups stored in a safe location.
- [Restoring Backups—Single Node Installs](./single-node-restoration/)
- [Restoring Backups—High Availablity Installs](./ha-restoration/)
content/rancher/v2.x/en/backups/restorations/ha-restoration/_index.md
-115
View File
@@ -1,115 +0,0 @@
---
title: Restoring Backups—High Availability Installs
weight: 370
aliases:
- /rancher/v2.x/en/installation/after-installation/ha-backup-and-restoration/
---
This procedure describes how to restore your a snapshot of `etcd` if you lose your Rancher data in a disaster scenario.
## Restoration Outline
Following a disaster scenario, restoration of your HA Rancher installation requires you to pull your snapshot from your chosen external location and then restore it.
1. [Create New Node and Pull Snapshot](#1-create-new-node-and-pull-snapshot)
If one of your `etcd` nodes goes down, create a new node, and then pull the most recent `etcd` snapshot to that node.
2. [Restore etcd Database](#2-restore-etcd-database)
After you pull the snapshot, run the RKE command to restore the `etcd` database.
<br/>
### 1. Create New Node and Pull Snapshot
If one of your `etcd` nodes go down, you need to replace it with a new node, and then pull the most recent working `etcd` snapshot to that node.
**To Create a New Node and Pull the Latest Snapshot:**
1. Create a new node of your choice—baremetal, on-prem virtual machine, cloud-based virtual machine, and so on. Provision it according to our [requirements]({{< baseurl >}}/rancher/v2.x/en/installation/ha-server-install/#host-requirements).
2. Log in to your new node using a remote Terminal connection.
3. Create a directory that mirrors your other nodes' snapshot directories:
```
root@newnode:~# mkdir -p /opt/rke/etcd-snapshots
```
4. Pull your most recent snapshot onto the node. Replace `<SNAPSHOT.db>` with the name of the snapshot you're restoring to.
```
root@newnode:~# s3cmd get s3://rke-etcd-snapshots/<SNAPSHOT.db> /opt/rke/etcd-snapshots/<SNAPSHOT.db>
```
>**Remember:** Our use of Amazon S3 is an example used for this documentation. The command for pulling your snapshot may vary.
After restoring the cluster you have to restart the kubernetes components on all nodes, otherwise there will be some conflicts with resource versions of objects stored in `etcd`, this will include restart to kubernetes components and the network components, for more information please refer to [kubernetes documentation](https://kubernetes.io/docs/tasks/administer-cluster/configure-upgrade-etcd/#etcd-upgrade-requirements), to do that you can run the following on each node:
```
docker restart kube-apiserver kubelet kube-controller-manager kube-scheduler kube-proxy
docker ps | grep flannel | cut -f 1 -d " " | xargs docker restart
docker ps | grep calico | cut -f 1 -d " " | xargs docker restart
```
### 2. Restore `etcd` Database
To restore the most recent `etcd` snapshot on your new node, run RKE the command `rke etcd snapshot-restore`. This command reverts to any snapshot stored in `/opt/rke/etcd-snapshots` that you explicitly define. When you run `rke etcd snapshot-restore`, RKE removes the old `etcd` container if it still exists. To restore operations, RKE creates a new `etcd` cluster using the snapshot you choose.
>**Warning:** Restoring an `etcd` snapshot deletes your current `etcd` cluster and replaces it with a new one. Before you run the `rke etcd snapshot-restore` command, backup any important data in your current cluster.
>
>**Prerequisites:** {{< requirements_rollback >}}
1. From your workstation, open `rancher-cluster.yml` in your favorite text editor.
2. Replace the unresponsive node (`3.3.3.3` in this example) with your new one (`4.4.4.4`). You IP addresses will be different obviously:
nodes:
- address: 1.1.1.1
user: root
role: [controlplane,etcd,worker]
ssh_key_path: ~/.ssh/id_rsa
- address: 2.2.2.2
user: root
role: [controlplane,etcd,worker]
ssh_key_path: ~/.ssh/id_rsa
# - address: 3.3.3.3 # UNRESPONSIVE NODE
# user: root
# role: [controlplane,etcd,worker]
# ssh_key_path: ~/.ssh/id_rsa
- address: 4.4.4.4 # NEW NODE
user: root
role: [controlplane,etcd,worker]
ssh_key_path: ~/.ssh/id_rsa
3. Save and close `rancher-cluster.yml`.
4. Open **Terminal** and change directory to the location of the RKE binary. Your `rancher-cluster.yml` file must reside in the same directory.
5. Run one of the following commands to restore the `etcd` database:
```
# MacOS
./rke_darwin-amd64 etcd snapshot-restore --name <SNAPSHOT.db> --config rancher-cluster.yml
# Linux
./rke_linux-amd64 etcd snapshot-restore --name <SNAPSHOT.db> --config rancher-cluster.yml
```
6. Run one of the following commands to bring your cluster back up:
```
# MacOS
./rke_darwin-amd64 up --config rancher-cluster.yml
# Linux
./rke_linux-amd64 up --config rancher-cluster.yml
```
7. Lastly, restart the Kubernetes components on all cluster nodes to prevent potential `etcd` conflicts. Run this command on each of your nodes.
```
docker restart kube-apiserver kubelet kube-controller-manager kube-scheduler kube-proxy
docker ps | grep flannel | cut -f 1 -d " " | xargs docker restart
docker ps | grep calico | cut -f 1 -d " " | xargs docker restart
```
content/rancher/v2.x/en/backups/restorations/single-node-restoration/_index.md
-31
View File
@@ -1,31 +0,0 @@
---
title: Restoring Backups—Single Node Installs
shortTitle: Singe Node Installs
weight: 365
aliases:
- /rancher/v2.x/en/installation/after-installation/single-node-backup-and-restoration/
---
Restoring to a backup for your Rancher install if you encounter issues in your Rancher setup.
1. Stop the container currently running Rancher Server. Replace `<RANCHER_CONTAINER_ID>` with the ID of your Rancher container.
```
docker stop <RANCHER_CONTAINER_ID>
```
2. Go to the location where you saved your [backup tar balls]({{< baseurl >}}/rancher/v2.x/en/backups/backups/single-node-backups/#backup). Run the following command to delete your current state data and start your backup data:
```
docker run --volumes-from <RANCHER_CONTAINER_ID> -v $PWD:/backup \
alpine sh -c "rm /var/lib/rancher/* -rf && \
tar zxvf /backup/<BACKUP_FILENAME>.tar.gz"
```
>**Warning!** Running this command will delete ALL current state data from your Rancher Server container. Any changes that happened after the backup point you are restoring will be lost.
3. Start you rancher server container back. The container will start with the data from the restored backup.
```
docker start <RANCHER_CONTAINER_ID>
```
content/rancher/v2.x/en/backups/rollbacks/ha-server-rollbacks/_index.md
+115 -2
View File
@@ -1,10 +1,21 @@
---
title: Rolling Back—High Availability Installs
title: Roll Backs—High Availability Installs
weight: 50
aliases:
- /rancher/v2.x/en/upgrades/rollbacks/ha-server-rollbacks/
---
There are two scenarios where you'll need to rollback to prior image of Rancher. The instructions change slightly for each scenario.
<!-- TOC -->
- [Recovering from an Unsuccessful Upgrade](#recovering-from-an-unsuccessful-upgrade): In this scenario, you restore both the prior installation of Rancher and your backup data.
- [Recovering from a Disaster Scenario](#recovering-from-a-disaster-scenario): In this scenario, you replace a problematic node with a new one, and then restore your backup data.
<!-- /TOC -->
## Recovering from an Unsuccessful Upgrade
If you upgrade Rancher, but the upgrade does not complete successfully, you may need to roll back your Rancher Server to its last healthy state before upgrade. To restore Rancher:
- Restore the `etcd` snapshot taken before upgrade.
@@ -53,4 +64,106 @@ kubectl --kubeconfig=kube_config-rancher-cluster.yml set image deployment/cattle
**Step Result:** The rollback begins. Rancher Server may be unavailable for a few minutes.
1. Log into Rancher. Confirm that the rollback succeeded by checking the version displayed in the bottom-left corner of the browser window.
1. Log into Rancher. Confirm that the rollback succeeded by checking the version displayed in the bottom-left corner of the browser window.
## Recovering from a Disaster Scenario
If you experience a disaster scenario where one of your cluster nodes fails, you'll need to replace the node and rollback to backup data.
- Create a new node and image it with a backup snapshot.
- Restore the `etcd` snapshot last taken before failure
### 1. Create New Node and Pull Snapshot
If one of your `etcd` nodes go down, you need to replace it with a new node, and then pull the most recent working `etcd` snapshot to that node.
**To Create a New Node and Pull the Latest Snapshot:**
1. Create a new node of your choice—baremetal, on-prem virtual machine, cloud-based virtual machine, and so on. Provision it according to our [requirements]({{< baseurl >}}/rancher/v2.x/en/installation/ha-server-install/#host-requirements).
2. Log in to your new node using a remote Terminal connection.
3. Create a directory that mirrors your other nodes' snapshot directories:
```
root@newnode:~# mkdir -p /opt/rke/etcd-snapshots
```
4. Pull your most recent snapshot onto the node. Replace `<SNAPSHOT.db>` with the name of the snapshot you're restoring to.
```
root@newnode:~# s3cmd get s3://rke-etcd-snapshots/<SNAPSHOT.db> /opt/rke/etcd-snapshots/<SNAPSHOT.db>
```
>**Remember:** Our use of Amazon S3 is an example used for this documentation. The command for pulling your snapshot may vary.
After restoring the cluster you have to restart the kubernetes components on all nodes, otherwise there will be some conflicts with resource versions of objects stored in `etcd`, this will include restart to kubernetes components and the network components, for more information please refer to [kubernetes documentation](https://kubernetes.io/docs/tasks/administer-cluster/configure-upgrade-etcd/#etcd-upgrade-requirements), to do that you can run the following on each node:
```
docker restart kube-apiserver kubelet kube-controller-manager kube-scheduler kube-proxy
docker ps | grep flannel | cut -f 1 -d " " | xargs docker restart
docker ps | grep calico | cut -f 1 -d " " | xargs docker restart
```
### 2. Restore `etcd` Database
To restore the most recent `etcd` snapshot on your new node, run RKE the command `rke etcd snapshot-restore`. This command reverts to any snapshot stored in `/opt/rke/etcd-snapshots` that you explicitly define. When you run `rke etcd snapshot-restore`, RKE removes the old `etcd` container if it still exists. To restore operations, RKE creates a new `etcd` cluster using the snapshot you choose.
>**Warning:** Restoring an `etcd` snapshot deletes your current `etcd` cluster and replaces it with a new one. Before you run the `rke etcd snapshot-restore` command, backup any important data in your current cluster.
>
>**Prerequisites:** {{< requirements_rollback >}}
1. From your workstation, open `rancher-cluster.yml` in your favorite text editor.
2. Replace the unresponsive node (`3.3.3.3` in this example) with your new one (`4.4.4.4`). You IP addresses will be different obviously:
nodes:
- address: 1.1.1.1
user: root
role: [controlplane,etcd,worker]
ssh_key_path: ~/.ssh/id_rsa
- address: 2.2.2.2
user: root
role: [controlplane,etcd,worker]
ssh_key_path: ~/.ssh/id_rsa
# - address: 3.3.3.3 # UNRESPONSIVE NODE
# user: root
# role: [controlplane,etcd,worker]
# ssh_key_path: ~/.ssh/id_rsa
- address: 4.4.4.4 # NEW NODE
user: root
role: [controlplane,etcd,worker]
ssh_key_path: ~/.ssh/id_rsa
3. Save and close `rancher-cluster.yml`.
4. Open **Terminal** and change directory to the location of the RKE binary. Your `rancher-cluster.yml` file must reside in the same directory.
5. Run one of the following commands to restore the `etcd` database:
```
# MacOS
./rke_darwin-amd64 etcd snapshot-restore --name <SNAPSHOT.db> --config rancher-cluster.yml
# Linux
./rke_linux-amd64 etcd snapshot-restore --name <SNAPSHOT.db> --config rancher-cluster.yml
```
6. Run one of the following commands to bring your cluster back up:
```
# MacOS
./rke_darwin-amd64 up --config rancher-cluster.yml
# Linux
./rke_linux-amd64 up --config rancher-cluster.yml
```
7. Lastly, restart the Kubernetes components on all cluster nodes to prevent potential `etcd` conflicts. Run this command on each of your nodes.
```
docker restart kube-apiserver kubelet kube-controller-manager kube-scheduler kube-proxy
docker ps | grep flannel | cut -f 1 -d " " | xargs docker restart
docker ps | grep calico | cut -f 1 -d " " | xargs docker restart
```
content/rancher/v2.x/en/backups/rollbacks/single-node-rollbacks/_index.md
+40 -1
View File
@@ -1,8 +1,22 @@
---
title: Rolling Back—Single Node Installs
title: Roll Backs—Single Node Installs
weight: 25
aliases:
- /rancher/v2.x/en/installation/after-installation/single-node-backup-and-restoration/
- /rancher/v2.x/en/backups/restorations/single-node-restoration/
---
There are two scenarios where you'll need to rollback to prior image of Rancher. The instructions change slightly for each scenario.
<!-- TOC -->
- [Recovering from an Unsuccessful Upgrade](#recovering-from-an-unsuccessful-upgrade): In this scenario, you need to restore both the prior version of Rancher and your state data.
- [Recovering from a Disaster Scenario](#recovering-from-a-disaster-scenario): In this scenario, you'll only need to restore you state data.
<!-- /TOC -->
## Recovering from an Unsuccessful Upgrade
Rolling back an unsuccessful Rancher upgrade requires you to
restore the backup you created while completing [Single Node Upgrade]({{< baseurl >}}/rancher/v2.x/en/upgrades/upgrade-scenarios/single-node-upgrade/).
@@ -34,3 +48,28 @@ docker run -d --volumes-from rancher-data-<PRIOR_VERSION> \
--restart=unless-stopped -p 80:80 -p 443:443 rancher/rancher:<PRIOR_VERSION>
```
>**Note:** _Do not_ stop the rollback after initiating it, even if the rollback process seems longer than expected. Stopping the rollback may result in database issues during future upgrades.
## Recovering from a Disaster Scenario
1. Stop the container currently running Rancher Server. Replace `<RANCHER_CONTAINER_ID>` with the ID of your Rancher container.
```
docker stop <RANCHER_CONTAINER_ID>
```
2. Go to the location where you saved your [backup tar balls]({{< baseurl >}}/rancher/v2.x/en/backups/backups/single-node-backups/#backup). Run the following command to delete your current state data and start your backup data:
```
docker run --volumes-from <RANCHER_CONTAINER_ID> -v $PWD:/backup \
alpine sh -c "rm /var/lib/rancher/* -rf && \
tar zxvf /backup/<BACKUP_FILENAME>.tar.gz"
```
>**Warning!** Running this command will delete ALL current state data from your Rancher Server container. Any changes that happened after the backup point you are restoring will be lost.
3. Start you rancher server container back. The container will start with the data from the restored backup.
```
docker start <RANCHER_CONTAINER_ID>
```