diff --git a/content/rancher/v2.x/en/backups/_index.md b/content/rancher/v2.x/en/backups/_index.md index 6c9c7fed1f1..19d1071b556 100644 --- a/content/rancher/v2.x/en/backups/_index.md +++ b/content/rancher/v2.x/en/backups/_index.md @@ -5,14 +5,11 @@ weight: 1000 This section is devoted to protecting your Rancher Server data in a disaster scenario. -- [Backups](./backups/) +- [Backups]({{< baseurl >}}/rancher/v2.x/en/backups/backups) To protect yourself from a disaster scenario, you should create Rancher backups on a regular basis. -- [Restorations](./restorations/) - In a disaster scenario, you can restore your `etcd` database by restoring a backup. +- [Rollbacks]({{< baseurl >}}/rancher/v2.x/en/backups/rollbacks) -- [Rollbacks](./rollbacks/) - - If an upgrade does not complete successfully, you can revert you Rancher Server to its previous version along with its data prior to upgrade. + If you experience a disaster scenario, or an upgrade does not complete successfully, you can restore Rancher to its last healthy backup. diff --git a/content/rancher/v2.x/en/backups/restorations/_index.md b/content/rancher/v2.x/en/backups/restorations/_index.md deleted file mode 100644 index 88982fcc4bf..00000000000 --- a/content/rancher/v2.x/en/backups/restorations/_index.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -title: Restorations -weight: 1010 ---- -If you lose the data on your Rancher Server, you can restore it if you have backups stored in a safe location. - -- [Restoring Backups—Single Node Installs](./single-node-restoration/) -- [Restoring Backups—High Availablity Installs](./ha-restoration/) \ No newline at end of file diff --git a/content/rancher/v2.x/en/backups/restorations/ha-restoration/_index.md b/content/rancher/v2.x/en/backups/restorations/ha-restoration/_index.md deleted file mode 100644 index e4906b85a29..00000000000 --- a/content/rancher/v2.x/en/backups/restorations/ha-restoration/_index.md +++ /dev/null @@ -1,115 +0,0 @@ ---- -title: Restoring Backups—High Availability Installs -weight: 370 -aliases: - - /rancher/v2.x/en/installation/after-installation/ha-backup-and-restoration/ ---- -This procedure describes how to restore your a snapshot of `etcd` if you lose your Rancher data in a disaster scenario. - -## Restoration Outline - -Following a disaster scenario, restoration of your HA Rancher installation requires you to pull your snapshot from your chosen external location and then restore it. - -1. [Create New Node and Pull Snapshot](#1-create-new-node-and-pull-snapshot) - - If one of your `etcd` nodes goes down, create a new node, and then pull the most recent `etcd` snapshot to that node. - -2. [Restore etcd Database](#2-restore-etcd-database) - - After you pull the snapshot, run the RKE command to restore the `etcd` database. - -
-### 1. Create New Node and Pull Snapshot - -If one of your `etcd` nodes go down, you need to replace it with a new node, and then pull the most recent working `etcd` snapshot to that node. - -**To Create a New Node and Pull the Latest Snapshot:** - -1. Create a new node of your choice—baremetal, on-prem virtual machine, cloud-based virtual machine, and so on. Provision it according to our [requirements]({{< baseurl >}}/rancher/v2.x/en/installation/ha-server-install/#host-requirements). - -2. Log in to your new node using a remote Terminal connection. - - -3. Create a directory that mirrors your other nodes' snapshot directories: - - ``` - root@newnode:~# mkdir -p /opt/rke/etcd-snapshots - ``` - -4. Pull your most recent snapshot onto the node. Replace `` with the name of the snapshot you're restoring to. - - ``` - root@newnode:~# s3cmd get s3://rke-etcd-snapshots/ /opt/rke/etcd-snapshots/ - ``` - - >**Remember:** Our use of Amazon S3 is an example used for this documentation. The command for pulling your snapshot may vary. - - -After restoring the cluster you have to restart the kubernetes components on all nodes, otherwise there will be some conflicts with resource versions of objects stored in `etcd`, this will include restart to kubernetes components and the network components, for more information please refer to [kubernetes documentation](https://kubernetes.io/docs/tasks/administer-cluster/configure-upgrade-etcd/#etcd-upgrade-requirements), to do that you can run the following on each node: -``` -docker restart kube-apiserver kubelet kube-controller-manager kube-scheduler kube-proxy -docker ps | grep flannel | cut -f 1 -d " " | xargs docker restart -docker ps | grep calico | cut -f 1 -d " " | xargs docker restart -``` - -### 2. Restore `etcd` Database - -To restore the most recent `etcd` snapshot on your new node, run RKE the command `rke etcd snapshot-restore`. This command reverts to any snapshot stored in `/opt/rke/etcd-snapshots` that you explicitly define. When you run `rke etcd snapshot-restore`, RKE removes the old `etcd` container if it still exists. To restore operations, RKE creates a new `etcd` cluster using the snapshot you choose. - ->**Warning:** Restoring an `etcd` snapshot deletes your current `etcd` cluster and replaces it with a new one. Before you run the `rke etcd snapshot-restore` command, backup any important data in your current cluster. -> ->**Prerequisites:** {{< requirements_rollback >}} - - -1. From your workstation, open `rancher-cluster.yml` in your favorite text editor. - -2. Replace the unresponsive node (`3.3.3.3` in this example) with your new one (`4.4.4.4`). You IP addresses will be different obviously: - - nodes: - - address: 1.1.1.1 - user: root - role: [controlplane,etcd,worker] - ssh_key_path: ~/.ssh/id_rsa - - address: 2.2.2.2 - user: root - role: [controlplane,etcd,worker] - ssh_key_path: ~/.ssh/id_rsa - # - address: 3.3.3.3 # UNRESPONSIVE NODE - # user: root - # role: [controlplane,etcd,worker] - # ssh_key_path: ~/.ssh/id_rsa - - address: 4.4.4.4 # NEW NODE - user: root - role: [controlplane,etcd,worker] - ssh_key_path: ~/.ssh/id_rsa - -3. Save and close `rancher-cluster.yml`. - -4. Open **Terminal** and change directory to the location of the RKE binary. Your `rancher-cluster.yml` file must reside in the same directory. - -5. Run one of the following commands to restore the `etcd` database: - - ``` - # MacOS - ./rke_darwin-amd64 etcd snapshot-restore --name --config rancher-cluster.yml - # Linux - ./rke_linux-amd64 etcd snapshot-restore --name --config rancher-cluster.yml - ``` - - -6. Run one of the following commands to bring your cluster back up: - - ``` - # MacOS - ./rke_darwin-amd64 up --config rancher-cluster.yml - # Linux - ./rke_linux-amd64 up --config rancher-cluster.yml - ``` - -7. Lastly, restart the Kubernetes components on all cluster nodes to prevent potential `etcd` conflicts. Run this command on each of your nodes. - - ``` - docker restart kube-apiserver kubelet kube-controller-manager kube-scheduler kube-proxy - docker ps | grep flannel | cut -f 1 -d " " | xargs docker restart - docker ps | grep calico | cut -f 1 -d " " | xargs docker restart - ``` diff --git a/content/rancher/v2.x/en/backups/restorations/single-node-restoration/_index.md b/content/rancher/v2.x/en/backups/restorations/single-node-restoration/_index.md deleted file mode 100644 index 2b057b1384f..00000000000 --- a/content/rancher/v2.x/en/backups/restorations/single-node-restoration/_index.md +++ /dev/null @@ -1,31 +0,0 @@ ---- -title: Restoring Backups—Single Node Installs -shortTitle: Singe Node Installs -weight: 365 -aliases: - - /rancher/v2.x/en/installation/after-installation/single-node-backup-and-restoration/ ---- - -Restoring to a backup for your Rancher install if you encounter issues in your Rancher setup. - -1. Stop the container currently running Rancher Server. Replace `` with the ID of your Rancher container. - - ``` -docker stop - ``` - -2. Go to the location where you saved your [backup tar balls]({{< baseurl >}}/rancher/v2.x/en/backups/backups/single-node-backups/#backup). Run the following command to delete your current state data and start your backup data: - - ``` -docker run --volumes-from -v $PWD:/backup \ -alpine sh -c "rm /var/lib/rancher/* -rf && \ -tar zxvf /backup/.tar.gz" - ``` - - >**Warning!** Running this command will delete ALL current state data from your Rancher Server container. Any changes that happened after the backup point you are restoring will be lost. - -3. Start you rancher server container back. The container will start with the data from the restored backup. - - ``` -docker start - ``` diff --git a/content/rancher/v2.x/en/backups/rollbacks/ha-server-rollbacks/_index.md b/content/rancher/v2.x/en/backups/rollbacks/ha-server-rollbacks/_index.md index 4a82b2de7b2..3c10d9d37e2 100644 --- a/content/rancher/v2.x/en/backups/rollbacks/ha-server-rollbacks/_index.md +++ b/content/rancher/v2.x/en/backups/rollbacks/ha-server-rollbacks/_index.md @@ -1,10 +1,21 @@ --- -title: Rolling Back—High Availability Installs +title: Roll Backs—High Availability Installs weight: 50 aliases: - /rancher/v2.x/en/upgrades/rollbacks/ha-server-rollbacks/ --- +There are two scenarios where you'll need to rollback to prior image of Rancher. The instructions change slightly for each scenario. + + + +- [Recovering from an Unsuccessful Upgrade](#recovering-from-an-unsuccessful-upgrade): In this scenario, you restore both the prior installation of Rancher and your backup data. +- [Recovering from a Disaster Scenario](#recovering-from-a-disaster-scenario): In this scenario, you replace a problematic node with a new one, and then restore your backup data. + + + +## Recovering from an Unsuccessful Upgrade + If you upgrade Rancher, but the upgrade does not complete successfully, you may need to roll back your Rancher Server to its last healthy state before upgrade. To restore Rancher: - Restore the `etcd` snapshot taken before upgrade. @@ -53,4 +64,106 @@ kubectl --kubeconfig=kube_config-rancher-cluster.yml set image deployment/cattle **Step Result:** The rollback begins. Rancher Server may be unavailable for a few minutes. -1. Log into Rancher. Confirm that the rollback succeeded by checking the version displayed in the bottom-left corner of the browser window. \ No newline at end of file +1. Log into Rancher. Confirm that the rollback succeeded by checking the version displayed in the bottom-left corner of the browser window. + +## Recovering from a Disaster Scenario + +If you experience a disaster scenario where one of your cluster nodes fails, you'll need to replace the node and rollback to backup data. + +- Create a new node and image it with a backup snapshot. +- Restore the `etcd` snapshot last taken before failure + +### 1. Create New Node and Pull Snapshot + +If one of your `etcd` nodes go down, you need to replace it with a new node, and then pull the most recent working `etcd` snapshot to that node. + +**To Create a New Node and Pull the Latest Snapshot:** + +1. Create a new node of your choice—baremetal, on-prem virtual machine, cloud-based virtual machine, and so on. Provision it according to our [requirements]({{< baseurl >}}/rancher/v2.x/en/installation/ha-server-install/#host-requirements). + +2. Log in to your new node using a remote Terminal connection. + + +3. Create a directory that mirrors your other nodes' snapshot directories: + + ``` + root@newnode:~# mkdir -p /opt/rke/etcd-snapshots + ``` + +4. Pull your most recent snapshot onto the node. Replace `` with the name of the snapshot you're restoring to. + + ``` + root@newnode:~# s3cmd get s3://rke-etcd-snapshots/ /opt/rke/etcd-snapshots/ + ``` + + >**Remember:** Our use of Amazon S3 is an example used for this documentation. The command for pulling your snapshot may vary. + + +After restoring the cluster you have to restart the kubernetes components on all nodes, otherwise there will be some conflicts with resource versions of objects stored in `etcd`, this will include restart to kubernetes components and the network components, for more information please refer to [kubernetes documentation](https://kubernetes.io/docs/tasks/administer-cluster/configure-upgrade-etcd/#etcd-upgrade-requirements), to do that you can run the following on each node: +``` +docker restart kube-apiserver kubelet kube-controller-manager kube-scheduler kube-proxy +docker ps | grep flannel | cut -f 1 -d " " | xargs docker restart +docker ps | grep calico | cut -f 1 -d " " | xargs docker restart +``` + +### 2. Restore `etcd` Database + +To restore the most recent `etcd` snapshot on your new node, run RKE the command `rke etcd snapshot-restore`. This command reverts to any snapshot stored in `/opt/rke/etcd-snapshots` that you explicitly define. When you run `rke etcd snapshot-restore`, RKE removes the old `etcd` container if it still exists. To restore operations, RKE creates a new `etcd` cluster using the snapshot you choose. + +>**Warning:** Restoring an `etcd` snapshot deletes your current `etcd` cluster and replaces it with a new one. Before you run the `rke etcd snapshot-restore` command, backup any important data in your current cluster. +> +>**Prerequisites:** {{< requirements_rollback >}} + + +1. From your workstation, open `rancher-cluster.yml` in your favorite text editor. + +2. Replace the unresponsive node (`3.3.3.3` in this example) with your new one (`4.4.4.4`). You IP addresses will be different obviously: + + nodes: + - address: 1.1.1.1 + user: root + role: [controlplane,etcd,worker] + ssh_key_path: ~/.ssh/id_rsa + - address: 2.2.2.2 + user: root + role: [controlplane,etcd,worker] + ssh_key_path: ~/.ssh/id_rsa + # - address: 3.3.3.3 # UNRESPONSIVE NODE + # user: root + # role: [controlplane,etcd,worker] + # ssh_key_path: ~/.ssh/id_rsa + - address: 4.4.4.4 # NEW NODE + user: root + role: [controlplane,etcd,worker] + ssh_key_path: ~/.ssh/id_rsa + +3. Save and close `rancher-cluster.yml`. + +4. Open **Terminal** and change directory to the location of the RKE binary. Your `rancher-cluster.yml` file must reside in the same directory. + +5. Run one of the following commands to restore the `etcd` database: + + ``` + # MacOS + ./rke_darwin-amd64 etcd snapshot-restore --name --config rancher-cluster.yml + # Linux + ./rke_linux-amd64 etcd snapshot-restore --name --config rancher-cluster.yml + ``` + + +6. Run one of the following commands to bring your cluster back up: + + ``` + # MacOS + ./rke_darwin-amd64 up --config rancher-cluster.yml + # Linux + ./rke_linux-amd64 up --config rancher-cluster.yml + ``` + +7. Lastly, restart the Kubernetes components on all cluster nodes to prevent potential `etcd` conflicts. Run this command on each of your nodes. + + ``` + docker restart kube-apiserver kubelet kube-controller-manager kube-scheduler kube-proxy + docker ps | grep flannel | cut -f 1 -d " " | xargs docker restart + docker ps | grep calico | cut -f 1 -d " " | xargs docker restart + ``` diff --git a/content/rancher/v2.x/en/backups/rollbacks/single-node-rollbacks/_index.md b/content/rancher/v2.x/en/backups/rollbacks/single-node-rollbacks/_index.md index 698bd9ea667..519a80d680e 100644 --- a/content/rancher/v2.x/en/backups/rollbacks/single-node-rollbacks/_index.md +++ b/content/rancher/v2.x/en/backups/rollbacks/single-node-rollbacks/_index.md @@ -1,8 +1,22 @@ --- -title: Rolling Back—Single Node Installs +title: Roll Backs—Single Node Installs weight: 25 +aliases: + - /rancher/v2.x/en/installation/after-installation/single-node-backup-and-restoration/ + - /rancher/v2.x/en/backups/restorations/single-node-restoration/ --- +There are two scenarios where you'll need to rollback to prior image of Rancher. The instructions change slightly for each scenario. + + + +- [Recovering from an Unsuccessful Upgrade](#recovering-from-an-unsuccessful-upgrade): In this scenario, you need to restore both the prior version of Rancher and your state data. +- [Recovering from a Disaster Scenario](#recovering-from-a-disaster-scenario): In this scenario, you'll only need to restore you state data. + + + +## Recovering from an Unsuccessful Upgrade + Rolling back an unsuccessful Rancher upgrade requires you to restore the backup you created while completing [Single Node Upgrade]({{< baseurl >}}/rancher/v2.x/en/upgrades/upgrade-scenarios/single-node-upgrade/). @@ -34,3 +48,28 @@ docker run -d --volumes-from rancher-data- \ --restart=unless-stopped -p 80:80 -p 443:443 rancher/rancher: ``` >**Note:** _Do not_ stop the rollback after initiating it, even if the rollback process seems longer than expected. Stopping the rollback may result in database issues during future upgrades. + +## Recovering from a Disaster Scenario + +1. Stop the container currently running Rancher Server. Replace `` with the ID of your Rancher container. + + ``` +docker stop + ``` + +2. Go to the location where you saved your [backup tar balls]({{< baseurl >}}/rancher/v2.x/en/backups/backups/single-node-backups/#backup). Run the following command to delete your current state data and start your backup data: + + ``` +docker run --volumes-from -v $PWD:/backup \ +alpine sh -c "rm /var/lib/rancher/* -rf && \ +tar zxvf /backup/.tar.gz" + ``` + + >**Warning!** Running this command will delete ALL current state data from your Rancher Server container. Any changes that happened after the backup point you are restoring will be lost. + +3. Start you rancher server container back. The container will start with the data from the restored backup. + + ``` +docker start + ``` +