restructuring upgrades with rollback content.

This commit is contained in:
Mark Bishop
2018-06-25 21:54:55 -07:00
committed by Denise
parent 13fd40d6a6
commit 06974aac73
19 changed files with 546 additions and 406 deletions
@@ -1,10 +0,0 @@
---
title: Backups and Restoration
weight: 345
aliases:
- /rancher/v2.x/en/installation/after-installation/
---
This section contains information about how to create backups of your Rancher data and how to restore them in a disaster scenario.
- [Single Node Install Backup and Restoration]({{< baseurl >}}/rancher/v2.x/en/installation/backups-and-restoration/single-node-backup-and-restoration/)
- [High Availability Install Backup and Restoration]({{< baseurl >}}/rancher/v2.x/en/installation/backups-and-restoration/ha-backup-and-restoration/)
@@ -1,237 +0,0 @@
---
title: High Availability Backup and Restoration
weight: 370
aliases:
- /rancher/v2.x/en/installation/after-installation/ha-backup-and-restoration/
---
This section describes how to:
- Create backups of your high-availability Rancher install.
- Restore the backups in a disaster scenario.
## Backup Workflow
Backing up your high-availability Rancher cluster is process that involves completing multiple tasks.
1. [Meet Backup Prerequisites](#1-meet-backup-prerequisites)
Before starting, make sure you have the files needed to create backups.
2. [Take Snapshots of the `etcd` Database](#2-take-snapshots-of-the-etcd-database)
Take snapshots of your current `etcd` database using Rancher Kubernetes Engine (RKE).
3. [Store Snapshot(s) Externally](#3-store-snapshots-externally)
After taking your snapshots, export them to a safe location that won't be affected if your cluster encounters issues.
<br/>
### 1. Meet Backup Prerequisites
Begin by gathering the files that you need to create backups of your Rancher install.
#### Prerequisites
- Rancher Kubernetes Engine v0.1.7 or later
The commands for taking `etcd` snapshots are only available in RKE v0.1.7 and later.
- rancher-cluster.yml
You'll need the RKE config file you used for Rancher install, `rancher-cluster.yml`. You created this file during your chosen high-availability installation scenario:
<br/>
<br/>
- [High Availability Installation]({{< baseurl >}}/rancher/v2.x/en/installation/ha-server-install)
- [High Availability Installation with External Load Balancer]({{< baseurl >}}/rancher/v2.x/en/installation/ha-server-install-external-lb)
<br/>
### 2. Take Snapshots of the `etcd` Database
Take snapshots of your `etcd` database. You can use these snapshots later to recover from a disaster scenario. There are two ways to take snapshots: recurringly, or as a one-off. Each option is better suited to a specific use case. Read the short description below each link to know when to use each option.
- [Option A: Recurring Snapshots](#option-a-recurring-snapshots)
After you stand up a high-availability Rancher install, we recommend configuring RKE to automatically take recurring snapshots so that you always have a safe restoration point available.
- [Option B: One-Time Snapshots](#option-b-one-time-snapshots)
We advise taking one-time snapshots before events like upgrades or restoration of another snapshot.
#### Option A: Recurring Snapshots
For all high-availability Rancher installs, we recommend taking recurring snapshots so that you always have a safe restoration point available.
To take recurring snapshots, enable the `etcd-snapshot` service, which is a service that's included with RKE. This service runs in a service container alongside the `etcd` container. You can enable this service by adding some code to `rancher-cluster.yml`.
**To Enable Recuring Snapshots:**
1. Open `rancher-cluster.yml` with your favorite text editor.
2. Add the following code block to the bottom of the file:
```
services:
etcd:
snapshot: true # enables recurring etcd snapshots
creation: 5m0s # time increment between shapshots
retention: 24h # time increment before snapshot purge
```
3. Edit the code according to your requirements.
4. Save and close `rancher-cluster.yml`.
5. Open **Terminal** and change directory to the location of the RKE binary. Your `rancher-cluster.yml` file must reside in the same directory.
6. Run one of the following commands:
```
# MacOS
./rke_darwin-amd64 up --config rancher-cluster.yml
# Linux
./rke_linux-amd64 up --config rancher-cluster.yml
```
**Result:** RKE is configured to take recurring snapshots of `etcd` on all nodes running the `etcd` role. Snapshots are saved to the following directory: `/opt/rke/etcd-snapshots/`.
#### Option B: One-Time Snapshots
When you're about to upgrade Rancher or restore it to a previous snapshot, you should snapshot your live image so that you have a backup of `etcd` in its last known state.
**To Take a One-Time Snapshot:**
1. Open **Terminal** and change directory to the location of the RKE binary. Your `rancher-cluster.yml` file must reside in the same directory.
2. Enter the following command. Replace `<SNAPSHOT.db>` with any name that you want to use for the snapshot (e.g. `upgrade.db`).
```
# MacOS
./rke_darwin-amd64 etcd snapshot-save --name <SNAPSHOT.db> --config rancher-cluster.yml
# Linux
./rke_linux-amd64 etcd snapshot-save --name <SNAPSHOT.db> --config rancher-cluster.yml
```
**Result:** RKE takes a snapshot of `etcd` running on each `etcd` node. The file is saved to `/opt/rke/etcd-snapshots`.
### 3. Backup Snapshots to a Safe Location
After taking the `etcd` snapshots, save them to a safe location so that they're unaffected if your cluster experiences a disaster scenrio. This location should be persistent.
In this documentation, as an example, we're using Amazon S3 as our safe location, and [S3cmd](http://s3tools.org/s3cmd) as our tool to create the backups. The backup location and tool that you use are ultimately your decision.
**Example:**
```
root@node:~# s3cmd mb s3://rke-etcd-snapshots
root@node:~# s3cmd /opt/rke/etcd-snapshots/snapshot.db s3://rke-etcd-snapshots/
```
## Restoration Workflow
Following a disaster scenario, restoration of your HA Rancher installation requires you to pull your snapshot from your chosen external location and then restore it.
1. [Create New Node and Pull Snapshot](#1-create-new-node-and-pull-snapshot)
If one of your `etcd` nodes goes down, create a new node, and then pull the most recent `etcd` snapshot to that node.
2. [Restore etcd Database](#2-restore-etcd-database)
After you pull the snapshot, run the RKE command to restore the `etcd` database.
<br/>
### 1. Create New Node and Pull Snapshot
If one of your `etcd` nodes go down, you need to replace it with a new node, and then pull the most recent working `etcd` snapshot to that node.
**To Create a New Node and Pull the Latest Snapshot:**
1. Create a new node of your choice—baremetal, on-prem virtual machine, cloud-based virtual machine, and so on. Provision it according to our [requirements]({{< baseurl >}}/rancher/v2.x/en/installation/ha-server-install/# host-requirements).
2. Log in to your new node using a remote Terminal connection.
3. Create a directory that mirrors your other nodes' snapshot directories:
```
root@newnode:~# mkdir -p /opt/rke/etcd-snapshots
```
4. Pull your most recent snapshot onto the node. Replace `<SNAPSHOT.db>` with the name of the snapshot you're restoring to.
```
root@newnode:~# s3cmd get s3://rke-etcd-snapshots/<SNAPSHOT.db> /opt/rke/etcd-snapshots/<SNAPSHOT.db>
```
>**Remember:** Our use of Amazon S3 is an example used for this documentation. The command for pulling your snapshot may vary.
After restoring the cluster you have to restart the kubernetes components on all nodes, otherwise there will be some conflicts with resource versions of objects stored in `etcd`, this will include restart to kubernetes components and the network components, for more information please refer to [kubernetes documentation](https://kubernetes.io/docs/tasks/administer-cluster/configure-upgrade-etcd/#etcd-upgrade-requirements), to do that you can run the following on each node:
```
docker restart kube-apiserver kubelet kube-controller-manager kube-scheduler kube-proxy
docker ps | grep flannel | cut -f 1 -d " " | xargs docker restart
docker ps | grep calico | cut -f 1 -d " " | xargs docker restart
```
### 2. Restore `etcd` Database
To restore the most recent `etcd` snapshot on your new node, run RKE the command `rke etcd snapshot-restore`. This command reverts to any snapshot stored in `/opt/rke/etcd-snapshots` that you explicitly define. When you run `rke etcd snapshot-restore`, RKE removes the old `etcd` container if it still exists. To restore operations, RKE creates a new `etcd` cluster using the snapshot you choose.
>**Important:** When restoring the etcd database, you must restore each `etcd` to the _same_ snapshot, this means the exact same copy, so to restore you have to copy the snapshot from one of the nodes to the others before doing the `etcd snapshot-restore`.
>**Warning:** Restoring an `etcd` snapshot deletes your current `etcd` cluster and replaces it with a new one. Before you run the `rke etcd snapshot-restore` command, backup any important data in your current cluster.
1. From your workstation, open `rancher-cluster.yml` in your favorite text editor.
2. Replace the unresponsive node (`3.3.3.3` in this example) with your new one (`4.4.4.4`). You IP addresses will be different obviously:
nodes:
- address: 1.1.1.1
user: root
role: [controlplane,etcd,worker]
ssh_key_path: ~/.ssh/id_rsa
- address: 2.2.2.2
user: root
role: [controlplane,etcd,worker]
ssh_key_path: ~/.ssh/id_rsa
# - address: 3.3.3.3 # UNRESPONSIVE NODE
# user: root
# role: [controlplane,etcd,worker]
# ssh_key_path: ~/.ssh/id_rsa
- address: 4.4.4.4 # NEW NODE
user: root
role: [controlplane,etcd,worker]
ssh_key_path: ~/.ssh/id_rsa
3. Save and close `rancher-cluster.yml`.
4. Open **Terminal** and change directory to the location of the RKE binary. Your `rancher-cluster.yml` file must reside in the same directory.
5. Run one of the following commands to restore the `etcd` database:
```
# MacOS
./rke_darwin-amd64 etcd snapshot-restore --name <SNAPSHOT.db> --config rancher-cluster.yml
# Linux
./rke_linux-amd64 etcd snapshot-restore --name <SNAPSHOT.db> --config rancher-cluster.yml
```
6. Run one of the following commands to bring your cluster back up:
```
# MacOS
./rke_darwin-amd64 up --config rancher-cluster.yml
# Linux
./rke_linux-amd64 up --config rancher-cluster.yml
```
7. Lastly, restart the Kubernetes components on all cluster nodes to prevent potential `etcd` conflicts. Run this command on each of your nodes.
```
docker restart kube-apiserver kubelet kube-controller-manager kube-scheduler kube-proxy
docker ps | grep flannel | cut -f 1 -d " " | xargs docker restart
docker ps | grep calico | cut -f 1 -d " " | xargs docker restart
```
@@ -1,52 +0,0 @@
---
title: Single Node Backup and Restoration
weight: 365
aliases:
- /rancher/v2.x/en/installation/after-installation/single-node-backup-and-restoration/
---
After completing your single node installation of Rancher, or before your upgrade to a newer version of Rancher, create a backup of your current installation. Use this backup as a restoration point for your Rancher install if you encounter issues when upgrading.
## Backing Up Your Rancher Server
>**Prerequisite:** Open Rancher and write down the version number displayed in the lower-left of the browser (example: `v2.0.0`). You'll need this number during the backup process.
1. Stop the container currently running Rancher Server. Replace `<RANCHER_CONTAINER_ID>` with the ID of your Rancher container.
```
docker stop <RANCHER_CONTAINER_ID>
```
>**Tip:** You can obtain the ID for your Rancher container by entering the following command: `docker ps`.
2. Create a backup container. This container backs up the data from your current Rancher Server, which you can use as a recovery point.
- Replace `<RANCHER_CONTAINER_ID>` with the same ID from the previous step.
- Replace `<RANCHER_CONTAINER_TAG>` and `<RANCHER_VERSION>` with the version of Rancher that you are currently running, as mentioned in the **Prerequisite** above.
```
docker create --volumes-from <RANCHER_CONTAINER_ID> \
--name rancher-backup-<RANCHER_VERSION> rancher/rancher:<RANCHER_CONTAINER_TAG>
```
3. Restart Rancher Server. Replace `<RANCHER_CONTAINER_ID>` with the ID of your Rancher container.
```
docker start <RANCHER_CONTAINER_ID>
```
## Restoring Your Rancher Server
1. Stop the container currently running Rancher Server. Replace `<RANCHER_CONTAINER_ID>` with the ID of your Rancher container.
```
docker stop <RANCHER_CONTAINER_ID>
```
2. Launch a new Rancher Server container using the most recent `rancher-backup-<RANCHER_VERSION>` container.
```
docker run -d --volumes-from rancher-backup-<RANCHER_VERSION> --restart=unless-stopped \
-p 80:80 -p 443:443 rancher/rancher:<CURRENT_RANCHER_VERSION>
```
+9 -11
View File
@@ -3,22 +3,20 @@ title: Upgrades
weight: 1000
---
This section includes instructions for upgrading Rancher from previous versions.
This section is devoted to upgrading your Rancher Server, as well as other procedures related to upgrades and system maintenance.
### Upgrading from Rancher 2.x.x
- [Upgrade Scenarios](./upgrade-scenarios/)
Each new version of Rancher 2.x.x supports upgrades from previous versions of Rancher 2.x.x. This section will be updated as soon as the first release post 2.0 is available.
When a new version of Rancher is released, follow these instructions to upgrade to the latest version.
Complete one of the upgrade procedures below based on your Rancher installation:
- [Backups](./backups/)
- [Single Node Upgrade](./single-node-upgrade/)
- [High Availability Upgrade](./ha-server-upgrade/)
- [Air Gap Upgrade](./air-gap-upgrade/)
To protect yourself from a disaster scenario, you should create Rancher backups on a regular basis.
### Migrating from Rancher 1.6.x
- [Restorations](./restorations/)
Until Rancher 2.1 is released, migrating to from Rancher 1.6.x to 2.x.x is not supported due to major code rewrites.
In a disaster scenario, you can restore your `etcd` database by restoring a backup.
For the 2.1 release, we plan to release a tool that converts Rancher Compose to Kubernetes YAML. This tool will help our Cattle users migrate from Rancher 1.6.x to 2.x.x. However, we understand that there is a learning curve switching from Cattle to Kubernetes as you deploy new workloads. Therefore, this release will include a cheatsheet for those that enjoy Cattle's simplicity but want to quickly create those workloads in Kubernetes.
- [Rollbacks](./rollbacks/)
We will continue support for Rancher 1.6.x for a minimum of one year after the 2.1 release so that 1.6.x users can plan and complete migration.
If an upgrade does not complete successfully, you can revert you Rancher Server to its previous version along with its data prior to upgrade.
@@ -0,0 +1,10 @@
---
title: Backups
weight: 50
aliases:
- /rancher/v2.x/en/installation/after-installation/
---
This section contains information about how to create backups of your Rancher data and how to restore them in a disaster scenario.
- [Single Node Install Backups](./single-node-backups/)
- [High Availability Install Backups](./ha-backups/)
@@ -0,0 +1,125 @@
---
title: Creating Backups—High Availability Installs
weight: 50
aliases:
- /rancher/v2.x/en/installation/after-installation/ha-backup-and-restoration/
---
This section describes how to create backups of your high-availability Rancher install.
## Backup Outline
Backing up your high-availability Rancher cluster is process that involves completing multiple tasks.
1. [Meet Backup Prerequisites](#1-meet-backup-prerequisites)
Before starting, make sure you have the files needed to create backups.
1. [Take Snapshots of the `etcd` Database](#2-take-snapshots-of-the-etcd-database)
Take snapshots of your current `etcd` database using Rancher Kubernetes Engine (RKE).
1. [Store Snapshot(s) Externally](#3-backup-snapshots-to-a-safe-location)
After taking your snapshots, export them to a safe location that won't be affected if your cluster encounters issues.
<br/>
### 1. Meet Backup Prerequisites
Begin by gathering the files that you need to create backups of your Rancher install.
#### Prerequisites
- Rancher Kubernetes Engine v0.1.7 or later
The commands for taking `etcd` snapshots are only available in RKE v0.1.7 and later.
- rancher-cluster.yml
You'll need the RKE config file you used for Rancher install, `rancher-cluster.yml`. You created this file during your chosen high-availability installation scenario:
<br/>
<br/>
- [High Availability Installation]({{< baseurl >}}/rancher/v2.x/en/installation/ha-server-install)
- [High Availability Installation with External Load Balancer]({{< baseurl >}}/rancher/v2.x/en/installation/ha-server-install-external-lb)
<br/>
### 2. Take Snapshots of the `etcd` Database
Take snapshots of your `etcd` database. You can use these snapshots later to recover from a disaster scenario. There are two ways to take snapshots: recurringly, or as a one-off. Each option is better suited to a specific use case. Read the short description below each link to know when to use each option.
- [Option A: Recurring Snapshots](#option-a-recurring-snapshots)
After you stand up a high-availability Rancher install, we recommend configuring RKE to automatically take recurring snapshots so that you always have a safe restoration point available.
- [Option B: One-Time Snapshots](#option-b-one-time-snapshots)
We advise taking one-time snapshots before events like upgrades or restoration of another snapshot.
#### Option A: Recurring Snapshots
For all high-availability Rancher installs, we recommend taking recurring snapshots so that you always have a safe restoration point available.
To take recurring snapshots, enable the `etcd-snapshot` service, which is a service that's included with RKE. This service runs in a service container alongside the `etcd` container. You can enable this service by adding some code to `rancher-cluster.yml`.
**To Enable Recurring Snapshots:**
1. Open `rancher-cluster.yml` with your favorite text editor.
2. Add the following code block to the bottom of the file:
```
services:
etcd:
snapshot: true # enables recurring etcd snapshots
creation: 5m0s # time increment between snapshots
retention: 24h # time increment before snapshot purge
```
3. Edit the code according to your requirements.
4. Save and close `rancher-cluster.yml`.
5. Open **Terminal** and change directory to the location of the RKE binary. Your `rancher-cluster.yml` file must reside in the same directory.
6. Run one of the following commands:
```
# MacOS
./rke_darwin-amd64 up --config rancher-cluster.yml
# Linux
./rke_linux-amd64 up --config rancher-cluster.yml
```
**Result:** RKE is configured to take recurring snapshots of `etcd` on all nodes running the `etcd` role. Snapshots are saved to the following directory: `/opt/rke/etcd-snapshots/`.
#### Option B: One-Time Snapshots
When you're about to upgrade Rancher or restore it to a previous snapshot, you should snapshot your live image so that you have a backup of `etcd` in its last known state.
**To Take a One-Time Snapshot:**
1. Open **Terminal** and change directory to the location of the RKE binary. Your `rancher-cluster.yml` file must reside in the same directory.
2. Enter the following command. Replace `<SNAPSHOT.db>` with any name that you want to use for the snapshot (e.g. `upgrade.db`).
```
# MacOS
./rke_darwin-amd64 etcd snapshot-save --name <SNAPSHOT.db> --config rancher-cluster.yml
# Linux
./rke_linux-amd64 etcd snapshot-save --name <SNAPSHOT.db> --config rancher-cluster.yml
```
**Result:** RKE takes a snapshot of `etcd` running on each `etcd` node. The file is saved to `/opt/rke/etcd-snapshots`.
### 3. Backup Snapshots to a Safe Location
After taking the `etcd` snapshots, save them to a safe location so that they're unaffected if your cluster experiences a disaster scenario. This location should be persistent.
In this documentation, as an example, we're using Amazon S3 as our safe location, and [S3cmd](http://s3tools.org/s3cmd) as our tool to create the backups. The backup location and tool that you use are ultimately your decision.
**Example:**
```
root@node:~# s3cmd mb s3://rke-etcd-snapshots
root@node:~# s3cmd /opt/rke/etcd-snapshots/snapshot.db s3://rke-etcd-snapshots/
```
@@ -0,0 +1,36 @@
---
title: Creating Backups—Single Node Installs
weight: 25
aliases:
- /rancher/v2.x/en/installation/after-installation/single-node-backup-and-restoration/
---
After completing your single node installation of Rancher create a backup of your current installation. Use this backup as a restoration point in disaster scenarios.
>**Prerequisite:** Open Rancher and write down the version number displayed in the lower-left of the browser (example: `v2.0.0`). You'll need this number during the backup process.
1. Stop the container currently running Rancher Server. Replace `<RANCHER_CONTAINER_ID>` with the ID of your Rancher container.
```
docker stop <RANCHER_CONTAINER_ID>
```
>**Tip:** You can obtain the ID for your Rancher container by entering the following command: `docker ps`.
2. <a id="backup"></a>Create a backup container. This container backs up the data from your current Rancher Server, which you can use as a recovery point.
- Replace `<RANCHER_CONTAINER_ID>` with the same ID from the previous step.
- Replace `<RANCHER_VERSION>` and `<RANCHER_CONTAINER_TAG>` with the version of Rancher that you are currently running, as mentioned in the **Prerequisite** above.
```
docker create --volumes-from <RANCHER_CONTAINER_ID> \
--name rancher-backup-<RANCHER_VERSION> rancher/rancher:<RANCHER_CONTAINER_TAG>
```
3. Restart Rancher Server. Replace `<RANCHER_CONTAINER_ID>` with the ID of your Rancher container.
```
docker start <RANCHER_CONTAINER_ID>
```
**Result:** A backup of your Rancher Server is created. If you ever need to restore your backup, see [Restoring Backups: Single Node Installs]({{< baseurl >}}/rancher/v2.x/en/upgrades/restorations/single-node-restoration).
@@ -1,27 +0,0 @@
---
title: High Availability Upgrade
weight: 1020
---
To upgrade Rancher 2.x running in a high availability configuration, run an upgrade command that points to your upgrade config file.
>**Prerequisites:**
>
>- Take a [one-time etcd snapshot]({{< baseurl >}}/rancher/v2.x/en/installation/backups-and-restoration/ha-backup-and-restoration). This snapshot is a restore point in case something goes wrong during upgrade.
>- Install [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/) on your workstation.
>- Confirm that the following path exists on your workstation: `~/.kube/`. If it doesn't, create it yourself.
>- Copy `kube_config_rancher-cluster.yml`, which is automatically generated after [Rancher Server installation]({{< baseurl >}}/rancher/v2.x/en/installation/ha-server-install#part-11-backup-kube-config-rancher-cluster-yml), to the `~/.kube/` directory.
1. From your workstation, open **Terminal**.
2. Enter the following command:
```
kubectl --kubeconfig=kube_config-rancher-cluster.yml set image deployment/cattle cattle-server=rancher/rancher:<VERSION_TAG> -n cattle-system
```
Replace `<VERSION_TAG>` with the version that you want to upgrade to. For a list of tags available, see [DockerHub](https://hub.docker.com/r/rancher/rancher/tags/). However, don't use any tag suffixed with `-rc`, as they are used for testing and are not officially supported.
**Step Result:** The upgrade begins. Rancher Server may be unavailable for a few minutes.
3. Log into Rancher. Confirm that the upgrade succeeded by checking the version displayed in the bottom-left corner of the browser window.
**Result:** Your Rancher Servers are upgraded.
@@ -0,0 +1,8 @@
---
title: Restorations
weight: 1010
---
If you lose the data on your Rancher Server, you can restore it if you have backups stored in a safe location.
- [Restoring Backups—Single Node Installs](./single-node-restoration/)
- [Restoring Backups—High Availablity Installs](./ha-restoration/)
@@ -0,0 +1,115 @@
---
title: Restoring Backups—High Availablity Installs
weight: 370
aliases:
- /rancher/v2.x/en/installation/after-installation/ha-backup-and-restoration/
---
This procedure describes how to restore your a snapshot of `etcd` if you lose your Rancher data in a disaster scenario.
## Restoration Outline
Following a disaster scenario, restoration of your HA Rancher installation requires you to pull your snapshot from your chosen external location and then restore it.
1. [Create New Node and Pull Snapshot](#1-create-new-node-and-pull-snapshot)
If one of your `etcd` nodes goes down, create a new node, and then pull the most recent `etcd` snapshot to that node.
2. [Restore etcd Database](#2-restore-etcd-database)
After you pull the snapshot, run the RKE command to restore the `etcd` database.
<br/>
### 1. Create New Node and Pull Snapshot
If one of your `etcd` nodes go down, you need to replace it with a new node, and then pull the most recent working `etcd` snapshot to that node.
**To Create a New Node and Pull the Latest Snapshot:**
1. Create a new node of your choice—baremetal, on-prem virtual machine, cloud-based virtual machine, and so on. Provision it according to our [requirements]({{< baseurl >}}/rancher/v2.x/en/installation/ha-server-install/#host-requirements).
2. Log in to your new node using a remote Terminal connection.
3. Create a directory that mirrors your other nodes' snapshot directories:
```
root@newnode:~# mkdir -p /opt/rke/etcd-snapshots
```
4. Pull your most recent snapshot onto the node. Replace `<SNAPSHOT.db>` with the name of the snapshot you're restoring to.
```
root@newnode:~# s3cmd get s3://rke-etcd-snapshots/<SNAPSHOT.db> /opt/rke/etcd-snapshots/<SNAPSHOT.db>
```
>**Remember:** Our use of Amazon S3 is an example used for this documentation. The command for pulling your snapshot may vary.
After restoring the cluster you have to restart the kubernetes components on all nodes, otherwise there will be some conflicts with resource versions of objects stored in `etcd`, this will include restart to kubernetes components and the network components, for more information please refer to [kubernetes documentation](https://kubernetes.io/docs/tasks/administer-cluster/configure-upgrade-etcd/#etcd-upgrade-requirements), to do that you can run the following on each node:
```
docker restart kube-apiserver kubelet kube-controller-manager kube-scheduler kube-proxy
docker ps | grep flannel | cut -f 1 -d " " | xargs docker restart
docker ps | grep calico | cut -f 1 -d " " | xargs docker restart
```
### 2. Restore `etcd` Database
To restore the most recent `etcd` snapshot on your new node, run RKE the command `rke etcd snapshot-restore`. This command reverts to any snapshot stored in `/opt/rke/etcd-snapshots` that you explicitly define. When you run `rke etcd snapshot-restore`, RKE removes the old `etcd` container if it still exists. To restore operations, RKE creates a new `etcd` cluster using the snapshot you choose.
>**Important:** When restoring the etcd database, you must restore each `etcd` to the _same_ snapshot, this means the exact same copy, so to restore you have to copy the snapshot from one of the nodes to the others before doing the `etcd snapshot-restore`.
>**Warning:** Restoring an `etcd` snapshot deletes your current `etcd` cluster and replaces it with a new one. Before you run the `rke etcd snapshot-restore` command, backup any important data in your current cluster.
1. From your workstation, open `rancher-cluster.yml` in your favorite text editor.
2. Replace the unresponsive node (`3.3.3.3` in this example) with your new one (`4.4.4.4`). You IP addresses will be different obviously:
nodes:
- address: 1.1.1.1
user: root
role: [controlplane,etcd,worker]
ssh_key_path: ~/.ssh/id_rsa
- address: 2.2.2.2
user: root
role: [controlplane,etcd,worker]
ssh_key_path: ~/.ssh/id_rsa
# - address: 3.3.3.3 # UNRESPONSIVE NODE
# user: root
# role: [controlplane,etcd,worker]
# ssh_key_path: ~/.ssh/id_rsa
- address: 4.4.4.4 # NEW NODE
user: root
role: [controlplane,etcd,worker]
ssh_key_path: ~/.ssh/id_rsa
3. Save and close `rancher-cluster.yml`.
4. Open **Terminal** and change directory to the location of the RKE binary. Your `rancher-cluster.yml` file must reside in the same directory.
5. Run one of the following commands to restore the `etcd` database:
```
# MacOS
./rke_darwin-amd64 etcd snapshot-restore --name <SNAPSHOT.db> --config rancher-cluster.yml
# Linux
./rke_linux-amd64 etcd snapshot-restore --name <SNAPSHOT.db> --config rancher-cluster.yml
```
6. Run one of the following commands to bring your cluster back up:
```
# MacOS
./rke_darwin-amd64 up --config rancher-cluster.yml
# Linux
./rke_linux-amd64 up --config rancher-cluster.yml
```
7. Lastly, restart the Kubernetes components on all cluster nodes to prevent potential `etcd` conflicts. Run this command on each of your nodes.
```
docker restart kube-apiserver kubelet kube-controller-manager kube-scheduler kube-proxy
docker ps | grep flannel | cut -f 1 -d " " | xargs docker restart
docker ps | grep calico | cut -f 1 -d " " | xargs docker restart
```
@@ -0,0 +1,24 @@
---
title: Restoring Backups—Single Node Installs
shortTitle: Singe Node Installs
weight: 365
aliases:
- /rancher/v2.x/en/installation/after-installation/single-node-backup-and-restoration/
---
Backup to a restoration point for your Rancher install if you encounter issues when upgrading.
1. Stop the container currently running Rancher Server. Replace `<RANCHER_CONTAINER_ID>` with the ID of your Rancher container.
```
docker stop <RANCHER_CONTAINER_ID>
```
1. Launch a new Rancher Server container using the most recent `rancher-backup-<RANCHER_VERSION>` container that you backed up.
For more information on obtaining this container name, see [Creating Backups—Single Node Installs](/Users/markbishop/Documents/GitHub/docs/content/rancher/v2.x/en/upgrades/backups/single-node-backups/#backup).
```
docker run -d --volumes-from rancher-backup-<RANCHER_VERSION> --restart=unless-stopped \
-p 80:80 -p 443:443 rancher/rancher:<CURRENT_RANCHER_VERSION>
```
@@ -0,0 +1,10 @@
---
title: Rollbacks
shortTitle: Rollbacks
weight: 1010
---
If you upgrade your Rancher Server and the upgrade is unsuccessful, you can use the backup you created during the upgrade process to restore the previous version of Rancher, along with its data.
- [Rolling Back—Single Node Installs](./single-node-rollbacks/)
- [Rolling Back—High Availablity Installs](./ha-server-rollbacks/)
@@ -0,0 +1,49 @@
---
title: Rolling Back—High Availability Installs
weight: 50
---
If you upgrade Rancher, but the upgrade does not complete successfully, you may need to roll back your Rancher Server to its last healthy state before upgrade. To restore Rancher:
- Restore the `etcd` snapshot taken before upgrade.
- Run the command to revert to your prior version of Rancher.
>**Warning!** Rolling back to a previous version of Rancher destroys any changes made to Rancher following the upgrade. Unrecoverable data loss may occur.
1. Open **Terminal** and change directory to the location of the RKE binary that you used during upgrade.
1. Run one of the following commands to restore the `etcd` snapshot [that you took]({{< baseurl >}}/rancher/v2.x/en/upgrades/upgrade-scenarios/ha-server-upgrade/#snapshot) before your unsuccessful upgrade. Replace the `<SNAPSHOT.db>` placeholder with your snapshot.
```
# MacOS
./rke_darwin-amd64 etcd snapshot-restore --name <SNAPSHOT.db> --config kube_config_rancher-cluster.yml
# Linux
./rke_linux-amd64 etcd snapshot-restore --name <SNAPSHOT.db> --config kube_config_rancher-cluster.yml
```
1. Run one of the following commands to bring your cluster back up:
```
# MacOS
./rke_darwin-amd64 up --config kube_config_rancher-cluster.yml
# Linux
./rke_linux-amd64 up --config kube_config_rancher-cluster.yml
```
1. Lastly, restart the Kubernetes components on all cluster nodes to prevent potential `etcd` conflicts. Run this command on each of your nodes.
```
docker restart kube-apiserver kubelet kube-controller-manager kube-scheduler kube-proxy
docker ps | grep flannel | cut -f 1 -d " " | xargs docker restart
docker ps | grep calico | cut -f 1 -d " " | xargs docker restart
```
1. Enter the following command:
```
kubectl --kubeconfig=kube_config-rancher-cluster.yml set image deployment/cattle cattle-server=rancher/rancher:<PRIOR_VERSION> -n cattle-system
```
Replace `<VERSION_TAG>` with the version that you were running before upgrade. For a list of tags available, see [DockerHub](https://hub.docker.com/r/rancher/rancher/tags/). However, don't use any tag suffixed with `-rc`, as they are used for testing and are not officially supported.
**Step Result:** The rollback begins. Rancher Server may be unavailable for a few minutes.
1. Log into Rancher. Confirm that the rollback succeeded by checking the version displayed in the bottom-left corner of the browser window.
@@ -0,0 +1,27 @@
---
title: Rolling Back—Single Node Installs
weight: 25
---
Rolling back an unsuccessful Rancher upgrade requires you to run the upgrade command again, with a couple of modifications:
- Run the upgrade command using the server tag for your prior Rancher install.
- Run the upgrade command while pointing toward the backup container you created while completing [Single Node Upgrade]({{< baseurl >}}/rancher/v2.x/en/upgrades/upgrade-scenarios/single-node-upgrade/).
>**Warning!** Rolling back to a previous version of Rancher destroys any changes made to Rancher following the upgrade. Unrecoverable data loss may occur.
1. Pull the version of Rancher that you were running prior to upgrade.
```
docker pull rancher/rancher:<PRIOR_VERSION>
```
1. Launch a Rancher Server container that you created while completing [Step 3 of Single Node Upgrade]({{< baseurl >}}/rancher/v2.x/en/upgrades/upgrade-scenarios/single-node-upgrade/#backup).
Replace the `<PRIOR_VERSION>` placeholders with the version number of Rancher that you're rolling back to before upgrade.
```
docker run -d --volumes-from rancher-data-snapshot-<PRIOR_VERSION> --restart=unless-stopped \
-p 80:80 -p 443:443 rancher/rancher:<PRIOR_VERSION>
```
>**Note:** _Do not_ stop the rollback after initiating it, even if the rollback process seems longer than expected. Stopping the rollback may result in database issues during future upgrades.
@@ -1,69 +0,0 @@
---
title: Single Node Upgrade
weight: 1010
---
To upgrade Rancher Server 2.x to the latest version, you need to enter only a few commands.
>**Prerequisites:**
>
> - Create a backup of your current Rancher installation. For more information, see [Backing Up Your Rancher Server]({{< baseurl >}}/rancher/v2.x/en/installation/after-installation/single-node-backup-and-restoration/#backing-up-your-rancher-server).
> - Open Rancher and write down the version number displayed in the lower-left of the browser (example: `v2.0.0`). You'll need this number during the upgrade process.
1. Stop the container currently running Rancher Server. Replace `<RANCHER_CONTAINER_ID>` with the ID of your Rancher container.
```
docker stop <RANCHER_CONTAINER_ID>
```
>**Tip:** You can obtain the ID for your Rancher container by entering the following command: `docker ps`.
2. Create a `rancher-data` container. This container backs up the data from your current Rancher Server, which you'll use to start the upgraded version of Rancher Server in step 4.
- Replace `<RANCHER_CONTAINER_ID>` with the same ID from the previous step.
- Replace `<RANCHER_CONTAINER_TAG>` with the version of Rancher that you are currently running, as mentioned in the **Prerequisite** above.
```
docker create --volumes-from <RANCHER_CONTAINER_ID> \
--name rancher-data rancher/rancher:<RANCHER_CONTAINER_TAG>
```
1. Create a different `rancher-data-snapshot` container. This container also backs up the data from your current Rancher Server, but you are creating this container to serve as a restoration point in case something goes wrong with your server upgrade.
- Replace `<RANCHER_CONTAINER_ID>` with the same ID from the previous step.
- Replace `<RANCHER_CONTAINER_TAG>` with the version of Rancher that you are currently running, as mentioned in the **Prerequisite** above.
```
docker create --volumes-from <RANCHER_CONTAINER_ID> \
--name rancher-data rancher/rancher:<RANCHER_CONTAINER_TAG>
```
3. Pull the most recent image of Rancher.
```
docker pull rancher/rancher:latest
```
>**Attention Air Gap Users:**
> If you are visiting this page to complete [Air Gap Upgrade]({{< baseurl >}}/rancher/v2.x/en/installation/air-gap-upgrade/), prepend your private registry URL to the image when running the `docker run` command.
>
> Example: `<registry.yourdomain.com:port>/rancher/rancher:latest`
>
4. Launch a new Rancher Server container using the `rancher-data` container.
```
docker run -d --volumes-from rancher-data --restart=unless-stopped \
-p 80:80 -p 443:443 rancher/rancher:latest
```
>**Note:** _Do not_ stop the upgrade after initiating it, even if the upgrade process seems longer than expected. Stopping the upgrade may result in database migration errors during future upgrades.
><br/>
><br/>
>**Note:** After upgrading Rancher Server, data from your upgraded server is now saved to the `rancher-data` container for use in future upgrades.
5. Remove the previous Rancher Server container.
If you only stop the previous Rancher Server container (and don't remove it), the container may restart after the next server reboot.
6. Log into Rancher. Confirm that the upgrade succeeded by checking the version displayed in the bottom-left corner of the browser window.
**Result:** Rancher Server is upgraded to the latest version.
@@ -0,0 +1,22 @@
---
title: Upgrade Scenarios
weight: 25
---
### Upgrading from Rancher 2.x.x
Each new version of Rancher 2.x.x supports upgrades from previous versions of Rancher 2.x.x. This section will be updated as soon as the first release post 2.0 is available.
Complete one of the upgrade procedures below based on your Rancher installation:
- [Single Node Upgrade](./single-node-upgrade/)
- [High Availability Upgrade](./ha-server-upgrade/)
- [Air Gap Upgrade](./air-gap-upgrade/)
### Migrating from Rancher 1.6.x
Until Rancher 2.1 is released, migrating to from Rancher 1.6.x to 2.x.x is not supported due to major code rewrites.
For the 2.1 release, we plan to release a tool that converts Rancher Compose to Kubernetes YAML. This tool will help our Cattle users migrate from Rancher 1.6.x to 2.x.x. However, we understand that there is a learning curve switching from Cattle to Kubernetes as you deploy new workloads. Therefore, this release will include a cheatsheet for those that enjoy Cattle's simplicity but want to quickly create those workloads in Kubernetes.
We will continue support for Rancher 1.6.x for a minimum of one year after the 2.1 release so that 1.6.x users can plan and complete migration.
@@ -0,0 +1,41 @@
---
title: High Availability Upgrade
weight: 1020
---
To upgrade Rancher 2.x running in a high availability configuration, run an upgrade command that points to your upgrade config file.
>**Prerequisites:**
>
>- Install [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/) on your workstation.
>- Confirm that the following path exists on your workstation: `~/.kube/`. If it doesn't, create it yourself.
>- Copy `kube_config_rancher-cluster.yml`, which is automatically generated after [Rancher Server installation]({{< baseurl >}}/rancher/v2.x/en/installation/ha-server-install#part-11-backup-kube-config-rancher-cluster-yml), to the `~/.kube/` directory.
1. From your workstation, open **Terminal**.
1. Change directory to the location of the RKE binary. Your `kube_config_rancher-cluster.yml` file must reside in the same directory.
1. <a id="snapshot"></a> Enter the following command. Replace `<SNAPSHOT.db>` with any name that you want to use for the snapshot (e.g. `upgrade.db`).
```
# MacOS
./rke_darwin-amd64 etcd snapshot-save --name <SNAPSHOT.db> --config kube_config_rancher-cluster.yml
# Linux
./rke_linux-amd64 etcd snapshot-save --name <SNAPSHOT.db> --config kube_config_rancher-cluster.yml
```
**Result:** RKE takes a snapshot of `etcd` running on each `etcd` node. The file is saved to `/opt/rke/etcd-snapshots`.
1. Enter the following command:
```
kubectl --kubeconfig=kube_config_rancher-cluster.yml set image deployment/cattle cattle-server=rancher/rancher:<VERSION_TAG> -n cattle-system
```
Replace `<VERSION_TAG>` with the version that you want to upgrade to. For a list of tags available, see [DockerHub](https://hub.docker.com/r/rancher/rancher/tags/). However, don't use any tag suffixed with `-rc`, as they are used for testing and are not officially supported.
**Step Result:** The upgrade begins. Rancher Server may be unavailable for a few minutes.
1. Log into Rancher. Confirm that the upgrade succeeded by checking the version displayed in the bottom-left corner of the browser window.
**Result:** Your Rancher Servers are upgraded.
>**Upgrade Issues?** You can restore your Rancher Server and data that was running prior to upgrade. For more information, see [Restoring Backups—High Availablity Installs]({{< baseurl >}}/rancher/v2.x/en/upgrades/rollbacks/ha-server-rollbacks).
@@ -0,0 +1,70 @@
---
title: Single Node Upgrade
weight: 1010
---
To upgrade Rancher Server 2.x after a new version is released, create a backup of your server and then run the upgrade command.
<a id="prereq"></a>
>**Prerequisites:** Open Rancher and write down the version number displayed in the lower-left of the browser (example: `v2.0.0`). You'll need this number during the upgrade process.
1. Stop the container currently running Rancher Server. Replace `<RANCHER_CONTAINER_ID>` with the ID of your Rancher container.
```
docker stop <RANCHER_CONTAINER_ID>
```
>**Tip:** You can obtain the ID for your Rancher container by entering the following command: `docker ps`.
1. Create a container of your current Rancher data for use in your upgraded Rancher Server. Name the container `rancher-data`.
- Replace `<RANCHER_CONTAINER_ID>` with the same ID from the previous step.
- Replace `<RANCHER_CONTAINER_TAG>` with the version of Rancher that you are currently running, as mentioned in the **Prerequisite** above.
```
docker create --volumes-from <RANCHER_CONTAINER_ID> \
--name rancher-data rancher/rancher:<RANCHER_CONTAINER_TAG>
```
1. <a id="backup"></a>Create another container of your current Rancher data. However, this container is a backup for restoring your Rancher Server if your upgrade is unsuccessful. Name the container `rancher-data-snapshot-<CURRENT_VERSION>`.
- Replace `<RANCHER_CONTAINER_ID>` with the same ID from the previous step.
- Replace `<CURRENT_VERSION>` with the tag for the version of Rancher currently installed.
- Replace `<RANCHER_CONTAINER_TAG>` with the version of Rancher that you are currently running, as mentioned in the [prerequisite](#prereq).
```
docker create --volumes-from <RANCHER_CONTAINER_ID> \
--name rancher-data-snapshot-<CURRENT_VERSION> rancher/rancher:<RANCHER_CONTAINER_TAG>
```
1. Pull the most recent image of Rancher.
```
docker pull rancher/rancher:latest
```
>**Attention Air Gap Users:**
> If you are visiting this page to complete [Air Gap Upgrade]({{< baseurl >}}/rancher/v2.x/en/upgrades/upgrade-scenarios/air-gap-upgrade/), prepend your private registry URL to the image when running the `docker run` command.
>
> Example: `<registry.yourdomain.com:port>/rancher/rancher:latest`
>
1. Launch a new Rancher Server container using the `rancher-data` container.
```
docker run -d --volumes-from rancher-data --restart=unless-stopped \
-p 80:80 -p 443:443 rancher/rancher:latest
```
>**Note:** _Do not_ stop the upgrade after initiating it, even if the upgrade process seems longer than expected. Stopping the upgrade may result in database migration errors during future upgrades.
><br/>
><br/>
>**Note:** After upgrading Rancher Server, data from your upgraded server is now saved to the `rancher-data` container for use in future upgrades.
1. Remove the previous Rancher Server container.
If you only stop the previous Rancher Server container (and don't remove it), the container may restart after the next server reboot.
1. Log into Rancher. Confirm that the upgrade succeeded by checking the version displayed in the bottom-left corner of the browser window.
**Result:** Rancher Server is upgraded to the latest version.
>**Note:** If your upgrade does not complete successfully, you can roll Rancher Server and its data back to its last healthy state. For more information, see [Restoring Backups—Single Node Installs]({{< baseurl >}}/rancher/v2.x/en/upgrades/restorations/single-node-restoration/).