public/rancher-docs
1
0
Fork 0
mirror of https://github.com/rancher/rancher-docs.git synced 2026-09-25 12:38:05 +00:00
Code Issues Packages Projects Releases Wiki Activity

Update the etcd restoration process in RKE docs (#1679)

Browse Source 
* Edit RKE backup and recovery docs

* Edit RKE backup and recovery docs

* Edit RKE restore docs

* Add manual steps to older RKE restore process

* Make example scenario consistent with restore doc

* Remove node3 from table in rke restore scenario

* Remove node3 from both tables in rke restore doc

* Fix typo

* Add numbers to example scenario steps
This commit is contained in:
Catherine Luse
2019-09-05 13:22:27 -07:00
committed by Denise
parent 4bf1fac1d0
commit 4bb7aa6c76
6 changed files with 585 additions and 282 deletions
Download Patch File Download Diff File Expand all files Collapse all files
content/rke/latest/en/etcd-snapshots/_index.md
+9 -282
View File
@@ -11,295 +11,22 @@ RKE clusters can be configured to automatically take snapshots of etcd. In a dis
_Available as of v0.2.0_
RKE can also upload your snapshots to a S3 compatible backend. Additionally, the **pki.bundle.tar.gz** file usage is no longer required as v0.2.0 has changed how the [Kubernetes cluster state is stored]({{< baseurl >}}/rke/latest/en/installation/#kubernetes-cluster-state).
RKE can upload your snapshots to a S3 compatible backend.
## One-Time Snapshots
**Note:** As of RKE v0.2.0, the `pki.bundle.tar.gz` file is no longer required because of a change in how the [Kubernetes cluster state is stored]({{< baseurl >}}/rke/latest/en/installation/#kubernetes-cluster-state).
The `rke etcd snapshot-save` command will save a snapshot of etcd from each etcd node in the cluster config file. The snapshot is saved in `/opt/rke/etcd-snapshots`. When running the command, an additional container is created to take the snapshot. When the snapshot is completed, the container is automatically removed.
# Backing Up a Cluster
Prior to v0.2.0, along with the individual snapshot, RKE saves a backup of the certificates, i.e. a file named `pki.bundle.tar.gz`, in the same location. The snapshot and pki bundle file are required for the restore process in versions prior to v0.2.0.
You can create [one-time snapshots]({{<baseurl>}}/rke/latest/en/etcd-snapshots/one-time-snapshots) to back up your cluster, and you can also configure [recurring snapshots]({{<baseurl>}}/rke/latest/en/etcd-snapshots/recurring-snapshots).
As of v0.2.0, the one-time snapshot can be uploaded to a S3 compatible backend by using the additional options to specify the S3 backend.
# Restoring a Cluster from Backup
### Options for `rke etcd snapshot-save`
You can use RKE to [restore your cluster from backup]({{<baseurl>}}/rke/latest/en/etcd-snapshots/restoring-from-backup).
| Option | Description | S3 Specific |
| --- | --- | --- |
| `--name` value | Specify snapshot name | |
| `--config` value | Specify an alternate cluster YAML file (default: "cluster.yml") [$RKE_CONFIG] | |
| `--s3` | Enabled backup to s3 | * |
| `--s3-endpoint` value | Specify s3 endpoint url (default: "s3.amazonaws.com") | * |
| `--access-key` value | Specify s3 accessKey | * |
| `--secret-key` value | Specify s3 secretKey | * |
| `--bucket-name` value | Specify s3 bucket name | * |
| `--region` value | Specify the s3 bucket location (optional) | * |
| `--ssh-agent-auth` | [Use SSH Agent Auth defined by SSH_AUTH_SOCK]({{< baseurl >}}/rke/latest/en/config-options/#ssh-agent) | |
| `--ignore-docker-version` | [Disable Docker version check]({{< baseurl >}}/rke/latest/en/config-options/#supported-docker-versions) |
# Example Scenarios
### IAM Support for Storing Snapshots in S3
In addition to API access keys, RKE supports using IAM roles for S3 authentication. The cluster etcd nodes must be assigned an IAM role that has read/write access to the designated backup bucket on S3. Also, the nodes must have network access to the S3 endpoint specified.
To give an application access to S3, refer to the AWS documentation on [Using an IAM Role to Grant Permissions to Applications Running on Amazon EC2 Instances.](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html)
### Local One-Time Snapshot Example
```
$ rke etcd snapshot-save --config cluster.yml --name snapshot-name
```
The snapshot is saved in `/opt/rke/etcd-snapshots`
### One-Time Snapshots uploaded to S3 Example
_Available as of v0.2.0_
```
$ rke etcd snapshot-save --config cluster.yml --name snapshot-name \
--s3 --access-key S3_ACCESS_KEY --secret-key S3_SECRET_KEY \
--bucket-name s3-bucket-name --s3-endpoint s3.amazonaws.com
```
The snapshot is saved in `/opt/rke/etcd-snapshots` as well as uploaded to the S3 backend.
## Recurring Snapshots
To schedule automatic recurring etcd snapshots, you can enable the `etcd-snapshot` service with [extra configuration options the etcd service](#options-for-the-etcd-snapshot-service). `etcd-snapshot` runs in a service container alongside the `etcd` container. By default, the `etcd-snapshot` service takes a snapshot for every node that has the `etcd` role and stores them to local disk in `/opt/rke/etcd-snapshots`. If you set up the [options for S3](#options-for-the-etcd-snapshot-service), the snapshot will also be uploaded to the S3 backend.
Prior to v0.2.0, along with the snapshots, RKE saves a backup of the certificates, i.e. a file named `pki.bundle.tar.gz`, in the same location. The snapshot and pki bundle file are required for the restore process in versions prior to v0.2.0.
When a cluster is launched with the `etcd-snapshot` service enabled, you can view the `etcd-rolling-snapshots` logs to confirm backups are being created automatically.
```
$ docker logs etcd-rolling-snapshots
time="2018-05-04T18:39:16Z" level=info msg="Initializing Rolling Backups" creation=1m0s retention=24h0m0s
time="2018-05-04T18:40:16Z" level=info msg="Created backup" name="2018-05-04T18:40:16Z_etcd" runtime=108.332814ms
time="2018-05-04T18:41:16Z" level=info msg="Created backup" name="2018-05-04T18:41:16Z_etcd" runtime=92.880112ms
time="2018-05-04T18:42:16Z" level=info msg="Created backup" name="2018-05-04T18:42:16Z_etcd" runtime=83.67642ms
time="2018-05-04T18:43:16Z" level=info msg="Created backup" name="2018-05-04T18:43:16Z_etcd" runtime=86.298499ms
```
### Options for the `Etcd-Snapshot` Service
Depending on your version of RKE, the options used to configure recurring snapshots may be different.
_Available as of v0.2.0_
|Option|Description| S3 Specific |
|---|---| --- |
|**interval_hours**| The duration in hours between recurring backups. This supercedes the `creation` option and will override it if both are specified.| |
|**retention**| The number of snapshots to retain before rotation. This supercedes the `retention` option and will override it if both are specified.| |
|**bucket_name**| S3 bucket name where backups will be stored| * |
|**access_key**| S3 access key with permission to access the backup bucket.| * |
|**secret_key** |S3 secret key with permission to access the backup bucket.| * |
|**region** |S3 region for the backup bucket. This is optional.| * |
|**endpoint** |S3 regions endpoint for the backup bucket.| * |
<br>
```yaml
services:
etcd:
backup_config:
interval_hours: 12
retention: 6
s3backupconfig:
access_key: S3_ACCESS_KEY
secret_key: S3_SECRET_KEY
bucket_name: s3-bucket-name
region: ""
endpoint: s3.amazonaws.com
```
#### Prior to v0.2.0
|Option|Description|
|---|---|
|**Snapshot**|By default, the recurring snapshot service is disabled. To enable the service, you need to define it as part of `etcd` and set it to `true`.|
|**Creation**|By default, the snapshot service will take snapshots every 5 minutes (`5m0s`). You can change the time between snapshots as part of the `creation` directive for the `etcd` service.|
|**Retention**|By default, all snapshots are saved for 24 hours (`24h`) before being deleted and purged. You can change how long to store a snapshot as part of the `retention` directive for the `etcd` service.|
```yaml
services:
etcd:
snapshot: true
creation: 5m0s
retention: 24h
```
## Etcd Disaster Recovery
If there is a disaster with your Kubernetes cluster, you can use `rke etcd snapshot-restore` to recover your etcd. This command reverts etcd to a specific snapshot. RKE also removes the old `etcd` container before creating a new `etcd` cluster using the snapshot that you have chosen.
>**Warning:** Restoring an etcd snapshot deletes your current etcd cluster and replaces it with a new one. Before you run the `rke etcd snapshot-restore` command, you should back up any important data in your cluster.
The snapshot used to restore your etcd cluster can either be stored locally in `/opt/rke/etcd-snapshots` or from a S3 compatible backend. The S3 backend option is available as of v0.2.0.
### Options for `rke etcd snapshot-restore`
| Option | Description | S3 Specific |
| --- | --- | ---|
| `--name` value | Specify snapshot name | |
| `--config` value | Specify an alternate cluster YAML file (default: "cluster.yml") [$RKE_CONFIG] | |
| `--s3` | Enabled backup to s3 |* |
| `--s3-endpoint` value | Specify s3 endpoint url (default: "s3.amazonaws.com") | * |
| `--access-key` value | Specify s3 accessKey | *|
| `--secret-key` value | Specify s3 secretKey | *|
| `--bucket-name` value | Specify s3 bucket name | *|
| `--region` value | Specify the s3 bucket location (optional) | *|
| `--ssh-agent-auth` | [Use SSH Agent Auth defined by SSH_AUTH_SOCK]({{< baseurl >}}/rke/latest/en/config-options/#ssh-agent) | |
| `--ignore-docker-version` | [Disable Docker version check]({{< baseurl >}}/rke/latest/en/config-options/#supported-docker-versions) |
### Example of Restoring from a Local Snapshot
When restoring etcd from a local snapshot, the snapshot is assumed to be located in `/opt/rke/etcd-snapshots`. In versions prior to v0.2.0, the `pki.bundle.tar.gz` file is also expected to be in the same location. As of v0.2.0, this file is no longer needed as v0.2.0 has changed how the [Kubernetes cluster state is stored]({{< baseurl >}}/rke/latest/en/installation/#kubernetes-cluster-state).
```
$ rke etcd snapshot-restore --config cluster.yml --name mysnapshot
```
### Example of Restoring from a Snapshot in S3
_Available as of v0.2.0_
> **Note:** Ensure your `cluster.rkestate` is present before starting the restore, as this contains your certificate data for the cluster
When restoring etcd from a snapshot located in S3, the command needs the S3 information in order to connect to the S3 backend and retrieve the snapshot.
```shell
$ rke etcd snapshot-restore --config cluster.yml --name snapshot-name \
--s3 --access-key S3_ACCESS_KEY --secret-key S3_SECRET_KEY \
--bucket-name s3-bucket-name --s3-endpoint s3.amazonaws.com
```
> **Note:** if you were restoring a cluster that had rancher installed the UI should start-up after a few minutes; you don't need to re-run helm.
### Example Scenario of restoring from a Local Snapshot
In this example, the Kubernetes cluster was deployed on two AWS nodes.
| Name | IP | Role |
|:-----:|:--------:|:----------------------:|
| node1 | 10.0.0.1 | [controlplane, worker] |
| node2 | 10.0.0.2 | [etcd] |
### Back up the `etcd` cluster
Take a local snapshot of the Kubernetes cluster. As of v0.2.0, you can also upload this snapshot directly to a S3 backend with the [S3 options](#options-for-rke-etcd-snapshot-save).
```
$ rke etcd snapshot-save --name snapshot.db --config cluster.yml
```
![etcd snapshot]({{< baseurl >}}/img/rke/rke-etcd-backup.png)
### Store the Snapshot Externally in S3
As of v0.2.0, this step is no longer required, as RKE can upload and download snapshots automatically from S3 by adding in [S3 options](#options-for-rke-etcd-snapshot-save) when running the `rke etcd snapshot-save` command.
After taking the etcd snapshot on `node2`, we recommend saving this backup in a persistence place. One of the options is to save the backup and `pki.bundle.tar.gz` file on a S3 bucket or tape backup.
> **Note:** As of v0.2.0, the file **pki.bundle.tar.gz** is no longer required for the restore process.
```
# If you're using an AWS host and have the ability to connect to S3
root@node2:~# s3cmd mb s3://rke-etcd-backup
root@node2:~# s3cmd /opt/rke/etcd-snapshots/snapshot.db /opt/rke/etcd-snapshots/pki.bundle.tar.gz s3://rke-etcd-backup/
```
### Place the backup on a new node
To simulate the failure, let's power down `node2`.
```
root@node2:~# poweroff
```
| Name | IP | Role |
|:-----:|:--------:|:----------------------:|
| node1 | 10.0.0.1 | [controlplane, worker] |
| ~~node2~~ | ~~10.0.0.2~~ | ~~[etcd]~~ |
| node3 | 10.0.0.3 | [etcd] |
| | | |
Before restoring etcd and running `rke up`, we need to retrieve the backup saved on S3 to a new node, e.g. `node3`. As of v0.2.0, you can directly retrieve the snapshot from S3 when running the restore command, so this step is for users who stored the snapshot externally without using the integrated S3 options.
```
# Make a Directory
root@node3:~# mkdir -p /opt/rke/etcdbackup
# Get the Backup from S3
root@node3:~# s3cmd get s3://rke-etcd-backup/snapshot.db /opt/rke/etcd-snapshots/snapshot.db
# Get the pki bundle from S3, only needed prior to v0.2.0
root@node3:~# s3cmd get s3://rke-etcd-backup/pki.bundle.tar.gz /opt/rke/etcd-snapshots/pki.bundle.tar.gz
```
### Restore `etcd` on the new node from the backup
Before updating and restoring etcd, you will need to add the new node into the Kubernetes cluster with the `etcd` role. In the `cluster.yml`, comment out the old node and add in the new node. `
```yaml
nodes:
- address: 10.0.0.1
hostname_override: node1
user: ubuntu
role:
- controlplane
- worker
# - address: 10.0.0.2
# hostname_override: node2
# user: ubuntu
# role:
# - etcd
- address: 10.0.0.3
hostname_override: node3
user: ubuntu
role:
- etcd
```
After the new node is added to the `cluster.yml`, run `rke etcd snapshot-restore` to launch `etcd` from the backup. The snapshot and `pki.bundle.tar.gz` file are expected to be saved at `/opt/rke/etcd-snapshots`.
As of v0.2.0, if you want to directly retrieve the snapshot from S3, add in the [S3 options](#options-for-rke-etcd-snapshot-restore).
> **Note:** As of v0.2.0, the file **pki.bundle.tar.gz** is no longer required for the restore process as the certificates required to restore are preserved within the `cluster.rkestate`
```
$ rke etcd snapshot-restore --name snapshot.db --config cluster.yml
```
Finally, we need to restore the operations on the cluster by making the Kubernetes API point to the new `etcd` by running `rke up` again using the new `cluster.yml`.
```
$ rke up --config cluster.yml
```
Confirm that your Kubernetes cluster is functional by checking the pods on your cluster.
```
> kubectl get pods
NAME READY STATUS RESTARTS AGE
nginx-65899c769f-kcdpr 1/1 Running 0 17s
nginx-65899c769f-pc45c 1/1 Running 0 17s
nginx-65899c769f-qkhml 1/1 Running 0 17s
```
These [example scenarios]({{<baseurl>}}/rke/latest/en/etcd-snapshots/example-scenarios) for backup and restore are different based on your version of RKE.
## Troubleshooting
As of **v0.1.9**, the **rke-bundle-cert** container is removed on both success and failure of a restore. To debug any issues, you will need to look at the **logs** generated from rke.
As of **v0.1.8** and below, the **rke-bundle-cert** container is left over from a failed etcd restore. If you are having an issue with restoring an **etcd snapshot** then you can do the following on each etcd nodes before attempting to do another restore:
```
docker container rm --force rke-bundle-cert
```
The rke-bundle-cert container is usually removed when a backup or restore of **etcd** succeeds. Whenever something goes wrong, the **rke-bundle-cert** container will be left over. You can look
at the logs or inspect the container to see what the issue is.
```
docker container logs --follow rke-bundle-cert
docker container inspect rke-bundle-cert
```
The important thing to note is the mounts of the container and location of the **pki.bundle.tar.gz**.
If you have trouble restoring your cluster, you can refer to the [troubleshooting]({{<baseurl>}}/rke/latest/en/etcd-snapshots/troubleshooting) page.
content/rke/latest/en/etcd-snapshots/example-scenarios/_index.md
+253
View File
@@ -0,0 +1,253 @@
---
title: Example Scenarios
weight: 4
---
These example scenarios for backup and restore are different based on your version of RKE.
{{% tabs %}}
{{% tab "RKE v0.2.0+" %}}
This walkthrough will demonstrate how to restore an etcd cluster from a local snapshot with the following steps:
1. [Back up the cluster](#1-back-up-the-cluster)
1. [Simulate a node failure](#2-simulate-a-node-failure)
1. [Add a new etcd node to the cluster](#3-add-a-new-etcd-node-to-the-kubernetes-cluster)
1. [Restore etcd on the new node from the backup](#4-restore-etcd-on-the-new-node-from-the-backup)
1. [Confirm that cluster operations are restored](#5-confirm-that-cluster-operations-are-restored)
In this example, the Kubernetes cluster was deployed on two AWS nodes.
| Name | IP | Role |
|:-----:|:--------:|:----------------------:|
| node1 | 10.0.0.1 | [controlplane, worker] |
| node2 | 10.0.0.2 | [etcd] |
### 1. Back Up the Cluster
Take a local snapshot of the Kubernetes cluster.
You can upload this snapshot directly to an S3 backend with the [S3 options]({{<baseurl>}}/rke/latest/en/etcd-snapshots/one-time-snapshots/#options-for-rke-etcd-snapshot-save).
```
$ rke etcd snapshot-save --name snapshot.db --config cluster.yml
```
![etcd snapshot]({{< baseurl >}}/img/rke/rke-etcd-backup.png)
### 2. Simulate a Node Failure
To simulate the failure, let's power down `node2`.
```
root@node2:~# poweroff
```
| Name | IP | Role |
|:-----:|:--------:|:----------------------:|
| node1 | 10.0.0.1 | [controlplane, worker] |
| ~~node2~~ | ~~10.0.0.2~~ | ~~[etcd]~~ |
### 3. Add a New etcd Node to the Kubernetes Cluster
Before updating and restoring etcd, you will need to add the new node into the Kubernetes cluster with the `etcd` role. In the `cluster.yml`, comment out the old node and add in the new node.
```yaml
nodes:
- address: 10.0.0.1
hostname_override: node1
user: ubuntu
role:
- controlplane
- worker
# - address: 10.0.0.2
# hostname_override: node2
# user: ubuntu
# role:
# - etcd
- address: 10.0.0.3
hostname_override: node3
user: ubuntu
role:
- etcd
```
### 4. Restore etcd on the New Node from the Backup
> **Prerequisite:** Ensure your `cluster.rkestate` is present before starting the restore, because this contains your certificate data for the cluster.
After the new node is added to the `cluster.yml`, run the `rke etcd snapshot-restore` to launch `etcd` from the backup:
```
$ rke etcd snapshot-restore --name snapshot.db --config cluster.yml
```
The snapshot is expected to be saved at `/opt/rke/etcd-snapshots`.
If you want to directly retrieve the snapshot from S3, add in the [S3 options](#options-for-rke-etcd-snapshot-restore).
> **Note:** As of v0.2.0, the file `pki.bundle.tar.gz` is no longer required for the restore process because the certificates required to restore are preserved within the `cluster.rkestate`.
### 5. Confirm that Cluster Operations are Restored
The `rke etcd snapshot-restore` command triggers `rke up` using the new `cluster.yml`. Confirm that your Kubernetes cluster is functional by checking the pods on your cluster.
```
> kubectl get pods
NAME READY STATUS RESTARTS AGE
nginx-65899c769f-kcdpr 1/1 Running 0 17s
nginx-65899c769f-pc45c 1/1 Running 0 17s
nginx-65899c769f-qkhml 1/1 Running 0 17s
```
{{% /tab %}}
{{% tab "RKE prior to v0.2.0" %}}
This walkthrough will demonstrate how to restore an etcd cluster from a local snapshot with the following steps:
1. [Take a local snapshot of the cluster](#take-a-local-snapshot-of-the-cluster-rke-prior-to-v0.2.0)
1. [Store the snapshot externally](#store-the-snapshot-externally-rke-prior-to-v0.2.0)
1. [Simulate a node failure](#simulate-a-node-failure-rke-prior-to-v0.2.0)
1. [Remove the Kubernetes cluster and clean the nodes](#remove-the-kubernetes-cluster-and-clean-the-nodes-rke-prior-to-v0.2.0)
1. [Retrieve the backup and place it on a new node](#retrieve-the-backup-and-place-it-on-a-new-node-rke-prior-to-v0.2.0)
1. [Add a new etcd node to the Kubernetes cluster](#add-a-new-etcd-node-to-the-kubernetes-cluster-rke-prior-to-v0.2.0)
1. [Restore etcd on the new node from the backup](#restore-etcd-on-the-new-node-from-the-backup-rke-prior-to-v0.2.0)
1. [Restore Operations on the Cluster](#restore-operations-on-the-cluster-rke-prior-to-v0.2.0)
### Example Scenario of restoring from a Local Snapshot
In this example, the Kubernetes cluster was deployed on two AWS nodes.
| Name | IP | Role |
|:-----:|:--------:|:----------------------:|
| node1 | 10.0.0.1 | [controlplane, worker] |
| node2 | 10.0.0.2 | [etcd] |
<a id="take-a-local-snapshot-of-the-cluster-rke-prior-to-v0.2.0"></a>
### 1. Take a Local Snapshot of the Cluster
Back up the Kubernetes cluster by taking a local snapshot:
```
$ rke etcd snapshot-save --name snapshot.db --config cluster.yml
```
![etcd snapshot]({{< baseurl >}}/img/rke/rke-etcd-backup.png)
<a id="store-the-snapshot-externally-rke-prior-to-v0.2.0"></a>
### 2. Store the Snapshot Externally
After taking the etcd snapshot on `node2`, we recommend saving this backup in a persistent place. One of the options is to save the backup and `pki.bundle.tar.gz` file on an S3 bucket or tape backup.
```
# If you're using an AWS host and have the ability to connect to S3
root@node2:~# s3cmd mb s3://rke-etcd-backup
root@node2:~# s3cmd \
/opt/rke/etcd-snapshots/snapshot.db \
/opt/rke/etcd-snapshots/pki.bundle.tar.gz \
s3://rke-etcd-backup/
```
<a id="simulate-a-node-failure-rke-prior-to-v0.2.0"></a>
### 3. Simulate a Node Failure
To simulate the failure, let's power down `node2`.
```
root@node2:~# poweroff
```
| Name | IP | Role |
|:-----:|:--------:|:----------------------:|
| node1 | 10.0.0.1 | [controlplane, worker] |
| ~~node2~~ | ~~10.0.0.2~~ | ~~[etcd]~~ |
<a id="remove-the-kubernetes-cluster-and-clean-the-nodes-rke-prior-to-v0.2.0"></a>
### 4. Remove the Kubernetes Cluster and Clean the Nodes
The following command removes your cluster and cleans the nodes so that the cluster can be restored without any conflicts:
```
rke remove --config rancher-cluster.yml
```
<a id="retrieve-the-backup-and-place-it-on-a-new-node-rke-prior-to-v0.2.0"></a>
### 5. Retrieve the Backup and Place it On a New Node
Before restoring etcd and running `rke up`, we need to retrieve the backup saved on S3 to a new node, e.g. `node3`.
```
# Make a Directory
root@node3:~# mkdir -p /opt/rke/etcdbackup
# Get the Backup from S3
root@node3:~# s3cmd get \
s3://rke-etcd-backup/snapshot.db \
/opt/rke/etcd-snapshots/snapshot.db
# Get the pki bundle from S3
root@node3:~# s3cmd get \
s3://rke-etcd-backup/pki.bundle.tar.gz \
/opt/rke/etcd-snapshots/pki.bundle.tar.gz
```
> **Note:** If you had multiple etcd nodes, you would have to manually sync the snapshot and `pki.bundle.tar.gz` across all of the etcd nodes in the cluster.
<a id="add-a-new-etcd-node-to-the-kubernetes-cluster-rke-prior-to-v0.2.0"></a>
### 6. Add a New etcd Node to the Kubernetes Cluster
Before updating and restoring etcd, you will need to add the new node into the Kubernetes cluster with the `etcd` role. In the `cluster.yml`, comment out the old node and add in the new node. `
```yaml
nodes:
- address: 10.0.0.1
hostname_override: node1
user: ubuntu
role:
- controlplane
- worker
# - address: 10.0.0.2
# hostname_override: node2
# user: ubuntu
# role:
# - etcd
- address: 10.0.0.3
hostname_override: node3
user: ubuntu
role:
- etcd
```
<a id="restore-etcd-on-the-new-node-from-the-backup-rke-prior-to-v0.2.0"></a>
### 7. Restore etcd on the New Node from the Backup
After the new node is added to the `cluster.yml`, run the `rke etcd snapshot-restore` command to launch `etcd` from the backup:
```
$ rke etcd snapshot-restore --name snapshot.db --config cluster.yml
```
The snapshot and `pki.bundle.tar.gz` file are expected to be saved at `/opt/rke/etcd-snapshots` on each etcd node.
<a id="restore-operations-on-the-cluster-rke-prior-to-v0.2.0"></a>
### 8. Restore Operations on the Cluster
Finally, we need to restore the operations on the cluster. We will make the Kubernetes API point to the new `etcd` by running `rke up` again using the new `cluster.yml`.
```
$ rke up --config cluster.yml
```
Confirm that your Kubernetes cluster is functional by checking the pods on your cluster.
```
> kubectl get pods
NAME READY STATUS RESTARTS AGE
nginx-65899c769f-kcdpr 1/1 Running 0 17s
nginx-65899c769f-pc45c 1/1 Running 0 17s
nginx-65899c769f-qkhml 1/1 Running 0 17s
```
{{% /tab %}}
{{% /tabs %}}
content/rke/latest/en/etcd-snapshots/one-time-snapshots/_index.md
+92
View File
@@ -0,0 +1,92 @@
---
title: One-time Snapshots
weight: 1
---
One-time snapshots are handled differently depending on your version of RKE.
{{% tabs %}}
{{% tab "RKE v0.2.0+" %}}
To save a snapshot of etcd from each etcd node in the cluster config file, run the `rke etcd snapshot-save` command.
The snapshot is saved in `/opt/rke/etcd-snapshots`.
When running the command, an additional container is created to take the snapshot. When the snapshot is completed, the container is automatically removed.
The one-time snapshot can be uploaded to a S3 compatible backend by using the additional options to specify the S3 backend.
To create a local one-time snapshot, run:
```
$ rke etcd snapshot-save --config cluster.yml --name snapshot-name
```
**Result:** The snapshot is saved in `/opt/rke/etcd-snapshots`.
To save a one-time snapshot to S3, run:
```
$ rke etcd snapshot-save \
--config cluster.yml \
--name snapshot-name \
--s3 \
--access-key S3_ACCESS_KEY \
--secret-key S3_SECRET_KEY \
--bucket-name s3-bucket-name \
--s3-endpoint s3.amazonaws.com
```
**Result:** The snapshot is saved in `/opt/rke/etcd-snapshots` as well as uploaded to the S3 backend.
### Options for `rke etcd snapshot-save`
| Option | Description | S3 Specific |
| --- | --- | --- |
| `--name` value | Specify snapshot name | |
| `--config` value | Specify an alternate cluster YAML file (default: `cluster.yml`) [$RKE_CONFIG] | |
| `--s3` | Enabled backup to s3 | * |
| `--s3-endpoint` value | Specify s3 endpoint url (default: "s3.amazonaws.com") | * |
| `--access-key` value | Specify s3 accessKey | * |
| `--secret-key` value | Specify s3 secretKey | * |
| `--bucket-name` value | Specify s3 bucket name | * |
| `--region` value | Specify the s3 bucket location (optional) | * |
| `--ssh-agent-auth` | [Use SSH Agent Auth defined by SSH_AUTH_SOCK]({{< baseurl >}}/rke/latest/en/config-options/#ssh-agent) | |
| `--ignore-docker-version` | [Disable Docker version check]({{< baseurl >}}/rke/latest/en/config-options/#supported-docker-versions) |
### IAM Support for Storing Snapshots in S3
In addition to API access keys, RKE supports using IAM roles for S3 authentication. The cluster etcd nodes must be assigned an IAM role that has read/write access to the designated backup bucket on S3. Also, the nodes must have network access to the S3 endpoint specified.
To give an application access to S3, refer to the AWS documentation on [Using an IAM Role to Grant Permissions to Applications Running on Amazon EC2 Instances.](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html)
{{% /tab %}}
{{% tab "RKE prior to v0.2.0" %}}
To save a snapshot of etcd from each etcd node in the cluster config file, run the `rke etcd snapshot-save` command.
When running the command, an additional container is created to take the snapshot. When the snapshot is completed, the container is automatically removed.
RKE saves a backup of the certificates, i.e. a file named `pki.bundle.tar.gz`, in the same location. The snapshot and pki bundle file are required for the restore process.
To create a local one-time snapshot, run:
```
$ rke etcd snapshot-save --config cluster.yml --name snapshot-name
```
**Result:** The snapshot is saved in `/opt/rke/etcd-snapshots`.
### Options for `rke etcd snapshot-save`
| Option | Description |
| --- | --- |
| `--name` value | Specify snapshot name |
| `--config` value | Specify an alternate cluster YAML file (default: `cluster.yml`) [$RKE_CONFIG] |
| `--ssh-agent-auth` | [Use SSH Agent Auth defined by SSH_AUTH_SOCK]({{< baseurl >}}/rke/latest/en/config-options/#ssh-agent) |
| `--ignore-docker-version` | [Disable Docker version check]({{< baseurl >}}/rke/latest/en/config-options/#supported-docker-versions) |
{{% /tab %}}
{{% /tabs %}}
content/rke/latest/en/etcd-snapshots/recurring-snapshots/_index.md
+97
View File
@@ -0,0 +1,97 @@
---
title: Recurring Snapshots
weight: 2
---
Recurring snapshots are handled differently based on your version of RKE.
{{% tabs %}}
{{% tab "RKE v0.2.0+"%}}
To schedule automatic recurring etcd snapshots, you can enable the `etcd-snapshot` service with [extra configuration options](#options-for-the-etcd-snapshot-service). `etcd-snapshot` runs in a service container alongside the `etcd` container. By default, the `etcd-snapshot` service takes a snapshot for every node that has the `etcd` role and stores them to local disk in `/opt/rke/etcd-snapshots`.
If you set up the [options for S3](#options-for-the-etcd-snapshot-service), the snapshot will also be uploaded to the S3 backend.
### Snapshot Service Logging
When a cluster is launched with the `etcd-snapshot` service enabled, you can view the `etcd-rolling-snapshots` logs to confirm backups are being created automatically.
```
$ docker logs etcd-rolling-snapshots
time="2018-05-04T18:39:16Z" level=info msg="Initializing Rolling Backups" creation=1m0s retention=24h0m0s
time="2018-05-04T18:40:16Z" level=info msg="Created backup" name="2018-05-04T18:40:16Z_etcd" runtime=108.332814ms
time="2018-05-04T18:41:16Z" level=info msg="Created backup" name="2018-05-04T18:41:16Z_etcd" runtime=92.880112ms
time="2018-05-04T18:42:16Z" level=info msg="Created backup" name="2018-05-04T18:42:16Z_etcd" runtime=83.67642ms
time="2018-05-04T18:43:16Z" level=info msg="Created backup" name="2018-05-04T18:43:16Z_etcd" runtime=86.298499ms
```
### Options for the `Etcd-Snapshot` Service
|Option|Description| S3 Specific |
|---|---| --- |
|**interval_hours**| The duration in hours between recurring backups. This supercedes the `creation` option (which was used in RKE prior to v0.2.0) and will override it if both are specified.| |
|**retention**| The number of snapshots to retain before rotation. This supercedes the `retention` option and will override it if both are specified.| |
|**bucket_name**| S3 bucket name where backups will be stored| * |
|**access_key**| S3 access key with permission to access the backup bucket.| * |
|**secret_key** |S3 secret key with permission to access the backup bucket.| * |
|**region** |S3 region for the backup bucket. This is optional.| * |
|**endpoint** |S3 regions endpoint for the backup bucket.| * |
### Configuring the Snapshot Service in YAML
```yaml
services:
etcd:
backup_config:
interval_hours: 12
retention: 6
s3backupconfig:
access_key: S3_ACCESS_KEY
secret_key: S3_SECRET_KEY
bucket_name: s3-bucket-name
region: ""
endpoint: s3.amazonaws.com
```
{{% /tab %}}
{{% tab "RKE prior to v0.2.0"%}}
To schedule automatic recurring etcd snapshots, you can enable the `etcd-snapshot` service with [extra configuration options](#options-for-the-local-etcd-snapshot-service). `etcd-snapshot` runs in a service container alongside the `etcd` container. By default, the `etcd-snapshot` service takes a snapshot for every node that has the `etcd` role and stores them to local disk in `/opt/rke/etcd-snapshots`.
RKE saves a backup of the certificates, i.e. a file named `pki.bundle.tar.gz`, in the same location. The snapshot and pki bundle file are required for the restore process in versions prior to v0.2.0.
### Snapshot Service Logging
When a cluster is launched with the `etcd-snapshot` service enabled, you can view the `etcd-rolling-snapshots` logs to confirm backups are being created automatically.
```
$ docker logs etcd-rolling-snapshots
time="2018-05-04T18:39:16Z" level=info msg="Initializing Rolling Backups" creation=1m0s retention=24h0m0s
time="2018-05-04T18:40:16Z" level=info msg="Created backup" name="2018-05-04T18:40:16Z_etcd" runtime=108.332814ms
time="2018-05-04T18:41:16Z" level=info msg="Created backup" name="2018-05-04T18:41:16Z_etcd" runtime=92.880112ms
time="2018-05-04T18:42:16Z" level=info msg="Created backup" name="2018-05-04T18:42:16Z_etcd" runtime=83.67642ms
time="2018-05-04T18:43:16Z" level=info msg="Created backup" name="2018-05-04T18:43:16Z_etcd" runtime=86.298499ms
```
### Options for the Local `Etcd-Snapshot` Service
|Option|Description|
|---|---|
|**Snapshot**|By default, the recurring snapshot service is disabled. To enable the service, you need to define it as part of `etcd` and set it to `true`.|
|**Creation**|By default, the snapshot service will take snapshots every 5 minutes (`5m0s`). You can change the time between snapshots as part of the `creation` directive for the `etcd` service.|
|**Retention**|By default, all snapshots are saved for 24 hours (`24h`) before being deleted and purged. You can change how long to store a snapshot as part of the `retention` directive for the `etcd` service.|
### Configuring the Snapshot Service in YAML
```yaml
services:
etcd:
snapshot: true
creation: 5m0s
retention: 24h
```
{{% /tab %}}
{{% /tabs %}}
content/rke/latest/en/etcd-snapshots/restoring-from-backup/_index.md
+112
View File
@@ -0,0 +1,112 @@
---
title: Restoring from Backup
weight: 3
---
The details of restoring your cluster from backup are different depending on your version of RKE.
{{% tabs %}}
{{% tab "RKE v0.2.0+"%}}
If there is a disaster with your Kubernetes cluster, you can use `rke etcd snapshot-restore` to recover your etcd. This command reverts the etcd to a specific snapshot. The following actions are included in the command:
- Syncs the snapshot or downloads the snapshot from S3, if necessary.
- Checks snapshot checksum across etcd nodes to make sure they are identical.
- Deletes your current cluster and cleans old data by running `rke remove`. This removes the entire Kubernetes cluster, not just the etcd cluster.
- Rebuilds the etcd cluster from the chosen snapshot.
- Creates a new cluster by running `rke up`.
- Restarts cluster system pods.
>**Warning:** You should back up any important data in your cluster before running `rke etcd snapshot-restore` because the command deletes your current Kubernetes cluster and replaces it with a new one.
The snapshot used to restore your etcd cluster can either be stored locally in `/opt/rke/etcd-snapshots` or from a S3 compatible backend.
### Example of Restoring from a Local Snapshot
To restore etcd from a local snapshot, run:
```
$ rke etcd snapshot-restore --config cluster.yml --name mysnapshot
```
The snapshot is assumed to be located in `/opt/rke/etcd-snapshots`.
**Note:** The `pki.bundle.tar.gz` file is not needed because RKE v0.2.0 changed how the [Kubernetes cluster state is stored]({{< baseurl >}}/rke/latest/en/installation/#kubernetes-cluster-state).
### Example of Restoring from a Snapshot in S3
> **Prerequisite:** Ensure your `cluster.rkestate` is present before starting the restore, because this contains your certificate data for the cluster.
When restoring etcd from a snapshot located in S3, the command needs the S3 information in order to connect to the S3 backend and retrieve the snapshot.
```shell
$ rke etcd snapshot-restore \
--config cluster.yml \
--name snapshot-name \
--s3 \
--access-key S3_ACCESS_KEY \
--secret-key S3_SECRET_KEY \
--bucket-name s3-bucket-name \
--s3-endpoint s3.amazonaws.com
```
**Note:** if you were restoring a cluster that had Rancher installed, the Rancher UI should start up after a few minutes; you don't need to re-run Helm.
### Options for `rke etcd snapshot-restore`
| Option | Description | S3 Specific |
| --- | --- | ---|
| `--name` value | Specify snapshot name | |
| `--config` value | Specify an alternate cluster YAML file (default: `cluster.yml`) [$RKE_CONFIG] | |
| `--s3` | Enabled backup to s3 |* |
| `--s3-endpoint` value | Specify s3 endpoint url (default: "s3.amazonaws.com") | * |
| `--access-key` value | Specify s3 accessKey | *|
| `--secret-key` value | Specify s3 secretKey | *|
| `--bucket-name` value | Specify s3 bucket name | *|
| `--region` value | Specify the s3 bucket location (optional) | *|
| `--ssh-agent-auth` | [Use SSH Agent Auth defined by SSH_AUTH_SOCK]({{< baseurl >}}/rke/latest/en/config-options/#ssh-agent) | |
| `--ignore-docker-version` | [Disable Docker version check]({{< baseurl >}}/rke/latest/en/config-options/#supported-docker-versions) |
{{% /tab %}}
{{% tab "RKE prior to v0.2.0"%}}
If there is a disaster with your Kubernetes cluster, you can use `rke etcd snapshot-restore` to recover your etcd. This command reverts etcd to a specific snapshot.
The following actions are included in `rke etcd snapshot-restore`:
- Removes the old etcd cluster
- Rebuilds the etcd cluster using the local snapshot
Before you run this command, you must:
- Run `rke remove` to remove your Kubernetes cluster and clean the nodes
- Download your etcd snapshot from S3, if applicable. Place the etcd snapshot and the `pki.bundle.tar.gz` file in `/opt/rke/etcd-snapshots`. Manually sync the snapshot across all `etcd` nodes.
After the restore, you must rebuild your Kubernetes cluster with `rke up`.
>**Warning:** You should back up any important data in your cluster before running `rke etcd snapshot-restore` because the command deletes your current etcd cluster and replaces it with a new one.
### Example of Restoring from a Local Snapshot
To restore etcd from a local snapshot, run:
```
$ rke etcd snapshot-restore --config cluster.yml --name mysnapshot
```
The snapshot is assumed to be located in `/opt/rke/etcd-snapshots`.
The snapshot must be manually synched across all `etcd` nodes.
The `pki.bundle.tar.gz` file is also expected to be in the same location.
### Options for `rke etcd snapshot-restore`
| Option | Description |
| --- | --- |
| `--name` value | Specify snapshot name |
| `--config` value | Specify an alternate cluster YAML file (default: `cluster.yml`) [$RKE_CONFIG] |
| `--ssh-agent-auth` | [Use SSH Agent Auth defined by SSH_AUTH_SOCK]({{< baseurl >}}/rke/latest/en/config-options/#ssh-agent) |
| `--ignore-docker-version` | [Disable Docker version check]({{< baseurl >}}/rke/latest/en/config-options/#supported-docker-versions) |
{{% /tab %}}
{{% /tabs %}}
content/rke/latest/en/etcd-snapshots/troubleshooting/_index.md
+22
View File
@@ -0,0 +1,22 @@
---
title: Troubleshooting
weight: 5
---
As of **v0.1.9**, the **rke-bundle-cert** container is removed on both success and failure of a restore. To debug any issues, you will need to look at the **logs** generated from rke.
As of **v0.1.8** and below, the **rke-bundle-cert** container is left over from a failed etcd restore. If you are having an issue with restoring an **etcd snapshot** then you can do the following on each etcd nodes before attempting to do another restore:
```
docker container rm --force rke-bundle-cert
```
The rke-bundle-cert container is usually removed when a backup or restore of **etcd** succeeds. Whenever something goes wrong, the **rke-bundle-cert** container will be left over. You can look
at the logs or inspect the container to see what the issue is.
```
docker container logs --follow rke-bundle-cert
docker container inspect rke-bundle-cert
```
The important thing to note is the mounts of the container and location of the `pki.bundle.tar.gz`.