public/rancher-docs
1
0
Fork 0
mirror of https://github.com/rancher/rancher-docs.git synced 2026-09-25 04:28:15 +00:00
Code Issues Packages Projects Releases Wiki Activity

rke snapshot fixes

Browse Source
This commit is contained in:
Denise Schannon
2019-03-22 10:23:26 -07:00
parent 664ab859df
commit 7f9e44164d
2 changed files with 106 additions and 151 deletions
Download Patch File Download Diff File Expand all files Collapse all files
content/rancher/v2.x/en/tools/snapshots/_index.md
+1 -1
View File
@@ -83,4 +83,4 @@ If your Kubernetes cluster is broken, you can restore the cluster from a snapsho
**Result:** The cluster will go into `updating` state and the process of restoring the `etcd` nodes from the snapshot will start. The cluster is restored when it returns to an `active` state.
> **Note**: If you are restoring a cluster with unavailable etcd nodes, it's recommended that all etcd nodes are removed from Rancher before attempting to restore. For clusters that were provisioned using [nodes hosted in an infrastructure provider]({{< baseurl >}}/rancher/v2.x/en/cluster-provisioning/rke-clusters/node-pools/), new etcd nodes will automatically be created. For [custom clusters]({{< baseurl >}}/rancher/v2.x/en/cluster-provisioning/rke-clusters/custom-nodes/), please ensure that you add new etcd nodes to the cluster.
> **Note:** If you are restoring a cluster with unavailable etcd nodes, it's recommended that all etcd nodes are removed from Rancher before attempting to restore. For clusters that were provisioned using [nodes hosted in an infrastructure provider]({{< baseurl >}}/rancher/v2.x/en/cluster-provisioning/rke-clusters/node-pools/), new etcd nodes will automatically be created. For [custom clusters]({{< baseurl >}}/rancher/v2.x/en/cluster-provisioning/rke-clusters/custom-nodes/), please ensure that you add new etcd nodes to the cluster.
content/rke/v0.1.x/en/etcd-snapshots/_index.md
+105 -150
View File
@@ -5,79 +5,60 @@ aliases:
- /rke/v0.1.x/en/installation/etcd-snapshots/
---
As of v0.1.7, you can configure a RKE cluster to automatically take snapshots of etcd. In a disaster scenario, you can restore these snapshots, which are stored on other nodes in the cluster.
_Available as of v0.1.7_
As of v0.2.0, you can use RKE to automatically upload your snapshots to an S3 compatible backend. Additionally, the **pki.bundle.tar.gz** file usage is no longer required as v0.2.0 uses a [newer state management model]({{< baseurl >}}/rke/v0.1.x/en/installation/#kubernetes-cluster-state).
RKE clusters can be configured to automatically take snapshots of etcd. In a disaster scenario, you can restore these snapshots, which are stored on other nodes in the cluster. Snapshots are always saved locally in `/opt/rke/etcd-snapshots`.
_Available as of v0.2.0_
RKE can also upload your snapshots to a S3 compatible backend. Additionally, the **pki.bundle.tar.gz** file usage is no longer required as v0.2.0 has changed how the [Kubernetes cluster state is stored]({{< baseurl >}}/rke/v0.1.x/en/installation/#kubernetes-cluster-state).
## One-Time Snapshots
RKE can take a one-time snapshot of a running etcd node in a RKE cluster. The snapshot is automatically saved in `/opt/rke/etcd-snapshots`.
The `rke etcd snapshot-save` command will save a snapshot of etcd from each etcd node in the cluster config file. The snapshot is saved in `/opt/rke/etcd-snapshots`. When running the command, an additional container is created to take the snapshot. When the snapshot is completed, the container is automatically removed.
As of v0.2.0, the one-time snapshot can be uploaded to a S3 compatible backend by using the additional options to specify the S3 backend.
### Options for `rke etcd snapshot-save`
| Option | Description | S3 Specific |
| --- | --- | --- |
| `--name` value | Specify snapshot name | |
| `--config` value | Specify an alternate cluster YAML file (default: "cluster.yml") [$RKE_CONFIG] | |
| `--s3` | Enabled backup to s3 | * |
| `--s3-endpoint` value | Specify s3 endpoint url (default: "s3.amazonaws.com") | * |
| `--access-key` value | Specify s3 accessKey | * |
| `--secret-key` value | Specify s3 secretKey | * |
| `--bucket-name` value | Specify s3 bucket name | * |
| `--region` value | Specify the s3 bucket location (optional) | * |
| `--ssh-agent-auth` | [Use SSH Agent Auth defined by SSH_AUTH_SOCK]({{< baseurl >}}/rke/v0.1.x/en/config-options/#ssh-agent) | |
| `--ignore-docker-version` | [Disable Docker version check]({{< baseurl >}}/rke/v0.1.x/en/config-options/#supported-docker-versions) |
### Local One-Time Snapshot Example
```
$ rke etcd snapshot-save --config cluster.yml
WARN[0000] Name of the snapshot is not specified using [rke_etcd_snapshot_2018-05-17T23:32:08+02:00]
INFO[0000] Starting saving snapshot on etcd hosts
INFO[0000] [dialer] Setup tunnel for host [x.x.x.x]
INFO[0001] [dialer] Setup tunnel for host [y.y.y.y]
INFO[0002] [dialer] Setup tunnel for host [z.z.z.z]
INFO[0003] [etcd] Saving snapshot [rke_etcd_snapshot_2018-05-17T23:32:08+02:00] on host [x.x.x.x]
INFO[0004] [etcd] Successfully started [etcd-snapshot-once] container on host [x.x.x.x]
INFO[0004] [etcd] Saving snapshot [rke_etcd_snapshot_2018-05-17T23:32:08+02:00] on host [y.y.y.y]
INFO[0005] [etcd] Successfully started [etcd-snapshot-once] container on host [y.y.y.y]
INFO[0005] [etcd] Saving snapshot [rke_etcd_snapshot_2018-05-17T23:32:08+02:00] on host [z.z.z.z]
INFO[0006] [etcd] Successfully started [etcd-snapshot-once] container on host [z.z.z.z]
INFO[0006] Finished saving snapshot [rke_etcd_snapshot_2018-05-17T23:32:08+02:00] on all etcd hosts
$ rke etcd snapshot-save --config cluster.yml --name snapshot-name
```
The command will save a snapshot of etcd from each etcd node in the cluster config file and will save it in `/opt/rke/etcd-snapshots`. When running the command, an additional container is created to take the snapshot. When the snapshot is completed, the container is automatically removed.
The snapshot is saved in `/opt/rke/etcd-snapshots`
### One-Time Snapshots uploaded to S3 Example
### S3 One-Time Snapshots
_Available as of v0.2.0_
As of v0.2.0, it's possible to use RKE to save and directly upload snapshots to S3.
```
$ rke etcd snapshot-save --help
NAME:
rke etcd snapshot-save - Take snapshot on all etcd hosts
USAGE:
rke etcd snapshot-save [command options] [arguments...]
OPTIONS:
--name value Specify snapshot name
--config value Specify an alternate cluster YAML file (default: "cluster.yml") [$RKE_CONFIG]
--s3 Enabled backup to s3
--s3-endpoint value Specify s3 endpoint url (default: "s3.amazonaws.com")
--access-key value Specify s3 accessKey
--secret-key value Specify s3 secretKey
--bucket-name value Specify s3 bucket name
--region value Specify the s3 bucket location (optional)
--ssh-agent-auth Use SSH Agent Auth defined by SSH_AUTH_SOCK
--ignore-docker-version Disable Docker version check
$ rke etcd snapshot-save --config cluster.yml --s3 --access-key S3_ACCESS_KEY --secret-key S3_SECRET_KEY --bucket-name s3-bucket-name --name snapshot-name --s3-endpoint s3.amazonaws.com
$ rke etcd snapshot-save --config cluster.yml --name snapshot-name \
--s3 --access-key S3_ACCESS_KEY --secret-key S3_SECRET_KEY \
--bucket-name s3-bucket-name --s3-endpoint s3.amazonaws.com
```
## Etcd Recurring Snapshots
The snapshot is saved in `/opt/rke/etcd-snapshots` as well as uploaded to the S3 backend.
To schedule a recurring automatic etcd snapshot save, you can enable the `etcd-snapshot` service. `etcd-snapshot` runs in a service container alongside the `etcd` container. `etcd-snapshot` automatically takes a snapshot of etcd and stores them to its local disk in `/opt/rke/etcd-snapshots`.
## Recurring Snapshots
To schedule automatic recurring etcd snapshots, you can enable the `etcd-snapshot` service. `etcd-snapshot` runs in a service container alongside the `etcd` container. In the `cluster.yml`, you need to turn enable `snapshot` as part of the `etcd service`. By default, `etcd-snapshot` service takes a snapshot for every node that has the `etcd` role and stores them to local disk in `/opt/rke/etcd-snapshots`. If you set up the [options for S3](#options-for-the-etcd-snapshot-service), the snapshot will also be uploaded to the S3 backend.
In the `cluster.yml`, you need to turn enable `snapshot` as part of the `etcd service`. Additionally, you want to specify `creation` and `retention` for the snapshot service.
```yaml
services:
etcd:
snapshot: true
creation: 5m0s
retention: 24h
```
When a cluster is launched with the etcd snapshot service enabled, you can view the `etcd-rolling-snapshots` logs to confirm backups are being created automatically.
When a cluster is launched with the `etcd-snapshot` service enabled, you can view the `etcd-rolling-snapshots` logs to confirm backups are being created automatically.
```
$ docker logs etcd-rolling-snapshots
@@ -89,22 +70,24 @@ time="2018-05-04T18:42:16Z" level=info msg="Created backup" name="2018-05-04T18:
time="2018-05-04T18:43:16Z" level=info msg="Created backup" name="2018-05-04T18:43:16Z_etcd" runtime=86.298499ms
```
For every node that has the `etcd` role, these `backups` are saved to `/opt/rke/etcd-snapshots/`.
### Options for the `Etcd-Snapshot` Service
### Snapshot Options
|Option|Description|
|---|---|
|**Snapshot**|By default, the recurring snapshot service is disabled. To enable the service, you need to define it as part of `etcd` and set it to `true`.|
|**Creation**|By default, the snapshot service will take snapshots every 5 minutes (`5m0s`). You can change the time between snapshots as part of the `creation` directive for the `etcd` service.|
|**Retention**|By default, all snapshots are saved for 24 hours (`24h`) before being deleted and purged. You can change how long to store a snapshot as part of the `retention` directive for the `etcd` service.|
Depending on your version of RKE, the options used to configure recurring snapshots may be different.
## S3 Recurring Backups
_Available as of v0.2.0_
As of v0.2.0, RKE support saving snapshots to S3 compatible backends. This is true for both recurring backups and one time snapshots.
|Option|Description| S3 Specific |
|---|---| --- |
|**interval_hours**| The duration in hours between recurring backups. This supercedes the `creation` option and will override it if both are specified.| |
|**retention**| The number of snapshots to retain before rotation. This supercedes the `retention` option and will override it if both are specified.| |
|**bucket_name**| S3 bucket name where backups will be stored| * |
|**access_key**| S3 access key with permission to access the backup bucket.| * |
|**secret_key** |S3 secret key with permission to access the backup bucket.| * |
|**region** |S3 region for the backup bucket. This is optional.| * |
|**endpoint** |S3 regions endpoint for the backup bucket.| * |
<br>
### S3 Snapshot Options
The new backup options replace the [legacy backup configuration]({{< baseurl >}}/rke/v0.1.x/en/etcd-snapshots/#etcd-recurring-snapshots) and should be used instead in the `cluster.yml` file:
```yaml
services:
@@ -120,89 +103,63 @@ services:
endpoint: s3.amazonaws.com
```
#### Prior to v0.2.0
|Option|Description|
|---|---|
|**interval_hours**| The duration in hours between recurring backups. This deprecates the `creation` legacy option and will override it if both are specified.|
|**retention**| The number of snapshots to retain before rotation. This deprecates the `retention` legacy option and will override it if both are specified.|
|**bucket_name**| S3 bucket name where backups will be stored|
|**access_key**| S3 access key with permission to access the backup bucket.|
|**secret_key** |S3 secret key with permission to access the backup bucket.|
|**region** |S3 region for the backup bucket. This is optional.|
|**endpoint** |S3 regions endpoint for the backup bucket.|
|**Snapshot**|By default, the recurring snapshot service is disabled. To enable the service, you need to define it as part of `etcd` and set it to `true`.|
|**Creation**|By default, the snapshot service will take snapshots every 5 minutes (`5m0s`). You can change the time between snapshots as part of the `creation` directive for the `etcd` service.|
|**Retention**|By default, all snapshots are saved for 24 hours (`24h`) before being deleted and purged. You can change how long to store a snapshot as part of the `retention` directive for the `etcd` service.|
## Etcd Disaster recovery
```yaml
services:
etcd:
snapshot: true
creation: 5m0s
retention: 24h
```
If there is a disaster with your Kubernetes cluster, you can use `rke etcd snapshot-restore` to recover your etcd. This command will revert to a specific snapshot stored in `/opt/rke/etcd-snapshots` that you explicitly define. During the restore process, RKE also removes the old `etcd` container before creating a new `etcd` cluster using the snapshot that you have chosen.
## Etcd Disaster Recovery
If there is a disaster with your Kubernetes cluster, you can use `rke etcd snapshot-restore` to recover your etcd. This command reverts etcd to a specific snapshot. RKE also removes the old `etcd` container before creating a new `etcd` cluster using the snapshot that you have chosen.
>**Warning:** Restoring an etcd snapshot deletes your current etcd cluster and replaces it with a new one. Before you run the `rke etcd snapshot-restore` command, you should back up any important data in your cluster.
The snapshot used to restore your etcd cluster can either be stored locally in `/opt/rke/etcd-snapshots` or from a S3 compatible backend. The S3 backend option is available as of v0.2.0.
### Options for `rke etcd snapshot-restore`
| Option | Description | S3 Specific |
| --- | --- | ---|
| `--name` value | Specify snapshot name | |
| `--config` value | Specify an alternate cluster YAML file (default: "cluster.yml") [$RKE_CONFIG] | |
| `--s3` | Enabled backup to s3 |* |
| `--s3-endpoint` value | Specify s3 endpoint url (default: "s3.amazonaws.com") | * |
| `--access-key` value | Specify s3 accessKey | *|
| `--secret-key` value | Specify s3 secretKey | *|
| `--bucket-name` value | Specify s3 bucket name | *|
| `--region` value | Specify the s3 bucket location (optional) | *|
| `--ssh-agent-auth` | [Use SSH Agent Auth defined by SSH_AUTH_SOCK]({{< baseurl >}}/rke/v0.1.x/en/config-options/#ssh-agent) | |
| `--ignore-docker-version` | [Disable Docker version check]({{< baseurl >}}/rke/v0.1.x/en/config-options/#supported-docker-versions) |
### Example of Restoring from a Local Snapshot
When restoring etcd from a local snapshot, the snapshot is assumed to be located in `/opt/rke/etcd-snapshots`.
```
$ rke etcd snapshot-restore --name mysnapshot --config cluster.yml
INFO[0000] Starting restore on etcd hosts
INFO[0000] [dialer] Setup tunnel for host [x.x.x.x]
INFO[0002] [dialer] Setup tunnel for host [y.y.y.y]
INFO[0005] [dialer] Setup tunnel for host [z.z.z.z]
INFO[0007] [hosts] Cleaning up host [x.x.x.x]
INFO[0007] [hosts] Running cleaner container on host [x.x.x.x]
INFO[0008] [kube-cleaner] Successfully started [kube-cleaner] container on host [x.x.x.x]
INFO[0008] [hosts] Removing cleaner container on host [x.x.x.x]
INFO[0008] [hosts] Successfully cleaned up host [x.x.x.x]
INFO[0009] [hosts] Cleaning up host [y.y.y.y]
INFO[0009] [hosts] Running cleaner container on host [y.y.y.y]
INFO[0010] [kube-cleaner] Successfully started [kube-cleaner] container on host [y.y.y.y]
INFO[0010] [hosts] Removing cleaner container on host [y.y.y.y]
INFO[0010] [hosts] Successfully cleaned up host [y.y.y.y]
INFO[0011] [hosts] Cleaning up host [z.z.z.z]
INFO[0011] [hosts] Running cleaner container on host [z.z.z.z]
INFO[0012] [kube-cleaner] Successfully started [kube-cleaner] container on host [z.z.z.z]
INFO[0012] [hosts] Removing cleaner container on host [z.z.z.z]
INFO[0012] [hosts] Successfully cleaned up host [z.z.z.z]
INFO[0012] [etcd] Restoring [snapshot] snapshot on etcd host [x.x.x.x]
INFO[0013] [etcd] Successfully started [etcd-restore] container on host [x.x.x.x]
INFO[0014] [etcd] Restoring [snapshot] snapshot on etcd host [y.y.y.y]
INFO[0015] [etcd] Successfully started [etcd-restore] container on host [y.y.y.y]
INFO[0015] [etcd] Restoring [snapshot] snapshot on etcd host [z.z.z.z]
INFO[0016] [etcd] Successfully started [etcd-restore] container on host [z.z.z.z]
INFO[0017] [etcd] Building up etcd plane..
INFO[0018] [etcd] Successfully started [etcd] container on host [x.x.x.x]
INFO[0020] [etcd] Successfully started [rke-log-linker] container on host [x.x.x.x]
INFO[0021] [remove/rke-log-linker] Successfully removed container on host [x.x.x.x]
INFO[0022] [etcd] Successfully started [etcd] container on host [y.y.y.y]
INFO[0023] [etcd] Successfully started [rke-log-linker] container on host [y.y.y.y]
INFO[0025] [remove/rke-log-linker] Successfully removed container on host [y.y.y.y]
INFO[0025] [etcd] Successfully started [etcd] container on host [z.z.z.z]
INFO[0027] [etcd] Successfully started [rke-log-linker] container on host [z.z.z.z]
INFO[0027] [remove/rke-log-linker] Successfully removed container on host [z.z.z.z]
INFO[0027] [etcd] Successfully started etcd plane..
INFO[0027] Finished restoring on all etcd hosts
$ rke etcd snapshot-restore --config cluster.yml --name mysnapshot
```
### S3 Snapshot Restore
### Example of Restoring from a Snapshot in S3
_Available as of v0.2.0_
As of v0.2.0, RKE support downloading snapshots from S3 compatible backends. This is true for both recurring backups and one time snapshots.
When restoring etcd from a snapshot located in S3, the command needs the S3 information in order to connect to the S3 backend and retrieve the snapshot.
In order to restore a cluster for a snapshot stored on S3, you need to run the following command:
```
$ rke etcd snapshot-restore --help
NAME:
rke etcd snapshot-restore - Restore existing snapshot
USAGE:
rke etcd snapshot-restore [command options] [arguments...]
OPTIONS:
--name value Specify snapshot name
--config value Specify an alternate cluster YAML file (default: "cluster.yml") [$RKE_CONFIG]
--s3 Enabled backup to s3
--s3-endpoint value Specify s3 endpoint url (default: "s3.amazonaws.com")
--access-key value Specify s3 accessKey
--secret-key value Specify s3 secretKey
--bucket-name value Specify s3 bucket name
--region value Specify the s3 bucket location (optional)
--ssh-agent-auth Use SSH Agent Auth defined by SSH_AUTH_SOCK
--ignore-docker-version Disable Docker version check
$ rke etcd snapshot-restore --config cluster.yml --s3 --access-key S3_ACCESS_KEY --secret-key S3_SECRET_KEY --bucket-name s3-bucket-name --name snapshot-name --s3-endpoint s3.amazonaws.com
$ rke etcd snapshot-restore --config cluster.yml --name snapshot-name \
--s3 --access-key S3_ACCESS_KEY --secret-key S3_SECRET_KEY \
--bucket-name s3-bucket-name --s3-endpoint s3.amazonaws.com
```
## Example
@@ -215,7 +172,7 @@ In this example, the Kubernetes cluster was deployed on two AWS nodes.
### Back up the `etcd` cluster
Take a snapshot of the Kubernetes cluster.
Take a local snapshot of the Kubernetes cluster. As of v0.2.0, you can also upload this snapshot directly to a S3 backend with the S3 options.
```
$ rke etcd snapshot-save --name snapshot.db --config cluster.yml
@@ -224,14 +181,14 @@ $ rke etcd snapshot-save --name snapshot.db --config cluster.yml
![etcd snapshot]({{< baseurl >}}/img/rke/rke-etcd-backup.png)
### Store the snapshot externally
### Store the Snapshot Externally
>**Note:** As of version 0.2.0, this step is no longer required, as RKE can upload and download snapshots automatically from S3 by adding in S3 options when running the `rke etcd snapshot-save` command.
>**Note:** As of version 0.2.0, this is no longer required, as RKE can upload and download snapshots automatically from S3.
>**Note:** As of version 0.2.0, the file **pki.bundle.tar.gz** is no longer required.
After taking the etcd snapshot on `node2`, we recommend saving this backup in a persistence place. One of the options is to save the backup on a S3 bucket or tape backup.
```
# If you're using an AWS host and have the ability to connect to S3
root@node2:~# s3cmd mb s3://rke-etcd-backup
@@ -246,9 +203,6 @@ To simulate the failure, let's power down `node2`.
root@node2:~# poweroff
```
Before restoring etcd and running `rke up`, we need to retrieve the backup saved on S3 to a new node, e.g. `node3`.
| Name | IP | Role |
|:-----:|:--------:|:----------------------:|
| node1 | 10.0.0.1 | [controlplane, worker] |
@@ -256,6 +210,9 @@ Before restoring etcd and running `rke up`, we need to retrieve the backup saved
| node3 | 10.0.0.3 | [etcd] |
| | | |
Before restoring etcd and running `rke up`, we need to retrieve the backup saved on S3 to a new node, e.g. `node3`. As of v0.2.0, you can directly retrieve the snapshot from S3 when running the restore command, so this step is for users who stored the snapshot externally without using the integrated S3 options.
```
# Make a Directory
root@node3:~# mkdir -p /opt/rke/etcdbackup
@@ -287,7 +244,7 @@ nodes:
- etcd
```
After the new node is added to the `cluster.yml`, run `rke etcd snapshot-restore` to launch `etcd` from the backup. ]
After the new node is added to the `cluster.yml`, run `rke etcd snapshot-restore` to launch `etcd` from the backup. As of v0.2.0, if you wnat to directly retrieve the snapshot from S3, add in the S3 options.
```
$ rke etcd snapshot-restore --name snapshot.db --config cluster.yml
@@ -311,14 +268,15 @@ nginx-65899c769f-qkhml 1/1 Running 0 17s
## Troubleshooting
As of **v0.1.9**, the **rke-bundle-cert** container is removed on both success and failure of a restore. To debug any issues, you will need to look at the **logs** generated from rke.
As of **v0.1.8** and below, the **rke-bundle-cert** container is left over from a failed etcd restore. If you are having an issue with restoring an **etcd snapshot** then you can do the following on each etcd nodes before attempting to do another restore:
```
docker container rm --force rke-bundle-cert
```
The rke-bundle-cert container is usually removed when a backup or restore of **etcd** succeeds.
Whenever something goes wrong, the **rke-bundle-cert** container will be left over. You can look
The rke-bundle-cert container is usually removed when a backup or restore of **etcd** succeeds. Whenever something goes wrong, the **rke-bundle-cert** container will be left over. You can look
at the logs or inspect the container to see what the issue is.
```
@@ -327,6 +285,3 @@ docker container inspect rke-bundle-cert
```
The important thing to note is the mounts of the container and location of the **pki.bundle.tar.gz**.
As of **v0.1.9**, the **rke-bundle-cert** container is removed on both success and
failure of a restore. To debug any issues, you will need to look at the **logs** generated from rke.