diff --git a/content/rke/latest/en/etcd-snapshots/_index.md b/content/rke/latest/en/etcd-snapshots/_index.md index efe05c2fb39..d973feb3d2f 100644 --- a/content/rke/latest/en/etcd-snapshots/_index.md +++ b/content/rke/latest/en/etcd-snapshots/_index.md @@ -11,295 +11,22 @@ RKE clusters can be configured to automatically take snapshots of etcd. In a dis _Available as of v0.2.0_ -RKE can also upload your snapshots to a S3 compatible backend. Additionally, the **pki.bundle.tar.gz** file usage is no longer required as v0.2.0 has changed how the [Kubernetes cluster state is stored]({{< baseurl >}}/rke/latest/en/installation/#kubernetes-cluster-state). +RKE can upload your snapshots to a S3 compatible backend. -## One-Time Snapshots +**Note:** As of RKE v0.2.0, the `pki.bundle.tar.gz` file is no longer required because of a change in how the [Kubernetes cluster state is stored]({{< baseurl >}}/rke/latest/en/installation/#kubernetes-cluster-state). -The `rke etcd snapshot-save` command will save a snapshot of etcd from each etcd node in the cluster config file. The snapshot is saved in `/opt/rke/etcd-snapshots`. When running the command, an additional container is created to take the snapshot. When the snapshot is completed, the container is automatically removed. +# Backing Up a Cluster -Prior to v0.2.0, along with the individual snapshot, RKE saves a backup of the certificates, i.e. a file named `pki.bundle.tar.gz`, in the same location. The snapshot and pki bundle file are required for the restore process in versions prior to v0.2.0. +You can create [one-time snapshots]({{}}/rke/latest/en/etcd-snapshots/one-time-snapshots) to back up your cluster, and you can also configure [recurring snapshots]({{}}/rke/latest/en/etcd-snapshots/recurring-snapshots). -As of v0.2.0, the one-time snapshot can be uploaded to a S3 compatible backend by using the additional options to specify the S3 backend. +# Restoring a Cluster from Backup -### Options for `rke etcd snapshot-save` +You can use RKE to [restore your cluster from backup]({{}}/rke/latest/en/etcd-snapshots/restoring-from-backup). -| Option | Description | S3 Specific | -| --- | --- | --- | -| `--name` value | Specify snapshot name | | -| `--config` value | Specify an alternate cluster YAML file (default: "cluster.yml") [$RKE_CONFIG] | | -| `--s3` | Enabled backup to s3 | * | -| `--s3-endpoint` value | Specify s3 endpoint url (default: "s3.amazonaws.com") | * | -| `--access-key` value | Specify s3 accessKey | * | -| `--secret-key` value | Specify s3 secretKey | * | -| `--bucket-name` value | Specify s3 bucket name | * | -| `--region` value | Specify the s3 bucket location (optional) | * | -| `--ssh-agent-auth` | [Use SSH Agent Auth defined by SSH_AUTH_SOCK]({{< baseurl >}}/rke/latest/en/config-options/#ssh-agent) | | -| `--ignore-docker-version` | [Disable Docker version check]({{< baseurl >}}/rke/latest/en/config-options/#supported-docker-versions) | +# Example Scenarios -### IAM Support for Storing Snapshots in S3 -In addition to API access keys, RKE supports using IAM roles for S3 authentication. The cluster etcd nodes must be assigned an IAM role that has read/write access to the designated backup bucket on S3. Also, the nodes must have network access to the S3 endpoint specified. - - To give an application access to S3, refer to the AWS documentation on [Using an IAM Role to Grant Permissions to Applications Running on Amazon EC2 Instances.](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html) - -### Local One-Time Snapshot Example - -``` -$ rke etcd snapshot-save --config cluster.yml --name snapshot-name -``` - -The snapshot is saved in `/opt/rke/etcd-snapshots` - -### One-Time Snapshots uploaded to S3 Example - -_Available as of v0.2.0_ - -``` -$ rke etcd snapshot-save --config cluster.yml --name snapshot-name \ ---s3 --access-key S3_ACCESS_KEY --secret-key S3_SECRET_KEY \ ---bucket-name s3-bucket-name --s3-endpoint s3.amazonaws.com -``` - -The snapshot is saved in `/opt/rke/etcd-snapshots` as well as uploaded to the S3 backend. - -## Recurring Snapshots - -To schedule automatic recurring etcd snapshots, you can enable the `etcd-snapshot` service with [extra configuration options the etcd service](#options-for-the-etcd-snapshot-service). `etcd-snapshot` runs in a service container alongside the `etcd` container. By default, the `etcd-snapshot` service takes a snapshot for every node that has the `etcd` role and stores them to local disk in `/opt/rke/etcd-snapshots`. If you set up the [options for S3](#options-for-the-etcd-snapshot-service), the snapshot will also be uploaded to the S3 backend. - -Prior to v0.2.0, along with the snapshots, RKE saves a backup of the certificates, i.e. a file named `pki.bundle.tar.gz`, in the same location. The snapshot and pki bundle file are required for the restore process in versions prior to v0.2.0. - -When a cluster is launched with the `etcd-snapshot` service enabled, you can view the `etcd-rolling-snapshots` logs to confirm backups are being created automatically. - -``` -$ docker logs etcd-rolling-snapshots - -time="2018-05-04T18:39:16Z" level=info msg="Initializing Rolling Backups" creation=1m0s retention=24h0m0s -time="2018-05-04T18:40:16Z" level=info msg="Created backup" name="2018-05-04T18:40:16Z_etcd" runtime=108.332814ms -time="2018-05-04T18:41:16Z" level=info msg="Created backup" name="2018-05-04T18:41:16Z_etcd" runtime=92.880112ms -time="2018-05-04T18:42:16Z" level=info msg="Created backup" name="2018-05-04T18:42:16Z_etcd" runtime=83.67642ms -time="2018-05-04T18:43:16Z" level=info msg="Created backup" name="2018-05-04T18:43:16Z_etcd" runtime=86.298499ms -``` - -### Options for the `Etcd-Snapshot` Service - -Depending on your version of RKE, the options used to configure recurring snapshots may be different. - -_Available as of v0.2.0_ - -|Option|Description| S3 Specific | -|---|---| --- | -|**interval_hours**| The duration in hours between recurring backups. This supercedes the `creation` option and will override it if both are specified.| | -|**retention**| The number of snapshots to retain before rotation. This supercedes the `retention` option and will override it if both are specified.| | -|**bucket_name**| S3 bucket name where backups will be stored| * | -|**access_key**| S3 access key with permission to access the backup bucket.| * | -|**secret_key** |S3 secret key with permission to access the backup bucket.| * | -|**region** |S3 region for the backup bucket. This is optional.| * | -|**endpoint** |S3 regions endpoint for the backup bucket.| * | - -
- - -```yaml -services: - etcd: - backup_config: - interval_hours: 12 - retention: 6 - s3backupconfig: - access_key: S3_ACCESS_KEY - secret_key: S3_SECRET_KEY - bucket_name: s3-bucket-name - region: "" - endpoint: s3.amazonaws.com -``` - -#### Prior to v0.2.0 - -|Option|Description| -|---|---| -|**Snapshot**|By default, the recurring snapshot service is disabled. To enable the service, you need to define it as part of `etcd` and set it to `true`.| -|**Creation**|By default, the snapshot service will take snapshots every 5 minutes (`5m0s`). You can change the time between snapshots as part of the `creation` directive for the `etcd` service.| -|**Retention**|By default, all snapshots are saved for 24 hours (`24h`) before being deleted and purged. You can change how long to store a snapshot as part of the `retention` directive for the `etcd` service.| - -```yaml -services: - etcd: - snapshot: true - creation: 5m0s - retention: 24h -``` - -## Etcd Disaster Recovery - -If there is a disaster with your Kubernetes cluster, you can use `rke etcd snapshot-restore` to recover your etcd. This command reverts etcd to a specific snapshot. RKE also removes the old `etcd` container before creating a new `etcd` cluster using the snapshot that you have chosen. - ->**Warning:** Restoring an etcd snapshot deletes your current etcd cluster and replaces it with a new one. Before you run the `rke etcd snapshot-restore` command, you should back up any important data in your cluster. - -The snapshot used to restore your etcd cluster can either be stored locally in `/opt/rke/etcd-snapshots` or from a S3 compatible backend. The S3 backend option is available as of v0.2.0. - -### Options for `rke etcd snapshot-restore` - -| Option | Description | S3 Specific | -| --- | --- | ---| -| `--name` value | Specify snapshot name | | -| `--config` value | Specify an alternate cluster YAML file (default: "cluster.yml") [$RKE_CONFIG] | | -| `--s3` | Enabled backup to s3 |* | -| `--s3-endpoint` value | Specify s3 endpoint url (default: "s3.amazonaws.com") | * | -| `--access-key` value | Specify s3 accessKey | *| -| `--secret-key` value | Specify s3 secretKey | *| -| `--bucket-name` value | Specify s3 bucket name | *| -| `--region` value | Specify the s3 bucket location (optional) | *| -| `--ssh-agent-auth` | [Use SSH Agent Auth defined by SSH_AUTH_SOCK]({{< baseurl >}}/rke/latest/en/config-options/#ssh-agent) | | -| `--ignore-docker-version` | [Disable Docker version check]({{< baseurl >}}/rke/latest/en/config-options/#supported-docker-versions) | - -### Example of Restoring from a Local Snapshot - -When restoring etcd from a local snapshot, the snapshot is assumed to be located in `/opt/rke/etcd-snapshots`. In versions prior to v0.2.0, the `pki.bundle.tar.gz` file is also expected to be in the same location. As of v0.2.0, this file is no longer needed as v0.2.0 has changed how the [Kubernetes cluster state is stored]({{< baseurl >}}/rke/latest/en/installation/#kubernetes-cluster-state). - -``` -$ rke etcd snapshot-restore --config cluster.yml --name mysnapshot -``` - -### Example of Restoring from a Snapshot in S3 - -_Available as of v0.2.0_ - -> **Note:** Ensure your `cluster.rkestate` is present before starting the restore, as this contains your certificate data for the cluster - -When restoring etcd from a snapshot located in S3, the command needs the S3 information in order to connect to the S3 backend and retrieve the snapshot. - -```shell -$ rke etcd snapshot-restore --config cluster.yml --name snapshot-name \ ---s3 --access-key S3_ACCESS_KEY --secret-key S3_SECRET_KEY \ ---bucket-name s3-bucket-name --s3-endpoint s3.amazonaws.com -``` -> **Note:** if you were restoring a cluster that had rancher installed the UI should start-up after a few minutes; you don't need to re-run helm. - -### Example Scenario of restoring from a Local Snapshot - -In this example, the Kubernetes cluster was deployed on two AWS nodes. - -| Name | IP | Role | -|:-----:|:--------:|:----------------------:| -| node1 | 10.0.0.1 | [controlplane, worker] | -| node2 | 10.0.0.2 | [etcd] | - -### Back up the `etcd` cluster - -Take a local snapshot of the Kubernetes cluster. As of v0.2.0, you can also upload this snapshot directly to a S3 backend with the [S3 options](#options-for-rke-etcd-snapshot-save). - -``` -$ rke etcd snapshot-save --name snapshot.db --config cluster.yml -``` - -![etcd snapshot]({{< baseurl >}}/img/rke/rke-etcd-backup.png) - - -### Store the Snapshot Externally in S3 - -As of v0.2.0, this step is no longer required, as RKE can upload and download snapshots automatically from S3 by adding in [S3 options](#options-for-rke-etcd-snapshot-save) when running the `rke etcd snapshot-save` command. - -After taking the etcd snapshot on `node2`, we recommend saving this backup in a persistence place. One of the options is to save the backup and `pki.bundle.tar.gz` file on a S3 bucket or tape backup. - -> **Note:** As of v0.2.0, the file **pki.bundle.tar.gz** is no longer required for the restore process. - -``` -# If you're using an AWS host and have the ability to connect to S3 -root@node2:~# s3cmd mb s3://rke-etcd-backup -root@node2:~# s3cmd /opt/rke/etcd-snapshots/snapshot.db /opt/rke/etcd-snapshots/pki.bundle.tar.gz s3://rke-etcd-backup/ -``` - -### Place the backup on a new node - -To simulate the failure, let's power down `node2`. - -``` -root@node2:~# poweroff -``` - -| Name | IP | Role | -|:-----:|:--------:|:----------------------:| -| node1 | 10.0.0.1 | [controlplane, worker] | -| ~~node2~~ | ~~10.0.0.2~~ | ~~[etcd]~~ | -| node3 | 10.0.0.3 | [etcd] | -| | | | - - -Before restoring etcd and running `rke up`, we need to retrieve the backup saved on S3 to a new node, e.g. `node3`. As of v0.2.0, you can directly retrieve the snapshot from S3 when running the restore command, so this step is for users who stored the snapshot externally without using the integrated S3 options. - -``` -# Make a Directory -root@node3:~# mkdir -p /opt/rke/etcdbackup -# Get the Backup from S3 -root@node3:~# s3cmd get s3://rke-etcd-backup/snapshot.db /opt/rke/etcd-snapshots/snapshot.db -# Get the pki bundle from S3, only needed prior to v0.2.0 -root@node3:~# s3cmd get s3://rke-etcd-backup/pki.bundle.tar.gz /opt/rke/etcd-snapshots/pki.bundle.tar.gz -``` - -### Restore `etcd` on the new node from the backup - -Before updating and restoring etcd, you will need to add the new node into the Kubernetes cluster with the `etcd` role. In the `cluster.yml`, comment out the old node and add in the new node. ` - -```yaml -nodes: - - address: 10.0.0.1 - hostname_override: node1 - user: ubuntu - role: - - controlplane - - worker -# - address: 10.0.0.2 -# hostname_override: node2 -# user: ubuntu -# role: -# - etcd - - address: 10.0.0.3 - hostname_override: node3 - user: ubuntu - role: - - etcd -``` - -After the new node is added to the `cluster.yml`, run `rke etcd snapshot-restore` to launch `etcd` from the backup. The snapshot and `pki.bundle.tar.gz` file are expected to be saved at `/opt/rke/etcd-snapshots`. -As of v0.2.0, if you want to directly retrieve the snapshot from S3, add in the [S3 options](#options-for-rke-etcd-snapshot-restore). - -> **Note:** As of v0.2.0, the file **pki.bundle.tar.gz** is no longer required for the restore process as the certificates required to restore are preserved within the `cluster.rkestate` - -``` -$ rke etcd snapshot-restore --name snapshot.db --config cluster.yml -``` - -Finally, we need to restore the operations on the cluster by making the Kubernetes API point to the new `etcd` by running `rke up` again using the new `cluster.yml`. - -``` -$ rke up --config cluster.yml -``` - -Confirm that your Kubernetes cluster is functional by checking the pods on your cluster. - -``` -> kubectl get pods -NAME READY STATUS RESTARTS AGE -nginx-65899c769f-kcdpr 1/1 Running 0 17s -nginx-65899c769f-pc45c 1/1 Running 0 17s -nginx-65899c769f-qkhml 1/1 Running 0 17s -``` +These [example scenarios]({{}}/rke/latest/en/etcd-snapshots/example-scenarios) for backup and restore are different based on your version of RKE. ## Troubleshooting -As of **v0.1.9**, the **rke-bundle-cert** container is removed on both success and failure of a restore. To debug any issues, you will need to look at the **logs** generated from rke. - -As of **v0.1.8** and below, the **rke-bundle-cert** container is left over from a failed etcd restore. If you are having an issue with restoring an **etcd snapshot** then you can do the following on each etcd nodes before attempting to do another restore: - -``` -docker container rm --force rke-bundle-cert -``` - -The rke-bundle-cert container is usually removed when a backup or restore of **etcd** succeeds. Whenever something goes wrong, the **rke-bundle-cert** container will be left over. You can look -at the logs or inspect the container to see what the issue is. - -``` -docker container logs --follow rke-bundle-cert -docker container inspect rke-bundle-cert -``` - -The important thing to note is the mounts of the container and location of the **pki.bundle.tar.gz**. +If you have trouble restoring your cluster, you can refer to the [troubleshooting]({{}}/rke/latest/en/etcd-snapshots/troubleshooting) page. diff --git a/content/rke/latest/en/etcd-snapshots/example-scenarios/_index.md b/content/rke/latest/en/etcd-snapshots/example-scenarios/_index.md new file mode 100644 index 00000000000..5ddd113ff71 --- /dev/null +++ b/content/rke/latest/en/etcd-snapshots/example-scenarios/_index.md @@ -0,0 +1,253 @@ +--- +title: Example Scenarios +weight: 4 +--- + +These example scenarios for backup and restore are different based on your version of RKE. + +{{% tabs %}} +{{% tab "RKE v0.2.0+" %}} + +This walkthrough will demonstrate how to restore an etcd cluster from a local snapshot with the following steps: + +1. [Back up the cluster](#1-back-up-the-cluster) +1. [Simulate a node failure](#2-simulate-a-node-failure) +1. [Add a new etcd node to the cluster](#3-add-a-new-etcd-node-to-the-kubernetes-cluster) +1. [Restore etcd on the new node from the backup](#4-restore-etcd-on-the-new-node-from-the-backup) +1. [Confirm that cluster operations are restored](#5-confirm-that-cluster-operations-are-restored) + +In this example, the Kubernetes cluster was deployed on two AWS nodes. + +| Name | IP | Role | +|:-----:|:--------:|:----------------------:| +| node1 | 10.0.0.1 | [controlplane, worker] | +| node2 | 10.0.0.2 | [etcd] | + + +### 1. Back Up the Cluster + +Take a local snapshot of the Kubernetes cluster. + +You can upload this snapshot directly to an S3 backend with the [S3 options]({{}}/rke/latest/en/etcd-snapshots/one-time-snapshots/#options-for-rke-etcd-snapshot-save). + +``` +$ rke etcd snapshot-save --name snapshot.db --config cluster.yml +``` + +![etcd snapshot]({{< baseurl >}}/img/rke/rke-etcd-backup.png) + +### 2. Simulate a Node Failure + +To simulate the failure, let's power down `node2`. + +``` +root@node2:~# poweroff +``` + +| Name | IP | Role | +|:-----:|:--------:|:----------------------:| +| node1 | 10.0.0.1 | [controlplane, worker] | +| ~~node2~~ | ~~10.0.0.2~~ | ~~[etcd]~~ | + +### 3. Add a New etcd Node to the Kubernetes Cluster + +Before updating and restoring etcd, you will need to add the new node into the Kubernetes cluster with the `etcd` role. In the `cluster.yml`, comment out the old node and add in the new node. + +```yaml +nodes: + - address: 10.0.0.1 + hostname_override: node1 + user: ubuntu + role: + - controlplane + - worker +# - address: 10.0.0.2 +# hostname_override: node2 +# user: ubuntu +# role: +# - etcd + - address: 10.0.0.3 + hostname_override: node3 + user: ubuntu + role: + - etcd +``` + +### 4. Restore etcd on the New Node from the Backup + +> **Prerequisite:** Ensure your `cluster.rkestate` is present before starting the restore, because this contains your certificate data for the cluster. + +After the new node is added to the `cluster.yml`, run the `rke etcd snapshot-restore` to launch `etcd` from the backup: + +``` +$ rke etcd snapshot-restore --name snapshot.db --config cluster.yml +``` + +The snapshot is expected to be saved at `/opt/rke/etcd-snapshots`. + +If you want to directly retrieve the snapshot from S3, add in the [S3 options](#options-for-rke-etcd-snapshot-restore). + +> **Note:** As of v0.2.0, the file `pki.bundle.tar.gz` is no longer required for the restore process because the certificates required to restore are preserved within the `cluster.rkestate`. + +### 5. Confirm that Cluster Operations are Restored + +The `rke etcd snapshot-restore` command triggers `rke up` using the new `cluster.yml`. Confirm that your Kubernetes cluster is functional by checking the pods on your cluster. + +``` +> kubectl get pods +NAME READY STATUS RESTARTS AGE +nginx-65899c769f-kcdpr 1/1 Running 0 17s +nginx-65899c769f-pc45c 1/1 Running 0 17s +nginx-65899c769f-qkhml 1/1 Running 0 17s +``` + +{{% /tab %}} +{{% tab "RKE prior to v0.2.0" %}} + +This walkthrough will demonstrate how to restore an etcd cluster from a local snapshot with the following steps: + +1. [Take a local snapshot of the cluster](#take-a-local-snapshot-of-the-cluster-rke-prior-to-v0.2.0) +1. [Store the snapshot externally](#store-the-snapshot-externally-rke-prior-to-v0.2.0) +1. [Simulate a node failure](#simulate-a-node-failure-rke-prior-to-v0.2.0) +1. [Remove the Kubernetes cluster and clean the nodes](#remove-the-kubernetes-cluster-and-clean-the-nodes-rke-prior-to-v0.2.0) +1. [Retrieve the backup and place it on a new node](#retrieve-the-backup-and-place-it-on-a-new-node-rke-prior-to-v0.2.0) +1. [Add a new etcd node to the Kubernetes cluster](#add-a-new-etcd-node-to-the-kubernetes-cluster-rke-prior-to-v0.2.0) +1. [Restore etcd on the new node from the backup](#restore-etcd-on-the-new-node-from-the-backup-rke-prior-to-v0.2.0) +1. [Restore Operations on the Cluster](#restore-operations-on-the-cluster-rke-prior-to-v0.2.0) + +### Example Scenario of restoring from a Local Snapshot + +In this example, the Kubernetes cluster was deployed on two AWS nodes. + +| Name | IP | Role | +|:-----:|:--------:|:----------------------:| +| node1 | 10.0.0.1 | [controlplane, worker] | +| node2 | 10.0.0.2 | [etcd] | + + +### 1. Take a Local Snapshot of the Cluster + +Back up the Kubernetes cluster by taking a local snapshot: + +``` +$ rke etcd snapshot-save --name snapshot.db --config cluster.yml +``` + +![etcd snapshot]({{< baseurl >}}/img/rke/rke-etcd-backup.png) + + +### 2. Store the Snapshot Externally + +After taking the etcd snapshot on `node2`, we recommend saving this backup in a persistent place. One of the options is to save the backup and `pki.bundle.tar.gz` file on an S3 bucket or tape backup. + +``` +# If you're using an AWS host and have the ability to connect to S3 +root@node2:~# s3cmd mb s3://rke-etcd-backup +root@node2:~# s3cmd \ + /opt/rke/etcd-snapshots/snapshot.db \ + /opt/rke/etcd-snapshots/pki.bundle.tar.gz \ + s3://rke-etcd-backup/ +``` + + +### 3. Simulate a Node Failure + +To simulate the failure, let's power down `node2`. + +``` +root@node2:~# poweroff +``` + +| Name | IP | Role | +|:-----:|:--------:|:----------------------:| +| node1 | 10.0.0.1 | [controlplane, worker] | +| ~~node2~~ | ~~10.0.0.2~~ | ~~[etcd]~~ | + + +### 4. Remove the Kubernetes Cluster and Clean the Nodes + +The following command removes your cluster and cleans the nodes so that the cluster can be restored without any conflicts: + +``` +rke remove --config rancher-cluster.yml +``` + + +### 5. Retrieve the Backup and Place it On a New Node + +Before restoring etcd and running `rke up`, we need to retrieve the backup saved on S3 to a new node, e.g. `node3`. + +``` +# Make a Directory +root@node3:~# mkdir -p /opt/rke/etcdbackup + +# Get the Backup from S3 +root@node3:~# s3cmd get \ + s3://rke-etcd-backup/snapshot.db \ + /opt/rke/etcd-snapshots/snapshot.db + +# Get the pki bundle from S3 +root@node3:~# s3cmd get \ + s3://rke-etcd-backup/pki.bundle.tar.gz \ + /opt/rke/etcd-snapshots/pki.bundle.tar.gz +``` + +> **Note:** If you had multiple etcd nodes, you would have to manually sync the snapshot and `pki.bundle.tar.gz` across all of the etcd nodes in the cluster. + + +### 6. Add a New etcd Node to the Kubernetes Cluster + +Before updating and restoring etcd, you will need to add the new node into the Kubernetes cluster with the `etcd` role. In the `cluster.yml`, comment out the old node and add in the new node. ` + +```yaml +nodes: + - address: 10.0.0.1 + hostname_override: node1 + user: ubuntu + role: + - controlplane + - worker +# - address: 10.0.0.2 +# hostname_override: node2 +# user: ubuntu +# role: +# - etcd + - address: 10.0.0.3 + hostname_override: node3 + user: ubuntu + role: + - etcd +``` + + +### 7. Restore etcd on the New Node from the Backup + +After the new node is added to the `cluster.yml`, run the `rke etcd snapshot-restore` command to launch `etcd` from the backup: + +``` +$ rke etcd snapshot-restore --name snapshot.db --config cluster.yml +``` + +The snapshot and `pki.bundle.tar.gz` file are expected to be saved at `/opt/rke/etcd-snapshots` on each etcd node. + + +### 8. Restore Operations on the Cluster + +Finally, we need to restore the operations on the cluster. We will make the Kubernetes API point to the new `etcd` by running `rke up` again using the new `cluster.yml`. + +``` +$ rke up --config cluster.yml +``` + +Confirm that your Kubernetes cluster is functional by checking the pods on your cluster. + +``` +> kubectl get pods +NAME READY STATUS RESTARTS AGE +nginx-65899c769f-kcdpr 1/1 Running 0 17s +nginx-65899c769f-pc45c 1/1 Running 0 17s +nginx-65899c769f-qkhml 1/1 Running 0 17s +``` + +{{% /tab %}} +{{% /tabs %}} \ No newline at end of file diff --git a/content/rke/latest/en/etcd-snapshots/one-time-snapshots/_index.md b/content/rke/latest/en/etcd-snapshots/one-time-snapshots/_index.md new file mode 100644 index 00000000000..e9215aa71f5 --- /dev/null +++ b/content/rke/latest/en/etcd-snapshots/one-time-snapshots/_index.md @@ -0,0 +1,92 @@ +--- +title: One-time Snapshots +weight: 1 +--- + +One-time snapshots are handled differently depending on your version of RKE. + +{{% tabs %}} +{{% tab "RKE v0.2.0+" %}} + +To save a snapshot of etcd from each etcd node in the cluster config file, run the `rke etcd snapshot-save` command. + +The snapshot is saved in `/opt/rke/etcd-snapshots`. + +When running the command, an additional container is created to take the snapshot. When the snapshot is completed, the container is automatically removed. + +The one-time snapshot can be uploaded to a S3 compatible backend by using the additional options to specify the S3 backend. + +To create a local one-time snapshot, run: + +``` +$ rke etcd snapshot-save --config cluster.yml --name snapshot-name +``` + +**Result:** The snapshot is saved in `/opt/rke/etcd-snapshots`. + +To save a one-time snapshot to S3, run: + +``` +$ rke etcd snapshot-save \ +--config cluster.yml \ +--name snapshot-name \ +--s3 \ +--access-key S3_ACCESS_KEY \ +--secret-key S3_SECRET_KEY \ +--bucket-name s3-bucket-name \ +--s3-endpoint s3.amazonaws.com +``` + +**Result:** The snapshot is saved in `/opt/rke/etcd-snapshots` as well as uploaded to the S3 backend. + +### Options for `rke etcd snapshot-save` + +| Option | Description | S3 Specific | +| --- | --- | --- | +| `--name` value | Specify snapshot name | | +| `--config` value | Specify an alternate cluster YAML file (default: `cluster.yml`) [$RKE_CONFIG] | | +| `--s3` | Enabled backup to s3 | * | +| `--s3-endpoint` value | Specify s3 endpoint url (default: "s3.amazonaws.com") | * | +| `--access-key` value | Specify s3 accessKey | * | +| `--secret-key` value | Specify s3 secretKey | * | +| `--bucket-name` value | Specify s3 bucket name | * | +| `--region` value | Specify the s3 bucket location (optional) | * | +| `--ssh-agent-auth` | [Use SSH Agent Auth defined by SSH_AUTH_SOCK]({{< baseurl >}}/rke/latest/en/config-options/#ssh-agent) | | +| `--ignore-docker-version` | [Disable Docker version check]({{< baseurl >}}/rke/latest/en/config-options/#supported-docker-versions) | + +### IAM Support for Storing Snapshots in S3 + +In addition to API access keys, RKE supports using IAM roles for S3 authentication. The cluster etcd nodes must be assigned an IAM role that has read/write access to the designated backup bucket on S3. Also, the nodes must have network access to the S3 endpoint specified. + + To give an application access to S3, refer to the AWS documentation on [Using an IAM Role to Grant Permissions to Applications Running on Amazon EC2 Instances.](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html) + + + +{{% /tab %}} +{{% tab "RKE prior to v0.2.0" %}} + +To save a snapshot of etcd from each etcd node in the cluster config file, run the `rke etcd snapshot-save` command. + +When running the command, an additional container is created to take the snapshot. When the snapshot is completed, the container is automatically removed. + +RKE saves a backup of the certificates, i.e. a file named `pki.bundle.tar.gz`, in the same location. The snapshot and pki bundle file are required for the restore process. + +To create a local one-time snapshot, run: + +``` +$ rke etcd snapshot-save --config cluster.yml --name snapshot-name +``` + +**Result:** The snapshot is saved in `/opt/rke/etcd-snapshots`. + +### Options for `rke etcd snapshot-save` + +| Option | Description | +| --- | --- | +| `--name` value | Specify snapshot name | +| `--config` value | Specify an alternate cluster YAML file (default: `cluster.yml`) [$RKE_CONFIG] | +| `--ssh-agent-auth` | [Use SSH Agent Auth defined by SSH_AUTH_SOCK]({{< baseurl >}}/rke/latest/en/config-options/#ssh-agent) | +| `--ignore-docker-version` | [Disable Docker version check]({{< baseurl >}}/rke/latest/en/config-options/#supported-docker-versions) | + +{{% /tab %}} +{{% /tabs %}} \ No newline at end of file diff --git a/content/rke/latest/en/etcd-snapshots/recurring-snapshots/_index.md b/content/rke/latest/en/etcd-snapshots/recurring-snapshots/_index.md new file mode 100644 index 00000000000..913200ac2f4 --- /dev/null +++ b/content/rke/latest/en/etcd-snapshots/recurring-snapshots/_index.md @@ -0,0 +1,97 @@ +--- +title: Recurring Snapshots +weight: 2 +--- + +Recurring snapshots are handled differently based on your version of RKE. + +{{% tabs %}} +{{% tab "RKE v0.2.0+"%}} + +To schedule automatic recurring etcd snapshots, you can enable the `etcd-snapshot` service with [extra configuration options](#options-for-the-etcd-snapshot-service). `etcd-snapshot` runs in a service container alongside the `etcd` container. By default, the `etcd-snapshot` service takes a snapshot for every node that has the `etcd` role and stores them to local disk in `/opt/rke/etcd-snapshots`. + +If you set up the [options for S3](#options-for-the-etcd-snapshot-service), the snapshot will also be uploaded to the S3 backend. + +### Snapshot Service Logging + +When a cluster is launched with the `etcd-snapshot` service enabled, you can view the `etcd-rolling-snapshots` logs to confirm backups are being created automatically. + +``` +$ docker logs etcd-rolling-snapshots + +time="2018-05-04T18:39:16Z" level=info msg="Initializing Rolling Backups" creation=1m0s retention=24h0m0s +time="2018-05-04T18:40:16Z" level=info msg="Created backup" name="2018-05-04T18:40:16Z_etcd" runtime=108.332814ms +time="2018-05-04T18:41:16Z" level=info msg="Created backup" name="2018-05-04T18:41:16Z_etcd" runtime=92.880112ms +time="2018-05-04T18:42:16Z" level=info msg="Created backup" name="2018-05-04T18:42:16Z_etcd" runtime=83.67642ms +time="2018-05-04T18:43:16Z" level=info msg="Created backup" name="2018-05-04T18:43:16Z_etcd" runtime=86.298499ms +``` + +### Options for the `Etcd-Snapshot` Service + +|Option|Description| S3 Specific | +|---|---| --- | +|**interval_hours**| The duration in hours between recurring backups. This supercedes the `creation` option (which was used in RKE prior to v0.2.0) and will override it if both are specified.| | +|**retention**| The number of snapshots to retain before rotation. This supercedes the `retention` option and will override it if both are specified.| | +|**bucket_name**| S3 bucket name where backups will be stored| * | +|**access_key**| S3 access key with permission to access the backup bucket.| * | +|**secret_key** |S3 secret key with permission to access the backup bucket.| * | +|**region** |S3 region for the backup bucket. This is optional.| * | +|**endpoint** |S3 regions endpoint for the backup bucket.| * | + +### Configuring the Snapshot Service in YAML + +```yaml +services: + etcd: + backup_config: + interval_hours: 12 + retention: 6 + s3backupconfig: + access_key: S3_ACCESS_KEY + secret_key: S3_SECRET_KEY + bucket_name: s3-bucket-name + region: "" + endpoint: s3.amazonaws.com +``` + +{{% /tab %}} +{{% tab "RKE prior to v0.2.0"%}} + +To schedule automatic recurring etcd snapshots, you can enable the `etcd-snapshot` service with [extra configuration options](#options-for-the-local-etcd-snapshot-service). `etcd-snapshot` runs in a service container alongside the `etcd` container. By default, the `etcd-snapshot` service takes a snapshot for every node that has the `etcd` role and stores them to local disk in `/opt/rke/etcd-snapshots`. + +RKE saves a backup of the certificates, i.e. a file named `pki.bundle.tar.gz`, in the same location. The snapshot and pki bundle file are required for the restore process in versions prior to v0.2.0. + +### Snapshot Service Logging + +When a cluster is launched with the `etcd-snapshot` service enabled, you can view the `etcd-rolling-snapshots` logs to confirm backups are being created automatically. + +``` +$ docker logs etcd-rolling-snapshots + +time="2018-05-04T18:39:16Z" level=info msg="Initializing Rolling Backups" creation=1m0s retention=24h0m0s +time="2018-05-04T18:40:16Z" level=info msg="Created backup" name="2018-05-04T18:40:16Z_etcd" runtime=108.332814ms +time="2018-05-04T18:41:16Z" level=info msg="Created backup" name="2018-05-04T18:41:16Z_etcd" runtime=92.880112ms +time="2018-05-04T18:42:16Z" level=info msg="Created backup" name="2018-05-04T18:42:16Z_etcd" runtime=83.67642ms +time="2018-05-04T18:43:16Z" level=info msg="Created backup" name="2018-05-04T18:43:16Z_etcd" runtime=86.298499ms +``` + +### Options for the Local `Etcd-Snapshot` Service + +|Option|Description| +|---|---| +|**Snapshot**|By default, the recurring snapshot service is disabled. To enable the service, you need to define it as part of `etcd` and set it to `true`.| +|**Creation**|By default, the snapshot service will take snapshots every 5 minutes (`5m0s`). You can change the time between snapshots as part of the `creation` directive for the `etcd` service.| +|**Retention**|By default, all snapshots are saved for 24 hours (`24h`) before being deleted and purged. You can change how long to store a snapshot as part of the `retention` directive for the `etcd` service.| + +### Configuring the Snapshot Service in YAML + +```yaml +services: + etcd: + snapshot: true + creation: 5m0s + retention: 24h +``` + +{{% /tab %}} +{{% /tabs %}} \ No newline at end of file diff --git a/content/rke/latest/en/etcd-snapshots/restoring-from-backup/_index.md b/content/rke/latest/en/etcd-snapshots/restoring-from-backup/_index.md new file mode 100644 index 00000000000..57f6f2ef008 --- /dev/null +++ b/content/rke/latest/en/etcd-snapshots/restoring-from-backup/_index.md @@ -0,0 +1,112 @@ +--- +title: Restoring from Backup +weight: 3 +--- + +The details of restoring your cluster from backup are different depending on your version of RKE. + +{{% tabs %}} +{{% tab "RKE v0.2.0+"%}} + +If there is a disaster with your Kubernetes cluster, you can use `rke etcd snapshot-restore` to recover your etcd. This command reverts the etcd to a specific snapshot. The following actions are included in the command: + +- Syncs the snapshot or downloads the snapshot from S3, if necessary. +- Checks snapshot checksum across etcd nodes to make sure they are identical. +- Deletes your current cluster and cleans old data by running `rke remove`. This removes the entire Kubernetes cluster, not just the etcd cluster. +- Rebuilds the etcd cluster from the chosen snapshot. +- Creates a new cluster by running `rke up`. +- Restarts cluster system pods. + +>**Warning:** You should back up any important data in your cluster before running `rke etcd snapshot-restore` because the command deletes your current Kubernetes cluster and replaces it with a new one. + +The snapshot used to restore your etcd cluster can either be stored locally in `/opt/rke/etcd-snapshots` or from a S3 compatible backend. + +### Example of Restoring from a Local Snapshot + +To restore etcd from a local snapshot, run: + +``` +$ rke etcd snapshot-restore --config cluster.yml --name mysnapshot +``` + +The snapshot is assumed to be located in `/opt/rke/etcd-snapshots`. + +**Note:** The `pki.bundle.tar.gz` file is not needed because RKE v0.2.0 changed how the [Kubernetes cluster state is stored]({{< baseurl >}}/rke/latest/en/installation/#kubernetes-cluster-state). + +### Example of Restoring from a Snapshot in S3 + +> **Prerequisite:** Ensure your `cluster.rkestate` is present before starting the restore, because this contains your certificate data for the cluster. + +When restoring etcd from a snapshot located in S3, the command needs the S3 information in order to connect to the S3 backend and retrieve the snapshot. + +```shell +$ rke etcd snapshot-restore \ +--config cluster.yml \ +--name snapshot-name \ +--s3 \ +--access-key S3_ACCESS_KEY \ +--secret-key S3_SECRET_KEY \ +--bucket-name s3-bucket-name \ +--s3-endpoint s3.amazonaws.com +``` +**Note:** if you were restoring a cluster that had Rancher installed, the Rancher UI should start up after a few minutes; you don't need to re-run Helm. + +### Options for `rke etcd snapshot-restore` + +| Option | Description | S3 Specific | +| --- | --- | ---| +| `--name` value | Specify snapshot name | | +| `--config` value | Specify an alternate cluster YAML file (default: `cluster.yml`) [$RKE_CONFIG] | | +| `--s3` | Enabled backup to s3 |* | +| `--s3-endpoint` value | Specify s3 endpoint url (default: "s3.amazonaws.com") | * | +| `--access-key` value | Specify s3 accessKey | *| +| `--secret-key` value | Specify s3 secretKey | *| +| `--bucket-name` value | Specify s3 bucket name | *| +| `--region` value | Specify the s3 bucket location (optional) | *| +| `--ssh-agent-auth` | [Use SSH Agent Auth defined by SSH_AUTH_SOCK]({{< baseurl >}}/rke/latest/en/config-options/#ssh-agent) | | +| `--ignore-docker-version` | [Disable Docker version check]({{< baseurl >}}/rke/latest/en/config-options/#supported-docker-versions) | + +{{% /tab %}} +{{% tab "RKE prior to v0.2.0"%}} + +If there is a disaster with your Kubernetes cluster, you can use `rke etcd snapshot-restore` to recover your etcd. This command reverts etcd to a specific snapshot. + +The following actions are included in `rke etcd snapshot-restore`: + +- Removes the old etcd cluster +- Rebuilds the etcd cluster using the local snapshot + +Before you run this command, you must: + +- Run `rke remove` to remove your Kubernetes cluster and clean the nodes +- Download your etcd snapshot from S3, if applicable. Place the etcd snapshot and the `pki.bundle.tar.gz` file in `/opt/rke/etcd-snapshots`. Manually sync the snapshot across all `etcd` nodes. + +After the restore, you must rebuild your Kubernetes cluster with `rke up`. + +>**Warning:** You should back up any important data in your cluster before running `rke etcd snapshot-restore` because the command deletes your current etcd cluster and replaces it with a new one. + +### Example of Restoring from a Local Snapshot + +To restore etcd from a local snapshot, run: + +``` +$ rke etcd snapshot-restore --config cluster.yml --name mysnapshot +``` + +The snapshot is assumed to be located in `/opt/rke/etcd-snapshots`. + +The snapshot must be manually synched across all `etcd` nodes. + +The `pki.bundle.tar.gz` file is also expected to be in the same location. + +### Options for `rke etcd snapshot-restore` + +| Option | Description | +| --- | --- | +| `--name` value | Specify snapshot name | +| `--config` value | Specify an alternate cluster YAML file (default: `cluster.yml`) [$RKE_CONFIG] | +| `--ssh-agent-auth` | [Use SSH Agent Auth defined by SSH_AUTH_SOCK]({{< baseurl >}}/rke/latest/en/config-options/#ssh-agent) | +| `--ignore-docker-version` | [Disable Docker version check]({{< baseurl >}}/rke/latest/en/config-options/#supported-docker-versions) | + +{{% /tab %}} +{{% /tabs %}} \ No newline at end of file diff --git a/content/rke/latest/en/etcd-snapshots/troubleshooting/_index.md b/content/rke/latest/en/etcd-snapshots/troubleshooting/_index.md new file mode 100644 index 00000000000..372142f649a --- /dev/null +++ b/content/rke/latest/en/etcd-snapshots/troubleshooting/_index.md @@ -0,0 +1,22 @@ +--- +title: Troubleshooting +weight: 5 +--- + +As of **v0.1.9**, the **rke-bundle-cert** container is removed on both success and failure of a restore. To debug any issues, you will need to look at the **logs** generated from rke. + +As of **v0.1.8** and below, the **rke-bundle-cert** container is left over from a failed etcd restore. If you are having an issue with restoring an **etcd snapshot** then you can do the following on each etcd nodes before attempting to do another restore: + +``` +docker container rm --force rke-bundle-cert +``` + +The rke-bundle-cert container is usually removed when a backup or restore of **etcd** succeeds. Whenever something goes wrong, the **rke-bundle-cert** container will be left over. You can look +at the logs or inspect the container to see what the issue is. + +``` +docker container logs --follow rke-bundle-cert +docker container inspect rke-bundle-cert +``` + +The important thing to note is the mounts of the container and location of the `pki.bundle.tar.gz`.