Update etcd backup/restore docs

2026-09-25 04:28:15 +00:00 · 2018-05-16 18:14:09 -07:00
parent d34a9861cd
commit c2b78e2461
2 changed files with 106 additions and 0 deletions
@@ -118,3 +118,109 @@ INFO[0027] [remove/rke-log-linker] Successfully removed container on host [z.z.z
 INFO[0027] [etcd] Successfully started etcd plane..
 INFO[0027] Finished restoring on all etcd hosts
 ```
+
+## Example
+
+In this example we will assume that you started RKE on two nodes:
+
+|  Name |    IP    |          Role          |
+|:-----:|:--------:|:----------------------:|
+| node1 | 10.0.0.1 | [controlplane, worker] |
+| node2 | 10.0.0.2 | [etcd]                 |
+
+### 1. Setting up rke cluster
+A minimal cluster configuration file for running k8s on these nodes should look something like the following:
+
+```
+nodes:
+  - address: 10.0.0.1
+    hostname_override: node1
+    user: ubuntu
+    role: [controlplane,worker]
+  - address: 10.0.0.2
+    hostname_override: node2
+    user: ubuntu
+    role: [etcd]
+```
+
+After running `rke up` you should be able to have a two node cluster, the next step is to run few pods on node1:
+
+```
+kubectl --kubeconfig=kube_config_cluster.yml run nginx --image=nginx --replicas=3
+```
+
+### 2. Backup etcd cluster
+
+Now lets take a snapshot backup using RKE:
+
+```
+rke etcd backup --name snapshot.db --config cluster.yml
+```
+
+![etcd backup]({{< baseurl >}}/img/rancher/rke-etcd-backup.png)
+
+### 3. Store backup externally
+
+After taking the etcd backup on node2 we should be able to save this backup in a persistence place, one of the options to do that is to save the backup taken on a s3 bucket or tape backup, for example:
+
+```
+root@node2:~# s3cmd mb s3://rke-etcd-backup
+root@node2:~# s3cmd /opt/rke/etcdbackup/snapshot.db s3://rke-etcd-backup/
+```
+
+### 4. Pull the backup on a new node
+
+To simulate the failure lets powerdown node2 completely:
+
+```
+root@node2:~# poweroff
+```
+
+Now its time to pull the backup saved on s3 on a new node:
+
+|  Name |    IP    |          Role          |
+|:-----:|:--------:|:----------------------:|
+| node1 | 10.0.0.1 | [controlplane, worker] |
+| ~~node2~~ | ~~10.0.0.2~~ | ~~[etcd]~~                 |
+| node3 | 10.0.0.3 | [etcd]                 |
+|   |   |   |
+```
+root@node3:~# mkdir -p /opt/rke/etcdbackup
+root@node3:~# s3cmd get s3://rke-etcd-backup/snapshot.db /opt/rke/etcdbackup/snapshot.db
+```
+
+### 5. Restore etcd on the new node
+
+Now lets do a restore to restore and run etcd on the third node, in order to do that you have first to add the third node to the cluster configuration file:
+```
+nodes:
+  - address: 10.0.0.1
+    hostname_override: node1
+    user: ubuntu
+    role: [controlplane,worker]
+#  - address: 10.0.0.2
+#    hostname_override: node2
+#    user: ubuntu
+#    role: [etcd]
+  - address: 10.0.0.3
+    hostname_override: node3
+    user: ubuntu
+    role: [etcd]
+```
+and then run `rke etcd restore`:
+```
+rke etcd restore --name snapshot.db --config cluster.yml
+```
+
+The previous command will restore the etcd data dir from the snapshot and run etcd container on this node, the final step is to restore the operations on the cluster by making the k8s api to point to the new etcd, to do that we run `rke up` again on the new cluster.yml file:
+```
+rke up --config cluster.yml
+```
+You can make sure that operations have been restored by checking the nginx deployment we created earlier:
+```
+> kubectl get pods                                                    
+NAME                     READY     STATUS    RESTARTS   AGE
+nginx-65899c769f-kcdpr   1/1       Running   0          17s
+nginx-65899c769f-pc45c   1/1       Running   0          17s
+nginx-65899c769f-qkhml   1/1       Running   0          17s
+```