diff --git a/content/rancher/v2.x/en/backups/restorations/_index.md b/content/rancher/v2.x/en/backups/restorations/_index.md index 88982fcc4bf..ccfb75a22e8 100644 --- a/content/rancher/v2.x/en/backups/restorations/_index.md +++ b/content/rancher/v2.x/en/backups/restorations/_index.md @@ -4,5 +4,5 @@ weight: 1010 --- If you lose the data on your Rancher Server, you can restore it if you have backups stored in a safe location. -- [Restoring Backups—Single Node Installs](./single-node-restoration/) -- [Restoring Backups—High Availablity Installs](./ha-restoration/) \ No newline at end of file +- [Restoring Backups—Single Node Installs]({{< baseurl >}}/rancher/v2.x/en/backups/restorations/single-node-restoration/) +- [Restoring Backups—High Availability Installs]({{< baseurl >}}/rancher/v2.x/en/backups/restorations/ha-restoration/) diff --git a/content/rancher/v2.x/en/backups/restorations/ha-restoration/_index.md b/content/rancher/v2.x/en/backups/restorations/ha-restoration/_index.md index e60dbcc1c59..1dea8ac3a97 100644 --- a/content/rancher/v2.x/en/backups/restorations/ha-restoration/_index.md +++ b/content/rancher/v2.x/en/backups/restorations/ha-restoration/_index.md @@ -4,132 +4,162 @@ weight: 370 aliases: - /rancher/v2.x/en/installation/after-installation/ha-backup-and-restoration/ --- -This procedure describes how to restore your a snapshot of `etcd` if you lose your Rancher data in a disaster scenario. -## Restoration Outline +This procedure describes how to use RKE to restore a snapshot of the Rancher Kubernetes cluster. The cluster snapshot will include Kubernetes configuration and the Rancher database and state. -Following a disaster scenario, restoration of your HA Rancher installation requires you to pull your snapshot from your chosen external location and then restore it. +## Restore Outline -1. [Create New Nodes and Pull Snapshot and Certificates Bundle](#1-create-new-nodes-and-pull-snapshot-and-certificate-bundle) +1. [Preparation](#1-preparation) - If `etcd` nodes failed or quorum lost for any reason, you can create new nodes and restore the etcd snapshot and certificate bundle to it, first create the new nodes and then pull the most recent `etcd` snapshot to that node, and the most recent certificates pki bundle `pki.bundle.tar.gz` + Install utilities and create new or clean existing nodes to prepare for restore. -2. [Restore etcd Database](#2-restore-etcd-database) +2. [Place Snapshot and PKI Bundle](#2-place-snapshot-and-pki-bundle) - After you pull the snapshot and the certificates bundle, run the RKE command to restore the `etcd` database. + Pick a node and place snapshot `.db` and `pki.bundle.tar.gz` files. + +3. [Configure RKE](#3-configure-rke) + + Configure RKE `cluster.yml`. Remove `addons:` section and point configuration to the clean nodes. + +4. [Restore Database](#4-restore-database) + + Run RKE command to restore the `etcd` database to a single node. + +5. [Bring Up the Cluster](#5-bring-up-the-cluster) + + Run RKE commands to bring up cluster one a single node. Clean up old nodes. Verify and add additional nodes.
-### 1. Pull Snapshot and Certificates Bundle On New Nodes -If `etcd` nodes failed, you can either: +### 1. Preparation -- Create a new nodes of your choice—baremetal, on-prem virtual machine, cloud-based virtual machine, and so on. Provision it according to our [requirements]({{< baseurl >}}/rancher/v2.x/en/installation/ha-server-install/#host-requirements). +You will need [RKE]({{< baseurl >}}/rke/v0.1.x/en/installation/) and [kubectl]({{< baseurl >}}/rancher/v2.x/en/faq/kubectl/) CLI utilities installed. -- Use the existing nodes, but you should be make sure to clean each node using the following commands: +Prepare by creating 3 new nodes to be the target for the restored Rancher instance. See [HA Install]({{< baseurl >}}/rancher/v2.x/en/installation/ha/create-nodes-lb/) for node requirements. + +We recommend that you start with fresh nodes and a clean state. Alternatively you can clear Kubernetes and Rancher configurations from the existing nodes. This will destroy the data on these nodes. See [Node Cleanup]({{< baseurl >}}/rancher/v2.x/en/faq/cleaning-cluster-nodes/) for the procedure. + +### 2. Place Snapshot and PKI Bundle + +Pick a one of the clean nodes. That node will be the "target node" for the initial restore. Place the snapshot and PKI certificate bundle files in the `/opt/rke/etcd-snapshots` directory on the "target node". + +* Snapshot - `.db` +* PKI Bundle - `pki.bundle.tar.gz` + +### 3. Configure RKE + +Make a copy of your original `rancher-cluster.yml` file. ``` -docker rm -fv $(docker ps -qa) -docker volume ls -q | xargs docker volume rm -rm -rf /etc/kubernetes/ssl -rm -rf /etc/kubernetes/.tmp -rm -rf /var/lib/etcd -rm -rf /etc/cni -rm -rf /etc/kube-flannel/ +cp rancher-cluster.yml rancher-cluster-restore.yml ``` -in both cases you will need to pull the most recent working `etcd` snapshot and the certificates bundle to these nodes. +Modify the copy and make the following changes. -**To Pull the Latest Snapshot and Certificates Bundle:** +* Remove or comment out entire the `addons:` section. The Rancher deployment and supporting configuration is already in the `etcd` database. +* Change your `nodes:` section to point to the restore nodes. +* Comment out the nodes that are not your "target node". We want the cluster to only start on that one node. -1. Log in to each new node using a remote Terminal connection. +*Example* `rancher-cluster-restore.yml` -2. Create a directory that mirrors your other nodes' snapshot directories: +```yaml +nodes: +- address: 52.15.238.179 # New Target Node + user: ubuntu + role: [ etcd, controlplane, worker ] +# - address: 52.15.23.24 +# user: ubuntu +# role: [ etcd, controlplane, worker ] +# - address: 52.15.238.133 +# user: ubuntu +# role: [ etcd, controlplane, worker ] - ``` - root@newnode:~# mkdir -p /opt/rke/etcd-snapshots - ``` +# addons: |- +# --- +# kind: Namespace +# apiVersion: v1 +# metadata: +# name: cattle-system +# --- +... -3. On each node, pull your most recent snapshot onto the node. Replace `` with the name of the snapshot you're restoring to. - - ``` - root@newnode:~# s3cmd get s3://rke-etcd-snapshots/ /opt/rke/etcd-snapshots/ - ``` - - >**Remember:** Our use of Amazon S3 is an example used for this documentation. The command for pulling your snapshot may vary. - -4. On each node, pull the most recent certificates pki bundle to the `/opt/rke/etcd-snapshots`: - - ``` - root@newnode:~# s3cmd get s3://rke-etcd-snapshots/pki.bundle.tar.gz /opt/rke/etcd-snapshots/pki.bundle.tar.gz - ``` - -### 2. Restore `etcd` Database - -To restore the most recent `etcd` snapshot on your new nodes, run RKE the command `rke etcd snapshot-restore`. This command reverts to any snapshot stored in `/opt/rke/etcd-snapshots` that you explicitly define. When you run `rke etcd snapshot-restore`, RKE removes the old `etcd` container if it still exists. To restore operations, RKE creates a new `etcd` cluster using the snapshot you choose, and extracts the certificate bundle to be used on the next `rke up` run. - ->**Warning:** Restoring an `etcd` snapshot deletes your current `etcd` cluster and replaces it with a new one. Before you run the `rke etcd snapshot-restore` command, backup any important data in your current cluster. -> ->**Prerequisites:** {{< requirements_rollback >}} - - -1. From your workstation, open `rancher-cluster.yml` in your favorite text editor. - -2. Replace the unresponsive nodes (`1.1.1.1`, `2.2.2.2`, `3.3.3.3` in this example) with your new one (`4.4.4.4`, `5.5.5.5`, and `6.6.6.6`). You IP addresses will be different obviously: -``` - nodes: - # - address: 1.1.1.1 # UNRESPONSIVE NODE - # user: root - # role: [controlplane,etcd,worker] - # ssh_key_path: ~/.ssh/id_rsa - # - address: 2.2.2.2 # UNRESPONSIVE NODE - # user: root - # role: [controlplane,etcd,worker] - # ssh_key_path: ~/.ssh/id_rsa - # - address: 3.3.3.3 # UNRESPONSIVE NODE - # user: root - # role: [controlplane,etcd,worker] - # ssh_key_path: ~/.ssh/id_rsa - - address: 4.4.4.4 # NEW NODE - user: root - role: [controlplane,etcd,worker] - ssh_key_path: ~/.ssh/id_rsa - - address: 5.5.5.5 # NEW NODE - user: root - role: [controlplane,etcd,worker] - ssh_key_path: ~/.ssh/id_rsa - - address: 6.6.6.6 # NEW NODE - user: root - role: [controlplane,etcd,worker] - ssh_key_path: ~/.ssh/id_rsa ``` -3. Save and close `rancher-cluster.yml`. +### 4. Restore Database -4. Open **Terminal** and change directory to the location of the RKE binary. Your `rancher-cluster.yml` file must reside in the same directory. +Use RKE with the new `rancher-cluster-restore.yml` configuration and restore the database to the single "target node". -5. Run one of the following commands to restore the `etcd` database: +``` +rke etcd snapshot-restore --name .db --config ./rancher-cluster-restore.yml +``` - ``` - # MacOS - ./rke_darwin-amd64 etcd snapshot-restore --name --config rancher-cluster.yml - # Linux - ./rke_linux-amd64 etcd snapshot-restore --name --config rancher-cluster.yml - ``` +> **NOTE:** RKE will create an `etcd` container with the restored database on the "target node". This container will not complete the `etcd` initialization and stay in a running state until the cluster brought up in the next step. +### 5. Bring Up the Cluster -6. Run one of the following commands to bring your cluster back up: +Use RKE and bring up the cluster on the single "target node". - ``` - # MacOS - ./rke_darwin-amd64 up --config rancher-cluster.yml - # Linux - ./rke_linux-amd64 up --config rancher-cluster.yml - ``` +``` +rke up --config ./rancher-cluster-restore.yml +``` -7. After restoring the cluster you have to restart the kubernetes components on all nodes, otherwise there will be some conflicts with resource versions of objects stored in `etcd`, this will include restart to kubernetes components and the network components, for more information please refer to [kubernetes documentation](https://kubernetes.io/docs/tasks/administer-cluster/configure-upgrade-etcd/#etcd-upgrade-requirements), to do that you can run the following on each node: +#### Testing the Cluster - ``` - docker restart kube-apiserver kubelet kube-controller-manager kube-scheduler kube-proxy - docker ps | grep flannel | cut -f 1 -d " " | xargs docker restart - docker ps | grep calico | cut -f 1 -d " " | xargs docker restart - ``` +Once RKE completes, the cluster should now be up and Rancher should be running. RKE will have created a credentials file in the local directory. Configure `kubectl` to use the `kube_config_rancher-cluster-restore.yml` credentials file and check on the state of the cluster. See [Installing and Configuring kubectl]({{< baseurl >}}/rancher/v2.x/en/faq/kubectl/#configuration) for details. + +You should see your new "target node" in `Ready` and three old nodes in `NotReady`. + +``` +kubectl get nodes + +NAME STATUS ROLES AGE VERSION +52.15.238.179 Ready controlplane,etcd,worker 1m v1.10.5 +18.217.82.189 NotReady controlplane,etcd,worker 16d v1.10.5 +18.222.22.56 NotReady controlplane,etcd,worker 16d v1.10.5 +18.191.222.99 NotReady controlplane,etcd,worker 16d v1.10.5 +``` + +#### Cleaning up Old Nodes + +Use `kubectl` to delete the old nodes from the cluster. Repeat for each old node. + +``` +kubectl delete node 18.217.82.189 +``` + +#### Adding in Additional Nodes + +Edit the `rancher-cluster-restore.yml` RKE config file and uncomment the additional nodes. + +*Example* `rancher-cluster-restore.yml` + +```yaml +nodes: +- address: 52.15.238.179 # New Target Node + user: ubuntu + role: [ etcd, controlplane, worker ] +- address: 52.15.23.24 + user: ubuntu + role: [ etcd, controlplane, worker ] +- address: 52.15.238.133 + user: ubuntu + role: [ etcd, controlplane, worker ] + +# addons: |- +# --- +# kind: Namespace +... + +``` + +Run RKE and add the nodes to the new cluster. + +``` +rke up --config ./rancher-cluster-restore.yml +``` + +#### Finishing Up + +Rancher should now be running and available to manage your Kubernetes clusters. Swap your Rancher DNS or Load Balancer endpoints to target the new cluster. Once this is done the agents on your managed clusters should automatically reconnect. This may take 10-15 minutes due to reconnect back off timeouts. + +> **IMPORTANT:** Remember to save your new RKE config (`rancher-cluster-restore.yml`) and `kubectl` credentials (`kube_config_rancher-cluster-restore.yml`) files in a safe place for future maintenance. diff --git a/content/rancher/v2.x/en/faq/kubectl/_index.md b/content/rancher/v2.x/en/faq/kubectl/_index.md new file mode 100644 index 00000000000..b4172ab0a40 --- /dev/null +++ b/content/rancher/v2.x/en/faq/kubectl/_index.md @@ -0,0 +1,30 @@ +--- +title: Installing and Configuring kubectl +weight: 100 +--- + +`kubectl` is a CLI utility for running commands against Kubernetes clusters. It's required for many maintenance and administrative tasks in Rancher 2.x. + +### Installation + +See [kubectl Installation](https://kubernetes.io/docs/tasks/tools/install-kubectl/) for installation on your operating system. + +### Configuration + +When you create a Kubernetes cluster with RKE, RKE creates a `kube_config_rancher-cluster.yml` in the local directory that contains credentials to connect to your new cluster with tools like `kubectl` or `helm`. + +You can copy this file to `$HOME/.kube/config` or if you are working with multiple Kubernetes clusters, set the `KUBECONFIG` environmental variable to the path of `kube_config_rancher-cluster.yml`. + +``` +export KUBECONFIG=$(pwd)/kube_config_rancher-cluster.yml +``` + +Test your connectivity with `kubectl` and see if you can get the list of nodes back. + +``` +kubectl get nodes + NAME STATUS ROLES AGE VERSION +165.227.114.63 Ready controlplane,etcd,worker 11m v1.10.1 +165.227.116.167 Ready controlplane,etcd,worker 11m v1.10.1 +165.227.127.226 Ready controlplane,etcd,worker 11m v1.10.1 +``` diff --git a/content/rancher/v2.x/en/upgrades/rollbacks/ha-server-rollbacks/_index.md b/content/rancher/v2.x/en/upgrades/rollbacks/ha-server-rollbacks/_index.md index 02745f1d2fc..1945c47315b 100644 --- a/content/rancher/v2.x/en/upgrades/rollbacks/ha-server-rollbacks/_index.md +++ b/content/rancher/v2.x/en/upgrades/rollbacks/ha-server-rollbacks/_index.md @@ -6,74 +6,10 @@ aliases: - /rancher/v2.x/en/upgrades/ha-server-rollbacks --- -If you upgrade Rancher, but the upgrade does not complete successfully, you may need to roll back your Rancher Server to its last healthy state before upgrade. To restore Rancher: +If you upgrade Rancher and the upgrade does not complete successfully, you may need to rollback your Rancher Server to its last healthy state. -- Restore the `etcd` snapshot taken before upgrade. -- Run the command to revert to your prior version of Rancher. +To restore Rancher follow the procedure detailed here: [Restoring Backups — High Availability Installs](/rancher/v2.x/en/backups/restorations/ha-restoration) ->**Warning!** -> -> Rolling back to a previous version of Rancher destroys any changes made to Rancher following the upgrade. Unrecoverable data loss may occur. +Restoring a snapshot of the Rancher Server cluster will revert Rancher to the version and state at the time of the snapshot. -> ->**Prerequisites:** {{< requirements_rollback >}} - -1. Open **Terminal** and change directory to the location of the RKE binary that you used during upgrade. - -1. Run one of the following commands to restore the `etcd` snapshot [that you took]({{< baseurl >}}/rancher/v2.x/en/upgrades/upgrade-scenarios/ha-server-upgrade/#snapshot) before your unsuccessful upgrade. Replace the `` placeholder with your snapshot. - - ``` - # MacOS - ./rke_darwin-amd64 etcd snapshot-restore --name --config rancher-cluster.yml - # Linux - ./rke_linux-amd64 etcd snapshot-restore --name --config rancher-cluster.yml - ``` - -1. Run one of the following commands to bring your cluster back up: - - ``` - # MacOS - ./rke_darwin-amd64 up --config rancher-cluster.yml - # Linux - ./rke_linux-amd64 up --config rancher-cluster.yml - ``` - -1. Lastly, restart the Kubernetes components on all cluster nodes to prevent potential `etcd` conflicts. Run this command on each of your nodes. - - ``` - docker restart kube-apiserver kubelet kube-controller-manager kube-scheduler kube-proxy - docker ps | grep flannel | cut -f 1 -d " " | xargs docker restart - docker ps | grep calico | cut -f 1 -d " " | xargs docker restart - ``` -1. Enter the following command: - - ``` -kubectl --kubeconfig=kube_config_rancher-cluster.yml set image deployment/cattle cattle-server=rancher/rancher: -n cattle-system - ``` - Replace `` with the version that you were running before upgrade. For a list of tags available, see the [Rancher Forum Announcements](https://forums.rancher.com/c/announcements). - - **Step Result:** The rollback begins. Rancher Server may be unavailable for a few minutes. - -1. Log into Rancher. Confirm that the rollback succeeded by checking the version displayed in the bottom-left corner of the browser window. - -## Troubleshooting - -With **RKE v0.1.8** and below, the **rke-bundle-cert** container is left over from a failed etcd restore. If you are having an issue with restoring an **etcd snapshot** then you can do the following on each etcd nodes before attempting to do another restore: - -``` -docker container rm --force rke-bundle-cert -``` - -The rke-bundle-cert container is usually removed when a backup or restore of **etcd** succeeds. -Whenever something goes wrong, the **rke-bundle-cert** container will be left over. You can look -at the logs or inspect the container to see what the issue is. - -``` -docker container logs --follow rke-bundle-cert -docker container inspect rke-bundle-cert -``` - -The important thing to note is the mounts of the container and location of the **pki.bundle.tar.gz**. - -As of **RKE v0.1.9**, the **rke-bundle-cert** container is removed on both success and -failure of a restore. To debug any issues, you will need to look at the **logs** generated from rke. +> **NOTE:** Managed cluster are authoritative for their state. This means restoring the rancher server will not revert workload deployments or changes made on managed clusters after the snapshot was taken.