Merge pull request #843 from MBishop17/clean-nodes-clusters

Clean nodes clusters
This commit is contained in:
Denise
2018-10-05 15:50:04 -07:00
committed by GitHub
4 changed files with 362 additions and 189 deletions
@@ -0,0 +1,16 @@
---
title: Removing Rancher
weight: 5000
---
When you deploy Rancher and use it to provision clusters, Rancher installs its components on the nodes you use. This section features instructions on how to remove Rancher's components from your nodes that you no longer want to use with Rancher.
There are two contexts in which you'd remove Rancher from a Kubernetes cluster node.
- [Removing Rancher Components from Rancher Launched Kubernetes Clusters]({{< baseurl >}}/rancher/v2.x/en/admin-settings/removing-rancher/user-cluster-nodes/)
In this context, you are removing Rancher components from Kubernetes clusters that you [launched using Rancher]({{< baseurl >}}/rancher/v2.x/en/cluster-provisioning/rke-clusters/).
- [Removing Rancher from Your Rancher Server Nodes]({{< baseurl >}}/rancher/v2.x/en/admin-settings/removing-rancher/rancher-cluster-nodes/)
In this context, you are removing Rancher from the Kubernetes cluster that you configured for your [Rancher installation]({{< baseurl >}}/rancher/v2.x/en/installation/ha/).
@@ -0,0 +1,81 @@
---
title: Removing Rancher from Your Rancher Server Nodes
weight: 2000
---
When you want to remove Rancher from your [installation cluster]({{< baseurl >}}/rancher/v2.x/en/installation/ha/) as part of a Rancher reinstall (or uninstall), follow the instructions below to download and run _system-tools_, a utility that removes all Rancher components from Rancher Server nodes provisioned by RKE.
### Download and Configuration
You can download the latest version of Rancher system-tools from its GitHub [releases page](https://github.com/rancher/system-tools/releases). Download the version of system-tools for the OS that you're running the tool from.
Operating System | File
-----------------|-----
MacOS | `system-tools_darwin-amd64`
Linux | `system-tools_linux-amd64`
Windows | `system-tools_windows-amd64.exe`
<br>
After you download the tools, complete the following actions:
1. Rename the file to `system-tools`.
1. Give the file executable permissions by running the following command:
```
chmod +x system-tools
```
1. Find the kubeconfig file that was generated during your Rancher installation, `kube_config_rancher-cluster.yml`. Move it to the `/.kube` on your workstation, if it isn't already there. Create this directory if it doesn't exist.
System-tools uses this file to access your installation cluster.
### Using the System-Tool
System-tools is a utility for running operational tasks on Rancher clusters. In this use case, it will help you remove the Rancher from your installation nodes.
#### Usage
After you move the `system-tools` and kubeconfig file to your workstation's `~/.kube` directory, you can run system-tools by changing to the `~/.kube` directory and entering the following command.
>**Warning:** This command will remove data from your etcd nodes. Make sure you have created a [backup of etcd]({{< baseurl >}}/rancher/v2.x/en/backups/backups) before executing the command.
```
./system-tools remove --kubeconfig <$KUBECONFIG> --namespace <NAMESPACE>
```
<br/>
When you run this command, the components listed in [What Gets Removed?](#what-gets-removed) are deleted.
##### Options
| Option | Description |
| ---------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------- |
| `--kubeconfig <KUBECONFIG_PATH>, -c <KUBECONFIG_PATH>` | The cluster's kubeconfig file absolute path, usually `~/.kube/kube_config_rancher-cluster.yml`.<sup>1</sup> |
| `--namespace <NAMESPACE>, -n cattle-system` | Rancher 2.x deployment namespace (`<NAMESPACE>`). If no namespace is defined, the options defaults to `cattle-system`. |
| `--force` | Skips the the interactive removal confirmation and removes the Rancher deployment without prompt. |
> <sup>1</sup> If you are working with multiple Kubernetes clusters, you can place `kube_config_rancher-cluster.yml` in another directory path and then set the `KUBECONFIG` environment variable to its path.
>```
export KUBECONFIG=$(pwd)/kube_config_rancher-cluster.yml
```
## What Gets Removed?
When removing Rancher from server nodes launched using RKE, the following components are deleted.
- The Rancher deployment namespace (`cattle-system` by default).
- Any `serviceAccount`, `clusterRoles`, and `clusterRoleBindings` that Rancher applied the `cattle.io/creator:norman` label to. Rancher applies this label to any resource that it creates as of v2.1.0.
- Labels, annotations, and finalizers.
- Rancher Deployment.
- Machines, clusters, projects, and user custom resource deployments (CRDs).
- All resources create under the `management.cattle.io` API Group.
- All CRDs created by Rancher v2.0.x.
>**Using 2.0.8 or Earlier?**
>
>These versions of Rancher do not automatically delete the `serviceAccount`, `clusterRole`, and `clusterRoleBindings` resources after the job runs. You'll have to delete them yourself.
@@ -0,0 +1,265 @@
---
title: Removing Rancher Components from Rancher Launched Kubernetes Nodes
weight: 375
aliases:
- /rancher/v2.x/en/installation/removing-rancher/cleaning-cluster-nodes/
- /rancher/v2.x/en/installation/removing-rancher/
- /rancher/v2.x/en/faq/cleaning-cluster-nodes/
---
When you use Rancher to [launch nodes for a cluster]({{< baseurl >}}rancher/v2.x/en/cluster-provisioning/#cluster-creation-in-rancher), resources (containers/virtual network interfaces) and configuration items (certificates/configuration files) are created.
When removing nodes from your Rancher-launched cluster (provided that they are in `Active` state), those resources automatically cleaned, and the only action needed is to restart the node. When a node has become unreachable and the automatic cleanup process cannot be used, we describe the steps that need to be executed before the node can be added to a cluster again.
## What Gets Removed?
When cleaning nodes provisioned using Rancher, the following components are deleted based on the type of cluster node you're removing.
| Removed Component | [IaaS Nodes][1] | [Custom Nodes][2] | [Hosted Cluster][3] | [Imported Nodes][4] |
| ------------------------------------------------------------------------------ | --------------- | ----------------- | ------------------- | ------------------- |
| The Rancher deployment namespace (`cattle-system` by default) | ✓ | ✓ | ✓ | ✓ |
| `serviceAccount`, `clusterRoles`, and `clusterRoleBindings` labeled by Rancher | ✓ | ✓ | ✓ | ✓ |
| Labels, Annotations, and Finalizers | ✓ | ✓ | ✓ | ✓ |
| Rancher Deployment | ✓ | ✓ | ✓ | |
| Machines, clusters, projects, and user custom resource deployments (CRDs) | ✓ | ✓ | ✓ | |
| All resources create under the `management.cattle.io` API Group | ✓ | ✓ | ✓ | |
| All CRDs created by Rancher v2.0.x | ✓ | ✓ | ✓ | |
[1]: {{< baseurl >}}/rancher/v2.x/en/cluster-provisioning/rke-clusters/node-pools/
[2]: {{< baseurl >}}/rancher/v2.x/en/cluster-provisioning/rke-clusters/custom-nodes/
[3]: {{< baseurl >}}/rancher/v2.x/en/cluster-provisioning/hosted-kubernetes-clusters/
[4]: {{< baseurl >}}/rancher/v2.x/en/cluster-provisioning/imported-clusters/
## Removing A Node from a Cluster by Rancher UI
When the node is in `Active` state, removing the node from a cluster will trigger a process to clean up the node. Please restart the node after the automatic cleanup process is done to make sure any non-persistent data is properly removed.
**To restart a node:**
```
# using reboot
reboot
# using shutdown
shutdown -r now
```
## Cleaning a Node Manually
When a node is unreachable and removed from the cluster, the automatic cleaning process can't be triggered because the node is unreachable. Please follow the steps below to manually clean the node.
>**Warning:** The commands listed below will remove data from the node. Make sure you have created a backup of files you want to keep before executing any of the commands as data will be lost.
## Imported Cluster Nodes
For imported clusters, the process for removing Rancher from its nodes is a little different. You can the option of simply deleting the cluster in the Rancher UI, or your can run a script that removes Rancher components from the nodes. Both options make the same deletions.
{{% tabs %}}
{{% tab "By UI / API" %}}
>**Warning:** This process will remove data from your nodes. Make sure you have created a backup of files you want to keep before executing the command, as data will be lost.
After you initiate the removal of an [imported cluster]({{< baseurl >}}/rancher/v2.x/en/cluster-provisioning/#import-existing-cluster) using the Rancher UI (or API), the following events occur.
1. Rancher creates a `serviceAccount` that it uses to remove the cluster. This account is assigned the [clusterRole](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#role-and-clusterrole) and [clusterRoleBinding](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#rolebinding-and-clusterrolebinding) permissions, which are required to remove the cluster.
1. Using the `serviceAccount`, Rancher schedules and runs a [job](https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/) that cleans the Rancher and Kubernetes components off of the node. This job also references the `serviceAccount` and its roles as dependencies, so the job deletes them before its completion.
1. Rancher is removed from the cluster nodes. However, the cluster persists, running the native version of Kubernetes.
**Result:** All components listed for imported clusters in [What Gets Removed?](#what-gets-removed) are deleted.
{{% /tab %}}
{{% tab "By Script" %}}
Rather than cleaning imported cluster nodes using the Rancher UI, you can run a script instead.
>**Prerequisite:**
>
>Install [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/).
1. Open a web browser, navigate to [GitHub](https://github.com/rancher/rancher/blob/master/cleanup/user-cluster.sh), and download `user-cluster.sh`.
1. Make the script executable by running the following command from the same directory as `user-cluster.sh`:
```
chmod +x user-cluster.sh
```
1. **Air Gap Users Only:** Open `user-cluster.sh` and replace `yaml_url` with the URL in `user-cluster.yml`.
If you aren't an air gap user, skip this step.
1. From the same directory, run the script:
>**Tip:**
>
>Add the `-dry-run` flag to preview the script's outcome without making changes.
```
./user-cluster.sh rancher/agent:latest
```
**Result:** The script runs. All components listed for imported clusters in [What Gets Removed?](#what-gets-removed) are deleted.
{{% /tab %}}
{{% /tabs %}}
### Docker Containers, Images, and Volumes
Based on what role you assigned to the node, Kubernetes components in containers, containers belonging to overlay networking, DNS, ingress controller and Rancher agent. (and pods you created that have been scheduled to this node)
**To clean all Docker containers, images and volumes:**
```
docker rm -f $(docker ps -qa)
docker rmi -f $(docker images -q)
docker volume rm $(docker volume ls -q)
```
### Mounts
Kubernetes components and secrets leave behind mounts on the system that need to be unmounted.
Mounts |
--------|
`/var/lib/kubelet/pods/XXX` (miscellaneous mounts) |
`/var/lib/kubelet` |
`/var/lib/rancher` |
**To unmount all mounts:**
```
for mount in $(mount | grep tmpfs | grep '/var/lib/kubelet' | awk '{ print $3 }') /var/lib/kubelet /var/lib/rancher; do umount $mount; done
```
### Directories and Files
The following directories are used when adding a node to a cluster, and should be removed. You can remove a directory using `rm -rf /directory_name`.
>**Note:** Depending on the role you assigned to the node, some of the directories will or won't be present on the node.
Directories |
--------|
`/etc/ceph` |
`/etc/cni` |
`/etc/kubernetes` |
`/opt/cni` |
`/opt/rke` |
`/run/secrets/kubernetes.io` |
`/run/calico` |
`/run/flannel` |
`/var/lib/calico` |
`/var/lib/etcd` |
`/var/lib/cni` |
`/var/lib/kubelet` |
`/var/lib/rancher/rke/log` |
`/var/log/containers` |
`/var/log/pods` |
`/var/run/calico` |
**To clean the directories:**
```
rm -rf /etc/ceph \
/etc/cni \
/etc/kubernetes \
/opt/cni \
/opt/rke \
/run/secrets/kubernetes.io \
/run/calico \
/run/flannel \
/var/lib/calico \
/var/lib/etcd \
/var/lib/cni \
/var/lib/kubelet \
/var/lib/rancher/rke/log \
/var/log/containers \
/var/log/pods \
/var/run/calico
```
### Network Interfaces and Iptables
The remaining two components that are changed/configured are (virtual) network interfaces and iptables rules. Both are non-persistent to the node, meaning that they will be cleared after a restart of the node.
This is the recommended method.
**To restart a node:**
```
# using reboot
reboot
# using shutdown
shutdown -r now
```
If you want to know more on (virtual) network interfaces or iptables rules, please see the specific subjects below.
### Network Interfaces
>**Note:** Depending on the network provider configured for the cluster the node was part of, some of the interfaces will or won't be present on the node.
Interfaces |
--------|
`flannel.1` |
`cni0` |
`tunl0` |
`caliXXXXXXXXXXX` (random interface names) |
`vethXXXXXXXX` (random interface names) |
**To list all interfaces:**
```
# Using ip
ip address show
# Using ifconfig
ifconfig -a
```
**To remove an interface:**
```
ip link delete interface_name
```
### Iptables
>**Note:** Depending on the network provider configured for the cluster the node was part of, some of the chains will or won't be present on the node.
Iptables rules are used to route traffic from and to containers. The created rules are not persistent, so restarting the node will restore iptables to it's original state.
Chains |
--------|
`cali-failsafe-in` |
`cali-failsafe-out` |
`cali-fip-dnat` |
`cali-fip-snat` |
`cali-from-hep-forward` |
`cali-from-host-endpoint` |
`cali-from-wl-dispatch` |
`cali-fw-caliXXXXXXXXXXX` (random chain names) |
`cali-nat-outgoing` |
`cali-pri-kns.NAMESPACE` (chain per namespace) |
`cali-pro-kns.NAMESPACE` (chain per namespace) |
`cali-to-hep-forward` |
`cali-to-host-endpoint` |
`cali-to-wl-dispatch` |
`cali-tw-caliXXXXXXXXXXX` (random chain names) |
`cali-wl-to-host` |
`KUBE-EXTERNAL-SERVICES` |
`KUBE-FIREWALL` |
`KUBE-MARK-DROP` |
`KUBE-MARK-MASQ` |
`KUBE-NODEPORTS` |
`KUBE-SEP-XXXXXXXXXXXXXXXX` (random chain names) |
`KUBE-SERVICES` |
`KUBE-SVC-XXXXXXXXXXXXXXXX` (random chain names) |
**To list all iptables rules:**
```
iptables -L -t nat
iptables -L -t mangle
iptables -L
```
@@ -1,189 +0,0 @@
---
title: Cleaning cluster nodes
weight: 375
aliases:
- /rancher/v2.x/en/installation/removing-rancher/cleaning-cluster-nodes/
- /rancher/v2.x/en/installation/removing-rancher/
---
When adding a node to a cluster, resources (containers/(virtual) network interfaces) and configuration items (certificates/configuration files) are created. When removing a node from a cluster (if it is in `Active` state), those resources will be automatically cleaned and the only action needed is to restart the node. When a node has become unreachable and the automatic cleanup process cannot be used, we describe the steps that need to be executed before the node can be added to a cluster again.
## Removing a node from a cluster via Rancher UI
When the node is in `Active` state, removing the node from a cluster will trigger a process to clean up the node. Please restart the node after the automatic cleanup process is done to make sure any non-persistent data is properly removed.
* How to restart a node
```
# using reboot
reboot
# using shutdown
shutdown -r now
```
## Cleaning a node manually
When a node is unreachable and removed from the cluster, the automatic cleaning process can't be triggered because the node is unreachable. Please follow the steps below to manually clean the node.
>**Warning:** The commands listed below will remove data from the node. Make sure you have created a backup of files you want to keep before executing any of the commands as data will be lost.
### Docker containers, images and volumes
Based on what role you assigned to the node, Kubernetes components in containers, containers belonging to overlay networking, DNS, ingress controller and Rancher agent. (and pods you created that have been scheduled to this node)
* How to clean all Docker containers, images and volumes:
```
docker rm -f $(docker ps -qa)
docker rmi -f $(docker images -q)
docker volume rm $(docker volume ls -q)
```
### Mounts
Kubernetes components and secrets leave behind mounts on the system that need to be unmounted.
Mounts |
--------|
`/var/lib/kubelet/pods/XXX` (miscellaneous mounts) |
`/var/lib/kubelet` |
`/var/lib/rancher` |
* How to unmount all mounts:
```
for mount in $(mount | grep tmpfs | grep '/var/lib/kubelet' | awk '{ print $3 }') /var/lib/kubelet /var/lib/rancher; do umount $mount; done
```
### Directories and files
The following directories are used when adding a node to a cluster, and should be removed. You can remove a directory using `rm -rf /directory_name`.
>**Note:** Depending on the role you assigned to the node, some of the directories will or won't be present on the node.
Directories |
--------|
`/etc/ceph` |
`/etc/cni` |
`/etc/kubernetes` |
`/opt/cni` |
`/opt/rke` |
`/run/secrets/kubernetes.io` |
`/run/calico` |
`/run/flannel` |
`/var/lib/calico` |
`/var/lib/etcd` |
`/var/lib/cni` |
`/var/lib/kubelet` |
`/var/lib/rancher/rke/log` |
`/var/log/containers` |
`/var/log/pods` |
`/var/run/calico` |
* How to clean the directories:
```
rm -rf /etc/ceph \
/etc/cni \
/etc/kubernetes \
/opt/cni \
/opt/rke \
/run/secrets/kubernetes.io \
/run/calico \
/run/flannel \
/var/lib/calico \
/var/lib/etcd \
/var/lib/cni \
/var/lib/kubelet \
/var/lib/rancher/rke/log \
/var/log/containers \
/var/log/pods \
/var/run/calico
```
### Network interfaces and iptables
The remaining two components that are changed/configured are (virtual) network interfaces and iptables rules. Both are non-persistent to the node, meaning that they will be cleared after a restart of the node.
This is the recommended method.
* How to restart a node
```
# using reboot
reboot
# using shutdown
shutdown -r now
```
If you want to know more on (virtual) network interfaces or iptables rules, please see the specific subjects below.
### Network interfaces
>**Note:** Depending on the network provider configured for the cluster the node was part of, some of the interfaces will or won't be present on the node.
Interfaces |
--------|
`flannel.1` |
`cni0` |
`tunl0` |
`caliXXXXXXXXXXX` (random interface names) |
`vethXXXXXXXX` (random interface names) |
* How to list all interfaces:
```
# Using ip
ip address show
# Using ifconfig
ifconfig -a
```
* How to remove an interface:
```
ip link delete interface_name
```
### Iptables
>**Note:** Depending on the network provider configured for the cluster the node was part of, some of the chains will or won't be present on the node.
Iptables rules are used to route traffic from and to containers. The created rules are not persistent, so restarting the node will restore iptables to it's original state.
Chains |
--------|
`cali-failsafe-in` |
`cali-failsafe-out` |
`cali-fip-dnat` |
`cali-fip-snat` |
`cali-from-hep-forward` |
`cali-from-host-endpoint` |
`cali-from-wl-dispatch` |
`cali-fw-caliXXXXXXXXXXX` (random chain names) |
`cali-nat-outgoing` |
`cali-pri-kns.NAMESPACE` (chain per namespace) |
`cali-pro-kns.NAMESPACE` (chain per namespace) |
`cali-to-hep-forward` |
`cali-to-host-endpoint` |
`cali-to-wl-dispatch` |
`cali-tw-caliXXXXXXXXXXX` (random chain names) |
`cali-wl-to-host` |
`KUBE-EXTERNAL-SERVICES` |
`KUBE-FIREWALL` |
`KUBE-MARK-DROP` |
`KUBE-MARK-MASQ` |
`KUBE-NODEPORTS` |
`KUBE-SEP-XXXXXXXXXXXXXXXX` (random chain names) |
`KUBE-SERVICES` |
`KUBE-SVC-XXXXXXXXXXXXXXXX` (random chain names) |
* How to list all iptables rules
```
iptables -L -t nat
iptables -L -t mangle
iptables -L
```