--- title: Kubernetes Resources --- The commands/steps listed on this page can be used to check the most important Kubernetes resources and apply to [Rancher Launched Kubernetes](../../how-to-guides/new-user-guides/launch-kubernetes-with-rancher/launch-kubernetes-with-rancher.md) clusters. Make sure you configured the correct kubeconfig (for example, `export KUBECONFIG=$PWD/kube_config_cluster.yml` for Rancher HA) or are using the embedded kubectl via the UI. ## Nodes ### Get nodes Run the command below and check the following: - All nodes in your cluster should be listed, make sure there is not one missing. - All nodes should have the **Ready** status (if not in **Ready** state, check the `kubelet` container logs on that node using `docker logs kubelet`) - Check if all nodes report the correct version. - Check if OS/Kernel/Docker values are shown as expected (possibly you can relate issues due to upgraded OS/Kernel/Docker) ``` kubectl get nodes -o wide ``` Example output: ``` NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME controlplane-0 Ready controlplane 31m v1.13.5 138.68.188.91 Ubuntu 18.04.2 LTS 4.15.0-47-generic docker://18.9.5 etcd-0 Ready etcd 31m v1.13.5 138.68.180.33 Ubuntu 18.04.2 LTS 4.15.0-47-generic docker://18.9.5 worker-0 Ready worker 30m v1.13.5 139.59.179.88 Ubuntu 18.04.2 LTS 4.15.0-47-generic docker://18.9.5 ``` ### Get node conditions Run the command below to list nodes with [Node Conditions](https://kubernetes.io/docs/concepts/architecture/nodes/#condition) ``` kubectl get nodes -o go-template='{{range .items}}{{$node := .}}{{range .status.conditions}}{{$node.metadata.name}}{{": "}}{{.type}}{{":"}}{{.status}}{{"\n"}}{{end}}{{end}}' ``` Run the command below to list nodes with [Node Conditions](https://kubernetes.io/docs/concepts/architecture/nodes/#condition) that are active that could prevent normal operation. ``` kubectl get nodes -o go-template='{{range .items}}{{$node := .}}{{range .status.conditions}}{{if ne .type "Ready"}}{{if eq .status "True"}}{{$node.metadata.name}}{{": "}}{{.type}}{{":"}}{{.status}}{{"\n"}}{{end}}{{else}}{{if ne .status "True"}}{{$node.metadata.name}}{{": "}}{{.type}}{{": "}}{{.status}}{{"\n"}}{{end}}{{end}}{{end}}{{end}}' ``` Example output: ``` worker-0: DiskPressure:True ``` ## Kubernetes leader election ### Kubernetes Controller Manager leader The leader is determined by a leader election process. After the leader has been determined, the leader (`holderIdentity`) is saved in the `kube-controller-manager` endpoint (in this example, `controlplane-0`). ``` kubectl -n kube-system get endpoints kube-controller-manager -o jsonpath='{.metadata.annotations.control-plane\.alpha\.kubernetes\.io/leader}' {"holderIdentity":"controlplane-0_xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx","leaseDurationSeconds":15,"acquireTime":"2018-12-27T08:59:45Z","renewTime":"2018-12-27T09:44:57Z","leaderTransitions":0}> ``` ### Kubernetes Scheduler leader The leader is determined by a leader election process. After the leader has been determined, the leader (`holderIdentity`) is saved in the `kube-scheduler` endpoint (in this example, `controlplane-0`). ``` kubectl -n kube-system get endpoints kube-scheduler -o jsonpath='{.metadata.annotations.control-plane\.alpha\.kubernetes\.io/leader}' {"holderIdentity":"controlplane-0_xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx","leaseDurationSeconds":15,"acquireTime":"2018-12-27T08:59:45Z","renewTime":"2018-12-27T09:44:57Z","leaderTransitions":0}> ``` ## Ingress Controller The default Ingress Controller is Traefik and is deployed as a DaemonSet in the `kube-system` namespace. The pods are only scheduled to nodes with the `worker` role. Check if the pods are running on all nodes: ``` kubectl -n kube-system get pods -o wide ``` Example RKE2 output: ``` kubectl -n kube-system get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE local-path-provisioner-xxxxxxxxxx-xxxxx 1/1 Running 0 17m x.x.x.x worker-1 rke2-traefik-xxxxxxxxxx-xxxxx 1/1 Running 0 14m x.x.x.x worker-1 svclb-rke2-traefik-xxxxxxxx-xxxxx 1/1 Running 0 13m x.x.x.x worker-0 ... ``` Example K3s output: ``` kubectl -n kube-system get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE local-path-provisioner-xxxxxxxxxx-xxxxx 1/1 Running 0 17m x.x.x.x worker-1 traefik-xxxxxxxxxx-xxxxx 1/1 Running 0 14m x.x.x.x worker-1 svclb-traefik-xxxxxxxx-xxxxx 1/1 Running 0 13m x.x.x.x worker-0 ... ``` If a pod is unable to run (Status is not **Running**, Ready status is not showing `1/1` or you see a high count of Restarts), check the pod details, logs and namespace events. ### Pod details RKE2 example: ``` kubectl -n kube-system describe pods -l app.kubernetes.io/name=rke2-traefik ``` K3s example: ``` kubectl -n kube-system describe pods -l app.kubernetes.io/name=traefik ``` ### Pod container logs The below command can show the logs of all the pods labeled "app.kubernetes.io/name=rke2-traefik" if using RKE2 or "app.kubernetes.io/name=traefik" if using K3s, but it will display only 10 lines of log because of the restrictions of the `kubectl logs` command. Refer to `--tail` of `kubectl logs -h` for more information. RKE2 example: ``` kubectl -n kube-system logs -l app.kubernetes.io/name=rke2-traefik ``` K3s example: ``` kubectl -n kube-system logs -l app.kubernetes.io/name=traefik ``` If the full log is needed, specify the pod name in the trailing command: ``` kubectl -n kube-system logs ``` ### Namespace events ``` kubectl -n kube-system get events ``` ### Debug logging To enable debug logging: RKE2 example: ``` cat <