mirror of
https://github.com/rancher/rancher-docs.git
synced 2026-09-26 13:08:11 +00:00
Docs: Migrate worker nodes troubleshooting guide from RKE/Docker to RKE2/K3s/containerd
This commit is contained in:
+68
-14
@@ -8,31 +8,85 @@ title: Troubleshooting Worker Nodes and Generic Components
|
|||||||
|
|
||||||
This section applies to every node as it includes components that run on nodes with any role.
|
This section applies to every node as it includes components that run on nodes with any role.
|
||||||
|
|
||||||
## Check if the Containers are Running
|
## Prerequisites
|
||||||
|
|
||||||
There are two specific containers launched on nodes with the `worker` role:
|
Since RKE2 and K3s utilize `containerd` as the container runtime, `crictl` serves as the primary tool for container management, replacing the Docker CLI. To allow `crictl` to communicate with `containerd`, you must configure your environment by exporting the following variables:
|
||||||
|
|
||||||
* kubelet
|
### RKE2
|
||||||
* kube-proxy
|
|
||||||
|
|
||||||
The containers should have status `Up`. The duration shown after `Up` is the time the container has been running.
|
|
||||||
|
|
||||||
|
```bash
|
||||||
|
export PATH=$PATH:/var/lib/rancher/rke2/bin/
|
||||||
|
export CRI_CONFIG_FILE=/var/lib/rancher/rke2/agent/etc/crictl.yaml
|
||||||
```
|
```
|
||||||
docker ps -a -f=name='kubelet|kube-proxy'
|
|
||||||
|
### K3s
|
||||||
|
|
||||||
|
```bash
|
||||||
|
export PATH=$PATH:/usr/local/bin
|
||||||
|
export CRI_CONFIG_FILE=/var/lib/rancher/k3s/agent/etc/crictl.yaml
|
||||||
|
```
|
||||||
|
|
||||||
|
## Check if the Components are Running
|
||||||
|
|
||||||
|
There are two specific components launched on nodes with the `worker` role:
|
||||||
|
|
||||||
|
* `kubelet`
|
||||||
|
* `kube-proxy`
|
||||||
|
|
||||||
|
### RKE2
|
||||||
|
|
||||||
|
The `kubelet` runs natively as part of the `rke2-agent` (or `rke2-server`) systemd process, while `kube-proxy` runs as a Static Pod managed by `containerd`.
|
||||||
|
|
||||||
|
Check the status of the `kubelet` via the agent service:
|
||||||
|
```bash
|
||||||
|
systemctl status rke2-agent
|
||||||
|
```
|
||||||
|
:::note
|
||||||
|
|
||||||
|
If you are checking a controlplane node, use `systemctl status rke2-server` instead.
|
||||||
|
|
||||||
|
:::
|
||||||
|
|
||||||
|
Check the status of `kube-proxy` using `crictl`:
|
||||||
|
```bash
|
||||||
|
crictl ps --name kube-proxy
|
||||||
```
|
```
|
||||||
|
|
||||||
Example output:
|
Example output:
|
||||||
```
|
```
|
||||||
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
|
CONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID
|
||||||
158d0dcc33a5 rancher/hyperkube:v1.11.5-rancher1 "/opt/rke-tools/en..." 3 hours ago Up 3 hours kube-proxy
|
26c7159abbcc rancher/hardened-kubernetes:v1.28.8-rke2r1-build20240404 3 hours ago Running kube-proxy 0 1a2b3c4d5e6f7
|
||||||
a30717ecfb55 rancher/hyperkube:v1.11.5-rancher1 "/opt/rke-tools/en..." 3 hours ago Up 3 hours kubelet
|
|
||||||
```
|
```
|
||||||
|
|
||||||
## Container Logging
|
### K3s
|
||||||
|
|
||||||
The logging of the containers can contain information on what the problem could be.
|
Both `kubelet` and `kube-proxy` run as embedded processes inside the `k3s-agent` (or `k3s` server) systemd service. There are no separate containers for them.
|
||||||
|
|
||||||
|
Check their status by checking the K3s service:
|
||||||
|
```bash
|
||||||
|
systemctl status k3s-agent
|
||||||
```
|
```
|
||||||
docker logs kubelet
|
:::note
|
||||||
docker logs kube-proxy
|
If you are checking a controlplane node, use `systemctl status k3s` instead.
|
||||||
|
:::
|
||||||
|
|
||||||
|
## Component Logging
|
||||||
|
|
||||||
|
The logging of the components can contain information on what the problem could be.
|
||||||
|
|
||||||
|
### RKE2
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# kubelet logs are part of the systemd service
|
||||||
|
journalctl -u rke2-agent -f | grep -i "kubelet"
|
||||||
|
|
||||||
|
# kube-proxy logs are retrieved from containerd
|
||||||
|
crictl logs $(crictl ps --name kube-proxy -q)
|
||||||
|
```
|
||||||
|
|
||||||
|
### K3s
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Both components log to the systemd service
|
||||||
|
journalctl -u k3s-agent -f | grep -iE "kubelet|kube-proxy"
|
||||||
```
|
```
|
||||||
|
|||||||
+68
-14
@@ -8,31 +8,85 @@ title: Troubleshooting Worker Nodes and Generic Components
|
|||||||
|
|
||||||
This section applies to every node as it includes components that run on nodes with any role.
|
This section applies to every node as it includes components that run on nodes with any role.
|
||||||
|
|
||||||
## Check if the Containers are Running
|
## Prerequisites
|
||||||
|
|
||||||
There are two specific containers launched on nodes with the `worker` role:
|
Since RKE2 and K3s utilize `containerd` as the container runtime, `crictl` serves as the primary tool for container management, replacing the Docker CLI. To allow `crictl` to communicate with `containerd`, you must configure your environment by exporting the following variables:
|
||||||
|
|
||||||
* kubelet
|
### RKE2
|
||||||
* kube-proxy
|
|
||||||
|
|
||||||
The containers should have status `Up`. The duration shown after `Up` is the time the container has been running.
|
|
||||||
|
|
||||||
|
```bash
|
||||||
|
export PATH=$PATH:/var/lib/rancher/rke2/bin/
|
||||||
|
export CRI_CONFIG_FILE=/var/lib/rancher/rke2/agent/etc/crictl.yaml
|
||||||
```
|
```
|
||||||
docker ps -a -f=name='kubelet|kube-proxy'
|
|
||||||
|
### K3s
|
||||||
|
|
||||||
|
```bash
|
||||||
|
export PATH=$PATH:/usr/local/bin
|
||||||
|
export CRI_CONFIG_FILE=/var/lib/rancher/k3s/agent/etc/crictl.yaml
|
||||||
|
```
|
||||||
|
|
||||||
|
## Check if the Components are Running
|
||||||
|
|
||||||
|
There are two specific components launched on nodes with the `worker` role:
|
||||||
|
|
||||||
|
* `kubelet`
|
||||||
|
* `kube-proxy`
|
||||||
|
|
||||||
|
### RKE2
|
||||||
|
|
||||||
|
The `kubelet` runs natively as part of the `rke2-agent` (or `rke2-server`) systemd process, while `kube-proxy` runs as a Static Pod managed by `containerd`.
|
||||||
|
|
||||||
|
Check the status of the `kubelet` via the agent service:
|
||||||
|
```bash
|
||||||
|
systemctl status rke2-agent
|
||||||
|
```
|
||||||
|
:::note
|
||||||
|
|
||||||
|
If you are checking a controlplane node, use `systemctl status rke2-server` instead.
|
||||||
|
|
||||||
|
:::
|
||||||
|
|
||||||
|
Check the status of `kube-proxy` using `crictl`:
|
||||||
|
```bash
|
||||||
|
crictl ps --name kube-proxy
|
||||||
```
|
```
|
||||||
|
|
||||||
Example output:
|
Example output:
|
||||||
```
|
```
|
||||||
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
|
CONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID
|
||||||
158d0dcc33a5 rancher/hyperkube:v1.11.5-rancher1 "/opt/rke-tools/en..." 3 hours ago Up 3 hours kube-proxy
|
26c7159abbcc rancher/hardened-kubernetes:v1.28.8-rke2r1-build20240404 3 hours ago Running kube-proxy 0 1a2b3c4d5e6f7
|
||||||
a30717ecfb55 rancher/hyperkube:v1.11.5-rancher1 "/opt/rke-tools/en..." 3 hours ago Up 3 hours kubelet
|
|
||||||
```
|
```
|
||||||
|
|
||||||
## Container Logging
|
### K3s
|
||||||
|
|
||||||
The logging of the containers can contain information on what the problem could be.
|
Both `kubelet` and `kube-proxy` run as embedded processes inside the `k3s-agent` (or `k3s` server) systemd service. There are no separate containers for them.
|
||||||
|
|
||||||
|
Check their status by checking the K3s service:
|
||||||
|
```bash
|
||||||
|
systemctl status k3s-agent
|
||||||
```
|
```
|
||||||
docker logs kubelet
|
:::note
|
||||||
docker logs kube-proxy
|
If you are checking a controlplane node, use `systemctl status k3s` instead.
|
||||||
|
:::
|
||||||
|
|
||||||
|
## Component Logging
|
||||||
|
|
||||||
|
The logging of the components can contain information on what the problem could be.
|
||||||
|
|
||||||
|
### RKE2
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# kubelet logs are part of the systemd service
|
||||||
|
journalctl -u rke2-agent -f | grep -i "kubelet"
|
||||||
|
|
||||||
|
# kube-proxy logs are retrieved from containerd
|
||||||
|
crictl logs $(crictl ps --name kube-proxy -q)
|
||||||
|
```
|
||||||
|
|
||||||
|
### K3s
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Both components log to the systemd service
|
||||||
|
journalctl -u k3s-agent -f | grep -iE "kubelet|kube-proxy"
|
||||||
```
|
```
|
||||||
|
|||||||
+68
-14
@@ -8,31 +8,85 @@ title: Troubleshooting Worker Nodes and Generic Components
|
|||||||
|
|
||||||
This section applies to every node as it includes components that run on nodes with any role.
|
This section applies to every node as it includes components that run on nodes with any role.
|
||||||
|
|
||||||
## Check if the Containers are Running
|
## Prerequisites
|
||||||
|
|
||||||
There are two specific containers launched on nodes with the `worker` role:
|
Since RKE2 and K3s utilize `containerd` as the container runtime, `crictl` serves as the primary tool for container management, replacing the Docker CLI. To allow `crictl` to communicate with `containerd`, you must configure your environment by exporting the following variables:
|
||||||
|
|
||||||
* kubelet
|
### RKE2
|
||||||
* kube-proxy
|
|
||||||
|
|
||||||
The containers should have status `Up`. The duration shown after `Up` is the time the container has been running.
|
|
||||||
|
|
||||||
|
```bash
|
||||||
|
export PATH=$PATH:/var/lib/rancher/rke2/bin/
|
||||||
|
export CRI_CONFIG_FILE=/var/lib/rancher/rke2/agent/etc/crictl.yaml
|
||||||
```
|
```
|
||||||
docker ps -a -f=name='kubelet|kube-proxy'
|
|
||||||
|
### K3s
|
||||||
|
|
||||||
|
```bash
|
||||||
|
export PATH=$PATH:/usr/local/bin
|
||||||
|
export CRI_CONFIG_FILE=/var/lib/rancher/k3s/agent/etc/crictl.yaml
|
||||||
|
```
|
||||||
|
|
||||||
|
## Check if the Components are Running
|
||||||
|
|
||||||
|
There are two specific components launched on nodes with the `worker` role:
|
||||||
|
|
||||||
|
* `kubelet`
|
||||||
|
* `kube-proxy`
|
||||||
|
|
||||||
|
### RKE2
|
||||||
|
|
||||||
|
The `kubelet` runs natively as part of the `rke2-agent` (or `rke2-server`) systemd process, while `kube-proxy` runs as a Static Pod managed by `containerd`.
|
||||||
|
|
||||||
|
Check the status of the `kubelet` via the agent service:
|
||||||
|
```bash
|
||||||
|
systemctl status rke2-agent
|
||||||
|
```
|
||||||
|
:::note
|
||||||
|
|
||||||
|
If you are checking a controlplane node, use `systemctl status rke2-server` instead.
|
||||||
|
|
||||||
|
:::
|
||||||
|
|
||||||
|
Check the status of `kube-proxy` using `crictl`:
|
||||||
|
```bash
|
||||||
|
crictl ps --name kube-proxy
|
||||||
```
|
```
|
||||||
|
|
||||||
Example output:
|
Example output:
|
||||||
```
|
```
|
||||||
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
|
CONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID
|
||||||
158d0dcc33a5 rancher/hyperkube:v1.11.5-rancher1 "/opt/rke-tools/en..." 3 hours ago Up 3 hours kube-proxy
|
26c7159abbcc rancher/hardened-kubernetes:v1.28.8-rke2r1-build20240404 3 hours ago Running kube-proxy 0 1a2b3c4d5e6f7
|
||||||
a30717ecfb55 rancher/hyperkube:v1.11.5-rancher1 "/opt/rke-tools/en..." 3 hours ago Up 3 hours kubelet
|
|
||||||
```
|
```
|
||||||
|
|
||||||
## Container Logging
|
### K3s
|
||||||
|
|
||||||
The logging of the containers can contain information on what the problem could be.
|
Both `kubelet` and `kube-proxy` run as embedded processes inside the `k3s-agent` (or `k3s` server) systemd service. There are no separate containers for them.
|
||||||
|
|
||||||
|
Check their status by checking the K3s service:
|
||||||
|
```bash
|
||||||
|
systemctl status k3s-agent
|
||||||
```
|
```
|
||||||
docker logs kubelet
|
:::note
|
||||||
docker logs kube-proxy
|
If you are checking a controlplane node, use `systemctl status k3s` instead.
|
||||||
|
:::
|
||||||
|
|
||||||
|
## Component Logging
|
||||||
|
|
||||||
|
The logging of the components can contain information on what the problem could be.
|
||||||
|
|
||||||
|
### RKE2
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# kubelet logs are part of the systemd service
|
||||||
|
journalctl -u rke2-agent -f | grep -i "kubelet"
|
||||||
|
|
||||||
|
# kube-proxy logs are retrieved from containerd
|
||||||
|
crictl logs $(crictl ps --name kube-proxy -q)
|
||||||
|
```
|
||||||
|
|
||||||
|
### K3s
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Both components log to the systemd service
|
||||||
|
journalctl -u k3s-agent -f | grep -iE "kubelet|kube-proxy"
|
||||||
```
|
```
|
||||||
|
|||||||
+68
-14
@@ -8,31 +8,85 @@ title: Troubleshooting Worker Nodes and Generic Components
|
|||||||
|
|
||||||
This section applies to every node as it includes components that run on nodes with any role.
|
This section applies to every node as it includes components that run on nodes with any role.
|
||||||
|
|
||||||
## Check if the Containers are Running
|
## Prerequisites
|
||||||
|
|
||||||
There are two specific containers launched on nodes with the `worker` role:
|
Since RKE2 and K3s utilize `containerd` as the container runtime, `crictl` serves as the primary tool for container management, replacing the Docker CLI. To allow `crictl` to communicate with `containerd`, you must configure your environment by exporting the following variables:
|
||||||
|
|
||||||
* kubelet
|
### RKE2
|
||||||
* kube-proxy
|
|
||||||
|
|
||||||
The containers should have status `Up`. The duration shown after `Up` is the time the container has been running.
|
|
||||||
|
|
||||||
|
```bash
|
||||||
|
export PATH=$PATH:/var/lib/rancher/rke2/bin/
|
||||||
|
export CRI_CONFIG_FILE=/var/lib/rancher/rke2/agent/etc/crictl.yaml
|
||||||
```
|
```
|
||||||
docker ps -a -f=name='kubelet|kube-proxy'
|
|
||||||
|
### K3s
|
||||||
|
|
||||||
|
```bash
|
||||||
|
export PATH=$PATH:/usr/local/bin
|
||||||
|
export CRI_CONFIG_FILE=/var/lib/rancher/k3s/agent/etc/crictl.yaml
|
||||||
|
```
|
||||||
|
|
||||||
|
## Check if the Components are Running
|
||||||
|
|
||||||
|
There are two specific components launched on nodes with the `worker` role:
|
||||||
|
|
||||||
|
* `kubelet`
|
||||||
|
* `kube-proxy`
|
||||||
|
|
||||||
|
### RKE2
|
||||||
|
|
||||||
|
The `kubelet` runs natively as part of the `rke2-agent` (or `rke2-server`) systemd process, while `kube-proxy` runs as a Static Pod managed by `containerd`.
|
||||||
|
|
||||||
|
Check the status of the `kubelet` via the agent service:
|
||||||
|
```bash
|
||||||
|
systemctl status rke2-agent
|
||||||
|
```
|
||||||
|
:::note
|
||||||
|
|
||||||
|
If you are checking a controlplane node, use `systemctl status rke2-server` instead.
|
||||||
|
|
||||||
|
:::
|
||||||
|
|
||||||
|
Check the status of `kube-proxy` using `crictl`:
|
||||||
|
```bash
|
||||||
|
crictl ps --name kube-proxy
|
||||||
```
|
```
|
||||||
|
|
||||||
Example output:
|
Example output:
|
||||||
```
|
```
|
||||||
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
|
CONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID
|
||||||
158d0dcc33a5 rancher/hyperkube:v1.11.5-rancher1 "/opt/rke-tools/en..." 3 hours ago Up 3 hours kube-proxy
|
26c7159abbcc rancher/hardened-kubernetes:v1.28.8-rke2r1-build20240404 3 hours ago Running kube-proxy 0 1a2b3c4d5e6f7
|
||||||
a30717ecfb55 rancher/hyperkube:v1.11.5-rancher1 "/opt/rke-tools/en..." 3 hours ago Up 3 hours kubelet
|
|
||||||
```
|
```
|
||||||
|
|
||||||
## Container Logging
|
### K3s
|
||||||
|
|
||||||
The logging of the containers can contain information on what the problem could be.
|
Both `kubelet` and `kube-proxy` run as embedded processes inside the `k3s-agent` (or `k3s` server) systemd service. There are no separate containers for them.
|
||||||
|
|
||||||
|
Check their status by checking the K3s service:
|
||||||
|
```bash
|
||||||
|
systemctl status k3s-agent
|
||||||
```
|
```
|
||||||
docker logs kubelet
|
:::note
|
||||||
docker logs kube-proxy
|
If you are checking a controlplane node, use `systemctl status k3s` instead.
|
||||||
|
:::
|
||||||
|
|
||||||
|
## Component Logging
|
||||||
|
|
||||||
|
The logging of the components can contain information on what the problem could be.
|
||||||
|
|
||||||
|
### RKE2
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# kubelet logs are part of the systemd service
|
||||||
|
journalctl -u rke2-agent -f | grep -i "kubelet"
|
||||||
|
|
||||||
|
# kube-proxy logs are retrieved from containerd
|
||||||
|
crictl logs $(crictl ps --name kube-proxy -q)
|
||||||
|
```
|
||||||
|
|
||||||
|
### K3s
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Both components log to the systemd service
|
||||||
|
journalctl -u k3s-agent -f | grep -iE "kubelet|kube-proxy"
|
||||||
```
|
```
|
||||||
|
|||||||
Reference in New Issue
Block a user