mirror of
https://github.com/rancher/rancher-docs.git
synced 2026-09-26 04:58:07 +00:00
Adding v2.15 documentation with prerelease banner
Signed-off-by: Sunil Singh <sunil.singh@suse.com>
This commit is contained in:
Sunil Singh
+21
@@ -0,0 +1,21 @@
|
||||
---
|
||||
title: Kubernetes Components
|
||||
---
|
||||
|
||||
<head>
|
||||
<link rel="canonical" href="https://ranchermanager.docs.rancher.com/troubleshooting/kubernetes-components"/>
|
||||
</head>
|
||||
|
||||
The commands and steps listed in this section apply to the core Kubernetes components on [Rancher Launched Kubernetes](../../how-to-guides/new-user-guides/launch-kubernetes-with-rancher/launch-kubernetes-with-rancher.md) clusters.
|
||||
|
||||
This section includes troubleshooting tips in the following categories:
|
||||
|
||||
- [Troubleshooting etcd Nodes](troubleshooting-etcd-nodes.md)
|
||||
- [Troubleshooting Controlplane Nodes](troubleshooting-controlplane-nodes.md)
|
||||
- [Troubleshooting nginx-proxy Nodes](troubleshooting-nginx-proxy.md)
|
||||
- [Troubleshooting Worker Nodes and Generic Components](troubleshooting-worker-nodes-and-generic-components.md)
|
||||
|
||||
## Kubernetes Component Diagram
|
||||
|
||||
<br/>
|
||||
<sup>Lines show the traffic flow between components. Colors are used purely for visual aid</sup>
|
||||
+93
@@ -0,0 +1,93 @@
|
||||
---
|
||||
title: Troubleshooting Controlplane Nodes
|
||||
---
|
||||
|
||||
<head>
|
||||
<link rel="canonical" href="https://ranchermanager.docs.rancher.com/troubleshooting/kubernetes-components/troubleshooting-controlplane-nodes"/>
|
||||
</head>
|
||||
|
||||
This section applies to nodes with the `controlplane` role in RKE2 and K3s clusters.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
As RKE2 and K3s rely on `containerd` as the container runtime, `crictl` replaces Docker for container management. Before proceeding with the troubleshooting commands, configure your environment by exporting the following variables:
|
||||
|
||||
### RKE2
|
||||
|
||||
```bash
|
||||
export PATH=$PATH:/var/lib/rancher/rke2/bin/
|
||||
export CRI_CONFIG_FILE=/var/lib/rancher/rke2/agent/etc/crictl.yaml
|
||||
```
|
||||
|
||||
### K3s
|
||||
|
||||
```bash
|
||||
export PATH=$PATH:/usr/local/bin
|
||||
export CRI_CONFIG_FILE=/var/lib/rancher/k3s/agent/etc/crictl.yaml
|
||||
```
|
||||
|
||||
## Check if the Controlplane Components are Running
|
||||
|
||||
**RKE2**: There are three specific containers launched on nodes with the `controlplane` role:
|
||||
|
||||
* `kube-apiserver`
|
||||
* `kube-controller-manager`
|
||||
* `kube-scheduler`
|
||||
|
||||
The containers should have state **Running**. You can check this using `crictl`:
|
||||
|
||||
```bash
|
||||
crictl ps | grep -E 'kube-apiserver|kube-controller-manager|kube-scheduler'
|
||||
```
|
||||
|
||||
Example output:
|
||||
```
|
||||
CONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID POD NAMESPACE
|
||||
deb8a96948594 138b1e685e151 11 days ago Running kube-controller-manager 0 0996426295dc5 kube-controller-manager kube-system
|
||||
f5abb4c7846e4 138b1e685e151 11 days ago Running kube-scheduler 0 80cd9f30af0be kube-scheduler kube-system
|
||||
ecd8a6991c22a 138b1e685e151 11 days ago Running kube-apiserver 0 58e042fabe78c kube-apiserver kube-system
|
||||
```
|
||||
|
||||
**K3s**: These components run as embedded processes within the K3s service. They do not run as separate containers, so their status is tied to the `k3s` systemd service:
|
||||
|
||||
```bash
|
||||
systemctl status k3s
|
||||
```
|
||||
|
||||
## Controlplane Logging
|
||||
|
||||
:::note
|
||||
|
||||
If you added multiple nodes with the `controlplane` role, both `kube-controller-manager` and `kube-scheduler` use a leader election process to determine the leader. Only the current leader will log the performed actions. See [Kubernetes leader election](../other-troubleshooting-tips/kubernetes-resources.md#kubernetes-leader-election) how to retrieve the current leader.
|
||||
|
||||
:::
|
||||
|
||||
The logs can contain information on what the problem could be.
|
||||
|
||||
**RKE2**:
|
||||
```bash
|
||||
crictl logs $(crictl ps --name kube-apiserver -q)
|
||||
crictl logs $(crictl ps --name kube-controller-manager -q)
|
||||
crictl logs $(crictl ps --name kube-scheduler -q)
|
||||
```
|
||||
|
||||
**K3s**:
|
||||
```bash
|
||||
journalctl -u k3s | grep -i "kube-apiserver"
|
||||
journalctl -u k3s | grep -i "kube-controller-manager"
|
||||
journalctl -u k3s | grep -i "kube-scheduler"
|
||||
```
|
||||
|
||||
## RKE2/K3s Server Logging
|
||||
|
||||
If Rancher provisions an RKE2 or K3s cluster that can't communicate with Rancher, you can run this command on a server node in the downstream cluster to get the server logs:
|
||||
|
||||
**RKE2**:
|
||||
```bash
|
||||
journalctl -u rke2-server -f
|
||||
```
|
||||
|
||||
**K3s**:
|
||||
```bash
|
||||
journalctl -u k3s -f
|
||||
```
|
||||
+390
@@ -0,0 +1,390 @@
|
||||
---
|
||||
title: Troubleshooting etcd Nodes
|
||||
---
|
||||
|
||||
<head>
|
||||
<link rel="canonical" href="https://ranchermanager.docs.rancher.com/troubleshooting/kubernetes-components/troubleshooting-etcd-nodes"/>
|
||||
</head>
|
||||
|
||||
This section contains commands and tips for troubleshooting nodes with the `etcd` role in RKE2 and K3s clusters.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
As RKE2 and K3s rely on `containerd` as the container runtime, `crictl` replaces Docker for container management. Before proceeding with the troubleshooting commands, configure your environment by exporting the following variables:
|
||||
|
||||
### RKE2
|
||||
|
||||
```bash
|
||||
export PATH=$PATH:/var/lib/rancher/rke2/bin/
|
||||
export CRI_CONFIG_FILE=/var/lib/rancher/rke2/agent/etc/crictl.yaml
|
||||
etcdcontainer=$(crictl ps --name etcd --quiet)
|
||||
```
|
||||
|
||||
### K3s
|
||||
|
||||
> ### ⚠️ **Warning**
|
||||
> K3s does not include `etcdctl` in the system PATH. If you need to perform etcd troubleshooting on a K3s cluster, you may need to install it or locate it within the K3s data directory.
|
||||
|
||||
```bash
|
||||
export PATH=$PATH:/usr/local/bin
|
||||
export CRI_CONFIG_FILE=/var/lib/rancher/k3s/agent/etc/crictl.yaml
|
||||
```
|
||||
|
||||
|
||||
## Checking if the etcd Container is Running
|
||||
|
||||
**RKE2**: The container for etcd should be in the **Running** state.
|
||||
|
||||
```bash
|
||||
crictl ps --name etcd
|
||||
```
|
||||
|
||||
Example output:
|
||||
```
|
||||
CONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID POD NAMESPACE
|
||||
f1e289d202ed0 11ad16872a9cf 58 minutes ago Running etcd 0 7b56aab8204ea etcd-cluster1 kube-system
|
||||
```
|
||||
|
||||
**K3s**: Etcd runs as an embedded process in the K3s service. Check the service status:
|
||||
|
||||
```bash
|
||||
systemctl status k3s
|
||||
```
|
||||
|
||||
## etcd Logging
|
||||
|
||||
The logs can contain information on what the problem could be.
|
||||
|
||||
**RKE2**:
|
||||
```bash
|
||||
crictl logs $etcdcontainer
|
||||
```
|
||||
|
||||
**K3s**:
|
||||
```bash
|
||||
journalctl -u k3s | grep -i etcd
|
||||
```
|
||||
| Log | Explanation |
|
||||
|-----|------------------|
|
||||
| `health check for peer xxx could not connect: dial tcp IP:2380: getsockopt: connection refused` | A connection to the address shown on port 2380 cannot be established. Check if the etcd container is running on the host with the address shown. |
|
||||
| `xxx is starting a new election at term x` | The etcd cluster has lost its quorum and is trying to establish a new leader. This can happen when the majority of the nodes running etcd go down/unreachable. |
|
||||
| `connection error: desc = "transport: Error while dialing dial tcp 0.0.0.0:2379: i/o timeout"; Reconnecting to {0.0.0.0:2379 0 <nil>}` | The host firewall is preventing network communication. |
|
||||
| `rafthttp: request cluster ID mismatch` | The node with the etcd instance logging `rafthttp: request cluster ID mismatch` is trying to join a cluster that has already been formed with another peer. The node should be removed from the cluster, and re-added. |
|
||||
| `rafthttp: failed to find member` | The cluster state (`/var/lib/etcd`) contains wrong information to join the cluster. The node should be removed from the cluster, the state directory should be cleaned and the node should be re-added.
|
||||
|
||||
## etcd Cluster and Connectivity Checks
|
||||
|
||||
The address where etcd is listening depends on the address configuration of the host etcd is running on. If an internal address is configured for the host etcd is running on, the endpoint for `etcdctl` needs to be specified explicitly. If any of the commands respond with `Error: context deadline exceeded`, the etcd instance is unhealthy (either quorum is lost or the instance is not correctly joined in the cluster)
|
||||
|
||||
### Check etcd Members on all Nodes
|
||||
|
||||
Output should contain all the nodes with the `etcd` role and the output should be identical on all nodes.
|
||||
|
||||
**RKE2**:
|
||||
Run the command inside the etcd container.
|
||||
|
||||
```bash
|
||||
crictl exec $etcdcontainer etcdctl member list \
|
||||
--cert /var/lib/rancher/rke2/server/tls/etcd/server-client.crt \
|
||||
--key /var/lib/rancher/rke2/server/tls/etcd/server-client.key \
|
||||
--cacert /var/lib/rancher/rke2/server/tls/etcd/server-ca.crt
|
||||
```
|
||||
|
||||
**K3s**:
|
||||
```bash
|
||||
etcdctl member list \
|
||||
--cert /var/lib/rancher/k3s/server/tls/etcd/server-client.crt \
|
||||
--key /var/lib/rancher/k3s/server/tls/etcd/server-client.key \
|
||||
--cacert /var/lib/rancher/k3s/server/tls/etcd/server-ca.crt
|
||||
```
|
||||
|
||||
Example output:
|
||||
```
|
||||
1c424074df86e854, started, cluster-node1-f289ac71, https://IP:2380, https://IP:2379, false
|
||||
45c68c44c5a792ff, started, cluster-node2-67e3cf6f, https://IP:2380, https://IP:2379, false
|
||||
7c584f77c5180258, started, cluster-node3-e976bc00, https://IP:2380, https://IP:2379, false
|
||||
```
|
||||
|
||||
### Check Endpoint Status
|
||||
|
||||
The values for `RAFT TERM` should be equal and `RAFT INDEX` should be not be too far apart from each other.
|
||||
|
||||
**RKE2**:
|
||||
```bash
|
||||
crictl exec $etcdcontainer etcdctl endpoint status --write-out table --endpoints=$(crictl exec $etcdcontainer etcdctl member list --cert /var/lib/rancher/rke2/server/tls/etcd/server-client.crt --key /var/lib/rancher/rke2/server/tls/etcd/server-client.key --cacert /var/lib/rancher/rke2/server/tls/etcd/server-ca.crt | cut -d, -f5 | sed -e 's/ //g' | paste -sd ',') --cert /var/lib/rancher/rke2/server/tls/etcd/server-client.crt --key /var/lib/rancher/rke2/server/tls/etcd/server-client.key --cacert /var/lib/rancher/rke2/server/tls/etcd/server-ca.crt
|
||||
```
|
||||
|
||||
**K3s**:
|
||||
```bash
|
||||
etcdctl endpoint status --write-out table --endpoints=$(etcdctl member list --cert /var/lib/rancher/k3s/server/tls/etcd/server-client.crt --key /var/lib/rancher/k3s/server/tls/etcd/server-client.key --cacert /var/lib/rancher/k3s/server/tls/etcd/server-ca.crt | cut -d, -f5 | sed -e 's/ //g' | paste -sd ',') --cert /var/lib/rancher/k3s/server/tls/etcd/server-client.crt --key /var/lib/rancher/k3s/server/tls/etcd/server-client.key --cacert /var/lib/rancher/k3s/server/tls/etcd/server-ca.crt
|
||||
```
|
||||
|
||||
Example output:
|
||||
```
|
||||
+-----------------+------------------+---------+---------+-----------+-----------+------------+
|
||||
| ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | RAFT TERM | RAFT INDEX |
|
||||
+-----------------+------------------+---------+---------+-----------+-----------+------------+
|
||||
| https://IP:2379 | 333ef673fc4add56 | 3.6.7 | 24 MB | false | 72 | 66887 |
|
||||
| https://IP:2379 | 5feed52d940ce4cf | 3.6.7 | 24 MB | true | 72 | 66887 |
|
||||
| https://IP:2379 | db6b3bdb559a848d | 3.6.7 | 25 MB | false | 72 | 66887 |
|
||||
+-----------------+------------------+---------+---------+-----------+-----------+------------+
|
||||
```
|
||||
|
||||
### Check Endpoint Health
|
||||
|
||||
**RKE2**:
|
||||
```bash
|
||||
crictl exec $etcdcontainer etcdctl endpoint health --endpoints=$(crictl exec $etcdcontainer etcdctl member list --cert /var/lib/rancher/rke2/server/tls/etcd/server-client.crt --key /var/lib/rancher/rke2/server/tls/etcd/server-client.key --cacert /var/lib/rancher/rke2/server/tls/etcd/server-ca.crt | cut -d, -f5 | sed -e 's/ //g' | paste -sd ',') --cert /var/lib/rancher/rke2/server/tls/etcd/server-client.crt --key /var/lib/rancher/rke2/server/tls/etcd/server-client.key --cacert /var/lib/rancher/rke2/server/tls/etcd/server-ca.crt
|
||||
```
|
||||
|
||||
**K3s**:
|
||||
```bash
|
||||
etcdctl endpoint health --endpoints=$(etcdctl member list --cert /var/lib/rancher/k3s/server/tls/etcd/server-client.crt --key /var/lib/rancher/k3s/server/tls/etcd/server-client.key --cacert /var/lib/rancher/k3s/server/tls/etcd/server-ca.crt | cut -d, -f5 | sed -e 's/ //g' | paste -sd ',') --cert /var/lib/rancher/k3s/server/tls/etcd/server-client.crt --key /var/lib/rancher/k3s/server/tls/etcd/server-client.key --cacert /var/lib/rancher/k3s/server/tls/etcd/server-ca.crt
|
||||
```
|
||||
|
||||
Example output:
|
||||
```
|
||||
https://IP:2379 is healthy: successfully committed proposal: took = 2.113189ms
|
||||
https://IP:2379 is healthy: successfully committed proposal: took = 2.649963ms
|
||||
https://IP:2379 is healthy: successfully committed proposal: took = 2.451201ms
|
||||
```
|
||||
### Check Connectivity on etcd Ports
|
||||
|
||||
> In modern versions of Kubernetes, the etcd database (versions 3.5 and newer) introduced significant architectural changes regarding network traffic handling. Previously, etcd permitted standard HTTP REST requests on its primary client port (`2379`). However, to enhance performance and security, etcd 3.5+ strictly enforces the gRPC protocol on this port.<br />
|
||||
If you attempt to use standard HTTP tools like `curl` to test connectivity on port `2379`, the etcd server will automatically terminate the connection or return an error. This behavior often leads administrators to misinterpret the result as a closed port or a node failure.
|
||||
|
||||
Since standard HTTP clients can no longer probe the primary etcd ports, the transport layer must be utilized for network troubleshooting. Using `openssl s_client` instead of `curl` bypasses the gRPC application requirement, allowing the raw TCP and TLS handshake to be tested directly.
|
||||
|
||||
These script isolate the network and security infrastructure from the database application. A successful `Verify return code: 0 (ok)` explicitly confirms four critical infrastructure components:
|
||||
|
||||
* **Network Path:** Routing is functional, and firewalls permit traffic on TCP port `2379` or `2380`.
|
||||
* **Process Availability:** The etcd service is running and actively listening on the designated port.
|
||||
* **Certificate Validity:** The TLS certificates are active, correctly formatted, and have not expired.
|
||||
* **Mutual Authentication (mTLS):** The node successfully authenticates against the cluster's specific Certificate Authority (CA).
|
||||
|
||||
**How these tests differ from the `etcdctl endpoint health` test**:
|
||||
|
||||
If `etcdctl endpoint health` test is failing, run these Connectivity Ports test scripts. If the scripts succeed, your network and certificates are intact, and the issue is likely confined to the etcd database itself. If these scripts fail, the issue is related to a firewall/network restriction, or certificate expiration.
|
||||
|
||||
#### Port TCP/2379
|
||||
|
||||
**RKE2**:
|
||||
```bash
|
||||
for endpoint in $(crictl exec $etcdcontainer etcdctl member list --cert /var/lib/rancher/rke2/server/tls/etcd/server-client.crt --key /var/lib/rancher/rke2/server/tls/etcd/server-client.key --cacert /var/lib/rancher/rke2/server/tls/etcd/server-ca.crt | cut -d, -f5); do
|
||||
echo "Validating connection to ${endpoint} (Client)";
|
||||
echo | openssl s_client -connect ${endpoint#https://} \
|
||||
-CAfile /var/lib/rancher/rke2/server/tls/etcd/server-ca.crt \
|
||||
-cert /var/lib/rancher/rke2/server/tls/etcd/server-client.crt \
|
||||
-key /var/lib/rancher/rke2/server/tls/etcd/server-client.key 2>/dev/null | grep -E 'Verify return code' || echo "Connection Failed/Timeout"
|
||||
done
|
||||
```
|
||||
|
||||
**K3s**:
|
||||
```bash
|
||||
for endpoint in $(etcdctl member list --cert /var/lib/rancher/k3s/server/tls/etcd/server-client.crt --key /var/lib/rancher/k3s/server/tls/etcd/server-client.key --cacert /var/lib/rancher/k3s/server/tls/etcd/server-ca.crt | cut -d, -f5); do
|
||||
echo "Validating connection to ${endpoint} (Client)";
|
||||
echo | openssl s_client -connect ${endpoint#https://} \
|
||||
-CAfile /var/lib/rancher/k3s/server/tls/etcd/server-ca.crt \
|
||||
-cert /var/lib/rancher/k3s/server/tls/etcd/server-client.crt \
|
||||
-key /var/lib/rancher/k3s/server/tls/etcd/server-client.key 2>/dev/null | grep -E 'Verify return code' || echo "Connection Failed/Timeout"
|
||||
done
|
||||
```
|
||||
|
||||
Example output:
|
||||
```
|
||||
Validating connection to https://IP:2379/health (Client)
|
||||
Verify return code: 0 (ok)
|
||||
Validating connection to https://IP:2379/health (Client)
|
||||
Verify return code: 0 (ok)
|
||||
Validating connection to https://IP:2379/health (Client)
|
||||
Verify return code: 0 (ok)
|
||||
```
|
||||
|
||||
#### Port TCP/2380
|
||||
|
||||
**RKE2**:
|
||||
```bash
|
||||
for endpoint in $(crictl exec $etcdcontainer etcdctl member list --cert /var/lib/rancher/rke2/server/tls/etcd/server-client.crt --key /var/lib/rancher/rke2/server/tls/etcd/server-client.key --cacert /var/lib/rancher/rke2/server/tls/etcd/server-ca.crt | cut -d, -f4); do
|
||||
echo "Validating connection to ${endpoint} (Peer)";
|
||||
echo | openssl s_client -connect ${endpoint#https://} \
|
||||
-CAfile /var/lib/rancher/rke2/server/tls/etcd/peer-ca.crt \
|
||||
-cert /var/lib/rancher/rke2/server/tls/etcd/peer-server-client.crt \
|
||||
-key /var/lib/rancher/rke2/server/tls/etcd/peer-server-client.key 2>/dev/null | grep -E 'Verify return code' || echo "Connection Failed/Timeout"
|
||||
done
|
||||
```
|
||||
|
||||
**K3s**:
|
||||
```bash
|
||||
for endpoint in $(etcdctl member list --cert /var/lib/rancher/k3s/server/tls/etcd/server-client.crt --key /var/lib/rancher/k3s/server/tls/etcd/server-client.key --cacert /var/lib/rancher/k3s/server/tls/etcd/server-ca.crt | cut -d, -f4); do
|
||||
echo "Validating connection to ${endpoint} (Peer)";
|
||||
echo | openssl s_client -connect ${endpoint#https://} \
|
||||
-CAfile /var/lib/rancher/k3s/server/tls/etcd/peer-ca.crt \
|
||||
-cert /var/lib/rancher/k3s/server/tls/etcd/peer-server-client.crt \
|
||||
-key /var/lib/rancher/k3s/server/tls/etcd/peer-server-client.key 2>/dev/null | grep -E 'Verify return code' || echo "Connection Failed/Timeout"
|
||||
done
|
||||
```
|
||||
|
||||
Example output:
|
||||
```
|
||||
Validating connection to https://IP:2380/version (Peer)
|
||||
Verify return code: 0 (ok)
|
||||
Validating connection to https://IP:2380/version (Peer)
|
||||
Verify return code: 0 (ok)
|
||||
Validating connection to https://IP:2380/version (Peer)
|
||||
Verify return code: 0 (ok)
|
||||
```
|
||||
|
||||
## etcd Alarms
|
||||
|
||||
etcd will trigger alarms, for instance when it runs out of space.
|
||||
|
||||
**RKE2**:
|
||||
```bash
|
||||
crictl exec $etcdcontainer etcdctl alarm list --cert /var/lib/rancher/rke2/server/tls/etcd/server-client.crt --key /var/lib/rancher/rke2/server/tls/etcd/server-client.key --cacert /var/lib/rancher/rke2/server/tls/etcd/server-ca.crt
|
||||
```
|
||||
|
||||
**K3s**:
|
||||
```bash
|
||||
etcdctl alarm list --cert /var/lib/rancher/k3s/server/tls/etcd/server-client.crt --key /var/lib/rancher/k3s/server/tls/etcd/server-client.key --cacert /var/lib/rancher/k3s/server/tls/etcd/server-ca.crt
|
||||
```
|
||||
|
||||
Example output when NOSPACE alarm is triggered:
|
||||
```
|
||||
memberID:x alarm:NOSPACE
|
||||
memberID:x alarm:NOSPACE
|
||||
memberID:x alarm:NOSPACE
|
||||
```
|
||||
|
||||
## etcd Space Errors
|
||||
|
||||
Related error messages are `etcdserver: mvcc: database space exceeded` or `applying raft message exceeded backend quota`. Alarm `NOSPACE` will be triggered.
|
||||
|
||||
Resolutions:
|
||||
|
||||
- [Compact the Keyspace](#compact-the-keyspace)
|
||||
- [Defrag All etcd Members](#defrag-all-etcd-members)
|
||||
- [Check Endpoint Status](#check-endpoint-status)
|
||||
- [Disarm Alarm](#disarm-alarm)
|
||||
|
||||
### Compact the Keyspace
|
||||
|
||||
**RKE2**:
|
||||
```bash
|
||||
rev=$(crictl exec $etcdcontainer etcdctl endpoint status --write-out json --cert /var/lib/rancher/rke2/server/tls/etcd/server-client.crt --key /var/lib/rancher/rke2/server/tls/etcd/server-client.key --cacert /var/lib/rancher/rke2/server/tls/etcd/server-ca.crt | egrep -o '"revision":[0-9]*' | egrep -o '[0-9]*' | head -1)
|
||||
crictl exec $etcdcontainer etcdctl compact "$rev" --cert /var/lib/rancher/rke2/server/tls/etcd/server-client.crt --key /var/lib/rancher/rke2/server/tls/etcd/server-client.key --cacert /var/lib/rancher/rke2/server/tls/etcd/server-ca.crt
|
||||
```
|
||||
|
||||
**K3s**:
|
||||
```bash
|
||||
rev=$(etcdctl endpoint status --write-out json --cert /var/lib/rancher/k3s/server/tls/etcd/server-client.crt --key /var/lib/rancher/k3s/server/tls/etcd/server-client.key --cacert /var/lib/rancher/k3s/server/tls/etcd/server-ca.crt | egrep -o '"revision":[0-9]*' | egrep -o '[0-9]*' | head -1)
|
||||
etcdctl compact "$rev" --cert /var/lib/rancher/k3s/server/tls/etcd/server-client.crt --key /var/lib/rancher/k3s/server/tls/etcd/server-client.key --cacert /var/lib/rancher/k3s/server/tls/etcd/server-ca.crt
|
||||
```
|
||||
|
||||
Example output:
|
||||
```
|
||||
compacted revision xxx
|
||||
```
|
||||
|
||||
### Defrag All etcd Members
|
||||
|
||||
**RKE2**:
|
||||
```bash
|
||||
crictl exec $etcdcontainer etcdctl defrag --endpoints=$(crictl exec $etcdcontainer etcdctl member list --cert /var/lib/rancher/rke2/server/tls/etcd/server-client.crt --key /var/lib/rancher/rke2/server/tls/etcd/server-client.key --cacert /var/lib/rancher/rke2/server/tls/etcd/server-ca.crt | cut -d, -f5 | sed -e 's/ //g' | paste -sd ',') --cert /var/lib/rancher/rke2/server/tls/etcd/server-client.crt --key /var/lib/rancher/rke2/server/tls/etcd/server-client.key --cacert /var/lib/rancher/rke2/server/tls/etcd/server-ca.crt
|
||||
```
|
||||
|
||||
**K3s**:
|
||||
```bash
|
||||
etcdctl defrag --endpoints=$(etcdctl member list --cert /var/lib/rancher/k3s/server/tls/etcd/server-client.crt --key /var/lib/rancher/k3s/server/tls/etcd/server-client.key --cacert /var/lib/rancher/k3s/server/tls/etcd/server-ca.crt | cut -d, -f5 | sed -e 's/ //g' | paste -sd ',') --cert /var/lib/rancher/k3s/server/tls/etcd/server-client.crt --key /var/lib/rancher/k3s/server/tls/etcd/server-client.key --cacert /var/lib/rancher/k3s/server/tls/etcd/server-ca.crt
|
||||
```
|
||||
|
||||
Example output:
|
||||
```
|
||||
Finished defragmenting etcd member[https://IP:2379]. took xx.xxxxxxms
|
||||
Finished defragmenting etcd member[https://IP:2379]. took xx.xxxxxxms
|
||||
Finished defragmenting etcd member[https://IP:2379]. took xx.xxxxxxms
|
||||
```
|
||||
|
||||
### Disarm Alarm
|
||||
|
||||
After verifying that the DB size went down after compaction and defragmenting, the alarm needs to be disarmed for etcd to allow writes again.
|
||||
|
||||
**RKE2**:
|
||||
```bash
|
||||
crictl exec $etcdcontainer etcdctl alarm list --cert /var/lib/rancher/rke2/server/tls/etcd/server-client.crt --key /var/lib/rancher/rke2/server/tls/etcd/server-client.key --cacert /var/lib/rancher/rke2/server/tls/etcd/server-ca.crt
|
||||
crictl exec $etcdcontainer etcdctl alarm disarm --cert /var/lib/rancher/rke2/server/tls/etcd/server-client.crt --key /var/lib/rancher/rke2/server/tls/etcd/server-client.key --cacert /var/lib/rancher/rke2/server/tls/etcd/server-ca.crt
|
||||
crictl exec $etcdcontainer etcdctl alarm list --cert /var/lib/rancher/rke2/server/tls/etcd/server-client.crt --key /var/lib/rancher/rke2/server/tls/etcd/server-client.key --cacert /var/lib/rancher/rke2/server/tls/etcd/server-ca.crt
|
||||
```
|
||||
|
||||
**K3s**:
|
||||
```bash
|
||||
etcdctl alarm list --cert /var/lib/rancher/k3s/server/tls/etcd/server-client.crt --key /var/lib/rancher/k3s/server/tls/etcd/server-client.key --cacert /var/lib/rancher/k3s/server/tls/etcd/server-ca.crt
|
||||
etcdctl alarm disarm --cert /var/lib/rancher/k3s/server/tls/etcd/server-client.crt --key /var/lib/rancher/k3s/server/tls/etcd/server-client.key --cacert /var/lib/rancher/k3s/server/tls/etcd/server-ca.crt
|
||||
etcdctl alarm list --cert /var/lib/rancher/k3s/server/tls/etcd/server-client.crt --key /var/lib/rancher/k3s/server/tls/etcd/server-client.key --cacert /var/lib/rancher/k3s/server/tls/etcd/server-ca.crt
|
||||
```
|
||||
|
||||
## Configure Log Level
|
||||
|
||||
:::note
|
||||
|
||||
You can no longer dynamically change the log level in etcd v3.5 or later.
|
||||
|
||||
:::
|
||||
|
||||
### etcd v3.5 And Later
|
||||
|
||||
To configure the log level for etcd, edit the cluster configuration YAML:
|
||||
|
||||
```
|
||||
services:
|
||||
etcd:
|
||||
extra_args:
|
||||
log-level: "debug"
|
||||
```
|
||||
|
||||
After modifying the configuration, restart the service (`systemctl restart rke2-server` or `systemctl restart k3s`) if you are configuring a stand-alone cluster.
|
||||
|
||||
## etcd Content
|
||||
|
||||
If you want to investigate the contents of your etcd, you can either watch streaming events or you can query etcd directly, see below for examples.
|
||||
|
||||
### Watch Streaming Events
|
||||
|
||||
**RKE2**:
|
||||
```bash
|
||||
crictl exec $etcdcontainer etcdctl watch --prefix /registry --cert /var/lib/rancher/rke2/server/tls/etcd/server-client.crt --key /var/lib/rancher/rke2/server/tls/etcd/server-client.key --cacert /var/lib/rancher/rke2/server/tls/etcd/server-ca.crt
|
||||
```
|
||||
|
||||
**K3s**:
|
||||
```bash
|
||||
etcdctl watch --prefix /registry --cert /var/lib/rancher/k3s/server/tls/etcd/server-client.crt --key /var/lib/rancher/k3s/server/tls/etcd/server-client.key --cacert /var/lib/rancher/k3s/server/tls/etcd/server-ca.crt
|
||||
```
|
||||
|
||||
If you only want to see the affected keys (and not the binary data), you can append `| grep -a ^/registry` to the command to filter for keys only.
|
||||
|
||||
### Query etcd Directly
|
||||
|
||||
**RKE2**:
|
||||
```bash
|
||||
crictl exec $etcdcontainer etcdctl get /registry --prefix=true --keys-only --cert /var/lib/rancher/rke2/server/tls/etcd/server-client.crt --key /var/lib/rancher/rke2/server/tls/etcd/server-client.key --cacert /var/lib/rancher/rke2/server/tls/etcd/server-ca.crt
|
||||
```
|
||||
|
||||
**K3s**:
|
||||
```bash
|
||||
etcdctl get /registry --prefix=true --keys-only --cert /var/lib/rancher/k3s/server/tls/etcd/server-client.crt --key /var/lib/rancher/k3s/server/tls/etcd/server-client.key --cacert /var/lib/rancher/k3s/server/tls/etcd/server-ca.crt
|
||||
```
|
||||
|
||||
You can process the data to get a summary of count per key, using the command below:
|
||||
|
||||
**RKE2**:
|
||||
```bash
|
||||
crictl exec $etcdcontainer etcdctl get /registry --prefix=true --keys-only --cert /var/lib/rancher/rke2/server/tls/etcd/server-client.crt --key /var/lib/rancher/rke2/server/tls/etcd/server-client.key --cacert /var/lib/rancher/rke2/server/tls/etcd/server-ca.crt | grep -v ^$ | awk -F'/' '{ if ($3 ~ /cattle.io/) {h[$3"/"$4]++} else { h[$3]++ }} END { for(k in h) print h[k], k }' | sort -nr
|
||||
```
|
||||
|
||||
**K3s**:
|
||||
```bash
|
||||
etcdctl get /registry --prefix=true --keys-only --cert /var/lib/rancher/k3s/server/tls/etcd/server-client.crt --key /var/lib/rancher/k3s/server/tls/etcd/server-client.key --cacert /var/lib/rancher/k3s/server/tls/etcd/server-ca.crt | grep -v ^$ | awk -F'/' '{ if ($3 ~ /cattle.io/) {h[$3"/"$4]++} else { h[$3]++ }} END { for(k in h) print h[k], k }' | sort -nr
|
||||
```
|
||||
|
||||
## Replacing Unhealthy etcd Nodes
|
||||
|
||||
When a node in your etcd cluster becomes unhealthy, the recommended approach is to fix or remove the failed or unhealthy node before adding a new etcd node to the cluster.
|
||||
+80
@@ -0,0 +1,80 @@
|
||||
---
|
||||
title: Troubleshooting nginx-proxy
|
||||
---
|
||||
|
||||
<head>
|
||||
<link rel="canonical" href="https://ranchermanager.docs.rancher.com/troubleshooting/kubernetes-components/troubleshooting-nginx-proxy"/>
|
||||
</head>
|
||||
|
||||
:::caution
|
||||
|
||||
The `nginx-proxy` container is an RKE1-specific component. If you are using RKE2 or K3s, this container is not deployed, as load balancing to the API servers is handled internally by a client-side load balancer within the agent process itself.
|
||||
|
||||
Additionally, please note that RKE1 has reached its [End of Life (EOL)](https://support.scc.suse.com/s/kb/RKE-EOL-what-when-why?language=en_US). Therefore, the information on this page is considered deprecated.
|
||||
|
||||
:::
|
||||
|
||||
The `nginx-proxy` container is deployed on every node that does not have the `controlplane` role. It provides access to all the nodes with the `controlplane` role by dynamically generating the NGINX configuration based on available nodes with the `controlplane` role.
|
||||
|
||||
## Check if the Container is Running
|
||||
|
||||
The container is called `nginx-proxy` and should have status `Up`. The duration shown after `Up` is the time the container has been running.
|
||||
|
||||
```
|
||||
docker ps -a -f=name=nginx-proxy
|
||||
```
|
||||
|
||||
Example output:
|
||||
|
||||
```
|
||||
docker ps -a -f=name=nginx-proxy
|
||||
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
|
||||
c3e933687c0e rancher/rke-tools:v0.1.15 "nginx-proxy CP_HO..." 3 hours ago Up 3 hours nginx-proxy
|
||||
```
|
||||
|
||||
## Check Generated NGINX Configuration
|
||||
|
||||
The generated configuration should include the IP addresses of the nodes with the `controlplane` role. The configuration can be checked using the following command:
|
||||
|
||||
```
|
||||
docker exec nginx-proxy cat /etc/nginx/nginx.conf
|
||||
```
|
||||
|
||||
Example output:
|
||||
```
|
||||
error_log stderr notice;
|
||||
|
||||
worker_processes auto;
|
||||
events {
|
||||
multi_accept on;
|
||||
use epoll;
|
||||
worker_connections 1024;
|
||||
}
|
||||
|
||||
stream {
|
||||
upstream kube_apiserver {
|
||||
|
||||
server ip_of_controlplane_node1:6443;
|
||||
|
||||
server ip_of_controlplane_node2:6443;
|
||||
|
||||
}
|
||||
|
||||
server {
|
||||
listen 6443;
|
||||
proxy_pass kube_apiserver;
|
||||
proxy_timeout 30;
|
||||
proxy_connect_timeout 2s;
|
||||
|
||||
}
|
||||
|
||||
}
|
||||
```
|
||||
|
||||
## nginx-proxy Container Logging
|
||||
|
||||
The logging of the containers can contain information on what the problem could be.
|
||||
|
||||
```
|
||||
docker logs nginx-proxy
|
||||
```
|
||||
+92
@@ -0,0 +1,92 @@
|
||||
---
|
||||
title: Troubleshooting Worker Nodes and Generic Components
|
||||
---
|
||||
|
||||
<head>
|
||||
<link rel="canonical" href="https://ranchermanager.docs.rancher.com/troubleshooting/kubernetes-components/troubleshooting-worker-nodes-and-generic-components"/>
|
||||
</head>
|
||||
|
||||
This section applies to every node as it includes components that run on nodes with any role.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
Since RKE2 and K3s utilize `containerd` as the container runtime, `crictl` serves as the primary tool for container management, replacing the Docker CLI. To allow `crictl` to communicate with `containerd`, you must configure your environment by exporting the following variables:
|
||||
|
||||
### RKE2
|
||||
|
||||
```bash
|
||||
export PATH=$PATH:/var/lib/rancher/rke2/bin/
|
||||
export CRI_CONFIG_FILE=/var/lib/rancher/rke2/agent/etc/crictl.yaml
|
||||
```
|
||||
|
||||
### K3s
|
||||
|
||||
```bash
|
||||
export PATH=$PATH:/usr/local/bin
|
||||
export CRI_CONFIG_FILE=/var/lib/rancher/k3s/agent/etc/crictl.yaml
|
||||
```
|
||||
|
||||
## Check if the Components are Running
|
||||
|
||||
There are two specific components launched on nodes with the `worker` role:
|
||||
|
||||
* `kubelet`
|
||||
* `kube-proxy`
|
||||
|
||||
### RKE2
|
||||
|
||||
The `kubelet` runs natively as part of the `rke2-agent` (or `rke2-server`) systemd process, while `kube-proxy` runs as a Static Pod managed by `containerd`.
|
||||
|
||||
Check the status of the `kubelet` via the agent service:
|
||||
```bash
|
||||
systemctl status rke2-agent
|
||||
```
|
||||
:::note
|
||||
|
||||
If you are checking a controlplane node, use `systemctl status rke2-server` instead.
|
||||
|
||||
:::
|
||||
|
||||
Check the status of `kube-proxy` using `crictl`:
|
||||
```bash
|
||||
crictl ps --name kube-proxy
|
||||
```
|
||||
|
||||
Example output:
|
||||
```
|
||||
CONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID
|
||||
26c7159abbcc rancher/hardened-kubernetes:v1.28.8-rke2r1-build20240404 3 hours ago Running kube-proxy 0 1a2b3c4d5e6f7
|
||||
```
|
||||
|
||||
### K3s
|
||||
|
||||
Both `kubelet` and `kube-proxy` run as embedded processes inside the `k3s-agent` (or `k3s` server) systemd service. There are no separate containers for them.
|
||||
|
||||
Check their status by checking the K3s service:
|
||||
```bash
|
||||
systemctl status k3s-agent
|
||||
```
|
||||
:::note
|
||||
If you are checking a controlplane node, use `systemctl status k3s` instead.
|
||||
:::
|
||||
|
||||
## Component Logging
|
||||
|
||||
The logging of the components can contain information on what the problem could be.
|
||||
|
||||
### RKE2
|
||||
|
||||
```bash
|
||||
# kubelet logs are part of the systemd service
|
||||
journalctl -u rke2-agent -f | grep -i "kubelet"
|
||||
|
||||
# kube-proxy logs are retrieved from containerd
|
||||
crictl logs $(crictl ps --name kube-proxy -q)
|
||||
```
|
||||
|
||||
### K3s
|
||||
|
||||
```bash
|
||||
# Both components log to the systemd service
|
||||
journalctl -u k3s-agent -f | grep -iE "kubelet|kube-proxy"
|
||||
```
|
||||
Reference in New Issue
Block a user