mirror of
https://github.com/rancher/rancher-docs.git
synced 2026-09-26 13:08:11 +00:00
Adding v2.15 documentation with prerelease banner
Signed-off-by: Sunil Singh <sunil.singh@suse.com>
This commit is contained in:
Sunil Singh
@@ -0,0 +1,48 @@
|
||||
---
|
||||
title: General Troubleshooting
|
||||
---
|
||||
|
||||
<head>
|
||||
<link rel="canonical" href="https://ranchermanager.docs.rancher.com/troubleshooting/general-troubleshooting"/>
|
||||
</head>
|
||||
|
||||
This section contains information to help you troubleshoot issues when using Rancher.
|
||||
|
||||
- [Kubernetes components](kubernetes-components/kubernetes-components.md)
|
||||
|
||||
If you need help troubleshooting core Kubernetes cluster components like:
|
||||
* `etcd`
|
||||
* `kube-apiserver`
|
||||
* `kube-controller-manager`
|
||||
* `kube-scheduler`
|
||||
* `kubelet`
|
||||
* `kube-proxy`
|
||||
* `nginx-proxy`
|
||||
|
||||
- [Kubernetes resources](other-troubleshooting-tips/kubernetes-resources.md)
|
||||
|
||||
Options for troubleshooting Kubernetes resources like Nodes, Ingress Controller and Rancher Agents are described in this section.
|
||||
|
||||
- [Networking](other-troubleshooting-tips/networking.md)
|
||||
|
||||
Steps to troubleshoot networking issues can be found here.
|
||||
|
||||
- [DNS](other-troubleshooting-tips/dns.md)
|
||||
|
||||
When you experience name resolution issues in your cluster.
|
||||
|
||||
- [Troubleshooting Rancher installed on Kubernetes](other-troubleshooting-tips/rancher-ha.md)
|
||||
|
||||
If you experience issues with your [Rancher server installed on Kubernetes](../getting-started/installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster/install-upgrade-on-a-kubernetes-cluster.md)
|
||||
|
||||
- [Logging](other-troubleshooting-tips/logging.md)
|
||||
|
||||
Read more about what log levels can be configured and how to configure a log level.
|
||||
|
||||
- [User ID Tracking in Audit Logs](other-troubleshooting-tips/user-id-tracking-in-audit-logs.md)
|
||||
|
||||
Read more about how a Rancher Admin can trace an event from the Rancher audit logs and into the Kubernetes audit logs using the external Identity Provider username.
|
||||
|
||||
- [Expired Webhook Certificates](other-troubleshooting-tips/expired-webhook-certificate-rotation.md)
|
||||
|
||||
Read more about how to rotate a Rancher webhook certificate secret after it expires on an annual basis.
|
||||
+21
@@ -0,0 +1,21 @@
|
||||
---
|
||||
title: Kubernetes Components
|
||||
---
|
||||
|
||||
<head>
|
||||
<link rel="canonical" href="https://ranchermanager.docs.rancher.com/troubleshooting/kubernetes-components"/>
|
||||
</head>
|
||||
|
||||
The commands and steps listed in this section apply to the core Kubernetes components on [Rancher Launched Kubernetes](../../how-to-guides/new-user-guides/launch-kubernetes-with-rancher/launch-kubernetes-with-rancher.md) clusters.
|
||||
|
||||
This section includes troubleshooting tips in the following categories:
|
||||
|
||||
- [Troubleshooting etcd Nodes](troubleshooting-etcd-nodes.md)
|
||||
- [Troubleshooting Controlplane Nodes](troubleshooting-controlplane-nodes.md)
|
||||
- [Troubleshooting nginx-proxy Nodes](troubleshooting-nginx-proxy.md)
|
||||
- [Troubleshooting Worker Nodes and Generic Components](troubleshooting-worker-nodes-and-generic-components.md)
|
||||
|
||||
## Kubernetes Component Diagram
|
||||
|
||||
<br/>
|
||||
<sup>Lines show the traffic flow between components. Colors are used purely for visual aid</sup>
|
||||
+93
@@ -0,0 +1,93 @@
|
||||
---
|
||||
title: Troubleshooting Controlplane Nodes
|
||||
---
|
||||
|
||||
<head>
|
||||
<link rel="canonical" href="https://ranchermanager.docs.rancher.com/troubleshooting/kubernetes-components/troubleshooting-controlplane-nodes"/>
|
||||
</head>
|
||||
|
||||
This section applies to nodes with the `controlplane` role in RKE2 and K3s clusters.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
As RKE2 and K3s rely on `containerd` as the container runtime, `crictl` replaces Docker for container management. Before proceeding with the troubleshooting commands, configure your environment by exporting the following variables:
|
||||
|
||||
### RKE2
|
||||
|
||||
```bash
|
||||
export PATH=$PATH:/var/lib/rancher/rke2/bin/
|
||||
export CRI_CONFIG_FILE=/var/lib/rancher/rke2/agent/etc/crictl.yaml
|
||||
```
|
||||
|
||||
### K3s
|
||||
|
||||
```bash
|
||||
export PATH=$PATH:/usr/local/bin
|
||||
export CRI_CONFIG_FILE=/var/lib/rancher/k3s/agent/etc/crictl.yaml
|
||||
```
|
||||
|
||||
## Check if the Controlplane Components are Running
|
||||
|
||||
**RKE2**: There are three specific containers launched on nodes with the `controlplane` role:
|
||||
|
||||
* `kube-apiserver`
|
||||
* `kube-controller-manager`
|
||||
* `kube-scheduler`
|
||||
|
||||
The containers should have state **Running**. You can check this using `crictl`:
|
||||
|
||||
```bash
|
||||
crictl ps | grep -E 'kube-apiserver|kube-controller-manager|kube-scheduler'
|
||||
```
|
||||
|
||||
Example output:
|
||||
```
|
||||
CONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID POD NAMESPACE
|
||||
deb8a96948594 138b1e685e151 11 days ago Running kube-controller-manager 0 0996426295dc5 kube-controller-manager kube-system
|
||||
f5abb4c7846e4 138b1e685e151 11 days ago Running kube-scheduler 0 80cd9f30af0be kube-scheduler kube-system
|
||||
ecd8a6991c22a 138b1e685e151 11 days ago Running kube-apiserver 0 58e042fabe78c kube-apiserver kube-system
|
||||
```
|
||||
|
||||
**K3s**: These components run as embedded processes within the K3s service. They do not run as separate containers, so their status is tied to the `k3s` systemd service:
|
||||
|
||||
```bash
|
||||
systemctl status k3s
|
||||
```
|
||||
|
||||
## Controlplane Logging
|
||||
|
||||
:::note
|
||||
|
||||
If you added multiple nodes with the `controlplane` role, both `kube-controller-manager` and `kube-scheduler` use a leader election process to determine the leader. Only the current leader will log the performed actions. See [Kubernetes leader election](../other-troubleshooting-tips/kubernetes-resources.md#kubernetes-leader-election) how to retrieve the current leader.
|
||||
|
||||
:::
|
||||
|
||||
The logs can contain information on what the problem could be.
|
||||
|
||||
**RKE2**:
|
||||
```bash
|
||||
crictl logs $(crictl ps --name kube-apiserver -q)
|
||||
crictl logs $(crictl ps --name kube-controller-manager -q)
|
||||
crictl logs $(crictl ps --name kube-scheduler -q)
|
||||
```
|
||||
|
||||
**K3s**:
|
||||
```bash
|
||||
journalctl -u k3s | grep -i "kube-apiserver"
|
||||
journalctl -u k3s | grep -i "kube-controller-manager"
|
||||
journalctl -u k3s | grep -i "kube-scheduler"
|
||||
```
|
||||
|
||||
## RKE2/K3s Server Logging
|
||||
|
||||
If Rancher provisions an RKE2 or K3s cluster that can't communicate with Rancher, you can run this command on a server node in the downstream cluster to get the server logs:
|
||||
|
||||
**RKE2**:
|
||||
```bash
|
||||
journalctl -u rke2-server -f
|
||||
```
|
||||
|
||||
**K3s**:
|
||||
```bash
|
||||
journalctl -u k3s -f
|
||||
```
|
||||
+390
@@ -0,0 +1,390 @@
|
||||
---
|
||||
title: Troubleshooting etcd Nodes
|
||||
---
|
||||
|
||||
<head>
|
||||
<link rel="canonical" href="https://ranchermanager.docs.rancher.com/troubleshooting/kubernetes-components/troubleshooting-etcd-nodes"/>
|
||||
</head>
|
||||
|
||||
This section contains commands and tips for troubleshooting nodes with the `etcd` role in RKE2 and K3s clusters.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
As RKE2 and K3s rely on `containerd` as the container runtime, `crictl` replaces Docker for container management. Before proceeding with the troubleshooting commands, configure your environment by exporting the following variables:
|
||||
|
||||
### RKE2
|
||||
|
||||
```bash
|
||||
export PATH=$PATH:/var/lib/rancher/rke2/bin/
|
||||
export CRI_CONFIG_FILE=/var/lib/rancher/rke2/agent/etc/crictl.yaml
|
||||
etcdcontainer=$(crictl ps --name etcd --quiet)
|
||||
```
|
||||
|
||||
### K3s
|
||||
|
||||
> ### ⚠️ **Warning**
|
||||
> K3s does not include `etcdctl` in the system PATH. If you need to perform etcd troubleshooting on a K3s cluster, you may need to install it or locate it within the K3s data directory.
|
||||
|
||||
```bash
|
||||
export PATH=$PATH:/usr/local/bin
|
||||
export CRI_CONFIG_FILE=/var/lib/rancher/k3s/agent/etc/crictl.yaml
|
||||
```
|
||||
|
||||
|
||||
## Checking if the etcd Container is Running
|
||||
|
||||
**RKE2**: The container for etcd should be in the **Running** state.
|
||||
|
||||
```bash
|
||||
crictl ps --name etcd
|
||||
```
|
||||
|
||||
Example output:
|
||||
```
|
||||
CONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID POD NAMESPACE
|
||||
f1e289d202ed0 11ad16872a9cf 58 minutes ago Running etcd 0 7b56aab8204ea etcd-cluster1 kube-system
|
||||
```
|
||||
|
||||
**K3s**: Etcd runs as an embedded process in the K3s service. Check the service status:
|
||||
|
||||
```bash
|
||||
systemctl status k3s
|
||||
```
|
||||
|
||||
## etcd Logging
|
||||
|
||||
The logs can contain information on what the problem could be.
|
||||
|
||||
**RKE2**:
|
||||
```bash
|
||||
crictl logs $etcdcontainer
|
||||
```
|
||||
|
||||
**K3s**:
|
||||
```bash
|
||||
journalctl -u k3s | grep -i etcd
|
||||
```
|
||||
| Log | Explanation |
|
||||
|-----|------------------|
|
||||
| `health check for peer xxx could not connect: dial tcp IP:2380: getsockopt: connection refused` | A connection to the address shown on port 2380 cannot be established. Check if the etcd container is running on the host with the address shown. |
|
||||
| `xxx is starting a new election at term x` | The etcd cluster has lost its quorum and is trying to establish a new leader. This can happen when the majority of the nodes running etcd go down/unreachable. |
|
||||
| `connection error: desc = "transport: Error while dialing dial tcp 0.0.0.0:2379: i/o timeout"; Reconnecting to {0.0.0.0:2379 0 <nil>}` | The host firewall is preventing network communication. |
|
||||
| `rafthttp: request cluster ID mismatch` | The node with the etcd instance logging `rafthttp: request cluster ID mismatch` is trying to join a cluster that has already been formed with another peer. The node should be removed from the cluster, and re-added. |
|
||||
| `rafthttp: failed to find member` | The cluster state (`/var/lib/etcd`) contains wrong information to join the cluster. The node should be removed from the cluster, the state directory should be cleaned and the node should be re-added.
|
||||
|
||||
## etcd Cluster and Connectivity Checks
|
||||
|
||||
The address where etcd is listening depends on the address configuration of the host etcd is running on. If an internal address is configured for the host etcd is running on, the endpoint for `etcdctl` needs to be specified explicitly. If any of the commands respond with `Error: context deadline exceeded`, the etcd instance is unhealthy (either quorum is lost or the instance is not correctly joined in the cluster)
|
||||
|
||||
### Check etcd Members on all Nodes
|
||||
|
||||
Output should contain all the nodes with the `etcd` role and the output should be identical on all nodes.
|
||||
|
||||
**RKE2**:
|
||||
Run the command inside the etcd container.
|
||||
|
||||
```bash
|
||||
crictl exec $etcdcontainer etcdctl member list \
|
||||
--cert /var/lib/rancher/rke2/server/tls/etcd/server-client.crt \
|
||||
--key /var/lib/rancher/rke2/server/tls/etcd/server-client.key \
|
||||
--cacert /var/lib/rancher/rke2/server/tls/etcd/server-ca.crt
|
||||
```
|
||||
|
||||
**K3s**:
|
||||
```bash
|
||||
etcdctl member list \
|
||||
--cert /var/lib/rancher/k3s/server/tls/etcd/server-client.crt \
|
||||
--key /var/lib/rancher/k3s/server/tls/etcd/server-client.key \
|
||||
--cacert /var/lib/rancher/k3s/server/tls/etcd/server-ca.crt
|
||||
```
|
||||
|
||||
Example output:
|
||||
```
|
||||
1c424074df86e854, started, cluster-node1-f289ac71, https://IP:2380, https://IP:2379, false
|
||||
45c68c44c5a792ff, started, cluster-node2-67e3cf6f, https://IP:2380, https://IP:2379, false
|
||||
7c584f77c5180258, started, cluster-node3-e976bc00, https://IP:2380, https://IP:2379, false
|
||||
```
|
||||
|
||||
### Check Endpoint Status
|
||||
|
||||
The values for `RAFT TERM` should be equal and `RAFT INDEX` should be not be too far apart from each other.
|
||||
|
||||
**RKE2**:
|
||||
```bash
|
||||
crictl exec $etcdcontainer etcdctl endpoint status --write-out table --endpoints=$(crictl exec $etcdcontainer etcdctl member list --cert /var/lib/rancher/rke2/server/tls/etcd/server-client.crt --key /var/lib/rancher/rke2/server/tls/etcd/server-client.key --cacert /var/lib/rancher/rke2/server/tls/etcd/server-ca.crt | cut -d, -f5 | sed -e 's/ //g' | paste -sd ',') --cert /var/lib/rancher/rke2/server/tls/etcd/server-client.crt --key /var/lib/rancher/rke2/server/tls/etcd/server-client.key --cacert /var/lib/rancher/rke2/server/tls/etcd/server-ca.crt
|
||||
```
|
||||
|
||||
**K3s**:
|
||||
```bash
|
||||
etcdctl endpoint status --write-out table --endpoints=$(etcdctl member list --cert /var/lib/rancher/k3s/server/tls/etcd/server-client.crt --key /var/lib/rancher/k3s/server/tls/etcd/server-client.key --cacert /var/lib/rancher/k3s/server/tls/etcd/server-ca.crt | cut -d, -f5 | sed -e 's/ //g' | paste -sd ',') --cert /var/lib/rancher/k3s/server/tls/etcd/server-client.crt --key /var/lib/rancher/k3s/server/tls/etcd/server-client.key --cacert /var/lib/rancher/k3s/server/tls/etcd/server-ca.crt
|
||||
```
|
||||
|
||||
Example output:
|
||||
```
|
||||
+-----------------+------------------+---------+---------+-----------+-----------+------------+
|
||||
| ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | RAFT TERM | RAFT INDEX |
|
||||
+-----------------+------------------+---------+---------+-----------+-----------+------------+
|
||||
| https://IP:2379 | 333ef673fc4add56 | 3.6.7 | 24 MB | false | 72 | 66887 |
|
||||
| https://IP:2379 | 5feed52d940ce4cf | 3.6.7 | 24 MB | true | 72 | 66887 |
|
||||
| https://IP:2379 | db6b3bdb559a848d | 3.6.7 | 25 MB | false | 72 | 66887 |
|
||||
+-----------------+------------------+---------+---------+-----------+-----------+------------+
|
||||
```
|
||||
|
||||
### Check Endpoint Health
|
||||
|
||||
**RKE2**:
|
||||
```bash
|
||||
crictl exec $etcdcontainer etcdctl endpoint health --endpoints=$(crictl exec $etcdcontainer etcdctl member list --cert /var/lib/rancher/rke2/server/tls/etcd/server-client.crt --key /var/lib/rancher/rke2/server/tls/etcd/server-client.key --cacert /var/lib/rancher/rke2/server/tls/etcd/server-ca.crt | cut -d, -f5 | sed -e 's/ //g' | paste -sd ',') --cert /var/lib/rancher/rke2/server/tls/etcd/server-client.crt --key /var/lib/rancher/rke2/server/tls/etcd/server-client.key --cacert /var/lib/rancher/rke2/server/tls/etcd/server-ca.crt
|
||||
```
|
||||
|
||||
**K3s**:
|
||||
```bash
|
||||
etcdctl endpoint health --endpoints=$(etcdctl member list --cert /var/lib/rancher/k3s/server/tls/etcd/server-client.crt --key /var/lib/rancher/k3s/server/tls/etcd/server-client.key --cacert /var/lib/rancher/k3s/server/tls/etcd/server-ca.crt | cut -d, -f5 | sed -e 's/ //g' | paste -sd ',') --cert /var/lib/rancher/k3s/server/tls/etcd/server-client.crt --key /var/lib/rancher/k3s/server/tls/etcd/server-client.key --cacert /var/lib/rancher/k3s/server/tls/etcd/server-ca.crt
|
||||
```
|
||||
|
||||
Example output:
|
||||
```
|
||||
https://IP:2379 is healthy: successfully committed proposal: took = 2.113189ms
|
||||
https://IP:2379 is healthy: successfully committed proposal: took = 2.649963ms
|
||||
https://IP:2379 is healthy: successfully committed proposal: took = 2.451201ms
|
||||
```
|
||||
### Check Connectivity on etcd Ports
|
||||
|
||||
> In modern versions of Kubernetes, the etcd database (versions 3.5 and newer) introduced significant architectural changes regarding network traffic handling. Previously, etcd permitted standard HTTP REST requests on its primary client port (`2379`). However, to enhance performance and security, etcd 3.5+ strictly enforces the gRPC protocol on this port.<br />
|
||||
If you attempt to use standard HTTP tools like `curl` to test connectivity on port `2379`, the etcd server will automatically terminate the connection or return an error. This behavior often leads administrators to misinterpret the result as a closed port or a node failure.
|
||||
|
||||
Since standard HTTP clients can no longer probe the primary etcd ports, the transport layer must be utilized for network troubleshooting. Using `openssl s_client` instead of `curl` bypasses the gRPC application requirement, allowing the raw TCP and TLS handshake to be tested directly.
|
||||
|
||||
These script isolate the network and security infrastructure from the database application. A successful `Verify return code: 0 (ok)` explicitly confirms four critical infrastructure components:
|
||||
|
||||
* **Network Path:** Routing is functional, and firewalls permit traffic on TCP port `2379` or `2380`.
|
||||
* **Process Availability:** The etcd service is running and actively listening on the designated port.
|
||||
* **Certificate Validity:** The TLS certificates are active, correctly formatted, and have not expired.
|
||||
* **Mutual Authentication (mTLS):** The node successfully authenticates against the cluster's specific Certificate Authority (CA).
|
||||
|
||||
**How these tests differ from the `etcdctl endpoint health` test**:
|
||||
|
||||
If `etcdctl endpoint health` test is failing, run these Connectivity Ports test scripts. If the scripts succeed, your network and certificates are intact, and the issue is likely confined to the etcd database itself. If these scripts fail, the issue is related to a firewall/network restriction, or certificate expiration.
|
||||
|
||||
#### Port TCP/2379
|
||||
|
||||
**RKE2**:
|
||||
```bash
|
||||
for endpoint in $(crictl exec $etcdcontainer etcdctl member list --cert /var/lib/rancher/rke2/server/tls/etcd/server-client.crt --key /var/lib/rancher/rke2/server/tls/etcd/server-client.key --cacert /var/lib/rancher/rke2/server/tls/etcd/server-ca.crt | cut -d, -f5); do
|
||||
echo "Validating connection to ${endpoint} (Client)";
|
||||
echo | openssl s_client -connect ${endpoint#https://} \
|
||||
-CAfile /var/lib/rancher/rke2/server/tls/etcd/server-ca.crt \
|
||||
-cert /var/lib/rancher/rke2/server/tls/etcd/server-client.crt \
|
||||
-key /var/lib/rancher/rke2/server/tls/etcd/server-client.key 2>/dev/null | grep -E 'Verify return code' || echo "Connection Failed/Timeout"
|
||||
done
|
||||
```
|
||||
|
||||
**K3s**:
|
||||
```bash
|
||||
for endpoint in $(etcdctl member list --cert /var/lib/rancher/k3s/server/tls/etcd/server-client.crt --key /var/lib/rancher/k3s/server/tls/etcd/server-client.key --cacert /var/lib/rancher/k3s/server/tls/etcd/server-ca.crt | cut -d, -f5); do
|
||||
echo "Validating connection to ${endpoint} (Client)";
|
||||
echo | openssl s_client -connect ${endpoint#https://} \
|
||||
-CAfile /var/lib/rancher/k3s/server/tls/etcd/server-ca.crt \
|
||||
-cert /var/lib/rancher/k3s/server/tls/etcd/server-client.crt \
|
||||
-key /var/lib/rancher/k3s/server/tls/etcd/server-client.key 2>/dev/null | grep -E 'Verify return code' || echo "Connection Failed/Timeout"
|
||||
done
|
||||
```
|
||||
|
||||
Example output:
|
||||
```
|
||||
Validating connection to https://IP:2379/health (Client)
|
||||
Verify return code: 0 (ok)
|
||||
Validating connection to https://IP:2379/health (Client)
|
||||
Verify return code: 0 (ok)
|
||||
Validating connection to https://IP:2379/health (Client)
|
||||
Verify return code: 0 (ok)
|
||||
```
|
||||
|
||||
#### Port TCP/2380
|
||||
|
||||
**RKE2**:
|
||||
```bash
|
||||
for endpoint in $(crictl exec $etcdcontainer etcdctl member list --cert /var/lib/rancher/rke2/server/tls/etcd/server-client.crt --key /var/lib/rancher/rke2/server/tls/etcd/server-client.key --cacert /var/lib/rancher/rke2/server/tls/etcd/server-ca.crt | cut -d, -f4); do
|
||||
echo "Validating connection to ${endpoint} (Peer)";
|
||||
echo | openssl s_client -connect ${endpoint#https://} \
|
||||
-CAfile /var/lib/rancher/rke2/server/tls/etcd/peer-ca.crt \
|
||||
-cert /var/lib/rancher/rke2/server/tls/etcd/peer-server-client.crt \
|
||||
-key /var/lib/rancher/rke2/server/tls/etcd/peer-server-client.key 2>/dev/null | grep -E 'Verify return code' || echo "Connection Failed/Timeout"
|
||||
done
|
||||
```
|
||||
|
||||
**K3s**:
|
||||
```bash
|
||||
for endpoint in $(etcdctl member list --cert /var/lib/rancher/k3s/server/tls/etcd/server-client.crt --key /var/lib/rancher/k3s/server/tls/etcd/server-client.key --cacert /var/lib/rancher/k3s/server/tls/etcd/server-ca.crt | cut -d, -f4); do
|
||||
echo "Validating connection to ${endpoint} (Peer)";
|
||||
echo | openssl s_client -connect ${endpoint#https://} \
|
||||
-CAfile /var/lib/rancher/k3s/server/tls/etcd/peer-ca.crt \
|
||||
-cert /var/lib/rancher/k3s/server/tls/etcd/peer-server-client.crt \
|
||||
-key /var/lib/rancher/k3s/server/tls/etcd/peer-server-client.key 2>/dev/null | grep -E 'Verify return code' || echo "Connection Failed/Timeout"
|
||||
done
|
||||
```
|
||||
|
||||
Example output:
|
||||
```
|
||||
Validating connection to https://IP:2380/version (Peer)
|
||||
Verify return code: 0 (ok)
|
||||
Validating connection to https://IP:2380/version (Peer)
|
||||
Verify return code: 0 (ok)
|
||||
Validating connection to https://IP:2380/version (Peer)
|
||||
Verify return code: 0 (ok)
|
||||
```
|
||||
|
||||
## etcd Alarms
|
||||
|
||||
etcd will trigger alarms, for instance when it runs out of space.
|
||||
|
||||
**RKE2**:
|
||||
```bash
|
||||
crictl exec $etcdcontainer etcdctl alarm list --cert /var/lib/rancher/rke2/server/tls/etcd/server-client.crt --key /var/lib/rancher/rke2/server/tls/etcd/server-client.key --cacert /var/lib/rancher/rke2/server/tls/etcd/server-ca.crt
|
||||
```
|
||||
|
||||
**K3s**:
|
||||
```bash
|
||||
etcdctl alarm list --cert /var/lib/rancher/k3s/server/tls/etcd/server-client.crt --key /var/lib/rancher/k3s/server/tls/etcd/server-client.key --cacert /var/lib/rancher/k3s/server/tls/etcd/server-ca.crt
|
||||
```
|
||||
|
||||
Example output when NOSPACE alarm is triggered:
|
||||
```
|
||||
memberID:x alarm:NOSPACE
|
||||
memberID:x alarm:NOSPACE
|
||||
memberID:x alarm:NOSPACE
|
||||
```
|
||||
|
||||
## etcd Space Errors
|
||||
|
||||
Related error messages are `etcdserver: mvcc: database space exceeded` or `applying raft message exceeded backend quota`. Alarm `NOSPACE` will be triggered.
|
||||
|
||||
Resolutions:
|
||||
|
||||
- [Compact the Keyspace](#compact-the-keyspace)
|
||||
- [Defrag All etcd Members](#defrag-all-etcd-members)
|
||||
- [Check Endpoint Status](#check-endpoint-status)
|
||||
- [Disarm Alarm](#disarm-alarm)
|
||||
|
||||
### Compact the Keyspace
|
||||
|
||||
**RKE2**:
|
||||
```bash
|
||||
rev=$(crictl exec $etcdcontainer etcdctl endpoint status --write-out json --cert /var/lib/rancher/rke2/server/tls/etcd/server-client.crt --key /var/lib/rancher/rke2/server/tls/etcd/server-client.key --cacert /var/lib/rancher/rke2/server/tls/etcd/server-ca.crt | egrep -o '"revision":[0-9]*' | egrep -o '[0-9]*' | head -1)
|
||||
crictl exec $etcdcontainer etcdctl compact "$rev" --cert /var/lib/rancher/rke2/server/tls/etcd/server-client.crt --key /var/lib/rancher/rke2/server/tls/etcd/server-client.key --cacert /var/lib/rancher/rke2/server/tls/etcd/server-ca.crt
|
||||
```
|
||||
|
||||
**K3s**:
|
||||
```bash
|
||||
rev=$(etcdctl endpoint status --write-out json --cert /var/lib/rancher/k3s/server/tls/etcd/server-client.crt --key /var/lib/rancher/k3s/server/tls/etcd/server-client.key --cacert /var/lib/rancher/k3s/server/tls/etcd/server-ca.crt | egrep -o '"revision":[0-9]*' | egrep -o '[0-9]*' | head -1)
|
||||
etcdctl compact "$rev" --cert /var/lib/rancher/k3s/server/tls/etcd/server-client.crt --key /var/lib/rancher/k3s/server/tls/etcd/server-client.key --cacert /var/lib/rancher/k3s/server/tls/etcd/server-ca.crt
|
||||
```
|
||||
|
||||
Example output:
|
||||
```
|
||||
compacted revision xxx
|
||||
```
|
||||
|
||||
### Defrag All etcd Members
|
||||
|
||||
**RKE2**:
|
||||
```bash
|
||||
crictl exec $etcdcontainer etcdctl defrag --endpoints=$(crictl exec $etcdcontainer etcdctl member list --cert /var/lib/rancher/rke2/server/tls/etcd/server-client.crt --key /var/lib/rancher/rke2/server/tls/etcd/server-client.key --cacert /var/lib/rancher/rke2/server/tls/etcd/server-ca.crt | cut -d, -f5 | sed -e 's/ //g' | paste -sd ',') --cert /var/lib/rancher/rke2/server/tls/etcd/server-client.crt --key /var/lib/rancher/rke2/server/tls/etcd/server-client.key --cacert /var/lib/rancher/rke2/server/tls/etcd/server-ca.crt
|
||||
```
|
||||
|
||||
**K3s**:
|
||||
```bash
|
||||
etcdctl defrag --endpoints=$(etcdctl member list --cert /var/lib/rancher/k3s/server/tls/etcd/server-client.crt --key /var/lib/rancher/k3s/server/tls/etcd/server-client.key --cacert /var/lib/rancher/k3s/server/tls/etcd/server-ca.crt | cut -d, -f5 | sed -e 's/ //g' | paste -sd ',') --cert /var/lib/rancher/k3s/server/tls/etcd/server-client.crt --key /var/lib/rancher/k3s/server/tls/etcd/server-client.key --cacert /var/lib/rancher/k3s/server/tls/etcd/server-ca.crt
|
||||
```
|
||||
|
||||
Example output:
|
||||
```
|
||||
Finished defragmenting etcd member[https://IP:2379]. took xx.xxxxxxms
|
||||
Finished defragmenting etcd member[https://IP:2379]. took xx.xxxxxxms
|
||||
Finished defragmenting etcd member[https://IP:2379]. took xx.xxxxxxms
|
||||
```
|
||||
|
||||
### Disarm Alarm
|
||||
|
||||
After verifying that the DB size went down after compaction and defragmenting, the alarm needs to be disarmed for etcd to allow writes again.
|
||||
|
||||
**RKE2**:
|
||||
```bash
|
||||
crictl exec $etcdcontainer etcdctl alarm list --cert /var/lib/rancher/rke2/server/tls/etcd/server-client.crt --key /var/lib/rancher/rke2/server/tls/etcd/server-client.key --cacert /var/lib/rancher/rke2/server/tls/etcd/server-ca.crt
|
||||
crictl exec $etcdcontainer etcdctl alarm disarm --cert /var/lib/rancher/rke2/server/tls/etcd/server-client.crt --key /var/lib/rancher/rke2/server/tls/etcd/server-client.key --cacert /var/lib/rancher/rke2/server/tls/etcd/server-ca.crt
|
||||
crictl exec $etcdcontainer etcdctl alarm list --cert /var/lib/rancher/rke2/server/tls/etcd/server-client.crt --key /var/lib/rancher/rke2/server/tls/etcd/server-client.key --cacert /var/lib/rancher/rke2/server/tls/etcd/server-ca.crt
|
||||
```
|
||||
|
||||
**K3s**:
|
||||
```bash
|
||||
etcdctl alarm list --cert /var/lib/rancher/k3s/server/tls/etcd/server-client.crt --key /var/lib/rancher/k3s/server/tls/etcd/server-client.key --cacert /var/lib/rancher/k3s/server/tls/etcd/server-ca.crt
|
||||
etcdctl alarm disarm --cert /var/lib/rancher/k3s/server/tls/etcd/server-client.crt --key /var/lib/rancher/k3s/server/tls/etcd/server-client.key --cacert /var/lib/rancher/k3s/server/tls/etcd/server-ca.crt
|
||||
etcdctl alarm list --cert /var/lib/rancher/k3s/server/tls/etcd/server-client.crt --key /var/lib/rancher/k3s/server/tls/etcd/server-client.key --cacert /var/lib/rancher/k3s/server/tls/etcd/server-ca.crt
|
||||
```
|
||||
|
||||
## Configure Log Level
|
||||
|
||||
:::note
|
||||
|
||||
You can no longer dynamically change the log level in etcd v3.5 or later.
|
||||
|
||||
:::
|
||||
|
||||
### etcd v3.5 And Later
|
||||
|
||||
To configure the log level for etcd, edit the cluster configuration YAML:
|
||||
|
||||
```
|
||||
services:
|
||||
etcd:
|
||||
extra_args:
|
||||
log-level: "debug"
|
||||
```
|
||||
|
||||
After modifying the configuration, restart the service (`systemctl restart rke2-server` or `systemctl restart k3s`) if you are configuring a stand-alone cluster.
|
||||
|
||||
## etcd Content
|
||||
|
||||
If you want to investigate the contents of your etcd, you can either watch streaming events or you can query etcd directly, see below for examples.
|
||||
|
||||
### Watch Streaming Events
|
||||
|
||||
**RKE2**:
|
||||
```bash
|
||||
crictl exec $etcdcontainer etcdctl watch --prefix /registry --cert /var/lib/rancher/rke2/server/tls/etcd/server-client.crt --key /var/lib/rancher/rke2/server/tls/etcd/server-client.key --cacert /var/lib/rancher/rke2/server/tls/etcd/server-ca.crt
|
||||
```
|
||||
|
||||
**K3s**:
|
||||
```bash
|
||||
etcdctl watch --prefix /registry --cert /var/lib/rancher/k3s/server/tls/etcd/server-client.crt --key /var/lib/rancher/k3s/server/tls/etcd/server-client.key --cacert /var/lib/rancher/k3s/server/tls/etcd/server-ca.crt
|
||||
```
|
||||
|
||||
If you only want to see the affected keys (and not the binary data), you can append `| grep -a ^/registry` to the command to filter for keys only.
|
||||
|
||||
### Query etcd Directly
|
||||
|
||||
**RKE2**:
|
||||
```bash
|
||||
crictl exec $etcdcontainer etcdctl get /registry --prefix=true --keys-only --cert /var/lib/rancher/rke2/server/tls/etcd/server-client.crt --key /var/lib/rancher/rke2/server/tls/etcd/server-client.key --cacert /var/lib/rancher/rke2/server/tls/etcd/server-ca.crt
|
||||
```
|
||||
|
||||
**K3s**:
|
||||
```bash
|
||||
etcdctl get /registry --prefix=true --keys-only --cert /var/lib/rancher/k3s/server/tls/etcd/server-client.crt --key /var/lib/rancher/k3s/server/tls/etcd/server-client.key --cacert /var/lib/rancher/k3s/server/tls/etcd/server-ca.crt
|
||||
```
|
||||
|
||||
You can process the data to get a summary of count per key, using the command below:
|
||||
|
||||
**RKE2**:
|
||||
```bash
|
||||
crictl exec $etcdcontainer etcdctl get /registry --prefix=true --keys-only --cert /var/lib/rancher/rke2/server/tls/etcd/server-client.crt --key /var/lib/rancher/rke2/server/tls/etcd/server-client.key --cacert /var/lib/rancher/rke2/server/tls/etcd/server-ca.crt | grep -v ^$ | awk -F'/' '{ if ($3 ~ /cattle.io/) {h[$3"/"$4]++} else { h[$3]++ }} END { for(k in h) print h[k], k }' | sort -nr
|
||||
```
|
||||
|
||||
**K3s**:
|
||||
```bash
|
||||
etcdctl get /registry --prefix=true --keys-only --cert /var/lib/rancher/k3s/server/tls/etcd/server-client.crt --key /var/lib/rancher/k3s/server/tls/etcd/server-client.key --cacert /var/lib/rancher/k3s/server/tls/etcd/server-ca.crt | grep -v ^$ | awk -F'/' '{ if ($3 ~ /cattle.io/) {h[$3"/"$4]++} else { h[$3]++ }} END { for(k in h) print h[k], k }' | sort -nr
|
||||
```
|
||||
|
||||
## Replacing Unhealthy etcd Nodes
|
||||
|
||||
When a node in your etcd cluster becomes unhealthy, the recommended approach is to fix or remove the failed or unhealthy node before adding a new etcd node to the cluster.
|
||||
+80
@@ -0,0 +1,80 @@
|
||||
---
|
||||
title: Troubleshooting nginx-proxy
|
||||
---
|
||||
|
||||
<head>
|
||||
<link rel="canonical" href="https://ranchermanager.docs.rancher.com/troubleshooting/kubernetes-components/troubleshooting-nginx-proxy"/>
|
||||
</head>
|
||||
|
||||
:::caution
|
||||
|
||||
The `nginx-proxy` container is an RKE1-specific component. If you are using RKE2 or K3s, this container is not deployed, as load balancing to the API servers is handled internally by a client-side load balancer within the agent process itself.
|
||||
|
||||
Additionally, please note that RKE1 has reached its [End of Life (EOL)](https://support.scc.suse.com/s/kb/RKE-EOL-what-when-why?language=en_US). Therefore, the information on this page is considered deprecated.
|
||||
|
||||
:::
|
||||
|
||||
The `nginx-proxy` container is deployed on every node that does not have the `controlplane` role. It provides access to all the nodes with the `controlplane` role by dynamically generating the NGINX configuration based on available nodes with the `controlplane` role.
|
||||
|
||||
## Check if the Container is Running
|
||||
|
||||
The container is called `nginx-proxy` and should have status `Up`. The duration shown after `Up` is the time the container has been running.
|
||||
|
||||
```
|
||||
docker ps -a -f=name=nginx-proxy
|
||||
```
|
||||
|
||||
Example output:
|
||||
|
||||
```
|
||||
docker ps -a -f=name=nginx-proxy
|
||||
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
|
||||
c3e933687c0e rancher/rke-tools:v0.1.15 "nginx-proxy CP_HO..." 3 hours ago Up 3 hours nginx-proxy
|
||||
```
|
||||
|
||||
## Check Generated NGINX Configuration
|
||||
|
||||
The generated configuration should include the IP addresses of the nodes with the `controlplane` role. The configuration can be checked using the following command:
|
||||
|
||||
```
|
||||
docker exec nginx-proxy cat /etc/nginx/nginx.conf
|
||||
```
|
||||
|
||||
Example output:
|
||||
```
|
||||
error_log stderr notice;
|
||||
|
||||
worker_processes auto;
|
||||
events {
|
||||
multi_accept on;
|
||||
use epoll;
|
||||
worker_connections 1024;
|
||||
}
|
||||
|
||||
stream {
|
||||
upstream kube_apiserver {
|
||||
|
||||
server ip_of_controlplane_node1:6443;
|
||||
|
||||
server ip_of_controlplane_node2:6443;
|
||||
|
||||
}
|
||||
|
||||
server {
|
||||
listen 6443;
|
||||
proxy_pass kube_apiserver;
|
||||
proxy_timeout 30;
|
||||
proxy_connect_timeout 2s;
|
||||
|
||||
}
|
||||
|
||||
}
|
||||
```
|
||||
|
||||
## nginx-proxy Container Logging
|
||||
|
||||
The logging of the containers can contain information on what the problem could be.
|
||||
|
||||
```
|
||||
docker logs nginx-proxy
|
||||
```
|
||||
+92
@@ -0,0 +1,92 @@
|
||||
---
|
||||
title: Troubleshooting Worker Nodes and Generic Components
|
||||
---
|
||||
|
||||
<head>
|
||||
<link rel="canonical" href="https://ranchermanager.docs.rancher.com/troubleshooting/kubernetes-components/troubleshooting-worker-nodes-and-generic-components"/>
|
||||
</head>
|
||||
|
||||
This section applies to every node as it includes components that run on nodes with any role.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
Since RKE2 and K3s utilize `containerd` as the container runtime, `crictl` serves as the primary tool for container management, replacing the Docker CLI. To allow `crictl` to communicate with `containerd`, you must configure your environment by exporting the following variables:
|
||||
|
||||
### RKE2
|
||||
|
||||
```bash
|
||||
export PATH=$PATH:/var/lib/rancher/rke2/bin/
|
||||
export CRI_CONFIG_FILE=/var/lib/rancher/rke2/agent/etc/crictl.yaml
|
||||
```
|
||||
|
||||
### K3s
|
||||
|
||||
```bash
|
||||
export PATH=$PATH:/usr/local/bin
|
||||
export CRI_CONFIG_FILE=/var/lib/rancher/k3s/agent/etc/crictl.yaml
|
||||
```
|
||||
|
||||
## Check if the Components are Running
|
||||
|
||||
There are two specific components launched on nodes with the `worker` role:
|
||||
|
||||
* `kubelet`
|
||||
* `kube-proxy`
|
||||
|
||||
### RKE2
|
||||
|
||||
The `kubelet` runs natively as part of the `rke2-agent` (or `rke2-server`) systemd process, while `kube-proxy` runs as a Static Pod managed by `containerd`.
|
||||
|
||||
Check the status of the `kubelet` via the agent service:
|
||||
```bash
|
||||
systemctl status rke2-agent
|
||||
```
|
||||
:::note
|
||||
|
||||
If you are checking a controlplane node, use `systemctl status rke2-server` instead.
|
||||
|
||||
:::
|
||||
|
||||
Check the status of `kube-proxy` using `crictl`:
|
||||
```bash
|
||||
crictl ps --name kube-proxy
|
||||
```
|
||||
|
||||
Example output:
|
||||
```
|
||||
CONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID
|
||||
26c7159abbcc rancher/hardened-kubernetes:v1.28.8-rke2r1-build20240404 3 hours ago Running kube-proxy 0 1a2b3c4d5e6f7
|
||||
```
|
||||
|
||||
### K3s
|
||||
|
||||
Both `kubelet` and `kube-proxy` run as embedded processes inside the `k3s-agent` (or `k3s` server) systemd service. There are no separate containers for them.
|
||||
|
||||
Check their status by checking the K3s service:
|
||||
```bash
|
||||
systemctl status k3s-agent
|
||||
```
|
||||
:::note
|
||||
If you are checking a controlplane node, use `systemctl status k3s` instead.
|
||||
:::
|
||||
|
||||
## Component Logging
|
||||
|
||||
The logging of the components can contain information on what the problem could be.
|
||||
|
||||
### RKE2
|
||||
|
||||
```bash
|
||||
# kubelet logs are part of the systemd service
|
||||
journalctl -u rke2-agent -f | grep -i "kubelet"
|
||||
|
||||
# kube-proxy logs are retrieved from containerd
|
||||
crictl logs $(crictl ps --name kube-proxy -q)
|
||||
```
|
||||
|
||||
### K3s
|
||||
|
||||
```bash
|
||||
# Both components log to the systemd service
|
||||
journalctl -u k3s-agent -f | grep -iE "kubelet|kube-proxy"
|
||||
```
|
||||
@@ -0,0 +1,224 @@
|
||||
---
|
||||
title: DNS
|
||||
---
|
||||
|
||||
<head>
|
||||
<link rel="canonical" href="https://ranchermanager.docs.rancher.com/troubleshooting/other-troubleshooting-tips/dns"/>
|
||||
</head>
|
||||
|
||||
The commands/steps listed on this page can be used to check name resolution issues in your cluster.
|
||||
|
||||
Make sure you configured the correct kubeconfig (for example, `export KUBECONFIG=$PWD/kube_config_cluster.yml` for Rancher HA) or are using the embedded kubectl via the UI.
|
||||
|
||||
Before running the DNS checks, make sure that [the overlay network is functioning correctly](networking.md#check-if-overlay-network-is-functioning-correctly) for your DNS provider as this can also be the reason why DNS resolution (partly) fails.
|
||||
|
||||
## Check if DNS pods are running
|
||||
|
||||
```
|
||||
kubectl -n kube-system get pods -l k8s-app=kube-dns
|
||||
```
|
||||
|
||||
Example output when using CoreDNS:
|
||||
```
|
||||
NAME READY STATUS RESTARTS AGE
|
||||
coredns-799dffd9c4-6jhlz 1/1 Running 0 76m
|
||||
```
|
||||
|
||||
Example output when using kube-dns:
|
||||
```
|
||||
NAME READY STATUS RESTARTS AGE
|
||||
kube-dns-5fd74c7488-h6f7n 3/3 Running 0 4m13s
|
||||
```
|
||||
|
||||
## Check if the DNS service is present with the correct cluster-ip
|
||||
|
||||
```
|
||||
kubectl -n kube-system get svc -l k8s-app=kube-dns
|
||||
```
|
||||
|
||||
```
|
||||
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
|
||||
service/kube-dns ClusterIP 10.43.0.10 <none> 53/UDP,53/TCP 4m13s
|
||||
```
|
||||
|
||||
## Check if domain names are resolving
|
||||
|
||||
Check if internal cluster names are resolving (in this example, `kubernetes.default`), the IP shown after `Server:` should be the same as the `CLUSTER-IP` from the `kube-dns` service.
|
||||
|
||||
```
|
||||
kubectl run -it --rm --restart=Never busybox --image=busybox:1.28 -- nslookup kubernetes.default
|
||||
```
|
||||
|
||||
Example output:
|
||||
```
|
||||
Server: 10.43.0.10
|
||||
Address 1: 10.43.0.10 kube-dns.kube-system.svc.cluster.local
|
||||
|
||||
Name: kubernetes.default
|
||||
Address 1: 10.43.0.1 kubernetes.default.svc.cluster.local
|
||||
pod "busybox" deleted
|
||||
```
|
||||
|
||||
Check if external names are resolving (in this example, `www.google.com`)
|
||||
|
||||
```
|
||||
kubectl run -it --rm --restart=Never busybox --image=busybox:1.28 -- nslookup www.google.com
|
||||
```
|
||||
|
||||
Example output:
|
||||
```
|
||||
Server: 10.43.0.10
|
||||
Address 1: 10.43.0.10 kube-dns.kube-system.svc.cluster.local
|
||||
|
||||
Name: www.google.com
|
||||
Address 1: 2a00:1450:4009:80b::2004 lhr35s04-in-x04.1e100.net
|
||||
Address 2: 216.58.211.100 ams15s32-in-f4.1e100.net
|
||||
pod "busybox" deleted
|
||||
```
|
||||
|
||||
If you want to check resolving of domain names on all of the hosts, execute the following steps:
|
||||
|
||||
1. Save the following file as `ds-dnstest.yml`
|
||||
|
||||
```
|
||||
apiVersion: apps/v1
|
||||
kind: DaemonSet
|
||||
metadata:
|
||||
name: dnstest
|
||||
spec:
|
||||
selector:
|
||||
matchLabels:
|
||||
name: dnstest
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
name: dnstest
|
||||
spec:
|
||||
tolerations:
|
||||
- operator: Exists
|
||||
containers:
|
||||
- image: busybox:1.28
|
||||
imagePullPolicy: Always
|
||||
name: alpine
|
||||
command: ["sleep", "infinity"]
|
||||
terminationMessagePath: /dev/termination-log
|
||||
```
|
||||
|
||||
2. Launch it using `kubectl create -f ds-dnstest.yml`
|
||||
3. Wait until `kubectl rollout status ds/dnstest -w` returns: `daemon set "dnstest" successfully rolled out`.
|
||||
4. Configure the environment variable `DOMAIN` to a fully qualified domain name (FQDN) that the host should be able to resolve (`www.google.com` is used as an example) and run the following command to let each container on every host resolve the configured domain name (it's a single line command).
|
||||
|
||||
```
|
||||
export DOMAIN=www.google.com; echo "=> Start DNS resolve test"; kubectl get pods -l name=dnstest --no-headers -o custom-columns=NAME:.metadata.name,HOSTIP:.status.hostIP | while read pod host; do kubectl exec $pod -- /bin/sh -c "nslookup $DOMAIN > /dev/null 2>&1"; RC=$?; if [ $RC -ne 0 ]; then echo $host cannot resolve $DOMAIN; fi; done; echo "=> End DNS resolve test"
|
||||
```
|
||||
|
||||
5. When this command has finished running, the output indicating everything is correct is:
|
||||
|
||||
```
|
||||
=> Start DNS resolve test
|
||||
=> End DNS resolve test
|
||||
```
|
||||
|
||||
If you see error in the output, that means that the mentioned host(s) is/are not able to resolve the given FQDN.
|
||||
|
||||
Example error output of a situation where host with IP 209.97.182.150 had the UDP ports blocked.
|
||||
|
||||
```
|
||||
=> Start DNS resolve test
|
||||
command terminated with exit code 1
|
||||
209.97.182.150 cannot resolve www.google.com
|
||||
=> End DNS resolve test
|
||||
```
|
||||
|
||||
Cleanup the alpine DaemonSet by running `kubectl delete ds/dnstest`.
|
||||
|
||||
## CoreDNS specific
|
||||
|
||||
### Check CoreDNS logging
|
||||
|
||||
```
|
||||
kubectl -n kube-system logs -l k8s-app=kube-dns
|
||||
```
|
||||
|
||||
### Check configuration
|
||||
|
||||
CoreDNS configuration is stored in the configmap `coredns` in the `kube-system` namespace.
|
||||
|
||||
```
|
||||
kubectl -n kube-system get configmap coredns -o go-template={{.data.Corefile}}
|
||||
```
|
||||
|
||||
### Check upstream nameservers in resolv.conf
|
||||
|
||||
By default, the configured nameservers on the host (in `/etc/resolv.conf`) will be used as upstream nameservers for CoreDNS. You can check this file on the host or run the following Pod with `dnsPolicy` set to `Default`, which will inherit the `/etc/resolv.conf` from the host it is running on.
|
||||
|
||||
```
|
||||
kubectl run -i --restart=Never --rm test-${RANDOM} --image=ubuntu --overrides='{"kind":"Pod", "apiVersion":"v1", "spec": {"dnsPolicy":"Default"}}' -- sh -c 'cat /etc/resolv.conf'
|
||||
```
|
||||
|
||||
### Enable query logging
|
||||
|
||||
Enabling query logging can be done by enabling the [log plugin](https://coredns.io/plugins/log/) in the Corefile configuration in the configmap `coredns`. You can do so by using `kubectl -n kube-system edit configmap coredns` or use the command below to replace the configuration in place:
|
||||
|
||||
```
|
||||
kubectl get configmap -n kube-system coredns -o json | sed -e 's_loadbalance_log\\n loadbalance_g' | kubectl apply -f -
|
||||
```
|
||||
|
||||
All queries will now be logged and can be checked using the command in [Check CoreDNS logging](#check-coredns-logging).
|
||||
|
||||
## kube-dns specific
|
||||
|
||||
### Check upstream nameservers in kubedns container
|
||||
|
||||
By default, the configured nameservers on the host (in `/etc/resolv.conf`) will be used as upstream nameservers for kube-dns. Sometimes the host will run a local caching DNS nameserver, which means the address in `/etc/resolv.conf` will point to an address in the loopback range (`127.0.0.0/8`) which will be unreachable by the container. In case of Ubuntu 18.04, this is done by `systemd-resolved`. We detect if `systemd-resolved` is running, and will automatically use the `/etc/resolv.conf` file with the correct upstream nameservers (which is located at `/run/systemd/resolve/resolv.conf`).
|
||||
|
||||
Use the following command to check the upstream nameservers used by the kubedns container:
|
||||
|
||||
```
|
||||
kubectl -n kube-system get pods -l k8s-app=kube-dns --no-headers -o custom-columns=NAME:.metadata.name,HOSTIP:.status.hostIP | while read pod host; do echo "Pod ${pod} on host ${host}"; kubectl -n kube-system exec $pod -c kubedns cat /etc/resolv.conf; done
|
||||
```
|
||||
|
||||
Example output:
|
||||
```
|
||||
Pod kube-dns-667c7cb9dd-z4dsf on host x.x.x.x
|
||||
nameserver 1.1.1.1
|
||||
nameserver 8.8.4.4
|
||||
```
|
||||
|
||||
If the output shows an address in the loopback range (`127.0.0.0/8`), you can correct this in two ways:
|
||||
|
||||
* Make sure the correct nameservers are listed in `/etc/resolv.conf` on your nodes in the cluster, please consult your operating system documentation on how to do this. Make sure you execute this before provisioning a cluster, or reboot the nodes after making the modification.
|
||||
* Configure the `kubelet` to use a different file for resolving names, by using `extra_args` as shown below (where `/run/resolvconf/resolv.conf` is the file with the correct nameservers):
|
||||
|
||||
```
|
||||
services:
|
||||
kubelet:
|
||||
extra_args:
|
||||
resolv-conf: "/run/resolvconf/resolv.conf"
|
||||
```
|
||||
|
||||
:::note
|
||||
|
||||
As the `kubelet` is running inside a container, the path for files located in `/etc` and `/usr` are in `/host/etc` and `/host/usr` inside the `kubelet` container.
|
||||
|
||||
:::
|
||||
|
||||
See Editing Cluster as YAML how to apply this change. When the provisioning of the cluster has finished, you have to remove the kube-dns pod to activate the new setting in the pod:
|
||||
|
||||
```
|
||||
kubectl delete pods -n kube-system -l k8s-app=kube-dns
|
||||
pod "kube-dns-5fd74c7488-6pwsf" deleted
|
||||
```
|
||||
|
||||
Try to resolve name again using [Check if domain names are resolving](#check-if-domain-names-are-resolving).
|
||||
|
||||
If you want to check the kube-dns configuration in your cluster (for example, to check if there are different upstream nameservers configured), you can run the following command to list the kube-dns configuration:
|
||||
|
||||
```
|
||||
kubectl -n kube-system get configmap kube-dns -o go-template='{{range $key, $value := .data}}{{ $key }}{{":"}}{{ $value }}{{"\n"}}{{end}}'
|
||||
```
|
||||
|
||||
Example output:
|
||||
```
|
||||
upstreamNameservers:["1.1.1.1"]
|
||||
```
|
||||
+33
@@ -0,0 +1,33 @@
|
||||
---
|
||||
title: Rotation of Expired Webhook Certificates
|
||||
---
|
||||
|
||||
<head>
|
||||
<link rel="canonical" href="https://ranchermanager.docs.rancher.com/troubleshooting/other-troubleshooting-tips/expired-webhook-certificate-rotation"/>
|
||||
</head>
|
||||
|
||||
For Rancher versions that have `rancher-webhook` installed, certain versions created certificates that will expire after one year. It will be necessary for you to rotate your webhook certificate if the certificate did not renew.
|
||||
|
||||
In Rancher v2.6.3 and up, rancher-webhook deployments will automatically renew their TLS certificate when it is within 30 or fewer days of its expiration date. If you are using v2.6.2 or below, there are two methods to work around this issue:
|
||||
|
||||
## 1. Users with Cluster Access, Run the Following Commands:
|
||||
|
||||
```
|
||||
kubectl delete secret -n cattle-system cattle-webhook-tls
|
||||
kubectl delete mutatingwebhookconfigurations.admissionregistration.k8s.io --ignore-not-found=true rancher.cattle.io
|
||||
kubectl delete pod -n cattle-system -l app=rancher-webhook
|
||||
```
|
||||
|
||||
## 2. Users with No Cluster Access Via `kubectl`:
|
||||
|
||||
1. Delete the `cattle-webhook-tls` secret in the `cattle-system` namespace in the local cluster.
|
||||
|
||||
2. Delete the `rancher.cattle.io` mutating webhook
|
||||
|
||||
3. Delete the `rancher-webhook` pod in the `cattle-system` namespace in the local cluster.
|
||||
|
||||
:::note
|
||||
|
||||
The webhook certificate expiration issue is not specific to `cattle-webhook-tls` as listed in the examples. You will fill in your expired certificate secret accordingly.
|
||||
|
||||
:::
|
||||
+330
@@ -0,0 +1,330 @@
|
||||
---
|
||||
title: Kubernetes Resources
|
||||
---
|
||||
|
||||
<head>
|
||||
<link rel="canonical" href="https://ranchermanager.docs.rancher.com/troubleshooting/other-troubleshooting-tips/kubernetes-resources"/>
|
||||
</head>
|
||||
|
||||
The commands/steps listed on this page can be used to check the most important Kubernetes resources and apply to [Rancher Launched Kubernetes](../../how-to-guides/new-user-guides/launch-kubernetes-with-rancher/launch-kubernetes-with-rancher.md) clusters.
|
||||
|
||||
Make sure you configured the correct kubeconfig (for example, `export KUBECONFIG=$PWD/kube_config_cluster.yml` for Rancher HA) or are using the embedded kubectl via the UI.
|
||||
|
||||
|
||||
## Nodes
|
||||
|
||||
### Get nodes
|
||||
|
||||
Run the command below and check the following:
|
||||
|
||||
- All nodes in your cluster should be listed, make sure there is not one missing.
|
||||
- All nodes should have the **Ready** status (if not in **Ready** state, check the `kubelet` container logs on that node using `docker logs kubelet`)
|
||||
- Check if all nodes report the correct version.
|
||||
- Check if OS/Kernel/Docker values are shown as expected (possibly you can relate issues due to upgraded OS/Kernel/Docker)
|
||||
|
||||
|
||||
```
|
||||
kubectl get nodes -o wide
|
||||
```
|
||||
|
||||
Example output:
|
||||
|
||||
```
|
||||
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
|
||||
controlplane-0 Ready controlplane 31m v1.13.5 138.68.188.91 <none> Ubuntu 18.04.2 LTS 4.15.0-47-generic docker://18.9.5
|
||||
etcd-0 Ready etcd 31m v1.13.5 138.68.180.33 <none> Ubuntu 18.04.2 LTS 4.15.0-47-generic docker://18.9.5
|
||||
worker-0 Ready worker 30m v1.13.5 139.59.179.88 <none> Ubuntu 18.04.2 LTS 4.15.0-47-generic docker://18.9.5
|
||||
```
|
||||
|
||||
### Get node conditions
|
||||
|
||||
Run the command below to list nodes with [Node Conditions](https://kubernetes.io/docs/concepts/architecture/nodes/#condition)
|
||||
|
||||
```
|
||||
kubectl get nodes -o go-template='{{range .items}}{{$node := .}}{{range .status.conditions}}{{$node.metadata.name}}{{": "}}{{.type}}{{":"}}{{.status}}{{"\n"}}{{end}}{{end}}'
|
||||
```
|
||||
|
||||
Run the command below to list nodes with [Node Conditions](https://kubernetes.io/docs/concepts/architecture/nodes/#condition) that are active that could prevent normal operation.
|
||||
|
||||
```
|
||||
kubectl get nodes -o go-template='{{range .items}}{{$node := .}}{{range .status.conditions}}{{if ne .type "Ready"}}{{if eq .status "True"}}{{$node.metadata.name}}{{": "}}{{.type}}{{":"}}{{.status}}{{"\n"}}{{end}}{{else}}{{if ne .status "True"}}{{$node.metadata.name}}{{": "}}{{.type}}{{": "}}{{.status}}{{"\n"}}{{end}}{{end}}{{end}}{{end}}'
|
||||
```
|
||||
|
||||
Example output:
|
||||
|
||||
```
|
||||
worker-0: DiskPressure:True
|
||||
```
|
||||
|
||||
## Kubernetes leader election
|
||||
|
||||
### Kubernetes Controller Manager leader
|
||||
|
||||
The leader is determined by a leader election process. After the leader has been determined, the leader (`holderIdentity`) is saved in the `kube-controller-manager` endpoint (in this example, `controlplane-0`).
|
||||
|
||||
```
|
||||
kubectl -n kube-system get endpoints kube-controller-manager -o jsonpath='{.metadata.annotations.control-plane\.alpha\.kubernetes\.io/leader}'
|
||||
{"holderIdentity":"controlplane-0_xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx","leaseDurationSeconds":15,"acquireTime":"2018-12-27T08:59:45Z","renewTime":"2018-12-27T09:44:57Z","leaderTransitions":0}>
|
||||
```
|
||||
|
||||
### Kubernetes Scheduler leader
|
||||
|
||||
The leader is determined by a leader election process. After the leader has been determined, the leader (`holderIdentity`) is saved in the `kube-scheduler` endpoint (in this example, `controlplane-0`).
|
||||
|
||||
```
|
||||
kubectl -n kube-system get endpoints kube-scheduler -o jsonpath='{.metadata.annotations.control-plane\.alpha\.kubernetes\.io/leader}'
|
||||
{"holderIdentity":"controlplane-0_xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx","leaseDurationSeconds":15,"acquireTime":"2018-12-27T08:59:45Z","renewTime":"2018-12-27T09:44:57Z","leaderTransitions":0}>
|
||||
```
|
||||
|
||||
## Ingress Controller
|
||||
|
||||
The default Ingress Controller is Traefik and is deployed as a DaemonSet in the `kube-system` namespace. The pods are only scheduled to nodes with the `worker` role.
|
||||
|
||||
Check if the pods are running on all nodes:
|
||||
|
||||
```
|
||||
kubectl -n kube-system get pods -o wide
|
||||
```
|
||||
|
||||
Example RKE2 output:
|
||||
|
||||
```
|
||||
kubectl -n kube-system get pods -o wide
|
||||
NAME READY STATUS RESTARTS AGE IP NODE
|
||||
local-path-provisioner-xxxxxxxxxx-xxxxx 1/1 Running 0 17m x.x.x.x worker-1
|
||||
rke2-traefik-xxxxxxxxxx-xxxxx 1/1 Running 0 14m x.x.x.x worker-1
|
||||
svclb-rke2-traefik-xxxxxxxx-xxxxx 1/1 Running 0 13m x.x.x.x worker-0
|
||||
...
|
||||
```
|
||||
|
||||
Example K3s output:
|
||||
|
||||
```
|
||||
kubectl -n kube-system get pods -o wide
|
||||
NAME READY STATUS RESTARTS AGE IP NODE
|
||||
local-path-provisioner-xxxxxxxxxx-xxxxx 1/1 Running 0 17m x.x.x.x worker-1
|
||||
traefik-xxxxxxxxxx-xxxxx 1/1 Running 0 14m x.x.x.x worker-1
|
||||
svclb-traefik-xxxxxxxx-xxxxx 1/1 Running 0 13m x.x.x.x worker-0
|
||||
...
|
||||
```
|
||||
|
||||
If a pod is unable to run (Status is not **Running**, Ready status is not showing `1/1` or you see a high count of Restarts), check the pod details, logs and namespace events.
|
||||
|
||||
### Pod details
|
||||
|
||||
RKE2 example:
|
||||
|
||||
```
|
||||
kubectl -n kube-system describe pods -l app.kubernetes.io/name=rke2-traefik
|
||||
```
|
||||
|
||||
K3s example:
|
||||
|
||||
```
|
||||
kubectl -n kube-system describe pods -l app.kubernetes.io/name=traefik
|
||||
```
|
||||
|
||||
### Pod container logs
|
||||
|
||||
The below command can show the logs of all the pods labeled "app.kubernetes.io/name=rke2-traefik" if using RKE2 or "app.kubernetes.io/name=traefik" if using K3s, but it will display only 10 lines of log because of the restrictions of the `kubectl logs` command. Refer to `--tail` of `kubectl logs -h` for more information.
|
||||
|
||||
RKE2 example:
|
||||
|
||||
```
|
||||
kubectl -n kube-system logs -l app.kubernetes.io/name=rke2-traefik
|
||||
```
|
||||
|
||||
K3s example:
|
||||
|
||||
```
|
||||
kubectl -n kube-system logs -l app.kubernetes.io/name=traefik
|
||||
```
|
||||
|
||||
If the full log is needed, specify the pod name in the trailing command:
|
||||
|
||||
```
|
||||
kubectl -n kube-system logs <pod name>
|
||||
```
|
||||
|
||||
### Namespace events
|
||||
|
||||
```
|
||||
kubectl -n kube-system get events
|
||||
```
|
||||
|
||||
### Debug logging
|
||||
|
||||
To enable debug logging:
|
||||
|
||||
RKE2 example:
|
||||
|
||||
```
|
||||
cat <<EOF | kubectl apply -f -
|
||||
apiVersion: helm.cattle.io/v1
|
||||
kind: HelmChartConfig
|
||||
metadata:
|
||||
name: rke2-traefik
|
||||
namespace: kube-system
|
||||
spec:
|
||||
valuesContent: |-
|
||||
additionalArguments:
|
||||
- "--log.level=DEBUG"
|
||||
EOF
|
||||
```
|
||||
|
||||
K3s example:
|
||||
|
||||
```
|
||||
cat <<EOF | kubectl apply -f -
|
||||
apiVersion: helm.cattle.io/v1
|
||||
kind: HelmChartConfig
|
||||
metadata:
|
||||
name: traefik
|
||||
namespace: kube-system
|
||||
spec:
|
||||
valuesContent: |-
|
||||
logs:
|
||||
general:
|
||||
level: "DEBUG"
|
||||
EOF
|
||||
```
|
||||
|
||||
### Check configuration
|
||||
|
||||
Retrieve generated configuration in each pod:
|
||||
|
||||
RKE2 example for manual Traefik configuration file check:
|
||||
|
||||
```
|
||||
kubectl exec -n kube-system pod/traefik-xxxxxxxxx-xxxxx -- cat /var/lib/rancher/rke2/server/manifests/rke2-traefik-config.yaml
|
||||
```
|
||||
|
||||
K3s example for manual Traefik configuration file check:
|
||||
|
||||
```
|
||||
kubectl exec -n kube-system pod/traefik-xxxxxxxxx-xxxxx -- cat /var/lib/rancher/k3s/server/manifests/k3s-traefik-config.yaml
|
||||
```
|
||||
|
||||
RKE2/K3s example for Traefik CLI argument configuration check:
|
||||
|
||||
```
|
||||
kubectl get pod traefik-xxxxxxxxx-xxxxx -n kube-system -o jsonpath='{.spec.containers[0].args}'
|
||||
```
|
||||
|
||||
## Rancher agents
|
||||
|
||||
Communication to the cluster (Kubernetes API via `cattle-cluster-agent`) and communication to the nodes (cluster provisioning via `cattle-node-agent`) is done through Rancher agents.
|
||||
|
||||
#### cattle-node-agent
|
||||
|
||||
Check if the cattle-node-agent pods are present on each node, have status **Running** and don't have a high count of Restarts:
|
||||
|
||||
```
|
||||
kubectl -n cattle-system get pods -l app=cattle-agent -o wide
|
||||
```
|
||||
|
||||
Example output:
|
||||
|
||||
```
|
||||
NAME READY STATUS RESTARTS AGE IP NODE
|
||||
cattle-node-agent-4gc2p 1/1 Running 0 2h x.x.x.x worker-1
|
||||
cattle-node-agent-8cxkk 1/1 Running 0 2h x.x.x.x etcd-1
|
||||
cattle-node-agent-kzrlg 1/1 Running 0 2h x.x.x.x etcd-0
|
||||
cattle-node-agent-nclz9 1/1 Running 0 2h x.x.x.x controlplane-0
|
||||
cattle-node-agent-pwxp7 1/1 Running 0 2h x.x.x.x worker-0
|
||||
cattle-node-agent-t5484 1/1 Running 0 2h x.x.x.x controlplane-1
|
||||
cattle-node-agent-t8mtz 1/1 Running 0 2h x.x.x.x etcd-2
|
||||
```
|
||||
|
||||
Check logging of a specific cattle-node-agent pod or all cattle-node-agent pods:
|
||||
|
||||
```
|
||||
kubectl -n cattle-system logs -l app=cattle-agent
|
||||
```
|
||||
|
||||
#### cattle-cluster-agent
|
||||
|
||||
Check if the cattle-cluster-agent pod is present in the cluster, has status **Running** and doesn't have a high count of Restarts:
|
||||
|
||||
```
|
||||
kubectl -n cattle-system get pods -l app=cattle-cluster-agent -o wide
|
||||
```
|
||||
|
||||
Example output:
|
||||
|
||||
```
|
||||
NAME READY STATUS RESTARTS AGE IP NODE
|
||||
cattle-cluster-agent-54d7c6c54d-ht9h4 1/1 Running 0 2h x.x.x.x worker-1
|
||||
```
|
||||
|
||||
Check logging of cattle-cluster-agent pod:
|
||||
|
||||
```
|
||||
kubectl -n cattle-system logs -l app=cattle-cluster-agent
|
||||
```
|
||||
|
||||
## Jobs and Pods
|
||||
|
||||
### Check that pods or jobs have status **Running**/**Completed**
|
||||
|
||||
To check, run the command:
|
||||
|
||||
```
|
||||
kubectl get pods --all-namespaces
|
||||
```
|
||||
|
||||
If a pod is not in **Running** state, you can dig into the root cause by running:
|
||||
|
||||
### Describe pod
|
||||
|
||||
```
|
||||
kubectl describe pod POD_NAME -n NAMESPACE
|
||||
```
|
||||
|
||||
### Pod container logs
|
||||
|
||||
```
|
||||
kubectl logs POD_NAME -n NAMESPACE
|
||||
```
|
||||
|
||||
If a job is not in **Completed** state, you can dig into the root cause by running:
|
||||
|
||||
### Describe job
|
||||
|
||||
```
|
||||
kubectl describe job JOB_NAME -n NAMESPACE
|
||||
```
|
||||
|
||||
### Logs from the containers of pods of the job
|
||||
|
||||
```
|
||||
kubectl logs -l job-name=JOB_NAME -n NAMESPACE
|
||||
```
|
||||
|
||||
### Evicted pods
|
||||
|
||||
Pods can be evicted based on [eviction signals](https://kubernetes.io/docs/tasks/administer-cluster/out-of-resource/#eviction-policy).
|
||||
|
||||
Retrieve a list of evicted pods (podname and namespace):
|
||||
|
||||
```
|
||||
kubectl get pods --all-namespaces -o go-template='{{range .items}}{{if eq .status.phase "Failed"}}{{if eq .status.reason "Evicted"}}{{.metadata.name}}{{" "}}{{.metadata.namespace}}{{"\n"}}{{end}}{{end}}{{end}}'
|
||||
```
|
||||
|
||||
To delete all evicted pods:
|
||||
|
||||
```
|
||||
kubectl get pods --all-namespaces -o go-template='{{range .items}}{{if eq .status.phase "Failed"}}{{if eq .status.reason "Evicted"}}{{.metadata.name}}{{" "}}{{.metadata.namespace}}{{"\n"}}{{end}}{{end}}{{end}}' | while read epod enamespace; do kubectl -n $enamespace delete pod $epod; done
|
||||
```
|
||||
|
||||
Retrieve a list of evicted pods, scheduled node and the reason:
|
||||
|
||||
```
|
||||
kubectl get pods --all-namespaces -o go-template='{{range .items}}{{if eq .status.phase "Failed"}}{{if eq .status.reason "Evicted"}}{{.metadata.name}}{{" "}}{{.metadata.namespace}}{{"\n"}}{{end}}{{end}}{{end}}' | while read epod enamespace; do kubectl -n $enamespace get pod $epod -o=custom-columns=NAME:.metadata.name,NODE:.spec.nodeName,MSG:.status.message; done
|
||||
```
|
||||
|
||||
### Job does not complete
|
||||
|
||||
If you have enabled Istio, and you are having issues with a Job you deployed not completing, you will need to add an annotation to your pod using [these steps.](../../how-to-guides/advanced-user-guides/istio-setup-guide/enable-istio-in-namespace.md)
|
||||
|
||||
Since Istio Sidecars run indefinitely, a Job cannot be considered complete even after its task has completed. This is a temporary workaround and will disable Istio for any traffic to/from the annotated Pod. Keep in mind this may not allow you to continue to use a Job for integration testing, as the Job will not have access to the service mesh.
|
||||
@@ -0,0 +1,96 @@
|
||||
---
|
||||
title: Logging
|
||||
---
|
||||
|
||||
<head>
|
||||
<link rel="canonical" href="https://ranchermanager.docs.rancher.com/troubleshooting/other-troubleshooting-tips/logging"/>
|
||||
</head>
|
||||
|
||||
## Log Levels
|
||||
|
||||
The following log levels are used in Rancher:
|
||||
|
||||
| Name | Description |
|
||||
|---------|-------------|
|
||||
| `info` | Logs informational messages. This is the default log level. |
|
||||
| `debug` | Logs more detailed messages that can be used to debug. |
|
||||
| `trace` | Logs very detailed messages on internal functions. This is very verbose and can contain sensitive information. |
|
||||
|
||||
### How to Configure a Log Level
|
||||
|
||||
#### Kubernetes Install
|
||||
|
||||
* Configure debug log level
|
||||
|
||||
```
|
||||
$ KUBECONFIG=./kube_config_cluster.yml
|
||||
$ kubectl -n cattle-system get pods -l app=rancher --no-headers -o custom-columns=name:.metadata.name | while read rancherpod; do kubectl -n cattle-system exec $rancherpod -c rancher -- loglevel --set debug; done
|
||||
OK
|
||||
OK
|
||||
OK
|
||||
$ kubectl -n cattle-system logs -l app=rancher -c rancher
|
||||
```
|
||||
|
||||
* Configure info log level
|
||||
|
||||
```
|
||||
$ KUBECONFIG=./kube_config_cluster.yml
|
||||
$ kubectl -n cattle-system get pods -l app=rancher --no-headers -o custom-columns=name:.metadata.name | while read rancherpod; do kubectl -n cattle-system exec $rancherpod -c rancher -- loglevel --set info; done
|
||||
OK
|
||||
OK
|
||||
OK
|
||||
```
|
||||
|
||||
#### Docker Install
|
||||
|
||||
* Configure debug log level
|
||||
|
||||
```
|
||||
$ docker exec -ti <container_id> loglevel --set debug
|
||||
OK
|
||||
$ docker logs -f <container_id>
|
||||
```
|
||||
|
||||
* Configure info log level
|
||||
|
||||
```
|
||||
$ docker exec -ti <container_id> loglevel --set info
|
||||
OK
|
||||
```
|
||||
|
||||
## Rancher Machine Debug Logs
|
||||
If you need to troubleshoot the creation of objects in your infrastructure provider of choice, `rancher-machine`
|
||||
debug logs might be helpful to you.
|
||||
|
||||
It's possible to enable debug logs for `rancher-machine` by setting environment variables when launching Rancher.
|
||||
|
||||
The `CATTLE_WHITELIST_ENVVARS` environment variable allows users to whitelist specific environment variables to be
|
||||
passed down to `rancher-machine` during provisioning.
|
||||
|
||||
The `MACHINE_DEBUG` variable enables debug logs in `rancher-machine`.
|
||||
|
||||
Thus, by setting `MACHINE_DEBUG=true` and adding `MACHINE_DEBUG` to the default list of variables in
|
||||
`CATTLE_WHITELIST_ENVVARS` (e.g. `CATTLE_WHITELIST_ENVVARS=HTTP_PROXY,HTTPS_PROXY,NO_PROXY,MACHINE_DEBUG`) it is
|
||||
possible to enable debug logs in `rancher-machine` when provisioning RKE1, RKE2 and k3s clusters.
|
||||
|
||||
:::caution
|
||||
|
||||
Just like the `trace` log level above, `rancher-machine` debug logs can contain sensitive information.
|
||||
|
||||
:::
|
||||
|
||||
|
||||
## Cattle-cluster-agent Debug Logs
|
||||
|
||||
The `cattle-cluster-agent` log levels can be set when you initialize downstream clusters.
|
||||
|
||||
When you create a cluster under **Cluster Configuration > Agent Environment Vars** you can set variables to define the log level.
|
||||
- Trace-level logging: Set `CATTLE_TRACE` or `RANCHER_TRACE` to `true`
|
||||
|
||||
- Debug-level logging: Set `CATTLE_DEBUG` or `RANCHER_DEBUG` to `true`
|
||||
|
||||
:::caution
|
||||
|
||||
The `cattle-cluster-agent` debug logs may contain sensitive information.
|
||||
|
||||
:::
|
||||
@@ -0,0 +1,117 @@
|
||||
---
|
||||
title: Networking
|
||||
---
|
||||
|
||||
<head>
|
||||
<link rel="canonical" href="https://ranchermanager.docs.rancher.com/troubleshooting/other-troubleshooting-tips/networking"/>
|
||||
</head>
|
||||
|
||||
The commands/steps listed on this page can be used to check networking related issues in your cluster.
|
||||
|
||||
Make sure you configured the correct kubeconfig (for example, `export KUBECONFIG=$PWD/kube_config_cluster.yml` for Rancher HA) or are using the embedded kubectl via the UI.
|
||||
|
||||
## Double Check if All the Required Ports are Opened in Your (Host) Firewall
|
||||
|
||||
Double check if all the [required ports](../../how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/node-requirements-for-rancher-managed-clusters.md#networking-requirements) are opened in your (host) firewall. The overlay network uses UDP in comparison to all other required ports which are TCP.
|
||||
|
||||
## Check if your downstream node can communicate to Rancher Manager
|
||||
|
||||
Rancher components with HTTP endpoints generally contain a `ping` liveness probe which you can use to test connectivity. Replace the `$RANCHER_URL` as appropriate and run the following from a node to check that it has connectivity to Rancher Manager's servers in the `local` cluster. If successful, it should return `pong`.
|
||||
|
||||
```
|
||||
curl -k https://$RANCHER_URL/ping
|
||||
```
|
||||
|
||||
## Check if Overlay Network is Functioning Correctly
|
||||
|
||||
The pod can be scheduled to any of the hosts you used for your cluster, but that means that the NGINX ingress controller needs to be able to route the request from `NODE_1` to `NODE_2`. This happens over the overlay network. If the overlay network is not functioning, you will experience intermittent TCP/HTTP connection failures due to the NGINX ingress controller not being able to route to the pod.
|
||||
|
||||
To test the overlay network, you can launch the following `DaemonSet` definition. This will run a `swiss-army-knife` container on every host (image was developed by Rancher engineers and can be found here: https://github.com/rancherlabs/swiss-army-knife), which we will use to run a `ping` test between containers on all hosts.
|
||||
|
||||
:::caution
|
||||
|
||||
The `swiss-army-knife` container does not support Windows nodes. It also [does not support ARM nodes](https://github.com/leodotcloud/swiss-army-knife/issues/18), such as a Raspberry Pi. When the test encounters incompatible nodes, this is recorded in the pod logs as an error message, such as `exec user process caused: exec format error` for ARM nodes, or `ImagePullBackOff (Back-off pulling image "rancherlabs/swiss-army-knife)` for Windows nodes.
|
||||
|
||||
:::
|
||||
|
||||
1. Save the following file as `overlaytest.yml`
|
||||
|
||||
```
|
||||
apiVersion: apps/v1
|
||||
kind: DaemonSet
|
||||
metadata:
|
||||
name: overlaytest
|
||||
spec:
|
||||
selector:
|
||||
matchLabels:
|
||||
name: overlaytest
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
name: overlaytest
|
||||
spec:
|
||||
tolerations:
|
||||
- operator: Exists
|
||||
containers:
|
||||
- image: rancherlabs/swiss-army-knife
|
||||
imagePullPolicy: Always
|
||||
name: overlaytest
|
||||
command: ["sleep", "infinity"]
|
||||
terminationMessagePath: /dev/termination-log
|
||||
|
||||
```
|
||||
|
||||
2. Launch it using `kubectl create -f overlaytest.yml`
|
||||
3. Wait until `kubectl rollout status ds/overlaytest -w` returns: `daemon set "overlaytest" successfully rolled out`.
|
||||
4. Run the following script, from the same location. It will have each `overlaytest` container on every host ping each other:
|
||||
```
|
||||
#!/bin/bash
|
||||
echo "=> Start network overlay test"
|
||||
kubectl get pods -l name=overlaytest -o jsonpath='{range .items[*]}{@.metadata.name}{" "}{@.spec.nodeName}{"\n"}{end}' |
|
||||
while read spod shost
|
||||
do kubectl get pods -l name=overlaytest -o jsonpath='{range .items[*]}{@.status.podIP}{" "}{@.spec.nodeName}{"\n"}{end}' |
|
||||
while read tip thost
|
||||
do kubectl --request-timeout='10s' exec $spod -c overlaytest -- /bin/sh -c "ping -c2 $tip > /dev/null 2>&1"
|
||||
RC=$?
|
||||
if [ $RC -ne 0 ]
|
||||
then echo FAIL: $spod on $shost cannot reach pod IP $tip on $thost
|
||||
else echo $shost can reach $thost
|
||||
fi
|
||||
done
|
||||
done
|
||||
echo "=> End network overlay test"
|
||||
```
|
||||
|
||||
5. When this command has finished running, it will output the state of each route:
|
||||
|
||||
```
|
||||
=> Start network overlay test
|
||||
Error from server (NotFound): pods "wk2" not found
|
||||
FAIL: overlaytest-5bglp on wk2 cannot reach pod IP 10.42.7.3 on wk2
|
||||
Error from server (NotFound): pods "wk2" not found
|
||||
FAIL: overlaytest-5bglp on wk2 cannot reach pod IP 10.42.0.5 on cp1
|
||||
Error from server (NotFound): pods "wk2" not found
|
||||
FAIL: overlaytest-5bglp on wk2 cannot reach pod IP 10.42.2.12 on wk1
|
||||
command terminated with exit code 1
|
||||
FAIL: overlaytest-v4qkl on cp1 cannot reach pod IP 10.42.7.3 on wk2
|
||||
cp1 can reach cp1
|
||||
cp1 can reach wk1
|
||||
command terminated with exit code 1
|
||||
FAIL: overlaytest-xpxwp on wk1 cannot reach pod IP 10.42.7.3 on wk2
|
||||
wk1 can reach cp1
|
||||
wk1 can reach wk1
|
||||
=> End network overlay test
|
||||
```
|
||||
If you see error in the output, there is some issue with the route between the pods on the two hosts. In the above output the node `wk2` has no connectivity over the overlay network. This could be because the [required ports](../../how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/node-requirements-for-rancher-managed-clusters.md#networking-requirements) for overlay networking are not opened for `wk2`.
|
||||
6. You can now clean up the DaemonSet by running `kubectl delete ds/overlaytest`.
|
||||
|
||||
|
||||
### Check if MTU is Correctly Configured on Hosts and on Peering/Tunnel Appliances/Devices
|
||||
|
||||
When the MTU is incorrectly configured (either on hosts running Rancher, nodes in created/imported clusters or on appliances/devices in between), error messages will be logged in Rancher and in the agents, similar to:
|
||||
|
||||
* `websocket: bad handshake`
|
||||
* `Failed to connect to proxy`
|
||||
* `read tcp: i/o timeout`
|
||||
|
||||
See [Google Cloud VPN: MTU Considerations](https://cloud.google.com/vpn/docs/concepts/mtu-considerations#gateway_mtu_vs_system_mtu) for an example how to configure MTU correctly when using Google Cloud VPN between Rancher and cluster nodes.
|
||||
@@ -0,0 +1,112 @@
|
||||
---
|
||||
title: Rancher HA
|
||||
---
|
||||
|
||||
<head>
|
||||
<link rel="canonical" href="https://ranchermanager.docs.rancher.com/troubleshooting/other-troubleshooting-tips/rancher-ha"/>
|
||||
</head>
|
||||
|
||||
The commands/steps listed on this page can be used to check your Rancher Kubernetes Installation.
|
||||
|
||||
Make sure you configured the correct kubeconfig (for example, `export KUBECONFIG=$PWD/kube_config_cluster.yml`).
|
||||
|
||||
## Check Rancher Pods
|
||||
|
||||
Rancher pods are deployed as a Deployment in the `cattle-system` namespace.
|
||||
|
||||
Check if the pods are running on all nodes:
|
||||
|
||||
```
|
||||
kubectl -n cattle-system get pods -l app=rancher -o wide
|
||||
```
|
||||
|
||||
Example output:
|
||||
|
||||
```
|
||||
NAME READY STATUS RESTARTS AGE IP NODE
|
||||
rancher-7dbd7875f7-n6t5t 1/1 Running 0 8m x.x.x.x x.x.x.x
|
||||
rancher-7dbd7875f7-qbj5k 1/1 Running 0 8m x.x.x.x x.x.x.x
|
||||
rancher-7dbd7875f7-qw7wb 1/1 Running 0 8m x.x.x.x x.x.x.x
|
||||
```
|
||||
|
||||
If a pod is unable to run (Status is not **Running**, Ready status is not showing `1/1` or you see a high count of Restarts), check the pod details, logs and namespace events.
|
||||
|
||||
### Pod Details
|
||||
|
||||
```
|
||||
kubectl -n cattle-system describe pods -l app=rancher
|
||||
```
|
||||
|
||||
### Pod Container Logs
|
||||
|
||||
```
|
||||
kubectl -n cattle-system logs -l app=rancher
|
||||
```
|
||||
|
||||
### Namespace Events
|
||||
|
||||
```
|
||||
kubectl -n cattle-system get events
|
||||
```
|
||||
|
||||
## Check Ingress
|
||||
|
||||
Ingress should have the correct `HOSTS` (showing the configured FQDN) and `ADDRESS` (host address(es) it will be routed to).
|
||||
|
||||
```
|
||||
kubectl -n cattle-system get ingress
|
||||
```
|
||||
|
||||
Example output:
|
||||
|
||||
```
|
||||
NAME HOSTS ADDRESS PORTS AGE
|
||||
rancher rancher.yourdomain.com x.x.x.x,x.x.x.x,x.x.x.x 80, 443 2m
|
||||
```
|
||||
|
||||
## Check Ingress Controller Logs
|
||||
|
||||
When accessing your configured Rancher FQDN does not show you the UI, check the ingress controller logging to see what happens when you try to access Rancher:
|
||||
|
||||
```
|
||||
kubectl -n traefik logs -l app=traefik
|
||||
```
|
||||
|
||||
## Leader Election
|
||||
|
||||
The leader is determined by a leader election process. After the leader has been determined, the leader (`holderIdentity`) is saved in the `cattle-controllers` Lease in the `kube-system` namespace (in this example, `rancher-dbc7ff869-gvg6k`).
|
||||
|
||||
```
|
||||
kubectl -n kube-system get lease cattle-controllers
|
||||
```
|
||||
|
||||
Example output:
|
||||
|
||||
```
|
||||
NAME HOLDER AGE
|
||||
cattle-controllers rancher-dbc7ff869-gvg6k 6h10m
|
||||
```
|
||||
|
||||
### Configuration
|
||||
|
||||
_Available as of Rancher 2.8.3_
|
||||
|
||||
If the Kubernetes API experiences latency, the Rancher replica holding the leader lock may not be able to renew the lease before the lease becomes invalid, which can be observed in the Rancher logs:
|
||||
```
|
||||
E0629 04:13:07.293461 34 leaderelection.go:364] Failed to update lock: Put "https://172.17.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/cattle-controllers?timeout=15m0s": context deadline exceeded
|
||||
I0629 04:13:07.293594 34 leaderelection.go:280] failed to renew lease kube-system/cattle-controllers: timed out waiting for the condition
|
||||
...
|
||||
2024/06/29 04:13:10 [FATAL] leaderelection lost for cattle-controllers
|
||||
```
|
||||
|
||||
To mitigate this, you can set environment variables in the `rancher` Deployment to modify the default parameters for leader election:
|
||||
- `CATTLE_ELECTION_LEASE_DURATION`: The [lease duration](https://pkg.go.dev/k8s.io/client-go/tools/leaderelection#LeaderElectionConfig.LeaseDuration). The default value is 45s.
|
||||
- `CATTLE_ELECTION_RENEW_DEADLINE`: The [renew deadline](https://pkg.go.dev/k8s.io/client-go/tools/leaderelection#LeaderElectionConfig.RenewDeadline). The default value is 30s.
|
||||
- `CATTLE_ELECTION_RETRY_PERIOD`: The [retry period](https://pkg.go.dev/k8s.io/client-go/tools/leaderelection#LeaderElectionConfig.RetryPeriod). The default value is 2s.
|
||||
|
||||
Example:
|
||||
```
|
||||
kubectl -n cattle-system set env deploy/rancher CATTLE_ELECTION_LEASE_DURATION=2m CATTLE_ELECTION_RENEW_DEADLINE=90s CATTLE_ELECTION_RETRY_PERIOD=10s
|
||||
```
|
||||
This will temporarily increase the lease duration, renew deadline and retry period to 120, 90 and 10 seconds respectively.
|
||||
Alternatively, in order to make such changes permanent, these environment variables can be set by [using Helm values](../../getting-started/installation-and-upgrade/installation-references/helm-chart-options.md#setting-extra-environment-variables) instead.
|
||||
+71
@@ -0,0 +1,71 @@
|
||||
---
|
||||
title: Registered Clusters
|
||||
---
|
||||
|
||||
<head>
|
||||
<link rel="canonical" href="https://ranchermanager.docs.rancher.com/troubleshooting/other-troubleshooting-tips/registered-clusters"/>
|
||||
</head>
|
||||
|
||||
The commands/steps listed on this page can be used to check clusters that you are registering or that are registered in Rancher.
|
||||
|
||||
Make sure you configured the correct kubeconfig (for example, `export KUBECONFIG=$PWD/kubeconfig_from_imported_cluster.yml`)
|
||||
|
||||
## Rancher Agents
|
||||
|
||||
Communication to the cluster (Kubernetes API via cattle-cluster-agent) and communication to the nodes is done through Rancher agents.
|
||||
|
||||
If the cattle-cluster-agent cannot connect to the configured `server-url`, the cluster will remain in **Pending** state, showing `Waiting for full cluster configuration`.
|
||||
|
||||
### cattle-node-agent
|
||||
|
||||
:::note
|
||||
|
||||
cattle-node-agents are only present in clusters created in Rancher with RKE.
|
||||
|
||||
:::
|
||||
|
||||
Check if the cattle-node-agent pods are present on each node, have status **Running** and don't have a high count of Restarts:
|
||||
|
||||
```
|
||||
kubectl -n cattle-system get pods -l app=cattle-agent -o wide
|
||||
```
|
||||
|
||||
Example output:
|
||||
|
||||
```
|
||||
NAME READY STATUS RESTARTS AGE IP NODE
|
||||
cattle-node-agent-4gc2p 1/1 Running 0 2h x.x.x.x worker-1
|
||||
cattle-node-agent-8cxkk 1/1 Running 0 2h x.x.x.x etcd-1
|
||||
cattle-node-agent-kzrlg 1/1 Running 0 2h x.x.x.x etcd-0
|
||||
cattle-node-agent-nclz9 1/1 Running 0 2h x.x.x.x controlplane-0
|
||||
cattle-node-agent-pwxp7 1/1 Running 0 2h x.x.x.x worker-0
|
||||
cattle-node-agent-t5484 1/1 Running 0 2h x.x.x.x controlplane-1
|
||||
cattle-node-agent-t8mtz 1/1 Running 0 2h x.x.x.x etcd-2
|
||||
```
|
||||
|
||||
Check logging of a specific cattle-node-agent pod or all cattle-node-agent pods:
|
||||
|
||||
```
|
||||
kubectl -n cattle-system logs -l app=cattle-agent
|
||||
```
|
||||
|
||||
### cattle-cluster-agent
|
||||
|
||||
Check if the cattle-cluster-agent pod is present in the cluster, has status **Running** and doesn't have a high count of Restarts:
|
||||
|
||||
```
|
||||
kubectl -n cattle-system get pods -l app=cattle-cluster-agent -o wide
|
||||
```
|
||||
|
||||
Example output:
|
||||
|
||||
```
|
||||
NAME READY STATUS RESTARTS AGE IP NODE
|
||||
cattle-cluster-agent-54d7c6c54d-ht9h4 1/1 Running 0 2h x.x.x.x worker-1
|
||||
```
|
||||
|
||||
Check logging of cattle-cluster-agent pod:
|
||||
|
||||
```
|
||||
kubectl -n cattle-system logs -l app=cattle-cluster-agent
|
||||
```
|
||||
+26
@@ -0,0 +1,26 @@
|
||||
---
|
||||
title: User ID Tracking in Audit Logs
|
||||
---
|
||||
|
||||
<head>
|
||||
<link rel="canonical" href="https://ranchermanager.docs.rancher.com/troubleshooting/other-troubleshooting-tips/user-id-tracking-in-audit-logs"/>
|
||||
</head>
|
||||
|
||||
The following audit logs are used in Rancher to track events occuring on the local and downstream clusters:
|
||||
|
||||
* [Kubernetes Audit Logs](https://rancher.com/docs/rke/latest/en/config-options/audit-log/)
|
||||
* [Rancher API Audit Logs](../../how-to-guides/advanced-user-guides/enable-api-audit-log.md)
|
||||
|
||||
Audit logs in Rancher v2.6 have been enhanced to include the external Identity Provider name (common name of the user in the external Auth provider) in both the Rancher and downstream Kubernetes audit logs.
|
||||
|
||||
Before v2.6, a Rancher Admin could not trace an event from the Rancher audit logs and into the Kubernetes audit logs without knowing the mapping of the external Identity Provider username to the userId (`u-xXXX`) used in Rancher.
|
||||
To know this mapping, the cluster admins needed to have access to Rancher API, UI, and the local management cluster.
|
||||
|
||||
Now with this feature, a downstream cluster admin should be able to look at the Kubernetes audit logs and know which specific external Identity Provider (IDP) user performed an action without needing to view anything in Rancher.
|
||||
If the audit logs are shipped off of the cluster, a user of the logging system should be able to identify the user in the external Identity Provider system.
|
||||
A Rancher Admin should now be able to view Rancher audit logs and follow through to the Kubernetes audit log by using the external Identity Provider username.
|
||||
|
||||
## Feature Description
|
||||
|
||||
- When Kubernetes Audit logs are enabled on the downstream cluster, in each event that is logged, the external Identity Provider's username is now logged for each request, at the "metadata" level.
|
||||
- When you enable Rancher API Audit logs for a Rancher installation, the external Identity Provider's username is also logged now at the `auditLog.level=0` for each request that hits the Rancher API server, including login requests.
|
||||
Reference in New Issue
Block a user