public/rancher-docs
1
0
Fork 0
mirror of https://github.com/rancher/rancher-docs.git synced 2026-09-26 13:08:11 +00:00
Code Issues Packages Projects Releases Wiki Activity

Overlay test updated

Browse Source 
Standardized names and etc. 
Used `operator: Exists` toleration from the rancher overlay test rather than the no schedule and execute from the RKE. 
Added some verbosity
This commit is contained in:
Tejeev
2020-11-17 18:24:10 +00:00
committed by GitHub
parent 43f4eadad3
commit e19ee79d96
1 changed files with 39 additions and 103 deletions
Download Patch File Download Diff File Expand all files Collapse all files
content/rancher/v2.x/en/troubleshooting/networking/_index.md
+39 -103
View File
@@ -14,9 +14,9 @@ Double check if all the [required ports]({{<baseurl>}}/rancher/v2.x/en/cluster-p
The pod can be scheduled to any of the hosts you used for your cluster, but that means that the NGINX ingress controller needs to be able to route the request from `NODE_1` to `NODE_2`. This happens over the overlay network. If the overlay network is not functioning, you will experience intermittent TCP/HTTP connection failures due to the NGINX ingress controller not being able to route to the pod. The pod can be scheduled to any of the hosts you used for your cluster, but that means that the NGINX ingress controller needs to be able to route the request from `NODE_1` to `NODE_2`. This happens over the overlay network. If the overlay network is not functioning, you will experience intermittent TCP/HTTP connection failures due to the NGINX ingress controller not being able to route to the pod.
To test the overlay network, you can launch the following `DaemonSet` definition. This will run a `swiss-army-knife` container on every host (image was developed by Rancher engineers), which we will use to run a `ping` test between containers on all hosts. To test the overlay network, you can launch the following `DaemonSet` definition. This will run a `swiss-army-knife` container on every host (image was developed by Rancher engineers and can be found here: https://github.com/leodotcloud/swiss-army-knife), which we will use to run a `ping` test between containers on all hosts.
1. Save the following file as `ds-overlaytest.yml` 1. Save the following file as `overlaytest.yml`
``` ```
apiVersion: apps/v1 apiVersion: apps/v1
@@ -30,126 +30,62 @@ To test the overlay network, you can launch the following `DaemonSet` definition
template: template:
metadata: metadata:
labels: labels:
name: ds-overlaytest name: overlaytest
spec: spec:
tolerations: tolerations:
- effect: NoExecute - operator: Exists
key: "node-role.kubernetes.io/etcd"
value: "true"
- effect: NoSchedule
key: "node-role.kubernetes.io/controlplane"
value: "true"
containers: containers:
- image: leodotcloud/swiss-army-knife - image: leodotcloud/swiss-army-knife
imagePullPolicy: Always imagePullPolicy: Always
name: overlaytest name: overlaytest
command: ["sh", "-c", "tail -f /dev/null"] command: ["sh", "-c", "tail -f /dev/null"]
terminationMessagePath: /dev/termination-log terminationMessagePath: /dev/termination-log
``` ```
2. Launch it using `kubectl create -f ds-overlaytest.yml` 2. Launch it using `kubectl create -f overlaytest.yml`
3. Wait until `kubectl rollout status ds/overlaytest -w` returns: `daemon set "overlaytest" successfully rolled out`. 3. Wait until `kubectl rollout status ds/overlaytest -w` returns: `daemon set "overlaytest" successfully rolled out`.
4. Run the following command, from the same location, to let each container on every host ping each other (it's a single line bash command). 4. Run the following script, from the same location. It will have each `overlaytest` container on every host ping each other:
``` ```
### Check if overlay network is functioning correctly #!/bin/bash
echo "=> Start network overlay test"
The pod can be scheduled to any of the hosts you used for your cluster, but that means that the NGINX ingress controller needs to be able to route the request from `NODE_1` to `NODE_2`. This happens over the overlay network. If the overlay network is not functioning, you will experience intermittent TCP/HTTP connection failures due to the NGINX ingress controller not being able to route to the pod. kubectl get pods -l name=overlaytest -o jsonpath='{range .items[*]}{@.metadata.name}{" "}{@.spec.nodeName}{"\n"}{end}' |
while read spod shost
To test the overlay network, you can launch the following `DaemonSet` definition. This will run a `swiss-army-knife` container on every host (image was developed by Rancher engineers), which we will use to run a `ping` test between containers on all hosts. do kubectl get pods -l name=overlaytest -o jsonpath='{range .items[*]}{@.status.podIP}{" "}{@.spec.nodeName}{"\n"}{end}' |
while read tip thost
1. Save the following file as `ds-overlaytest.yml` do kubectl --request-timeout='10s' exec $spod -c overlaytest -- /bin/sh -c "ping -c2 $tip > /dev/null 2>&1"
RC=$?
``` if [ $RC -ne 0 ]
apiVersion: apps/v1 then echo FAIL: $spod on $shost cannot reach pod IP $tip on $thost
kind: DaemonSet else echo $shost can reach $thost
metadata: fi
name: overlaytest done
spec: done
selector: echo "=> End network overlay test"
matchLabels:
name: overlaytest
template:
metadata:
labels:
name: ds-overlaytest
spec:
tolerations:
- effect: NoExecute
key: "node-role.kubernetes.io/etcd"
value: "true"
- effect: NoSchedule
key: "node-role.kubernetes.io/controlplane"
value: "true"
containers:
- image: leodotcloud/swiss-army-knife
imagePullPolicy: Always
name: overlaytest
command: ["sh", "-c", "tail -f /dev/null"]
terminationMessagePath: /dev/termination-log
``` ```
2. Launch it using `kubectl create -f ds-overlaytest.yml` 5. When this command has finished running, it will output the state of each route:
3. Wait until `kubectl rollout status ds/overlaytest -w` returns: `daemon set "overlaytest" successfully rolled out`.
4. Run the following command, from the same location, to let each container on every host ping each other (it's a single line bash command).
```
echo "=> Start network overlay test"; kubectl get pods -l name=ds-overlaytest -o jsonpath='{range .items[*]}{@.metadata.name}{" "}{@.spec.nodeName}{"\n"}{end}' | while read spod shost; do kubectl get pods -l name=ds-overlaytest -o jsonpath='{range .items[*]}{@.status.podIP}{" "}{@.spec.nodeName}{"\n"}{end}' | while read tip thost; do kubectl --request-timeout='10s' exec $spod -- /bin/sh -c "ping -c2 $tip > /dev/null 2>&1"; RC=$?; if [ $RC -ne 0 ]; then echo $shost cannot reach $thost; fi; done; done; echo "=> End network overlay test"
```
5. When this command has finished running, the output indicating everything is correct is:
``` ```
=> Start network overlay test => Start network overlay test
Pinging from pod overlaytest-4cpx5 on host NODE1 Error from server (NotFound): pods "wk2" not found
to pod ip 10.42.1.29 on host NODE1 FAIL: overlaytest-5bglp on wk2 cannot reach pod IP 10.42.7.3 on wk2
to pod ip 10.42.2.19 on host NODE2 Error from server (NotFound): pods "wk2" not found
to pod ip 10.42.4.12 on host NODE3 FAIL: overlaytest-5bglp on wk2 cannot reach pod IP 10.42.0.5 on cp1
Pinging from pod overlaytest-vms6w on host NODE2 Error from server (NotFound): pods "wk2" not found
to pod ip 10.42.1.29 on host NODE1 FAIL: overlaytest-5bglp on wk2 cannot reach pod IP 10.42.2.12 on wk1
to pod ip 10.42.2.19 on host NODE2 command terminated with exit code 1
to pod ip 10.42.4.12 on host NODE3 FAIL: overlaytest-v4qkl on cp1 cannot reach pod IP 10.42.7.3 on wk2
cp1 can reach cp1
cp1 can reach wk1
command terminated with exit code 1
FAIL: overlaytest-xpxwp on wk1 cannot reach pod IP 10.42.7.3 on wk2
wk1 can reach cp1
wk1 can reach wk1
=> End network overlay test => End network overlay test
``` ```
If you see error in the output, there is some issue with the route between the pods on the two hosts. In the above output the node `wk2` has no connectivity over the overlay network. This could be because the [required ports]({{<baseurl>}}/rancher/v2.x/en/cluster-provisioning/node-requirements/#networking-requirements/) for overlay networking are not opened for `wk2`.
If you see error in the output, that means that the [required ports]({{<baseurl>}}/rancher/v2.x/en/cluster-provisioning/node-requirements/#networking-requirements/) for overlay networking are not opened between the hosts indicated. 6. You can now clean up the DaemonSet by running `kubectl delete ds/overlaytest`.
If a path fails the overlay test, you will see errors like the following:
```
command terminated with exit code 1
NODE2 cannot reach NODE1
```
Cleanup the DaemonSet by running `kubectl delete ds/overlaytest`.
```
5. When this command has finished running, the output indicating everything is correct is:
```
=> Start network overlay test
=> End network overlay test
```
If you see error in the output, that means that the [required ports]({{<baseurl>}}/rancher/v2.x/en/cluster-provisioning/node-requirements/#networking-requirements/) for overlay networking are not opened between the hosts indicated.
Example error output of a situation where NODE1 had the UDP ports blocked.
```
=> Start network overlay test
command terminated with exit code 1
NODE2 cannot reach NODE1
command terminated with exit code 1
NODE3 cannot reach NODE1
command terminated with exit code 1
NODE1 cannot reach NODE2
command terminated with exit code 1
NODE1 cannot reach NODE3
=> End network overlay test
```
Cleanup the DaemonSet by running `kubectl delete ds/overlaytest`.
### Check if MTU is correctly configured on hosts and on peering/tunnel appliances/devices ### Check if MTU is correctly configured on hosts and on peering/tunnel appliances/devices