diff --git a/content/rancher/v2.x/en/installation/troubleshooting-ha/404-default-backend/_index.md b/content/rancher/v2.x/en/installation/troubleshooting-ha/404-default-backend/_index.md new file mode 100644 index 00000000000..9d606cc4461 --- /dev/null +++ b/content/rancher/v2.x/en/installation/troubleshooting-ha/404-default-backend/_index.md @@ -0,0 +1,28 @@ +--- +title: 404 - default backend +weight: 30 +--- + +To debug issues around this error, you will need to download the command-line tool `kubectl`. See [Install and Set Up kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/) how to download `kubectl` for your platform. + +When you have made changes to `rancher-cluster.yml`, you will have to run `rke remove --config rancher-cluster.yml` to clean the nodes, so it cannot conflict with previous configuration errors. + +### Possible causes + +The nginx ingress controller is not able to serve the configured host in `rancher-cluster.yml`. This should be the FQDN you configured to access Rancher. The logging of the nginx ingress controller will show why it cannot serve the requested host. To view the logs, you can run the following command + +``` +kubectl --kubeconfig kube_config_rancher-cluster.yml logs -l app=ingress-nginx -n ingress-nginx +``` + +Errors + +* `x509: certificate is valid for fqdn, not your_configured_fqdn` + +The used certificates do not contain the correct hostname. Generate new certificates that contain the chosen FQDN to access Rancher and redeploy. + +* `Port 80 is already in use. Please check the flag --http-port` + +There is a process on the node occupying port 80, this port is needed for the nginx ingress controller to route requests to Rancher. You can find the process by running the command: `netstat -plant | grep \:80`. + +Stop/kill the process and redeploy. diff --git a/content/rancher/v2.x/en/installation/troubleshooting-ha/_index.md b/content/rancher/v2.x/en/installation/troubleshooting-ha/_index.md index d7b48ddf8ed..00ce37f6927 100644 --- a/content/rancher/v2.x/en/installation/troubleshooting-ha/_index.md +++ b/content/rancher/v2.x/en/installation/troubleshooting-ha/_index.md @@ -6,6 +6,19 @@ This section contains common errors seen when setting up a High Availability Ins Choose from the following options: +- [Generic troubleshooting]({{< baseurl >}}/rancher/v2.x/en/installation/troubleshooting-ha/generic-troubleshooting) + + In this section, you can find generic ways to debug your Kubernetes cluster. + - [Failed to set up SSH tunneling for host]({{< baseurl >}}/rancher/v2.x/en/installation/troubleshooting-ha/ssh-tunneling) - In this section, you can find all errors related to SSH tunneling when you run the `rke` command to setup your nodes. + In this section, you can find errors related to SSH tunneling when you run the `rke` command to setup your nodes. + +- [Failed to get job complete status]({{< baseurl >}}/rancher/v2.x/en/installation/troubleshooting-ha/job-complete-status) + + In this section, you can find errors related to deploying addons. + +- [404 - default backend]({{< baseurl >}}/rancher/v2.x/en/installation/troubleshooting-ha/404-default-backend) + + In this section, you can find errors related to the `404 - default backend` page that is shown when trying to access Rancher. + diff --git a/content/rancher/v2.x/en/installation/troubleshooting-ha/generic-troubleshooting/_index.md b/content/rancher/v2.x/en/installation/troubleshooting-ha/generic-troubleshooting/_index.md new file mode 100644 index 00000000000..e77ecfa079b --- /dev/null +++ b/content/rancher/v2.x/en/installation/troubleshooting-ha/generic-troubleshooting/_index.md @@ -0,0 +1,60 @@ +--- +title: Generic troubleshooting +weight: 5 +--- + +Below are steps that you can follow to determine what is wrong in your cluster. + +* All nodes should be present and in **Ready** state + +To check, run the command: + +``` +kubectl --kubeconfig kube_config_rancher-cluster.yml get nodes +``` + +If a node is not shown in this output or a node is not in **Ready** state, you can check the logging of the `kubelet` container. Login to the node and run `docker logs kubelet`. + +* All pods/jobs should be in **Running**/**Completed** state + +To check, run the command: + +``` +kubectl --kubeconfig kube_config_rancher-cluster.yml get pods --all-namespaces +``` + +If a pod is not in **Running** state, you can dig into the root cause by running: + +
Describe pod
+ +``` +kubectl --kubeconfig kube_config_rancher-cluster.yml describe pod POD_NAME -n NAMESPACE +``` + +
Pod logs
+ +``` +kubectl --kubeconfig kube_config_rancher-cluster.yml logs POD_NAME -n NAMESPACE +``` + +If a job is not in **Completed** state, you can dig into the root cause by running: + +
Describe job
+ +``` +kubectl --kubeconfig kube_config_rancher-cluster.yml describe job JOB_NAME -n NAMESPACE +``` + +
Logs from the pods of the job
+ +``` +kubectl --kubeconfig kube_config_rancher-cluster.yml logs -l job-name=JOB_NAME -n NAMESPACE +``` + +* List all Kubernetes cluster events + +Kubernetes cluster events are stored, and can be retrieved by running: + +``` +kubectl --kubeconfig kube_config_rancher-cluster.yml get events --all-namespaces +``` diff --git a/content/rancher/v2.x/en/installation/troubleshooting-ha/job-complete-status/_index.md b/content/rancher/v2.x/en/installation/troubleshooting-ha/job-complete-status/_index.md new file mode 100644 index 00000000000..70bd8a3aee4 --- /dev/null +++ b/content/rancher/v2.x/en/installation/troubleshooting-ha/job-complete-status/_index.md @@ -0,0 +1,32 @@ +--- +title: Failed to get job complete status +weight: 20 +--- + +To debug issues around this error, you will need to download the command-line tool `kubectl`. See [Install and Set Up kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/) how to download `kubectl` for your platform. + +When you have made changes to `rancher-cluster.yml`, you will have to run `rke remove --config rancher-cluster.yml` to clean the nodes, so it cannot conflict with previous configuration errors. + +#### Failed to deploy addon execute job [rke-user-includes-addons]: Failed to get job complete status + +* The content in the `addons:` section of your `rancher-cluster.yml` is wrong. You can retrieve the root cause of this issue by running the following command: + +``` +kubectl --kubeconfig kube_config_rancher-cluster.yml logs -l job-name=rke-user-addon-deploy-job -n kube-system +``` + +Usually something is printed along the lines of `error: error converting YAML to JSON: yaml: line 9:`, where `line 9` references to the line number of the addon that is causing issues. + +Things to check + + +Other errors + +* `The Ingress "cattle-ingress-http" is invalid: spec.rules[0].host: Invalid value: "IP": must be a DNS name, not an IP address` + +The host value can only contain a host name, as it is needed by the ingress controller to match the hostname and pass to the correct backend.