From b5da180fc962c62262d17f562903280f73334007 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Manuel=20Sim=C3=B3n=20N=C3=B3voa?= Date: Wed, 15 Apr 2026 14:30:31 +0200 Subject: [PATCH] Docs: Migrate controlplane troubleshooting guide from RKE/Docker to RKE2/K3s/containerd --- .../publish-images.md | 37 +++++++-- .../troubleshooting-controlplane-nodes.md | 78 ++++++++++++++----- .../publish-images.md | 37 +++++++-- .../publish-images.md | 37 +++++++-- .../publish-images.md | 37 +++++++-- .../troubleshooting-controlplane-nodes.md | 78 ++++++++++++++----- .../publish-images.md | 37 +++++++-- .../troubleshooting-controlplane-nodes.md | 78 ++++++++++++++----- .../publish-images.md | 37 +++++++-- .../troubleshooting-controlplane-nodes.md | 78 ++++++++++++++----- 10 files changed, 412 insertions(+), 122 deletions(-) diff --git a/docs/getting-started/installation-and-upgrade/other-installation-methods/air-gapped-helm-cli-install/publish-images.md b/docs/getting-started/installation-and-upgrade/other-installation-methods/air-gapped-helm-cli-install/publish-images.md index 992129fc684..d546b027741 100644 --- a/docs/getting-started/installation-and-upgrade/other-installation-methods/air-gapped-helm-cli-install/publish-images.md +++ b/docs/getting-started/installation-and-upgrade/other-installation-methods/air-gapped-helm-cli-install/publish-images.md @@ -42,7 +42,7 @@ If you will use ARM64 hosts, the registry must support manifests. As of April 20 1. Go to our [releases page,](https://github.com/rancher/rancher/releases) find the Rancher v2.x.x release that you want to install, and click **Assets**. Note: Don't use releases marked `rc` or `Pre-release`, as they are not stable for production environments. -2. From the release's **Assets** section, download the following files, which are required to install Rancher in an air gap environment: +2. From the release's **Assets** section, download the following files, which are required to install Rancher in an air-gap environment: | Release File | Description | | ---------------- | -------------- | @@ -83,16 +83,39 @@ In a Kubernetes Install, if you elect to use the Rancher default self-signed TLS ### 3. Save the images to your workstation -1. Make `rancher-save-images.sh` an executable: - ``` +(Optional) Verify the image list before pulling: + +```bash +wc -l rancher-images.txt +head rancher-images.txt +``` + +1. Make `rancher-save-images.sh` executable: + ```bash chmod +x rancher-save-images.sh ``` 1. Run `rancher-save-images.sh` with the `rancher-images.txt` image list to create a tarball of all the required images: - ```plain + ```bash ./rancher-save-images.sh --image-list ./rancher-images.txt ``` - **Result:** Docker begins pulling the images used for an air gap install. Be patient. This process takes a few minutes. When the process completes, your current directory will output a tarball named `rancher-images.tar.gz`. Check that the output is in the directory. + +(Optional) Specify a custom output file: + +```bash +./rancher-save-images.sh \ + --image-list ./rancher-images.txt \ + --images rancher-images-custom.tar.gz +``` + +**Result:** Docker begins pulling the images required for an air-gap installation. The process may take several minutes. + +1. Verify that the tarball was created: + ```bash + ls -lh rancher-images.tar.gz + ``` + +If some images fail to pull, review the output and retry after resolving any issues. ### 4. Populate the private registry @@ -163,7 +186,7 @@ Your registry must support manifests. As of April 2020, Amazon Elastic Container ./rancher-save-images.ps1 ``` - **Result:** Docker begins pulling the images used for an air gap install. Be patient. This process takes a few minutes. When the process completes, your current directory will output a tarball named `rancher-windows-images.tar.gz`. Check that the output is in the directory. + **Result:** Docker begins pulling the images used for an air-gap install. Be patient. This process takes a few minutes. When the process completes, your current directory will output a tarball named `rancher-windows-images.tar.gz`. Check that the output is in the directory. @@ -273,7 +296,7 @@ The workstation must have Docker 18.02+ in order to support manifests, which are ./rancher-save-images.sh --image-list ./rancher-images.txt ``` -**Result:** Docker begins pulling the images used for an air gap install. Be patient. This process takes a few minutes. When the process completes, your current directory will output a tarball named `rancher-images.tar.gz`. Check that the output is in the directory. +**Result:** Docker begins pulling the images used for an air-gap install. Be patient. This process takes a few minutes. When the process completes, your current directory will output a tarball named `rancher-images.tar.gz`. Check that the output is in the directory. diff --git a/docs/troubleshooting/kubernetes-components/troubleshooting-controlplane-nodes.md b/docs/troubleshooting/kubernetes-components/troubleshooting-controlplane-nodes.md index 84877bc8fab..487506c45cf 100644 --- a/docs/troubleshooting/kubernetes-components/troubleshooting-controlplane-nodes.md +++ b/docs/troubleshooting/kubernetes-components/troubleshooting-controlplane-nodes.md @@ -6,31 +6,55 @@ title: Troubleshooting Controlplane Nodes -This section applies to nodes with the `controlplane` role. +This section applies to nodes with the `controlplane` role in RKE2 and K3s clusters. -## Check if the Controlplane Containers are Running +## Prerequisites -There are three specific containers launched on nodes with the `controlplane` role: +As RKE2 and K3s rely on `containerd` as the container runtime, `crictl` replaces Docker for container management. Before proceeding with the troubleshooting commands, configure your environment by exporting the following variables: + +### RKE2 + +```bash +export PATH=$PATH:/var/lib/rancher/rke2/bin/ +export CRI_CONFIG_FILE=/var/lib/rancher/rke2/agent/etc/crictl.yaml +``` + +### K3s + +```bash +export PATH=$PATH:/usr/local/bin +export CRI_CONFIG_FILE=/var/lib/rancher/k3s/agent/etc/crictl.yaml +``` + +## Check if the Controlplane Components are Running + +**RKE2**: There are three specific containers launched on nodes with the `controlplane` role: * `kube-apiserver` * `kube-controller-manager` * `kube-scheduler` -The containers should have status **Up**. The duration shown after **Up** is the time the container has been running. +The containers should have state **Running**. You can check this using `crictl`: -``` -docker ps -a -f=name='kube-apiserver|kube-controller-manager|kube-scheduler' +```bash +crictl ps | grep -E 'kube-apiserver|kube-controller-manager|kube-scheduler' ``` Example output: ``` -CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES -26c7159abbcc rancher/hyperkube:v1.11.5-rancher1 "/opt/rke-tools/en..." 3 hours ago Up 3 hours kube-apiserver -f3d287ca4549 rancher/hyperkube:v1.11.5-rancher1 "/opt/rke-tools/en..." 3 hours ago Up 3 hours kube-scheduler -bdf3898b8063 rancher/hyperkube:v1.11.5-rancher1 "/opt/rke-tools/en..." 3 hours ago Up 3 hours kube-controller-manager +CONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID POD NAMESPACE +deb8a96948594 138b1e685e151 11 days ago Running kube-controller-manager 0 0996426295dc5 kube-controller-manager kube-system +f5abb4c7846e4 138b1e685e151 11 days ago Running kube-scheduler 0 80cd9f30af0be kube-scheduler kube-system +ecd8a6991c22a 138b1e685e151 11 days ago Running kube-apiserver 0 58e042fabe78c kube-apiserver kube-system ``` -## Controlplane Container Logging +**K3s**: These components run as embedded processes within the K3s service. They do not run as separate containers, so their status is tied to the `k3s` systemd service: + +```bash +systemctl status k3s +``` + +## Controlplane Logging :::note @@ -38,18 +62,32 @@ If you added multiple nodes with the `controlplane` role, both `kube-controller- ::: -The logging of the containers can contain information on what the problem could be. +The logs can contain information on what the problem could be. -``` -docker logs kube-apiserver -docker logs kube-controller-manager -docker logs kube-scheduler +**RKE2**: +```bash +crictl logs $(crictl ps --name kube-apiserver -q) +crictl logs $(crictl ps --name kube-controller-manager -q) +crictl logs $(crictl ps --name kube-scheduler -q) ``` -## RKE2 Server Logging - -If Rancher provisions an RKE2 cluster that can't communicate with Rancher, you can run this command on a server node in the downstream cluster to get the RKE2 server logs: - +**K3s**: +```bash +journalctl -u k3s | grep -i "kube-apiserver" +journalctl -u k3s | grep -i "kube-controller-manager" +journalctl -u k3s | grep -i "kube-scheduler" ``` + +## RKE2/K3s Server Logging + +If Rancher provisions an RKE2 or K3s cluster that can't communicate with Rancher, you can run this command on a server node in the downstream cluster to get the server logs: + +**RKE2**: +```bash journalctl -u rke2-server -f +``` + +**K3s**: +```bash +journalctl -u k3s -f ``` \ No newline at end of file diff --git a/versioned_docs/version-2.10/getting-started/installation-and-upgrade/other-installation-methods/air-gapped-helm-cli-install/publish-images.md b/versioned_docs/version-2.10/getting-started/installation-and-upgrade/other-installation-methods/air-gapped-helm-cli-install/publish-images.md index 992129fc684..d546b027741 100644 --- a/versioned_docs/version-2.10/getting-started/installation-and-upgrade/other-installation-methods/air-gapped-helm-cli-install/publish-images.md +++ b/versioned_docs/version-2.10/getting-started/installation-and-upgrade/other-installation-methods/air-gapped-helm-cli-install/publish-images.md @@ -42,7 +42,7 @@ If you will use ARM64 hosts, the registry must support manifests. As of April 20 1. Go to our [releases page,](https://github.com/rancher/rancher/releases) find the Rancher v2.x.x release that you want to install, and click **Assets**. Note: Don't use releases marked `rc` or `Pre-release`, as they are not stable for production environments. -2. From the release's **Assets** section, download the following files, which are required to install Rancher in an air gap environment: +2. From the release's **Assets** section, download the following files, which are required to install Rancher in an air-gap environment: | Release File | Description | | ---------------- | -------------- | @@ -83,16 +83,39 @@ In a Kubernetes Install, if you elect to use the Rancher default self-signed TLS ### 3. Save the images to your workstation -1. Make `rancher-save-images.sh` an executable: - ``` +(Optional) Verify the image list before pulling: + +```bash +wc -l rancher-images.txt +head rancher-images.txt +``` + +1. Make `rancher-save-images.sh` executable: + ```bash chmod +x rancher-save-images.sh ``` 1. Run `rancher-save-images.sh` with the `rancher-images.txt` image list to create a tarball of all the required images: - ```plain + ```bash ./rancher-save-images.sh --image-list ./rancher-images.txt ``` - **Result:** Docker begins pulling the images used for an air gap install. Be patient. This process takes a few minutes. When the process completes, your current directory will output a tarball named `rancher-images.tar.gz`. Check that the output is in the directory. + +(Optional) Specify a custom output file: + +```bash +./rancher-save-images.sh \ + --image-list ./rancher-images.txt \ + --images rancher-images-custom.tar.gz +``` + +**Result:** Docker begins pulling the images required for an air-gap installation. The process may take several minutes. + +1. Verify that the tarball was created: + ```bash + ls -lh rancher-images.tar.gz + ``` + +If some images fail to pull, review the output and retry after resolving any issues. ### 4. Populate the private registry @@ -163,7 +186,7 @@ Your registry must support manifests. As of April 2020, Amazon Elastic Container ./rancher-save-images.ps1 ``` - **Result:** Docker begins pulling the images used for an air gap install. Be patient. This process takes a few minutes. When the process completes, your current directory will output a tarball named `rancher-windows-images.tar.gz`. Check that the output is in the directory. + **Result:** Docker begins pulling the images used for an air-gap install. Be patient. This process takes a few minutes. When the process completes, your current directory will output a tarball named `rancher-windows-images.tar.gz`. Check that the output is in the directory. @@ -273,7 +296,7 @@ The workstation must have Docker 18.02+ in order to support manifests, which are ./rancher-save-images.sh --image-list ./rancher-images.txt ``` -**Result:** Docker begins pulling the images used for an air gap install. Be patient. This process takes a few minutes. When the process completes, your current directory will output a tarball named `rancher-images.tar.gz`. Check that the output is in the directory. +**Result:** Docker begins pulling the images used for an air-gap install. Be patient. This process takes a few minutes. When the process completes, your current directory will output a tarball named `rancher-images.tar.gz`. Check that the output is in the directory. diff --git a/versioned_docs/version-2.11/getting-started/installation-and-upgrade/other-installation-methods/air-gapped-helm-cli-install/publish-images.md b/versioned_docs/version-2.11/getting-started/installation-and-upgrade/other-installation-methods/air-gapped-helm-cli-install/publish-images.md index 992129fc684..d546b027741 100644 --- a/versioned_docs/version-2.11/getting-started/installation-and-upgrade/other-installation-methods/air-gapped-helm-cli-install/publish-images.md +++ b/versioned_docs/version-2.11/getting-started/installation-and-upgrade/other-installation-methods/air-gapped-helm-cli-install/publish-images.md @@ -42,7 +42,7 @@ If you will use ARM64 hosts, the registry must support manifests. As of April 20 1. Go to our [releases page,](https://github.com/rancher/rancher/releases) find the Rancher v2.x.x release that you want to install, and click **Assets**. Note: Don't use releases marked `rc` or `Pre-release`, as they are not stable for production environments. -2. From the release's **Assets** section, download the following files, which are required to install Rancher in an air gap environment: +2. From the release's **Assets** section, download the following files, which are required to install Rancher in an air-gap environment: | Release File | Description | | ---------------- | -------------- | @@ -83,16 +83,39 @@ In a Kubernetes Install, if you elect to use the Rancher default self-signed TLS ### 3. Save the images to your workstation -1. Make `rancher-save-images.sh` an executable: - ``` +(Optional) Verify the image list before pulling: + +```bash +wc -l rancher-images.txt +head rancher-images.txt +``` + +1. Make `rancher-save-images.sh` executable: + ```bash chmod +x rancher-save-images.sh ``` 1. Run `rancher-save-images.sh` with the `rancher-images.txt` image list to create a tarball of all the required images: - ```plain + ```bash ./rancher-save-images.sh --image-list ./rancher-images.txt ``` - **Result:** Docker begins pulling the images used for an air gap install. Be patient. This process takes a few minutes. When the process completes, your current directory will output a tarball named `rancher-images.tar.gz`. Check that the output is in the directory. + +(Optional) Specify a custom output file: + +```bash +./rancher-save-images.sh \ + --image-list ./rancher-images.txt \ + --images rancher-images-custom.tar.gz +``` + +**Result:** Docker begins pulling the images required for an air-gap installation. The process may take several minutes. + +1. Verify that the tarball was created: + ```bash + ls -lh rancher-images.tar.gz + ``` + +If some images fail to pull, review the output and retry after resolving any issues. ### 4. Populate the private registry @@ -163,7 +186,7 @@ Your registry must support manifests. As of April 2020, Amazon Elastic Container ./rancher-save-images.ps1 ``` - **Result:** Docker begins pulling the images used for an air gap install. Be patient. This process takes a few minutes. When the process completes, your current directory will output a tarball named `rancher-windows-images.tar.gz`. Check that the output is in the directory. + **Result:** Docker begins pulling the images used for an air-gap install. Be patient. This process takes a few minutes. When the process completes, your current directory will output a tarball named `rancher-windows-images.tar.gz`. Check that the output is in the directory. @@ -273,7 +296,7 @@ The workstation must have Docker 18.02+ in order to support manifests, which are ./rancher-save-images.sh --image-list ./rancher-images.txt ``` -**Result:** Docker begins pulling the images used for an air gap install. Be patient. This process takes a few minutes. When the process completes, your current directory will output a tarball named `rancher-images.tar.gz`. Check that the output is in the directory. +**Result:** Docker begins pulling the images used for an air-gap install. Be patient. This process takes a few minutes. When the process completes, your current directory will output a tarball named `rancher-images.tar.gz`. Check that the output is in the directory. diff --git a/versioned_docs/version-2.12/getting-started/installation-and-upgrade/other-installation-methods/air-gapped-helm-cli-install/publish-images.md b/versioned_docs/version-2.12/getting-started/installation-and-upgrade/other-installation-methods/air-gapped-helm-cli-install/publish-images.md index 992129fc684..d546b027741 100644 --- a/versioned_docs/version-2.12/getting-started/installation-and-upgrade/other-installation-methods/air-gapped-helm-cli-install/publish-images.md +++ b/versioned_docs/version-2.12/getting-started/installation-and-upgrade/other-installation-methods/air-gapped-helm-cli-install/publish-images.md @@ -42,7 +42,7 @@ If you will use ARM64 hosts, the registry must support manifests. As of April 20 1. Go to our [releases page,](https://github.com/rancher/rancher/releases) find the Rancher v2.x.x release that you want to install, and click **Assets**. Note: Don't use releases marked `rc` or `Pre-release`, as they are not stable for production environments. -2. From the release's **Assets** section, download the following files, which are required to install Rancher in an air gap environment: +2. From the release's **Assets** section, download the following files, which are required to install Rancher in an air-gap environment: | Release File | Description | | ---------------- | -------------- | @@ -83,16 +83,39 @@ In a Kubernetes Install, if you elect to use the Rancher default self-signed TLS ### 3. Save the images to your workstation -1. Make `rancher-save-images.sh` an executable: - ``` +(Optional) Verify the image list before pulling: + +```bash +wc -l rancher-images.txt +head rancher-images.txt +``` + +1. Make `rancher-save-images.sh` executable: + ```bash chmod +x rancher-save-images.sh ``` 1. Run `rancher-save-images.sh` with the `rancher-images.txt` image list to create a tarball of all the required images: - ```plain + ```bash ./rancher-save-images.sh --image-list ./rancher-images.txt ``` - **Result:** Docker begins pulling the images used for an air gap install. Be patient. This process takes a few minutes. When the process completes, your current directory will output a tarball named `rancher-images.tar.gz`. Check that the output is in the directory. + +(Optional) Specify a custom output file: + +```bash +./rancher-save-images.sh \ + --image-list ./rancher-images.txt \ + --images rancher-images-custom.tar.gz +``` + +**Result:** Docker begins pulling the images required for an air-gap installation. The process may take several minutes. + +1. Verify that the tarball was created: + ```bash + ls -lh rancher-images.tar.gz + ``` + +If some images fail to pull, review the output and retry after resolving any issues. ### 4. Populate the private registry @@ -163,7 +186,7 @@ Your registry must support manifests. As of April 2020, Amazon Elastic Container ./rancher-save-images.ps1 ``` - **Result:** Docker begins pulling the images used for an air gap install. Be patient. This process takes a few minutes. When the process completes, your current directory will output a tarball named `rancher-windows-images.tar.gz`. Check that the output is in the directory. + **Result:** Docker begins pulling the images used for an air-gap install. Be patient. This process takes a few minutes. When the process completes, your current directory will output a tarball named `rancher-windows-images.tar.gz`. Check that the output is in the directory. @@ -273,7 +296,7 @@ The workstation must have Docker 18.02+ in order to support manifests, which are ./rancher-save-images.sh --image-list ./rancher-images.txt ``` -**Result:** Docker begins pulling the images used for an air gap install. Be patient. This process takes a few minutes. When the process completes, your current directory will output a tarball named `rancher-images.tar.gz`. Check that the output is in the directory. +**Result:** Docker begins pulling the images used for an air-gap install. Be patient. This process takes a few minutes. When the process completes, your current directory will output a tarball named `rancher-images.tar.gz`. Check that the output is in the directory. diff --git a/versioned_docs/version-2.12/troubleshooting/kubernetes-components/troubleshooting-controlplane-nodes.md b/versioned_docs/version-2.12/troubleshooting/kubernetes-components/troubleshooting-controlplane-nodes.md index 84877bc8fab..487506c45cf 100644 --- a/versioned_docs/version-2.12/troubleshooting/kubernetes-components/troubleshooting-controlplane-nodes.md +++ b/versioned_docs/version-2.12/troubleshooting/kubernetes-components/troubleshooting-controlplane-nodes.md @@ -6,31 +6,55 @@ title: Troubleshooting Controlplane Nodes -This section applies to nodes with the `controlplane` role. +This section applies to nodes with the `controlplane` role in RKE2 and K3s clusters. -## Check if the Controlplane Containers are Running +## Prerequisites -There are three specific containers launched on nodes with the `controlplane` role: +As RKE2 and K3s rely on `containerd` as the container runtime, `crictl` replaces Docker for container management. Before proceeding with the troubleshooting commands, configure your environment by exporting the following variables: + +### RKE2 + +```bash +export PATH=$PATH:/var/lib/rancher/rke2/bin/ +export CRI_CONFIG_FILE=/var/lib/rancher/rke2/agent/etc/crictl.yaml +``` + +### K3s + +```bash +export PATH=$PATH:/usr/local/bin +export CRI_CONFIG_FILE=/var/lib/rancher/k3s/agent/etc/crictl.yaml +``` + +## Check if the Controlplane Components are Running + +**RKE2**: There are three specific containers launched on nodes with the `controlplane` role: * `kube-apiserver` * `kube-controller-manager` * `kube-scheduler` -The containers should have status **Up**. The duration shown after **Up** is the time the container has been running. +The containers should have state **Running**. You can check this using `crictl`: -``` -docker ps -a -f=name='kube-apiserver|kube-controller-manager|kube-scheduler' +```bash +crictl ps | grep -E 'kube-apiserver|kube-controller-manager|kube-scheduler' ``` Example output: ``` -CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES -26c7159abbcc rancher/hyperkube:v1.11.5-rancher1 "/opt/rke-tools/en..." 3 hours ago Up 3 hours kube-apiserver -f3d287ca4549 rancher/hyperkube:v1.11.5-rancher1 "/opt/rke-tools/en..." 3 hours ago Up 3 hours kube-scheduler -bdf3898b8063 rancher/hyperkube:v1.11.5-rancher1 "/opt/rke-tools/en..." 3 hours ago Up 3 hours kube-controller-manager +CONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID POD NAMESPACE +deb8a96948594 138b1e685e151 11 days ago Running kube-controller-manager 0 0996426295dc5 kube-controller-manager kube-system +f5abb4c7846e4 138b1e685e151 11 days ago Running kube-scheduler 0 80cd9f30af0be kube-scheduler kube-system +ecd8a6991c22a 138b1e685e151 11 days ago Running kube-apiserver 0 58e042fabe78c kube-apiserver kube-system ``` -## Controlplane Container Logging +**K3s**: These components run as embedded processes within the K3s service. They do not run as separate containers, so their status is tied to the `k3s` systemd service: + +```bash +systemctl status k3s +``` + +## Controlplane Logging :::note @@ -38,18 +62,32 @@ If you added multiple nodes with the `controlplane` role, both `kube-controller- ::: -The logging of the containers can contain information on what the problem could be. +The logs can contain information on what the problem could be. -``` -docker logs kube-apiserver -docker logs kube-controller-manager -docker logs kube-scheduler +**RKE2**: +```bash +crictl logs $(crictl ps --name kube-apiserver -q) +crictl logs $(crictl ps --name kube-controller-manager -q) +crictl logs $(crictl ps --name kube-scheduler -q) ``` -## RKE2 Server Logging - -If Rancher provisions an RKE2 cluster that can't communicate with Rancher, you can run this command on a server node in the downstream cluster to get the RKE2 server logs: - +**K3s**: +```bash +journalctl -u k3s | grep -i "kube-apiserver" +journalctl -u k3s | grep -i "kube-controller-manager" +journalctl -u k3s | grep -i "kube-scheduler" ``` + +## RKE2/K3s Server Logging + +If Rancher provisions an RKE2 or K3s cluster that can't communicate with Rancher, you can run this command on a server node in the downstream cluster to get the server logs: + +**RKE2**: +```bash journalctl -u rke2-server -f +``` + +**K3s**: +```bash +journalctl -u k3s -f ``` \ No newline at end of file diff --git a/versioned_docs/version-2.13/getting-started/installation-and-upgrade/other-installation-methods/air-gapped-helm-cli-install/publish-images.md b/versioned_docs/version-2.13/getting-started/installation-and-upgrade/other-installation-methods/air-gapped-helm-cli-install/publish-images.md index 992129fc684..d546b027741 100644 --- a/versioned_docs/version-2.13/getting-started/installation-and-upgrade/other-installation-methods/air-gapped-helm-cli-install/publish-images.md +++ b/versioned_docs/version-2.13/getting-started/installation-and-upgrade/other-installation-methods/air-gapped-helm-cli-install/publish-images.md @@ -42,7 +42,7 @@ If you will use ARM64 hosts, the registry must support manifests. As of April 20 1. Go to our [releases page,](https://github.com/rancher/rancher/releases) find the Rancher v2.x.x release that you want to install, and click **Assets**. Note: Don't use releases marked `rc` or `Pre-release`, as they are not stable for production environments. -2. From the release's **Assets** section, download the following files, which are required to install Rancher in an air gap environment: +2. From the release's **Assets** section, download the following files, which are required to install Rancher in an air-gap environment: | Release File | Description | | ---------------- | -------------- | @@ -83,16 +83,39 @@ In a Kubernetes Install, if you elect to use the Rancher default self-signed TLS ### 3. Save the images to your workstation -1. Make `rancher-save-images.sh` an executable: - ``` +(Optional) Verify the image list before pulling: + +```bash +wc -l rancher-images.txt +head rancher-images.txt +``` + +1. Make `rancher-save-images.sh` executable: + ```bash chmod +x rancher-save-images.sh ``` 1. Run `rancher-save-images.sh` with the `rancher-images.txt` image list to create a tarball of all the required images: - ```plain + ```bash ./rancher-save-images.sh --image-list ./rancher-images.txt ``` - **Result:** Docker begins pulling the images used for an air gap install. Be patient. This process takes a few minutes. When the process completes, your current directory will output a tarball named `rancher-images.tar.gz`. Check that the output is in the directory. + +(Optional) Specify a custom output file: + +```bash +./rancher-save-images.sh \ + --image-list ./rancher-images.txt \ + --images rancher-images-custom.tar.gz +``` + +**Result:** Docker begins pulling the images required for an air-gap installation. The process may take several minutes. + +1. Verify that the tarball was created: + ```bash + ls -lh rancher-images.tar.gz + ``` + +If some images fail to pull, review the output and retry after resolving any issues. ### 4. Populate the private registry @@ -163,7 +186,7 @@ Your registry must support manifests. As of April 2020, Amazon Elastic Container ./rancher-save-images.ps1 ``` - **Result:** Docker begins pulling the images used for an air gap install. Be patient. This process takes a few minutes. When the process completes, your current directory will output a tarball named `rancher-windows-images.tar.gz`. Check that the output is in the directory. + **Result:** Docker begins pulling the images used for an air-gap install. Be patient. This process takes a few minutes. When the process completes, your current directory will output a tarball named `rancher-windows-images.tar.gz`. Check that the output is in the directory. @@ -273,7 +296,7 @@ The workstation must have Docker 18.02+ in order to support manifests, which are ./rancher-save-images.sh --image-list ./rancher-images.txt ``` -**Result:** Docker begins pulling the images used for an air gap install. Be patient. This process takes a few minutes. When the process completes, your current directory will output a tarball named `rancher-images.tar.gz`. Check that the output is in the directory. +**Result:** Docker begins pulling the images used for an air-gap install. Be patient. This process takes a few minutes. When the process completes, your current directory will output a tarball named `rancher-images.tar.gz`. Check that the output is in the directory. diff --git a/versioned_docs/version-2.13/troubleshooting/kubernetes-components/troubleshooting-controlplane-nodes.md b/versioned_docs/version-2.13/troubleshooting/kubernetes-components/troubleshooting-controlplane-nodes.md index 84877bc8fab..487506c45cf 100644 --- a/versioned_docs/version-2.13/troubleshooting/kubernetes-components/troubleshooting-controlplane-nodes.md +++ b/versioned_docs/version-2.13/troubleshooting/kubernetes-components/troubleshooting-controlplane-nodes.md @@ -6,31 +6,55 @@ title: Troubleshooting Controlplane Nodes -This section applies to nodes with the `controlplane` role. +This section applies to nodes with the `controlplane` role in RKE2 and K3s clusters. -## Check if the Controlplane Containers are Running +## Prerequisites -There are three specific containers launched on nodes with the `controlplane` role: +As RKE2 and K3s rely on `containerd` as the container runtime, `crictl` replaces Docker for container management. Before proceeding with the troubleshooting commands, configure your environment by exporting the following variables: + +### RKE2 + +```bash +export PATH=$PATH:/var/lib/rancher/rke2/bin/ +export CRI_CONFIG_FILE=/var/lib/rancher/rke2/agent/etc/crictl.yaml +``` + +### K3s + +```bash +export PATH=$PATH:/usr/local/bin +export CRI_CONFIG_FILE=/var/lib/rancher/k3s/agent/etc/crictl.yaml +``` + +## Check if the Controlplane Components are Running + +**RKE2**: There are three specific containers launched on nodes with the `controlplane` role: * `kube-apiserver` * `kube-controller-manager` * `kube-scheduler` -The containers should have status **Up**. The duration shown after **Up** is the time the container has been running. +The containers should have state **Running**. You can check this using `crictl`: -``` -docker ps -a -f=name='kube-apiserver|kube-controller-manager|kube-scheduler' +```bash +crictl ps | grep -E 'kube-apiserver|kube-controller-manager|kube-scheduler' ``` Example output: ``` -CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES -26c7159abbcc rancher/hyperkube:v1.11.5-rancher1 "/opt/rke-tools/en..." 3 hours ago Up 3 hours kube-apiserver -f3d287ca4549 rancher/hyperkube:v1.11.5-rancher1 "/opt/rke-tools/en..." 3 hours ago Up 3 hours kube-scheduler -bdf3898b8063 rancher/hyperkube:v1.11.5-rancher1 "/opt/rke-tools/en..." 3 hours ago Up 3 hours kube-controller-manager +CONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID POD NAMESPACE +deb8a96948594 138b1e685e151 11 days ago Running kube-controller-manager 0 0996426295dc5 kube-controller-manager kube-system +f5abb4c7846e4 138b1e685e151 11 days ago Running kube-scheduler 0 80cd9f30af0be kube-scheduler kube-system +ecd8a6991c22a 138b1e685e151 11 days ago Running kube-apiserver 0 58e042fabe78c kube-apiserver kube-system ``` -## Controlplane Container Logging +**K3s**: These components run as embedded processes within the K3s service. They do not run as separate containers, so their status is tied to the `k3s` systemd service: + +```bash +systemctl status k3s +``` + +## Controlplane Logging :::note @@ -38,18 +62,32 @@ If you added multiple nodes with the `controlplane` role, both `kube-controller- ::: -The logging of the containers can contain information on what the problem could be. +The logs can contain information on what the problem could be. -``` -docker logs kube-apiserver -docker logs kube-controller-manager -docker logs kube-scheduler +**RKE2**: +```bash +crictl logs $(crictl ps --name kube-apiserver -q) +crictl logs $(crictl ps --name kube-controller-manager -q) +crictl logs $(crictl ps --name kube-scheduler -q) ``` -## RKE2 Server Logging - -If Rancher provisions an RKE2 cluster that can't communicate with Rancher, you can run this command on a server node in the downstream cluster to get the RKE2 server logs: - +**K3s**: +```bash +journalctl -u k3s | grep -i "kube-apiserver" +journalctl -u k3s | grep -i "kube-controller-manager" +journalctl -u k3s | grep -i "kube-scheduler" ``` + +## RKE2/K3s Server Logging + +If Rancher provisions an RKE2 or K3s cluster that can't communicate with Rancher, you can run this command on a server node in the downstream cluster to get the server logs: + +**RKE2**: +```bash journalctl -u rke2-server -f +``` + +**K3s**: +```bash +journalctl -u k3s -f ``` \ No newline at end of file diff --git a/versioned_docs/version-2.14/getting-started/installation-and-upgrade/other-installation-methods/air-gapped-helm-cli-install/publish-images.md b/versioned_docs/version-2.14/getting-started/installation-and-upgrade/other-installation-methods/air-gapped-helm-cli-install/publish-images.md index 992129fc684..d546b027741 100644 --- a/versioned_docs/version-2.14/getting-started/installation-and-upgrade/other-installation-methods/air-gapped-helm-cli-install/publish-images.md +++ b/versioned_docs/version-2.14/getting-started/installation-and-upgrade/other-installation-methods/air-gapped-helm-cli-install/publish-images.md @@ -42,7 +42,7 @@ If you will use ARM64 hosts, the registry must support manifests. As of April 20 1. Go to our [releases page,](https://github.com/rancher/rancher/releases) find the Rancher v2.x.x release that you want to install, and click **Assets**. Note: Don't use releases marked `rc` or `Pre-release`, as they are not stable for production environments. -2. From the release's **Assets** section, download the following files, which are required to install Rancher in an air gap environment: +2. From the release's **Assets** section, download the following files, which are required to install Rancher in an air-gap environment: | Release File | Description | | ---------------- | -------------- | @@ -83,16 +83,39 @@ In a Kubernetes Install, if you elect to use the Rancher default self-signed TLS ### 3. Save the images to your workstation -1. Make `rancher-save-images.sh` an executable: - ``` +(Optional) Verify the image list before pulling: + +```bash +wc -l rancher-images.txt +head rancher-images.txt +``` + +1. Make `rancher-save-images.sh` executable: + ```bash chmod +x rancher-save-images.sh ``` 1. Run `rancher-save-images.sh` with the `rancher-images.txt` image list to create a tarball of all the required images: - ```plain + ```bash ./rancher-save-images.sh --image-list ./rancher-images.txt ``` - **Result:** Docker begins pulling the images used for an air gap install. Be patient. This process takes a few minutes. When the process completes, your current directory will output a tarball named `rancher-images.tar.gz`. Check that the output is in the directory. + +(Optional) Specify a custom output file: + +```bash +./rancher-save-images.sh \ + --image-list ./rancher-images.txt \ + --images rancher-images-custom.tar.gz +``` + +**Result:** Docker begins pulling the images required for an air-gap installation. The process may take several minutes. + +1. Verify that the tarball was created: + ```bash + ls -lh rancher-images.tar.gz + ``` + +If some images fail to pull, review the output and retry after resolving any issues. ### 4. Populate the private registry @@ -163,7 +186,7 @@ Your registry must support manifests. As of April 2020, Amazon Elastic Container ./rancher-save-images.ps1 ``` - **Result:** Docker begins pulling the images used for an air gap install. Be patient. This process takes a few minutes. When the process completes, your current directory will output a tarball named `rancher-windows-images.tar.gz`. Check that the output is in the directory. + **Result:** Docker begins pulling the images used for an air-gap install. Be patient. This process takes a few minutes. When the process completes, your current directory will output a tarball named `rancher-windows-images.tar.gz`. Check that the output is in the directory. @@ -273,7 +296,7 @@ The workstation must have Docker 18.02+ in order to support manifests, which are ./rancher-save-images.sh --image-list ./rancher-images.txt ``` -**Result:** Docker begins pulling the images used for an air gap install. Be patient. This process takes a few minutes. When the process completes, your current directory will output a tarball named `rancher-images.tar.gz`. Check that the output is in the directory. +**Result:** Docker begins pulling the images used for an air-gap install. Be patient. This process takes a few minutes. When the process completes, your current directory will output a tarball named `rancher-images.tar.gz`. Check that the output is in the directory. diff --git a/versioned_docs/version-2.14/troubleshooting/kubernetes-components/troubleshooting-controlplane-nodes.md b/versioned_docs/version-2.14/troubleshooting/kubernetes-components/troubleshooting-controlplane-nodes.md index 84877bc8fab..487506c45cf 100644 --- a/versioned_docs/version-2.14/troubleshooting/kubernetes-components/troubleshooting-controlplane-nodes.md +++ b/versioned_docs/version-2.14/troubleshooting/kubernetes-components/troubleshooting-controlplane-nodes.md @@ -6,31 +6,55 @@ title: Troubleshooting Controlplane Nodes -This section applies to nodes with the `controlplane` role. +This section applies to nodes with the `controlplane` role in RKE2 and K3s clusters. -## Check if the Controlplane Containers are Running +## Prerequisites -There are three specific containers launched on nodes with the `controlplane` role: +As RKE2 and K3s rely on `containerd` as the container runtime, `crictl` replaces Docker for container management. Before proceeding with the troubleshooting commands, configure your environment by exporting the following variables: + +### RKE2 + +```bash +export PATH=$PATH:/var/lib/rancher/rke2/bin/ +export CRI_CONFIG_FILE=/var/lib/rancher/rke2/agent/etc/crictl.yaml +``` + +### K3s + +```bash +export PATH=$PATH:/usr/local/bin +export CRI_CONFIG_FILE=/var/lib/rancher/k3s/agent/etc/crictl.yaml +``` + +## Check if the Controlplane Components are Running + +**RKE2**: There are three specific containers launched on nodes with the `controlplane` role: * `kube-apiserver` * `kube-controller-manager` * `kube-scheduler` -The containers should have status **Up**. The duration shown after **Up** is the time the container has been running. +The containers should have state **Running**. You can check this using `crictl`: -``` -docker ps -a -f=name='kube-apiserver|kube-controller-manager|kube-scheduler' +```bash +crictl ps | grep -E 'kube-apiserver|kube-controller-manager|kube-scheduler' ``` Example output: ``` -CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES -26c7159abbcc rancher/hyperkube:v1.11.5-rancher1 "/opt/rke-tools/en..." 3 hours ago Up 3 hours kube-apiserver -f3d287ca4549 rancher/hyperkube:v1.11.5-rancher1 "/opt/rke-tools/en..." 3 hours ago Up 3 hours kube-scheduler -bdf3898b8063 rancher/hyperkube:v1.11.5-rancher1 "/opt/rke-tools/en..." 3 hours ago Up 3 hours kube-controller-manager +CONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID POD NAMESPACE +deb8a96948594 138b1e685e151 11 days ago Running kube-controller-manager 0 0996426295dc5 kube-controller-manager kube-system +f5abb4c7846e4 138b1e685e151 11 days ago Running kube-scheduler 0 80cd9f30af0be kube-scheduler kube-system +ecd8a6991c22a 138b1e685e151 11 days ago Running kube-apiserver 0 58e042fabe78c kube-apiserver kube-system ``` -## Controlplane Container Logging +**K3s**: These components run as embedded processes within the K3s service. They do not run as separate containers, so their status is tied to the `k3s` systemd service: + +```bash +systemctl status k3s +``` + +## Controlplane Logging :::note @@ -38,18 +62,32 @@ If you added multiple nodes with the `controlplane` role, both `kube-controller- ::: -The logging of the containers can contain information on what the problem could be. +The logs can contain information on what the problem could be. -``` -docker logs kube-apiserver -docker logs kube-controller-manager -docker logs kube-scheduler +**RKE2**: +```bash +crictl logs $(crictl ps --name kube-apiserver -q) +crictl logs $(crictl ps --name kube-controller-manager -q) +crictl logs $(crictl ps --name kube-scheduler -q) ``` -## RKE2 Server Logging - -If Rancher provisions an RKE2 cluster that can't communicate with Rancher, you can run this command on a server node in the downstream cluster to get the RKE2 server logs: - +**K3s**: +```bash +journalctl -u k3s | grep -i "kube-apiserver" +journalctl -u k3s | grep -i "kube-controller-manager" +journalctl -u k3s | grep -i "kube-scheduler" ``` + +## RKE2/K3s Server Logging + +If Rancher provisions an RKE2 or K3s cluster that can't communicate with Rancher, you can run this command on a server node in the downstream cluster to get the server logs: + +**RKE2**: +```bash journalctl -u rke2-server -f +``` + +**K3s**: +```bash +journalctl -u k3s -f ``` \ No newline at end of file