Docs: Migrate controlplane troubleshooting guide from RKE/Docker to RKE2/K3s/containerd

2026-09-25 04:28:15 +00:00 · 2026-04-15 14:30:31 +02:00
parent b25d27891d
commit b5da180fc9
10 changed files with 412 additions and 122 deletions
@@ -42,7 +42,7 @@ If you will use ARM64 hosts, the registry must support manifests. As of April 20

 1. Go to our [releases page,](https://github.com/rancher/rancher/releases) find the Rancher v2.x.x release that you want to install, and click **Assets**. Note: Don't use releases marked `rc` or `Pre-release`, as they are not stable for production environments.

-2. From the release's **Assets** section, download the following files, which are required to install Rancher in an air gap environment:
+2. From the release's **Assets** section, download the following files, which are required to install Rancher in an air-gap environment:

 | Release File   | Description  |
 | ---------------- | -------------- |
@@ -83,16 +83,39 @@ In a Kubernetes Install, if you elect to use the Rancher default self-signed TLS

 ### 3. Save the images to your workstation

-1. Make `rancher-save-images.sh` an executable:
-   ```
+(Optional) Verify the image list before pulling:
+
+```bash
+wc -l rancher-images.txt
+head rancher-images.txt
+```
+
+1. Make `rancher-save-images.sh` executable:
+   ```bash
   chmod +x rancher-save-images.sh
   ```

 1. Run `rancher-save-images.sh` with the `rancher-images.txt` image list to create a tarball of all the required images:
-   ```plain
+   ```bash
   ./rancher-save-images.sh --image-list ./rancher-images.txt
   ```
-   **Result:** Docker begins pulling the images used for an air gap install. Be patient. This process takes a few minutes. When the process completes, your current directory will output a tarball named `rancher-images.tar.gz`. Check that the output is in the directory.
+
+(Optional) Specify a custom output file:
+
+```bash
+./rancher-save-images.sh \
+  --image-list ./rancher-images.txt \
+  --images rancher-images-custom.tar.gz
+```
+
+**Result:** Docker begins pulling the images required for an air-gap installation. The process may take several minutes.
+
+1. Verify that the tarball was created:
+   ```bash
+   ls -lh rancher-images.tar.gz
+   ```
+
+If some images fail to pull, review the output and retry after resolving any issues.

 ### 4. Populate the private registry

@@ -163,7 +186,7 @@ Your registry must support manifests. As of April 2020, Amazon Elastic Container
   ./rancher-save-images.ps1
   ```

-   **Result:** Docker begins pulling the images used for an air gap install. Be patient. This process takes a few minutes. When the process completes, your current directory will output a tarball named `rancher-windows-images.tar.gz`. Check that the output is in the directory.
+   **Result:** Docker begins pulling the images used for an air-gap install. Be patient. This process takes a few minutes. When the process completes, your current directory will output a tarball named `rancher-windows-images.tar.gz`. Check that the output is in the directory.

 <a name="windows-3"></a>

@@ -273,7 +296,7 @@ The workstation must have Docker 18.02+ in order to support manifests, which are
   ./rancher-save-images.sh --image-list ./rancher-images.txt
   ```

-**Result:** Docker begins pulling the images used for an air gap install. Be patient. This process takes a few minutes. When the process completes, your current directory will output a tarball named `rancher-images.tar.gz`. Check that the output is in the directory.
+**Result:** Docker begins pulling the images used for an air-gap install. Be patient. This process takes a few minutes. When the process completes, your current directory will output a tarball named `rancher-images.tar.gz`. Check that the output is in the directory.

 <a name="linux-4"></a>

@@ -6,31 +6,55 @@ title: Troubleshooting Controlplane Nodes
  <link rel="canonical" href="https://ranchermanager.docs.rancher.com/troubleshooting/kubernetes-components/troubleshooting-controlplane-nodes"/>
 </head>

-This section applies to nodes with the `controlplane` role.
+This section applies to nodes with the `controlplane` role in RKE2 and K3s clusters.

-## Check if the Controlplane Containers are Running
+## Prerequisites

-There are three specific containers launched on nodes with the `controlplane` role:
+As RKE2 and K3s rely on `containerd` as the container runtime, `crictl` replaces Docker for container management. Before proceeding with the troubleshooting commands, configure your environment by exporting the following variables:
+
+### RKE2
+
+```bash
+export PATH=$PATH:/var/lib/rancher/rke2/bin/
+export CRI_CONFIG_FILE=/var/lib/rancher/rke2/agent/etc/crictl.yaml
+```
+
+### K3s
+
+```bash
+export PATH=$PATH:/usr/local/bin
+export CRI_CONFIG_FILE=/var/lib/rancher/k3s/agent/etc/crictl.yaml
+```
+
+## Check if the Controlplane Components are Running
+
+**RKE2**: There are three specific containers launched on nodes with the `controlplane` role:

 * `kube-apiserver`
 * `kube-controller-manager`
 * `kube-scheduler`

-The containers should have status **Up**. The duration shown after **Up** is the time the container has been running.
+The containers should have state **Running**. You can check this using `crictl`:

-```
-docker ps -a -f=name='kube-apiserver|kube-controller-manager|kube-scheduler'
+```bash
+crictl ps | grep -E 'kube-apiserver|kube-controller-manager|kube-scheduler'
 ```

 Example output:
 ```
-CONTAINER ID        IMAGE                                COMMAND                  CREATED             STATUS              PORTS               NAMES
-26c7159abbcc        rancher/hyperkube:v1.11.5-rancher1   "/opt/rke-tools/en..."   3 hours ago         Up 3 hours                              kube-apiserver
-f3d287ca4549        rancher/hyperkube:v1.11.5-rancher1   "/opt/rke-tools/en..."   3 hours ago         Up 3 hours                              kube-scheduler
-bdf3898b8063        rancher/hyperkube:v1.11.5-rancher1   "/opt/rke-tools/en..."   3 hours ago         Up 3 hours                              kube-controller-manager
+CONTAINER           IMAGE               CREATED             STATE               NAME                            ATTEMPT             POD ID              POD                                     NAMESPACE
+deb8a96948594       138b1e685e151       11 days ago         Running             kube-controller-manager         0                   0996426295dc5       kube-controller-manager                 kube-system
+f5abb4c7846e4       138b1e685e151       11 days ago         Running             kube-scheduler                  0                   80cd9f30af0be       kube-scheduler                          kube-system
+ecd8a6991c22a       138b1e685e151       11 days ago         Running             kube-apiserver                  0                   58e042fabe78c       kube-apiserver                          kube-system
 ```

-## Controlplane Container Logging
+**K3s**: These components run as embedded processes within the K3s service. They do not run as separate containers, so their status is tied to the `k3s` systemd service:
+
+```bash
+systemctl status k3s
+```
+
+## Controlplane Logging

 :::note

@@ -38,18 +62,32 @@ If you added multiple nodes with the `controlplane` role, both `kube-controller-

 :::

-The logging of the containers can contain information on what the problem could be.
+The logs can contain information on what the problem could be.

-```
-docker logs kube-apiserver
-docker logs kube-controller-manager
-docker logs kube-scheduler
+**RKE2**:
+```bash
+crictl logs $(crictl ps --name kube-apiserver -q)
+crictl logs $(crictl ps --name kube-controller-manager -q)
+crictl logs $(crictl ps --name kube-scheduler -q)
 ```

-## RKE2 Server Logging
-
-If Rancher provisions an RKE2 cluster that can't communicate with Rancher, you can run this command on a server node in the downstream cluster to get the RKE2 server logs:
-
+**K3s**:
+```bash
+journalctl -u k3s | grep -i "kube-apiserver"
+journalctl -u k3s | grep -i "kube-controller-manager"
+journalctl -u k3s | grep -i "kube-scheduler"
 ```
+
+## RKE2/K3s Server Logging
+
+If Rancher provisions an RKE2 or K3s cluster that can't communicate with Rancher, you can run this command on a server node in the downstream cluster to get the server logs:
+
+**RKE2**:
+```bash
 journalctl -u rke2-server -f
+```
+
+**K3s**:
+```bash
+journalctl -u k3s -f
 ```
@@ -42,7 +42,7 @@ If you will use ARM64 hosts, the registry must support manifests. As of April 20

 1. Go to our [releases page,](https://github.com/rancher/rancher/releases) find the Rancher v2.x.x release that you want to install, and click **Assets**. Note: Don't use releases marked `rc` or `Pre-release`, as they are not stable for production environments.

-2. From the release's **Assets** section, download the following files, which are required to install Rancher in an air gap environment:
+2. From the release's **Assets** section, download the following files, which are required to install Rancher in an air-gap environment:

 | Release File   | Description  |
 | ---------------- | -------------- |
@@ -83,16 +83,39 @@ In a Kubernetes Install, if you elect to use the Rancher default self-signed TLS

 ### 3. Save the images to your workstation

-1. Make `rancher-save-images.sh` an executable:
-   ```
+(Optional) Verify the image list before pulling:
+
+```bash
+wc -l rancher-images.txt
+head rancher-images.txt
+```
+
+1. Make `rancher-save-images.sh` executable:
+   ```bash
   chmod +x rancher-save-images.sh
   ```

 1. Run `rancher-save-images.sh` with the `rancher-images.txt` image list to create a tarball of all the required images:
-   ```plain
+   ```bash
   ./rancher-save-images.sh --image-list ./rancher-images.txt
   ```
-   **Result:** Docker begins pulling the images used for an air gap install. Be patient. This process takes a few minutes. When the process completes, your current directory will output a tarball named `rancher-images.tar.gz`. Check that the output is in the directory.
+
+(Optional) Specify a custom output file:
+
+```bash
+./rancher-save-images.sh \
+  --image-list ./rancher-images.txt \
+  --images rancher-images-custom.tar.gz
+```
+
+**Result:** Docker begins pulling the images required for an air-gap installation. The process may take several minutes.
+
+1. Verify that the tarball was created:
+   ```bash
+   ls -lh rancher-images.tar.gz
+   ```
+
+If some images fail to pull, review the output and retry after resolving any issues.

 ### 4. Populate the private registry

@@ -163,7 +186,7 @@ Your registry must support manifests. As of April 2020, Amazon Elastic Container
   ./rancher-save-images.ps1
   ```

-   **Result:** Docker begins pulling the images used for an air gap install. Be patient. This process takes a few minutes. When the process completes, your current directory will output a tarball named `rancher-windows-images.tar.gz`. Check that the output is in the directory.
+   **Result:** Docker begins pulling the images used for an air-gap install. Be patient. This process takes a few minutes. When the process completes, your current directory will output a tarball named `rancher-windows-images.tar.gz`. Check that the output is in the directory.

 <a name="windows-3"></a>

@@ -273,7 +296,7 @@ The workstation must have Docker 18.02+ in order to support manifests, which are
   ./rancher-save-images.sh --image-list ./rancher-images.txt
   ```

-**Result:** Docker begins pulling the images used for an air gap install. Be patient. This process takes a few minutes. When the process completes, your current directory will output a tarball named `rancher-images.tar.gz`. Check that the output is in the directory.
+**Result:** Docker begins pulling the images used for an air-gap install. Be patient. This process takes a few minutes. When the process completes, your current directory will output a tarball named `rancher-images.tar.gz`. Check that the output is in the directory.

 <a name="linux-4"></a>

@@ -42,7 +42,7 @@ If you will use ARM64 hosts, the registry must support manifests. As of April 20

 1. Go to our [releases page,](https://github.com/rancher/rancher/releases) find the Rancher v2.x.x release that you want to install, and click **Assets**. Note: Don't use releases marked `rc` or `Pre-release`, as they are not stable for production environments.

-2. From the release's **Assets** section, download the following files, which are required to install Rancher in an air gap environment:
+2. From the release's **Assets** section, download the following files, which are required to install Rancher in an air-gap environment:

 | Release File   | Description  |
 | ---------------- | -------------- |
@@ -83,16 +83,39 @@ In a Kubernetes Install, if you elect to use the Rancher default self-signed TLS

 ### 3. Save the images to your workstation

-1. Make `rancher-save-images.sh` an executable:
-   ```
+(Optional) Verify the image list before pulling:
+
+```bash
+wc -l rancher-images.txt
+head rancher-images.txt
+```
+
+1. Make `rancher-save-images.sh` executable:
+   ```bash
   chmod +x rancher-save-images.sh
   ```

 1. Run `rancher-save-images.sh` with the `rancher-images.txt` image list to create a tarball of all the required images:
-   ```plain
+   ```bash
   ./rancher-save-images.sh --image-list ./rancher-images.txt
   ```
-   **Result:** Docker begins pulling the images used for an air gap install. Be patient. This process takes a few minutes. When the process completes, your current directory will output a tarball named `rancher-images.tar.gz`. Check that the output is in the directory.
+
+(Optional) Specify a custom output file:
+
+```bash
+./rancher-save-images.sh \
+  --image-list ./rancher-images.txt \
+  --images rancher-images-custom.tar.gz
+```
+
+**Result:** Docker begins pulling the images required for an air-gap installation. The process may take several minutes.
+
+1. Verify that the tarball was created:
+   ```bash
+   ls -lh rancher-images.tar.gz
+   ```
+
+If some images fail to pull, review the output and retry after resolving any issues.

 ### 4. Populate the private registry

@@ -163,7 +186,7 @@ Your registry must support manifests. As of April 2020, Amazon Elastic Container
   ./rancher-save-images.ps1
   ```

-   **Result:** Docker begins pulling the images used for an air gap install. Be patient. This process takes a few minutes. When the process completes, your current directory will output a tarball named `rancher-windows-images.tar.gz`. Check that the output is in the directory.
+   **Result:** Docker begins pulling the images used for an air-gap install. Be patient. This process takes a few minutes. When the process completes, your current directory will output a tarball named `rancher-windows-images.tar.gz`. Check that the output is in the directory.

 <a name="windows-3"></a>

@@ -273,7 +296,7 @@ The workstation must have Docker 18.02+ in order to support manifests, which are
   ./rancher-save-images.sh --image-list ./rancher-images.txt
   ```

-**Result:** Docker begins pulling the images used for an air gap install. Be patient. This process takes a few minutes. When the process completes, your current directory will output a tarball named `rancher-images.tar.gz`. Check that the output is in the directory.
+**Result:** Docker begins pulling the images used for an air-gap install. Be patient. This process takes a few minutes. When the process completes, your current directory will output a tarball named `rancher-images.tar.gz`. Check that the output is in the directory.

 <a name="linux-4"></a>

@@ -42,7 +42,7 @@ If you will use ARM64 hosts, the registry must support manifests. As of April 20

 1. Go to our [releases page,](https://github.com/rancher/rancher/releases) find the Rancher v2.x.x release that you want to install, and click **Assets**. Note: Don't use releases marked `rc` or `Pre-release`, as they are not stable for production environments.

-2. From the release's **Assets** section, download the following files, which are required to install Rancher in an air gap environment:
+2. From the release's **Assets** section, download the following files, which are required to install Rancher in an air-gap environment:

 | Release File   | Description  |
 | ---------------- | -------------- |
@@ -83,16 +83,39 @@ In a Kubernetes Install, if you elect to use the Rancher default self-signed TLS

 ### 3. Save the images to your workstation

-1. Make `rancher-save-images.sh` an executable:
-   ```
+(Optional) Verify the image list before pulling:
+
+```bash
+wc -l rancher-images.txt
+head rancher-images.txt
+```
+
+1. Make `rancher-save-images.sh` executable:
+   ```bash
   chmod +x rancher-save-images.sh
   ```

 1. Run `rancher-save-images.sh` with the `rancher-images.txt` image list to create a tarball of all the required images:
-   ```plain
+   ```bash
   ./rancher-save-images.sh --image-list ./rancher-images.txt
   ```
-   **Result:** Docker begins pulling the images used for an air gap install. Be patient. This process takes a few minutes. When the process completes, your current directory will output a tarball named `rancher-images.tar.gz`. Check that the output is in the directory.
+
+(Optional) Specify a custom output file:
+
+```bash
+./rancher-save-images.sh \
+  --image-list ./rancher-images.txt \
+  --images rancher-images-custom.tar.gz
+```
+
+**Result:** Docker begins pulling the images required for an air-gap installation. The process may take several minutes.
+
+1. Verify that the tarball was created:
+   ```bash
+   ls -lh rancher-images.tar.gz
+   ```
+
+If some images fail to pull, review the output and retry after resolving any issues.

 ### 4. Populate the private registry

@@ -163,7 +186,7 @@ Your registry must support manifests. As of April 2020, Amazon Elastic Container
   ./rancher-save-images.ps1
   ```

-   **Result:** Docker begins pulling the images used for an air gap install. Be patient. This process takes a few minutes. When the process completes, your current directory will output a tarball named `rancher-windows-images.tar.gz`. Check that the output is in the directory.
+   **Result:** Docker begins pulling the images used for an air-gap install. Be patient. This process takes a few minutes. When the process completes, your current directory will output a tarball named `rancher-windows-images.tar.gz`. Check that the output is in the directory.

 <a name="windows-3"></a>

@@ -273,7 +296,7 @@ The workstation must have Docker 18.02+ in order to support manifests, which are
   ./rancher-save-images.sh --image-list ./rancher-images.txt
   ```

-**Result:** Docker begins pulling the images used for an air gap install. Be patient. This process takes a few minutes. When the process completes, your current directory will output a tarball named `rancher-images.tar.gz`. Check that the output is in the directory.
+**Result:** Docker begins pulling the images used for an air-gap install. Be patient. This process takes a few minutes. When the process completes, your current directory will output a tarball named `rancher-images.tar.gz`. Check that the output is in the directory.

 <a name="linux-4"></a>

@@ -6,31 +6,55 @@ title: Troubleshooting Controlplane Nodes
  <link rel="canonical" href="https://ranchermanager.docs.rancher.com/troubleshooting/kubernetes-components/troubleshooting-controlplane-nodes"/>
 </head>

-This section applies to nodes with the `controlplane` role.
+This section applies to nodes with the `controlplane` role in RKE2 and K3s clusters.

-## Check if the Controlplane Containers are Running
+## Prerequisites

-There are three specific containers launched on nodes with the `controlplane` role:
+As RKE2 and K3s rely on `containerd` as the container runtime, `crictl` replaces Docker for container management. Before proceeding with the troubleshooting commands, configure your environment by exporting the following variables:
+
+### RKE2
+
+```bash
+export PATH=$PATH:/var/lib/rancher/rke2/bin/
+export CRI_CONFIG_FILE=/var/lib/rancher/rke2/agent/etc/crictl.yaml
+```
+
+### K3s
+
+```bash
+export PATH=$PATH:/usr/local/bin
+export CRI_CONFIG_FILE=/var/lib/rancher/k3s/agent/etc/crictl.yaml
+```
+
+## Check if the Controlplane Components are Running
+
+**RKE2**: There are three specific containers launched on nodes with the `controlplane` role:

 * `kube-apiserver`
 * `kube-controller-manager`
 * `kube-scheduler`

-The containers should have status **Up**. The duration shown after **Up** is the time the container has been running.
+The containers should have state **Running**. You can check this using `crictl`:

-```
-docker ps -a -f=name='kube-apiserver|kube-controller-manager|kube-scheduler'
+```bash
+crictl ps | grep -E 'kube-apiserver|kube-controller-manager|kube-scheduler'
 ```

 Example output:
 ```
-CONTAINER ID        IMAGE                                COMMAND                  CREATED             STATUS              PORTS               NAMES
-26c7159abbcc        rancher/hyperkube:v1.11.5-rancher1   "/opt/rke-tools/en..."   3 hours ago         Up 3 hours                              kube-apiserver
-f3d287ca4549        rancher/hyperkube:v1.11.5-rancher1   "/opt/rke-tools/en..."   3 hours ago         Up 3 hours                              kube-scheduler
-bdf3898b8063        rancher/hyperkube:v1.11.5-rancher1   "/opt/rke-tools/en..."   3 hours ago         Up 3 hours                              kube-controller-manager
+CONTAINER           IMAGE               CREATED             STATE               NAME                            ATTEMPT             POD ID              POD                                     NAMESPACE
+deb8a96948594       138b1e685e151       11 days ago         Running             kube-controller-manager         0                   0996426295dc5       kube-controller-manager                 kube-system
+f5abb4c7846e4       138b1e685e151       11 days ago         Running             kube-scheduler                  0                   80cd9f30af0be       kube-scheduler                          kube-system
+ecd8a6991c22a       138b1e685e151       11 days ago         Running             kube-apiserver                  0                   58e042fabe78c       kube-apiserver                          kube-system
 ```

-## Controlplane Container Logging
+**K3s**: These components run as embedded processes within the K3s service. They do not run as separate containers, so their status is tied to the `k3s` systemd service:
+
+```bash
+systemctl status k3s
+```
+
+## Controlplane Logging

 :::note

@@ -38,18 +62,32 @@ If you added multiple nodes with the `controlplane` role, both `kube-controller-

 :::

-The logging of the containers can contain information on what the problem could be.
+The logs can contain information on what the problem could be.

-```
-docker logs kube-apiserver
-docker logs kube-controller-manager
-docker logs kube-scheduler
+**RKE2**:
+```bash
+crictl logs $(crictl ps --name kube-apiserver -q)
+crictl logs $(crictl ps --name kube-controller-manager -q)
+crictl logs $(crictl ps --name kube-scheduler -q)
 ```

-## RKE2 Server Logging
-
-If Rancher provisions an RKE2 cluster that can't communicate with Rancher, you can run this command on a server node in the downstream cluster to get the RKE2 server logs:
-
+**K3s**:
+```bash
+journalctl -u k3s | grep -i "kube-apiserver"
+journalctl -u k3s | grep -i "kube-controller-manager"
+journalctl -u k3s | grep -i "kube-scheduler"
 ```
+
+## RKE2/K3s Server Logging
+
+If Rancher provisions an RKE2 or K3s cluster that can't communicate with Rancher, you can run this command on a server node in the downstream cluster to get the server logs:
+
+**RKE2**:
+```bash
 journalctl -u rke2-server -f
+```
+
+**K3s**:
+```bash
+journalctl -u k3s -f
 ```
@@ -42,7 +42,7 @@ If you will use ARM64 hosts, the registry must support manifests. As of April 20

 1. Go to our [releases page,](https://github.com/rancher/rancher/releases) find the Rancher v2.x.x release that you want to install, and click **Assets**. Note: Don't use releases marked `rc` or `Pre-release`, as they are not stable for production environments.

-2. From the release's **Assets** section, download the following files, which are required to install Rancher in an air gap environment:
+2. From the release's **Assets** section, download the following files, which are required to install Rancher in an air-gap environment:

 | Release File   | Description  |
 | ---------------- | -------------- |
@@ -83,16 +83,39 @@ In a Kubernetes Install, if you elect to use the Rancher default self-signed TLS

 ### 3. Save the images to your workstation

-1. Make `rancher-save-images.sh` an executable:
-   ```
+(Optional) Verify the image list before pulling:
+
+```bash
+wc -l rancher-images.txt
+head rancher-images.txt
+```
+
+1. Make `rancher-save-images.sh` executable:
+   ```bash
   chmod +x rancher-save-images.sh
   ```

 1. Run `rancher-save-images.sh` with the `rancher-images.txt` image list to create a tarball of all the required images:
-   ```plain
+   ```bash
   ./rancher-save-images.sh --image-list ./rancher-images.txt
   ```
-   **Result:** Docker begins pulling the images used for an air gap install. Be patient. This process takes a few minutes. When the process completes, your current directory will output a tarball named `rancher-images.tar.gz`. Check that the output is in the directory.
+
+(Optional) Specify a custom output file:
+
+```bash
+./rancher-save-images.sh \
+  --image-list ./rancher-images.txt \
+  --images rancher-images-custom.tar.gz
+```
+
+**Result:** Docker begins pulling the images required for an air-gap installation. The process may take several minutes.
+
+1. Verify that the tarball was created:
+   ```bash
+   ls -lh rancher-images.tar.gz
+   ```
+
+If some images fail to pull, review the output and retry after resolving any issues.

 ### 4. Populate the private registry

@@ -163,7 +186,7 @@ Your registry must support manifests. As of April 2020, Amazon Elastic Container
   ./rancher-save-images.ps1
   ```

-   **Result:** Docker begins pulling the images used for an air gap install. Be patient. This process takes a few minutes. When the process completes, your current directory will output a tarball named `rancher-windows-images.tar.gz`. Check that the output is in the directory.
+   **Result:** Docker begins pulling the images used for an air-gap install. Be patient. This process takes a few minutes. When the process completes, your current directory will output a tarball named `rancher-windows-images.tar.gz`. Check that the output is in the directory.

 <a name="windows-3"></a>

@@ -273,7 +296,7 @@ The workstation must have Docker 18.02+ in order to support manifests, which are
   ./rancher-save-images.sh --image-list ./rancher-images.txt
   ```

-**Result:** Docker begins pulling the images used for an air gap install. Be patient. This process takes a few minutes. When the process completes, your current directory will output a tarball named `rancher-images.tar.gz`. Check that the output is in the directory.
+**Result:** Docker begins pulling the images used for an air-gap install. Be patient. This process takes a few minutes. When the process completes, your current directory will output a tarball named `rancher-images.tar.gz`. Check that the output is in the directory.

 <a name="linux-4"></a>

@@ -6,31 +6,55 @@ title: Troubleshooting Controlplane Nodes
  <link rel="canonical" href="https://ranchermanager.docs.rancher.com/troubleshooting/kubernetes-components/troubleshooting-controlplane-nodes"/>
 </head>

-This section applies to nodes with the `controlplane` role.
+This section applies to nodes with the `controlplane` role in RKE2 and K3s clusters.

-## Check if the Controlplane Containers are Running
+## Prerequisites

-There are three specific containers launched on nodes with the `controlplane` role:
+As RKE2 and K3s rely on `containerd` as the container runtime, `crictl` replaces Docker for container management. Before proceeding with the troubleshooting commands, configure your environment by exporting the following variables:
+
+### RKE2
+
+```bash
+export PATH=$PATH:/var/lib/rancher/rke2/bin/
+export CRI_CONFIG_FILE=/var/lib/rancher/rke2/agent/etc/crictl.yaml
+```
+
+### K3s
+
+```bash
+export PATH=$PATH:/usr/local/bin
+export CRI_CONFIG_FILE=/var/lib/rancher/k3s/agent/etc/crictl.yaml
+```
+
+## Check if the Controlplane Components are Running
+
+**RKE2**: There are three specific containers launched on nodes with the `controlplane` role:

 * `kube-apiserver`
 * `kube-controller-manager`
 * `kube-scheduler`

-The containers should have status **Up**. The duration shown after **Up** is the time the container has been running.
+The containers should have state **Running**. You can check this using `crictl`:

-```
-docker ps -a -f=name='kube-apiserver|kube-controller-manager|kube-scheduler'
+```bash
+crictl ps | grep -E 'kube-apiserver|kube-controller-manager|kube-scheduler'
 ```

 Example output:
 ```
-CONTAINER ID        IMAGE                                COMMAND                  CREATED             STATUS              PORTS               NAMES
-26c7159abbcc        rancher/hyperkube:v1.11.5-rancher1   "/opt/rke-tools/en..."   3 hours ago         Up 3 hours                              kube-apiserver
-f3d287ca4549        rancher/hyperkube:v1.11.5-rancher1   "/opt/rke-tools/en..."   3 hours ago         Up 3 hours                              kube-scheduler
-bdf3898b8063        rancher/hyperkube:v1.11.5-rancher1   "/opt/rke-tools/en..."   3 hours ago         Up 3 hours                              kube-controller-manager
+CONTAINER           IMAGE               CREATED             STATE               NAME                            ATTEMPT             POD ID              POD                                     NAMESPACE
+deb8a96948594       138b1e685e151       11 days ago         Running             kube-controller-manager         0                   0996426295dc5       kube-controller-manager                 kube-system
+f5abb4c7846e4       138b1e685e151       11 days ago         Running             kube-scheduler                  0                   80cd9f30af0be       kube-scheduler                          kube-system
+ecd8a6991c22a       138b1e685e151       11 days ago         Running             kube-apiserver                  0                   58e042fabe78c       kube-apiserver                          kube-system
 ```

-## Controlplane Container Logging
+**K3s**: These components run as embedded processes within the K3s service. They do not run as separate containers, so their status is tied to the `k3s` systemd service:
+
+```bash
+systemctl status k3s
+```
+
+## Controlplane Logging

 :::note

@@ -38,18 +62,32 @@ If you added multiple nodes with the `controlplane` role, both `kube-controller-

 :::

-The logging of the containers can contain information on what the problem could be.
+The logs can contain information on what the problem could be.

-```
-docker logs kube-apiserver
-docker logs kube-controller-manager
-docker logs kube-scheduler
+**RKE2**:
+```bash
+crictl logs $(crictl ps --name kube-apiserver -q)
+crictl logs $(crictl ps --name kube-controller-manager -q)
+crictl logs $(crictl ps --name kube-scheduler -q)
 ```

-## RKE2 Server Logging
-
-If Rancher provisions an RKE2 cluster that can't communicate with Rancher, you can run this command on a server node in the downstream cluster to get the RKE2 server logs:
-
+**K3s**:
+```bash
+journalctl -u k3s | grep -i "kube-apiserver"
+journalctl -u k3s | grep -i "kube-controller-manager"
+journalctl -u k3s | grep -i "kube-scheduler"
 ```
+
+## RKE2/K3s Server Logging
+
+If Rancher provisions an RKE2 or K3s cluster that can't communicate with Rancher, you can run this command on a server node in the downstream cluster to get the server logs:
+
+**RKE2**:
+```bash
 journalctl -u rke2-server -f
+```
+
+**K3s**:
+```bash
+journalctl -u k3s -f
 ```
@@ -42,7 +42,7 @@ If you will use ARM64 hosts, the registry must support manifests. As of April 20

 1. Go to our [releases page,](https://github.com/rancher/rancher/releases) find the Rancher v2.x.x release that you want to install, and click **Assets**. Note: Don't use releases marked `rc` or `Pre-release`, as they are not stable for production environments.

-2. From the release's **Assets** section, download the following files, which are required to install Rancher in an air gap environment:
+2. From the release's **Assets** section, download the following files, which are required to install Rancher in an air-gap environment:

 | Release File   | Description  |
 | ---------------- | -------------- |
@@ -83,16 +83,39 @@ In a Kubernetes Install, if you elect to use the Rancher default self-signed TLS

 ### 3. Save the images to your workstation

-1. Make `rancher-save-images.sh` an executable:
-   ```
+(Optional) Verify the image list before pulling:
+
+```bash
+wc -l rancher-images.txt
+head rancher-images.txt
+```
+
+1. Make `rancher-save-images.sh` executable:
+   ```bash
   chmod +x rancher-save-images.sh
   ```

 1. Run `rancher-save-images.sh` with the `rancher-images.txt` image list to create a tarball of all the required images:
-   ```plain
+   ```bash
   ./rancher-save-images.sh --image-list ./rancher-images.txt
   ```
-   **Result:** Docker begins pulling the images used for an air gap install. Be patient. This process takes a few minutes. When the process completes, your current directory will output a tarball named `rancher-images.tar.gz`. Check that the output is in the directory.
+
+(Optional) Specify a custom output file:
+
+```bash
+./rancher-save-images.sh \
+  --image-list ./rancher-images.txt \
+  --images rancher-images-custom.tar.gz
+```
+
+**Result:** Docker begins pulling the images required for an air-gap installation. The process may take several minutes.
+
+1. Verify that the tarball was created:
+   ```bash
+   ls -lh rancher-images.tar.gz
+   ```
+
+If some images fail to pull, review the output and retry after resolving any issues.

 ### 4. Populate the private registry

@@ -163,7 +186,7 @@ Your registry must support manifests. As of April 2020, Amazon Elastic Container
   ./rancher-save-images.ps1
   ```

-   **Result:** Docker begins pulling the images used for an air gap install. Be patient. This process takes a few minutes. When the process completes, your current directory will output a tarball named `rancher-windows-images.tar.gz`. Check that the output is in the directory.
+   **Result:** Docker begins pulling the images used for an air-gap install. Be patient. This process takes a few minutes. When the process completes, your current directory will output a tarball named `rancher-windows-images.tar.gz`. Check that the output is in the directory.

 <a name="windows-3"></a>

@@ -273,7 +296,7 @@ The workstation must have Docker 18.02+ in order to support manifests, which are
   ./rancher-save-images.sh --image-list ./rancher-images.txt
   ```

-**Result:** Docker begins pulling the images used for an air gap install. Be patient. This process takes a few minutes. When the process completes, your current directory will output a tarball named `rancher-images.tar.gz`. Check that the output is in the directory.
+**Result:** Docker begins pulling the images used for an air-gap install. Be patient. This process takes a few minutes. When the process completes, your current directory will output a tarball named `rancher-images.tar.gz`. Check that the output is in the directory.

 <a name="linux-4"></a>

@@ -6,31 +6,55 @@ title: Troubleshooting Controlplane Nodes
  <link rel="canonical" href="https://ranchermanager.docs.rancher.com/troubleshooting/kubernetes-components/troubleshooting-controlplane-nodes"/>
 </head>

-This section applies to nodes with the `controlplane` role.
+This section applies to nodes with the `controlplane` role in RKE2 and K3s clusters.

-## Check if the Controlplane Containers are Running
+## Prerequisites

-There are three specific containers launched on nodes with the `controlplane` role:
+As RKE2 and K3s rely on `containerd` as the container runtime, `crictl` replaces Docker for container management. Before proceeding with the troubleshooting commands, configure your environment by exporting the following variables:
+
+### RKE2
+
+```bash
+export PATH=$PATH:/var/lib/rancher/rke2/bin/
+export CRI_CONFIG_FILE=/var/lib/rancher/rke2/agent/etc/crictl.yaml
+```
+
+### K3s
+
+```bash
+export PATH=$PATH:/usr/local/bin
+export CRI_CONFIG_FILE=/var/lib/rancher/k3s/agent/etc/crictl.yaml
+```
+
+## Check if the Controlplane Components are Running
+
+**RKE2**: There are three specific containers launched on nodes with the `controlplane` role:

 * `kube-apiserver`
 * `kube-controller-manager`
 * `kube-scheduler`

-The containers should have status **Up**. The duration shown after **Up** is the time the container has been running.
+The containers should have state **Running**. You can check this using `crictl`:

-```
-docker ps -a -f=name='kube-apiserver|kube-controller-manager|kube-scheduler'
+```bash
+crictl ps | grep -E 'kube-apiserver|kube-controller-manager|kube-scheduler'
 ```

 Example output:
 ```
-CONTAINER ID        IMAGE                                COMMAND                  CREATED             STATUS              PORTS               NAMES
-26c7159abbcc        rancher/hyperkube:v1.11.5-rancher1   "/opt/rke-tools/en..."   3 hours ago         Up 3 hours                              kube-apiserver
-f3d287ca4549        rancher/hyperkube:v1.11.5-rancher1   "/opt/rke-tools/en..."   3 hours ago         Up 3 hours                              kube-scheduler
-bdf3898b8063        rancher/hyperkube:v1.11.5-rancher1   "/opt/rke-tools/en..."   3 hours ago         Up 3 hours                              kube-controller-manager
+CONTAINER           IMAGE               CREATED             STATE               NAME                            ATTEMPT             POD ID              POD                                     NAMESPACE
+deb8a96948594       138b1e685e151       11 days ago         Running             kube-controller-manager         0                   0996426295dc5       kube-controller-manager                 kube-system
+f5abb4c7846e4       138b1e685e151       11 days ago         Running             kube-scheduler                  0                   80cd9f30af0be       kube-scheduler                          kube-system
+ecd8a6991c22a       138b1e685e151       11 days ago         Running             kube-apiserver                  0                   58e042fabe78c       kube-apiserver                          kube-system
 ```

-## Controlplane Container Logging
+**K3s**: These components run as embedded processes within the K3s service. They do not run as separate containers, so their status is tied to the `k3s` systemd service:
+
+```bash
+systemctl status k3s
+```
+
+## Controlplane Logging

 :::note

@@ -38,18 +62,32 @@ If you added multiple nodes with the `controlplane` role, both `kube-controller-

 :::

-The logging of the containers can contain information on what the problem could be.
+The logs can contain information on what the problem could be.

-```
-docker logs kube-apiserver
-docker logs kube-controller-manager
-docker logs kube-scheduler
+**RKE2**:
+```bash
+crictl logs $(crictl ps --name kube-apiserver -q)
+crictl logs $(crictl ps --name kube-controller-manager -q)
+crictl logs $(crictl ps --name kube-scheduler -q)
 ```

-## RKE2 Server Logging
-
-If Rancher provisions an RKE2 cluster that can't communicate with Rancher, you can run this command on a server node in the downstream cluster to get the RKE2 server logs:
-
+**K3s**:
+```bash
+journalctl -u k3s | grep -i "kube-apiserver"
+journalctl -u k3s | grep -i "kube-controller-manager"
+journalctl -u k3s | grep -i "kube-scheduler"
 ```
+
+## RKE2/K3s Server Logging
+
+If Rancher provisions an RKE2 or K3s cluster that can't communicate with Rancher, you can run this command on a server node in the downstream cluster to get the server logs:
+
+**RKE2**:
+```bash
 journalctl -u rke2-server -f
+```
+
+**K3s**:
+```bash
+journalctl -u k3s -f
 ```