From 48dab2c8403531f2720a75e31529de36da2a2750 Mon Sep 17 00:00:00 2001 From: Nick Gerace Date: Wed, 3 Mar 2021 17:49:46 -0500 Subject: [PATCH 1/3] Add cattle-ogging ns recreation troubleshooting --- .../rancher/v2.x/en/logging/v2.5/_index.md | 38 +++++++++++++++++-- 1 file changed, 34 insertions(+), 4 deletions(-) diff --git a/content/rancher/v2.x/en/logging/v2.5/_index.md b/content/rancher/v2.x/en/logging/v2.5/_index.md index 5a839ff1d70..8bf71dc23e3 100644 --- a/content/rancher/v2.x/en/logging/v2.5/_index.md +++ b/content/rancher/v2.x/en/logging/v2.5/_index.md @@ -56,7 +56,7 @@ Rancher logging has two roles, `logging-admin` and `logging-view`. - `logging-admin` gives users full access to namespaced flows and outputs - `logging-view` allows users to *view* namespaced flows and outputs, and cluster flows and outputs -> **Why choose one role over the other?** Edit access to cluster flow and cluster output resources is powerful. Any user with it has edit access for all logs in the cluster. +> **Why choose one role over the other?** Edit access to cluster flow and cluster output resources is powerful. Any user with it has edit access for all logs in the cluster. In Rancher, the cluster administrator role is the only role with full access to all `rancher-logging` resources. Cluster members are not able to edit or read any logging resources. Project owners and members have the following privileges: @@ -116,7 +116,7 @@ metadata: spec: globalOutputRefs: - "example-es" -``` +``` We should now see our configured index with logs in it. @@ -152,7 +152,7 @@ spec: containers: - name: generator image: paynejacob/loggenerator:latest -``` +``` With `coolapp` running, we will follow a similar path as when we created a cluster output. However, unlike cluster outputs, we create our output in our application's namespace. @@ -301,7 +301,7 @@ spec: - key: cattle.io/os operator: "Equal" value: "linux" - effect: NoSchedule + effect: NoSchedule nodeSelector: kubernetes.io/os: linux ``` @@ -334,3 +334,33 @@ However, if you would like to add tolerations for *only* the `fluentbit` contain fluentbit_tolerations: # insert tolerations list for fluentbit containers only... ``` + +# Troubleshooting + +### The `cattle-logging` Namespace Being Recreated + +If your cluster previously deployed logging from the Cluster Manager UI, you may encounter an issue where its `cattle-logging` namespace is continually being recreated. + +The solution is to delete all `clusterloggings.management.cattle.io` and `projectloggings.management.cattle.io` custom resources from the cluster specific namespace in the management cluster. +The existence of these custom resources causes Rancher to create the `cattle-logging` namespace in the downstream cluster if it does not exist. + +The cluster namespace matches the cluster ID, so we need to find the cluster ID for each cluster. + +1. In your web browser, navigate to your cluster(s) in either the Cluster Manager UI or the Cluster Explorer UI. +2. Copy the `` portion from one of the URLs below. The `` portion is the cluster namespace name. + +```bash +# Cluster Management UI +https:///c// + +# Cluster Explorer UI (Dashboard) +https:///dashboard/c// +``` + +Now that we have the `` namespace, we can delete the CRs that cause `cattle-logging` to be continually recreated. +*Warning:* ensure that logging, the version installed from the Cluster Manager UI, is not currently in use. + +```bash +kubectl delete clusterloggings.management.cattle.io -n +kubectl delete projectloggings.management.cattle.io -n +``` From 76efcfa39c3f7d79223528337e172697e2a50eb1 Mon Sep 17 00:00:00 2001 From: Catherine Luse Date: Mon, 5 Apr 2021 17:10:07 -0700 Subject: [PATCH 2/3] Add cattle-logging ns recreation troubleshooting #3064 --- content/rancher/v2.5/en/logging/_index.md | 52 +++++++++++++++-------- 1 file changed, 34 insertions(+), 18 deletions(-) diff --git a/content/rancher/v2.5/en/logging/_index.md b/content/rancher/v2.5/en/logging/_index.md index b39293e58f7..541b5774e9d 100644 --- a/content/rancher/v2.5/en/logging/_index.md +++ b/content/rancher/v2.5/en/logging/_index.md @@ -62,7 +62,7 @@ Rancher logging has two roles, `logging-admin` and `logging-view`. - `logging-admin` gives users full access to namespaced flows and outputs - `logging-view` allows users to *view* namespaced flows and outputs, and cluster flows and outputs -> **Why choose one role over the other?** Edit access to cluster flow and cluster output resources is powerful. Any user with it has edit access for all logs in the cluster. +> **Why choose one role over the other?** Edit access to cluster flow and cluster output resources is powerful. Any user with it has edit access for all logs in the cluster. In Rancher, the cluster administrator role is the only role with full access to all `rancher-logging` resources. Cluster members are not able to edit or read any logging resources. Project owners and members have the following privileges: @@ -122,7 +122,7 @@ metadata: spec: globalOutputRefs: - "example-es" -``` +``` We should now see our configured index with logs in it. @@ -158,7 +158,7 @@ spec: containers: - name: generator image: paynejacob/loggenerator:latest -``` +``` With `coolapp` running, we will follow a similar path as when we created a cluster output. However, unlike cluster outputs, we create our output in our application's namespace. @@ -282,13 +282,6 @@ Let's break down what is happening here. First, we create a deployment of a cont > **Note on syslog** Official `syslog` support is coming in Rancher v2.5.4. However, this example still provides an overview on using unsupported plugins. -# Working with a Custom Docker Root Directory - -_Applies to v2.5.6+_ - -If using a custom Docker root directory, you can set `global.dockerRootDirectory` in `values.yaml`. -This will ensure that the Logging CRs created will use your specified path rather than the default Docker `data-root` location. - # Working with Taints and Tolerations "Tainting" a Kubernetes node causes pods to repel running on that node. @@ -314,7 +307,7 @@ spec: - key: cattle.io/os operator: "Equal" value: "linux" - effect: NoSchedule + effect: NoSchedule nodeSelector: kubernetes.io/os: linux ``` @@ -322,13 +315,6 @@ spec: In the above example, we ensure that our pod only runs on Linux nodes, and we add a `toleration` for the taint we have on all of our Linux nodes. You can do the same with Rancher's existing taints, or with your own custom ones. -# Working with a Custom Docker Root Directory - -_Applies to v2.5.6+_ - -If using a custom Docker root directory, you can set `global.dockerRootDirectory` in `values.yaml`. -This will ensure that the Logging CRs created will use your specified path rather than the default Docker `data-root` location. - ### Windows Support Clusters with Windows workers support exporting logs from Linux nodes, but Windows node logs are currently unable to be exported. @@ -354,3 +340,33 @@ However, if you would like to add tolerations for *only* the `fluentbit` contain fluentbit_tolerations: # insert tolerations list for fluentbit containers only... ``` + +# Troubleshooting + +### The `cattle-logging` Namespace Being Recreated + +If your cluster previously deployed logging from the Cluster Manager UI, you may encounter an issue where its `cattle-logging` namespace is continually being recreated. + +The solution is to delete all `clusterloggings.management.cattle.io` and `projectloggings.management.cattle.io` custom resources from the cluster specific namespace in the management cluster. +The existence of these custom resources causes Rancher to create the `cattle-logging` namespace in the downstream cluster if it does not exist. + +The cluster namespace matches the cluster ID, so we need to find the cluster ID for each cluster. + +1. In your web browser, navigate to your cluster(s) in either the Cluster Manager UI or the Cluster Explorer UI. +2. Copy the `` portion from one of the URLs below. The `` portion is the cluster namespace name. + +```bash +# Cluster Management UI +https:///c// + +# Cluster Explorer UI (Dashboard) +https:///dashboard/c// +``` + +Now that we have the `` namespace, we can delete the CRs that cause `cattle-logging` to be continually recreated. +*Warning:* ensure that logging, the version installed from the Cluster Manager UI, is not currently in use. + +```bash +kubectl delete clusterloggings.management.cattle.io -n +kubectl delete projectloggings.management.cattle.io -n +``` From 3708ab43a1d0ccfad5f19109a33154ed000d09e2 Mon Sep 17 00:00:00 2001 From: Catherine Luse Date: Tue, 6 Apr 2021 09:41:18 -0700 Subject: [PATCH 3/3] Add 'Working with a Custom Docker Root Directory' to toc and v2.5 section --- content/rancher/v2.5/en/logging/_index.md | 8 ++++++++ content/rancher/v2.x/en/logging/v2.5/_index.md | 1 + 2 files changed, 9 insertions(+) diff --git a/content/rancher/v2.5/en/logging/_index.md b/content/rancher/v2.5/en/logging/_index.md index 541b5774e9d..6aa6c4be2c5 100644 --- a/content/rancher/v2.5/en/logging/_index.md +++ b/content/rancher/v2.5/en/logging/_index.md @@ -15,6 +15,7 @@ aliases: - [Uninstall Logging](#uninstall-logging) - [Role-based Access Control](#role-based-access-control) - [Configuring the Logging Application](#configuring-the-logging-application) +- [Working with a Custom Docker Root Directory](#working-with-a-custom-docker-root-directory) - [Working with Taints and Tolerations](#working-with-taints-and-tolerations) # Changes in Rancher v2.5 @@ -282,6 +283,13 @@ Let's break down what is happening here. First, we create a deployment of a cont > **Note on syslog** Official `syslog` support is coming in Rancher v2.5.4. However, this example still provides an overview on using unsupported plugins. +# Working with a Custom Docker Root Directory + +_Applies to v2.5.6+_ + +If using a custom Docker root directory, you can set `global.dockerRootDirectory` in `values.yaml`. +This will ensure that the Logging CRs created will use your specified path rather than the default Docker `data-root` location. + # Working with Taints and Tolerations "Tainting" a Kubernetes node causes pods to repel running on that node. diff --git a/content/rancher/v2.x/en/logging/v2.5/_index.md b/content/rancher/v2.x/en/logging/v2.5/_index.md index 48d096cb223..851d3ba6c25 100644 --- a/content/rancher/v2.x/en/logging/v2.5/_index.md +++ b/content/rancher/v2.x/en/logging/v2.5/_index.md @@ -9,6 +9,7 @@ weight: 1 - [Uninstall Logging](#uninstall-logging) - [Role-based Access Control](#role-based-access-control) - [Configuring the Logging Application](#configuring-the-logging-application) +- [Working with a Custom Docker Root Directory](#working-with-a-custom-docker-root-directory) - [Working with Taints and Tolerations](#working-with-taints-and-tolerations) # Changes in Rancher v2.5