From b6a83f0a86da46955172d56a9520adb7ed5b6b3d Mon Sep 17 00:00:00 2001 From: Billy Tat Date: Mon, 26 Apr 2021 19:58:41 -0700 Subject: [PATCH 1/2] Logging V2 update --- content/rancher/v2.5/en/logging/_index.md | 100 ++++++++++++++++++++-- 1 file changed, 91 insertions(+), 9 deletions(-) diff --git a/content/rancher/v2.5/en/logging/_index.md b/content/rancher/v2.5/en/logging/_index.md index 6aa6c4be2c5..38feea88c14 100644 --- a/content/rancher/v2.5/en/logging/_index.md +++ b/content/rancher/v2.5/en/logging/_index.md @@ -13,10 +13,14 @@ aliases: - [Changes in Rancher v2.5](#changes-in-rancher-v2-5) - [Enabling Logging for Rancher Managed Clusters](#enabling-logging-for-rancher-managed-clusters) - [Uninstall Logging](#uninstall-logging) +- [Windows Support](#windows-support) - [Role-based Access Control](#role-based-access-control) - [Configuring the Logging Application](#configuring-the-logging-application) +- [Examples](#examples) - [Working with a Custom Docker Root Directory](#working-with-a-custom-docker-root-directory) - [Working with Taints and Tolerations](#working-with-taints-and-tolerations) +- [Additional Logging Sources](#additional-logging-sources) +- [Troubleshooting](#troubleshooting) # Changes in Rancher v2.5 @@ -56,6 +60,29 @@ You can enable the logging for a Rancher managed cluster by going to the Apps pa **Result** `rancher-logging` is uninstalled. +# Windows Support + +{{% tabs %}} +{{% tab "Rancher v2.5.8" %}} +As of Rancher v2.5.8, logging support for Windows clusters has been added and logs can be collected from Windows nodes. + +### Enabling and Disabling Windows Node Logging + +You can enable or disable Windows node logging by setting `global.cattle.windows.enabled` to either `true` or `false` in the `values.yaml`. +By default, Windows node logging will be enabled if the logging application is installed on a Windows cluster. +In this scenario, setting `global.cattle.windows.enabled` to `false` will disable Windows node logging on the cluster. +When disabled, logs will still be collected from Linux nodes within the Windows cluster. + +> Note: Currently an [issue](https://github.com/rancher/rancher/issues/32325) exists where Windows nodeAgents are not deleted when performing a `helm upgrade` after disabling Windows logging in a Windows cluster. In this scenario, users may need to manually remove the Windows nodeAgents if they are already installed. + +{{% /tab %}} +{{% tab "Rancher v2.5.0-2.5.7" %}} +Clusters with Windows workers support exporting logs from Linux nodes, but Windows node logs are currently unable to be exported. +Only Linux node logs are able to be exported. + +{{% /tab %}} +{{% /tabs %}} + # Role-based Access Control Rancher logging has two roles, `logging-admin` and `logging-view`. @@ -189,9 +216,50 @@ spec: - "devteam-splunk" ``` +### Output to Syslog + +Let's say you wanted to send all logs in your cluster to an `syslog` server. First, we create a cluster output. + +```yaml +apiVersion: logging.banzaicloud.io/v1beta1 + kind: ClusterOutput + metadata: + name: "example-syslog" + namespace: "cattle-logging-system" + spec: + syslog: + buffer: + timekey: 30s + timekey_use_utc: true + timekey_wait: 10s + flush_interval: 5s + format: + type: json + app_name_field: test + host: syslog.example.com + insecure: true + port: 514 + transport: tcp +``` + +Now that we have configured where we want the logs to go, let's configure all logs to go to that output. + +```yaml +apiVersion: logging.banzaicloud.io/v1beta1 + kind: ClusterFlow + metadata: + name: "all-logs" + namespace: cattle-logging-system + spec: + globalOutputRefs: + - "example-syslog" +``` + ### Unsupported Output -For the final example, we create an output to write logs to a destination that is not supported out of the box (e.g. syslog): +For the final example, we create an output to write logs to a destination that is not supported out of the box: + +> **Note on syslog** As of Rancher v2.5.4, `syslog` is a supported output. However, this example still provides an overview on using unsupported plugins. ```yaml apiVersion: v1 @@ -281,14 +349,14 @@ spec: Let's break down what is happening here. First, we create a deployment of a container that has the additional `syslog` plugin and accepts logs forwarded from another `fluentd`. Next we create an output configured as a forwarder to our deployment. The deployment `fluentd` will then forward all logs to the configured `syslog` destination. -> **Note on syslog** Official `syslog` support is coming in Rancher v2.5.4. However, this example still provides an overview on using unsupported plugins. - # Working with a Custom Docker Root Directory _Applies to v2.5.6+_ If using a custom Docker root directory, you can set `global.dockerRootDirectory` in `values.yaml`. This will ensure that the Logging CRs created will use your specified path rather than the default Docker `data-root` location. +Note that this is only affects Linux nodes. +If there are any Windows nodes in the cluster, the change will not be applicable to those nodes. # Working with Taints and Tolerations @@ -302,7 +370,7 @@ Both provide choice for the what node(s) the pod will run on. By default, Rancher taints all Linux nodes with `cattle.io/os=linux`, and does not taint Windows nodes. The logging stack pods have `tolerations` for this taint, which enables them to run on Linux nodes. -Moreover, we can populate the `nodeSelector` to ensure that our pods *only* run on Linux nodes. +Moreover, the logging stack pods have a `nodeSelector` added to ensure that they *only* run on Linux nodes. Let's look at an example pod YAML file with these settings... ```yaml @@ -323,11 +391,6 @@ spec: In the above example, we ensure that our pod only runs on Linux nodes, and we add a `toleration` for the taint we have on all of our Linux nodes. You can do the same with Rancher's existing taints, or with your own custom ones. -### Windows Support - -Clusters with Windows workers support exporting logs from Linux nodes, but Windows node logs are currently unable to be exported. -Only Linux node logs are able to be exported. - ### Adding NodeSelector Settings and Tolerations for Custom Taints If you would like to add your own `nodeSelector` settings, or if you would like to add `tolerations` for additional taints, you can pass the following to the chart's values. @@ -349,6 +412,25 @@ fluentbit_tolerations: # insert tolerations list for fluentbit containers only... ``` +# Additional Logging Sources + +Logs for [control plane components](https://kubernetes.io/docs/concepts/overview/components/#control-plane-components) and [node components](https://kubernetes.io/docs/concepts/overview/components/#node-components) can be collected from different sources. + +The following table summarizes the supported logging sources for each node type and the path to the logs: + +| Logging Source | Linux Nodes (including in Windows cluster) | Windows Nodes | +| --- | --- | ---| +| RKE | ✓ | ✓ | +| RKE2 | ✓ | | +| K3s | ✓ | | +| AKS | ✓ | | +| EKS | ✓ | | +| GKE | ✓ | | + +To enable hosted Kubernetes providers as additional logging sources, go to **Cluster Exploere > Logging > Chart Options** and select the **Enable enhanced cloud provider logging** option. +When enabled, Rancher collects all node and control plane logs the provider has made available, which may vary between providers. +If you're already using a cloud provider's own logging solution such as AWS CloudWatch or Google Cloud operations suite (formerly Stackdriver), it is not necessary to enable this option as the native solution will have unrestricted access to all logs. + # Troubleshooting ### The `cattle-logging` Namespace Being Recreated From fd37e756954330d4236026430025c4349e17b4a1 Mon Sep 17 00:00:00 2001 From: Billy Tat Date: Fri, 30 Apr 2021 15:45:31 -0700 Subject: [PATCH 2/2] address feedback --- content/rancher/v2.5/en/logging/_index.md | 28 +++++++++++++++++------ 1 file changed, 21 insertions(+), 7 deletions(-) diff --git a/content/rancher/v2.5/en/logging/_index.md b/content/rancher/v2.5/en/logging/_index.md index 38feea88c14..4dab871ff80 100644 --- a/content/rancher/v2.5/en/logging/_index.md +++ b/content/rancher/v2.5/en/logging/_index.md @@ -69,7 +69,7 @@ As of Rancher v2.5.8, logging support for Windows clusters has been added and lo ### Enabling and Disabling Windows Node Logging You can enable or disable Windows node logging by setting `global.cattle.windows.enabled` to either `true` or `false` in the `values.yaml`. -By default, Windows node logging will be enabled if the logging application is installed on a Windows cluster. +By default, Windows node logging will be enabled if the Cluster Explorer UI is used to install the logging application on a Windows cluster. In this scenario, setting `global.cattle.windows.enabled` to `false` will disable Windows node logging on the cluster. When disabled, logs will still be collected from Linux nodes within the Windows cluster. @@ -80,6 +80,7 @@ When disabled, logs will still be collected from Linux nodes within the Windows Clusters with Windows workers support exporting logs from Linux nodes, but Windows node logs are currently unable to be exported. Only Linux node logs are able to be exported. +To allow the logging pods to be scheduled on Linux nodes, tolerations must be added to the pods. Refer to the [Working with Taints and Tolerations](#working-with-taints-and-tolerations) section for details and an example. {{% /tab %}} {{% /tabs %}} @@ -355,7 +356,7 @@ _Applies to v2.5.6+_ If using a custom Docker root directory, you can set `global.dockerRootDirectory` in `values.yaml`. This will ensure that the Logging CRs created will use your specified path rather than the default Docker `data-root` location. -Note that this is only affects Linux nodes. +Note that this only affects Linux nodes. If there are any Windows nodes in the cluster, the change will not be applicable to those nodes. # Working with Taints and Tolerations @@ -366,11 +367,23 @@ Unless the pods have a `toleration` for that node's taint, they will run on othe Using `nodeSelector` gives pods an affinity towards certain nodes. Both provide choice for the what node(s) the pod will run on. + ### Default Implementation in Rancher's Logging Stack +{{% tabs %}} +{{% tab "Rancher v2.5.8" %}} By default, Rancher taints all Linux nodes with `cattle.io/os=linux`, and does not taint Windows nodes. The logging stack pods have `tolerations` for this taint, which enables them to run on Linux nodes. -Moreover, the logging stack pods have a `nodeSelector` added to ensure that they *only* run on Linux nodes. +Moreover, most logging stack pods run on Linux only and have a `nodeSelector` added to ensure they run on Linux nodes. + +{{% /tab %}} +{{% tab "Rancher v2.5.0-2.5.7" %}} +By default, Rancher taints all Linux nodes with `cattle.io/os=linux`, and does not taint Windows nodes. +The logging stack pods have `tolerations` for this taint, which enables them to run on Linux nodes. +Moreover, we can populate the `nodeSelector` to ensure that our pods *only* run on Linux nodes. + +{{% /tab %}} +{{% /tabs %}} Let's look at an example pod YAML file with these settings... ```yaml @@ -414,9 +427,10 @@ fluentbit_tolerations: # Additional Logging Sources -Logs for [control plane components](https://kubernetes.io/docs/concepts/overview/components/#control-plane-components) and [node components](https://kubernetes.io/docs/concepts/overview/components/#node-components) can be collected from different sources. +By default, Rancher collects logs for [control plane components](https://kubernetes.io/docs/concepts/overview/components/#control-plane-components) and [node components](https://kubernetes.io/docs/concepts/overview/components/#node-components) for all cluster types. +In some cases, Rancher may be able to collect additional logs. -The following table summarizes the supported logging sources for each node type and the path to the logs: +The following table summarizes the sources where additional logs may be collected for each node types: | Logging Source | Linux Nodes (including in Windows cluster) | Windows Nodes | | --- | --- | ---| @@ -427,8 +441,8 @@ The following table summarizes the supported logging sources for each node type | EKS | ✓ | | | GKE | ✓ | | -To enable hosted Kubernetes providers as additional logging sources, go to **Cluster Exploere > Logging > Chart Options** and select the **Enable enhanced cloud provider logging** option. -When enabled, Rancher collects all node and control plane logs the provider has made available, which may vary between providers. +To enable hosted Kubernetes providers as additional logging sources, go to **Cluster Explorer > Logging > Chart Options** and select the **Enable enhanced cloud provider logging** option. +When enabled, Rancher collects all additional node and control plane logs the provider has made available, which may vary between providers. If you're already using a cloud provider's own logging solution such as AWS CloudWatch or Google Cloud operations suite (formerly Stackdriver), it is not necessary to enable this option as the native solution will have unrestricted access to all logs. # Troubleshooting