mirror of
https://github.com/rancher/rancher-docs.git
synced 2026-09-26 13:08:11 +00:00
Updating site structure per feedback
This commit is contained in:
Jennifer Travinski
committed by
Billy Tat
Billy Tat
parent
47c1e6e048
commit
0ffe457668
+74
@@ -0,0 +1,74 @@
|
||||
---
|
||||
title: Flows and ClusterFlows
|
||||
---
|
||||
|
||||
For the full details on configuring `Flows` and `ClusterFlows`, see the [Banzai Cloud Logging operator documentation.](https://banzaicloud.com/docs/one-eye/logging-operator/configuration/flow/)
|
||||
|
||||
|
||||
## Flows
|
||||
|
||||
A `Flow` defines which logs to collect and filter and which output to send the logs to.
|
||||
|
||||
The `Flow` is a namespaced resource, which means logs will only be collected from the namespace that the `Flow` is deployed in.
|
||||
|
||||
`Flows` can be configured by filling out forms in the Rancher UI.
|
||||
|
||||
For more details about the `Flow` custom resource, see [FlowSpec.](https://banzaicloud.com/docs/one-eye/logging-operator/configuration/crds/v1beta1/flow_types/)
|
||||
|
||||
### Matches
|
||||
|
||||
Match statements are used to select which containers to pull logs from.
|
||||
|
||||
You can specify match statements to select or exclude logs according to Kubernetes labels, container and host names. Match statements are evaluated in the order they are defined and processed only until the first matching select or exclude rule applies.
|
||||
|
||||
Matches can be configured by filling out the `Flow` or `ClusterFlow` forms in the Rancher UI.
|
||||
|
||||
For detailed examples on using the match statement, see the [official documentation on log routing.](https://banzaicloud.com/docs/one-eye/logging-operator/configuration/log-routing/)
|
||||
|
||||
### Filters
|
||||
|
||||
You can define one or more filters within a `Flow`. Filters can perform various actions on the logs, for example, add additional data, transform the logs, or parse values from the records. The filters in the `Flow` are applied in the order in the definition.
|
||||
|
||||
For a list of filters supported by the Banzai Cloud Logging operator, see [this page.](https://banzaicloud.com/docs/one-eye/logging-operator/configuration/plugins/filters/)
|
||||
|
||||
Filters need to be configured in YAML.
|
||||
|
||||
### Outputs
|
||||
|
||||
This `Output` will receive logs from the `Flow`. Because the `Flow` is a namespaced resource, the `Output` must reside in same namespace as the `Flow`.
|
||||
|
||||
`Outputs` can be referenced when filling out the `Flow` or `ClusterFlow` forms in the Rancher UI.
|
||||
|
||||
## ClusterFlows
|
||||
|
||||
Matches, filters and `Outputs` are configured for `ClusterFlows` in the same way that they are configured for `Flows`. The key difference is that the `ClusterFlow` is scoped at the cluster level and can configure log collection across all namespaces.
|
||||
|
||||
`ClusterFlows` can be configured by filling out forms in the Rancher UI.
|
||||
|
||||
After `ClusterFlow` selects logs from all namespaces in the cluster, logs from the cluster will be collected and logged to the selected `ClusterOutput`.
|
||||
|
||||
## YAML Example
|
||||
|
||||
The following example `Flow` transforms the log messages from the default namespace and sends them to an S3 `Output`:
|
||||
|
||||
```yaml
|
||||
apiVersion: logging.banzaicloud.io/v1beta1
|
||||
kind: Flow
|
||||
metadata:
|
||||
name: flow-sample
|
||||
namespace: default
|
||||
spec:
|
||||
filters:
|
||||
- parser:
|
||||
remove_key_name_field: true
|
||||
parse:
|
||||
type: nginx
|
||||
- tag_normaliser:
|
||||
format: ${namespace_name}.${pod_name}.${container_name}
|
||||
localOutputRefs:
|
||||
- s3-output
|
||||
match:
|
||||
- select:
|
||||
labels:
|
||||
app: nginx
|
||||
```
|
||||
+293
@@ -0,0 +1,293 @@
|
||||
---
|
||||
title: Outputs and ClusterOutputs
|
||||
---
|
||||
|
||||
For the full details on configuring `Outputs` and `ClusterOutputs`, see the [Banzai Cloud Logging operator documentation.](https://banzaicloud.com/docs/one-eye/logging-operator/configuration/output/)
|
||||
|
||||
## Outputs
|
||||
|
||||
The `Output` resource defines where your `Flows` can send the log messages. `Outputs` are the final stage for a logging `Flow`.
|
||||
|
||||
The `Output` is a namespaced resource, which means only a `Flow` within the same namespace can access it.
|
||||
|
||||
You can use secrets in these definitions, but they must also be in the same namespace.
|
||||
|
||||
`Outputs` can be configured by filling out forms in the Rancher UI.
|
||||
|
||||
For the details of `Output` custom resource, see [OutputSpec.](https://banzaicloud.com/docs/one-eye/logging-operator/configuration/crds/v1beta1/output_types/)
|
||||
|
||||
The Rancher UI provides forms for configuring the following `Output` types:
|
||||
|
||||
- Amazon ElasticSearch
|
||||
- Azure Storage
|
||||
- Cloudwatch
|
||||
- Datadog
|
||||
- Elasticsearch
|
||||
- File
|
||||
- Fluentd
|
||||
- GCS
|
||||
- Kafka
|
||||
- Kinesis Stream
|
||||
- LogDNA
|
||||
- LogZ
|
||||
- Loki
|
||||
- New Relic
|
||||
- Splunk
|
||||
- SumoLogic
|
||||
- Syslog
|
||||
|
||||
The Rancher UI provides forms for configuring the `Output` type, target, and access credentials if applicable.
|
||||
|
||||
For example configuration for each logging plugin supported by the logging operator, see the [logging operator documentation.](https://banzaicloud.com/docs/one-eye/logging-operator/configuration/plugins/outputs/)
|
||||
|
||||
## ClusterOutputs
|
||||
|
||||
`ClusterOutput` defines an `Output` without namespace restrictions. It is only effective when deployed in the same namespace as the logging operator.
|
||||
|
||||
`ClusterOutputs` can be configured by filling out forms in the Rancher UI.
|
||||
|
||||
For the details of the `ClusterOutput` custom resource, see [ClusterOutput.](https://banzaicloud.com/docs/one-eye/logging-operator/configuration/crds/v1beta1/clusteroutput_types/)
|
||||
|
||||
## YAML Examples
|
||||
|
||||
Once logging is installed, you can use these examples to help craft your own logging pipeline.
|
||||
|
||||
- [Cluster Output to ElasticSearch](#cluster-output-to-elasticsearch)
|
||||
- [Output to Splunk](#output-to-splunk)
|
||||
- [Output to Syslog](#output-to-syslog)
|
||||
- [Unsupported Outputs](#unsupported-outputs)
|
||||
|
||||
### Cluster Output to ElasticSearch
|
||||
|
||||
Let's say you wanted to send all logs in your cluster to an `elasticsearch` cluster. First, we create a cluster `Output`.
|
||||
|
||||
```yaml
|
||||
apiVersion: logging.banzaicloud.io/v1beta1
|
||||
kind: ClusterOutput
|
||||
metadata:
|
||||
name: "example-es"
|
||||
namespace: "cattle-logging-system"
|
||||
spec:
|
||||
elasticsearch:
|
||||
host: elasticsearch.example.com
|
||||
port: 9200
|
||||
scheme: http
|
||||
```
|
||||
|
||||
We have created this `ClusterOutput`, without elasticsearch configuration, in the same namespace as our operator: `cattle-logging-system.`. Any time we create a `ClusterFlow` or `ClusterOutput`, we have to put it in the `cattle-logging-system` namespace.
|
||||
|
||||
Now that we have configured where we want the logs to go, let's configure all logs to go to that `ClusterOutput`.
|
||||
|
||||
```yaml
|
||||
apiVersion: logging.banzaicloud.io/v1beta1
|
||||
kind: ClusterFlow
|
||||
metadata:
|
||||
name: "all-logs"
|
||||
namespace: "cattle-logging-system"
|
||||
spec:
|
||||
globalOutputRefs:
|
||||
- "example-es"
|
||||
```
|
||||
|
||||
We should now see our configured index with logs in it.
|
||||
|
||||
|
||||
### Output to Splunk
|
||||
|
||||
What if we have an application team who only wants logs from a specific namespaces sent to a `splunk` server? For this case, we can use namespaced `Outputs` and `Flows`.
|
||||
|
||||
Before we start, let's set up that team's application: `coolapp`.
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: Namespace
|
||||
metadata:
|
||||
name: devteam
|
||||
---
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: coolapp
|
||||
namespace: devteam
|
||||
labels:
|
||||
app: coolapp
|
||||
spec:
|
||||
replicas: 2
|
||||
selector:
|
||||
matchLabels:
|
||||
app: coolapp
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: coolapp
|
||||
spec:
|
||||
containers:
|
||||
- name: generator
|
||||
image: paynejacob/loggenerator:latest
|
||||
```
|
||||
|
||||
With `coolapp` running, we will follow a similar path as when we created a `ClusterOutput`. However, unlike `ClusterOutputs`, we create our `Output` in our application's namespace.
|
||||
|
||||
```yaml
|
||||
apiVersion: logging.banzaicloud.io/v1beta1
|
||||
kind: Output
|
||||
metadata:
|
||||
name: "devteam-splunk"
|
||||
namespace: "devteam"
|
||||
spec:
|
||||
splunkHec:
|
||||
hec_host: splunk.example.com
|
||||
hec_port: 8088
|
||||
protocol: http
|
||||
```
|
||||
|
||||
Once again, let's feed our `Output` some logs:
|
||||
|
||||
```yaml
|
||||
apiVersion: logging.banzaicloud.io/v1beta1
|
||||
kind: Flow
|
||||
metadata:
|
||||
name: "devteam-logs"
|
||||
namespace: "devteam"
|
||||
spec:
|
||||
localOutputRefs:
|
||||
- "devteam-splunk"
|
||||
```
|
||||
|
||||
|
||||
### Output to Syslog
|
||||
|
||||
Let's say you wanted to send all logs in your cluster to an `syslog` server. First, we create a `ClusterOutput`:
|
||||
|
||||
```yaml
|
||||
apiVersion: logging.banzaicloud.io/v1beta1
|
||||
kind: ClusterOutput
|
||||
metadata:
|
||||
name: "example-syslog"
|
||||
namespace: "cattle-logging-system"
|
||||
spec:
|
||||
syslog:
|
||||
buffer:
|
||||
timekey: 30s
|
||||
timekey_use_utc: true
|
||||
timekey_wait: 10s
|
||||
flush_interval: 5s
|
||||
format:
|
||||
type: json
|
||||
app_name_field: test
|
||||
host: syslog.example.com
|
||||
insecure: true
|
||||
port: 514
|
||||
transport: tcp
|
||||
```
|
||||
|
||||
Now that we have configured where we want the logs to go, let's configure all logs to go to that `Output`.
|
||||
|
||||
```yaml
|
||||
apiVersion: logging.banzaicloud.io/v1beta1
|
||||
kind: ClusterFlow
|
||||
metadata:
|
||||
name: "all-logs"
|
||||
namespace: cattle-logging-system
|
||||
spec:
|
||||
globalOutputRefs:
|
||||
- "example-syslog"
|
||||
```
|
||||
|
||||
### Unsupported Outputs
|
||||
|
||||
For the final example, we create an `Output` to write logs to a destination that is not supported out of the box:
|
||||
|
||||
:::note Note on syslog:
|
||||
|
||||
`syslog` is a supported `Output`. However, this example still provides an overview on using unsupported plugins.
|
||||
|
||||
:::
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: syslog-config
|
||||
namespace: cattle-logging-system
|
||||
type: Opaque
|
||||
stringData:
|
||||
fluent-bit.conf: |
|
||||
[INPUT]
|
||||
Name forward
|
||||
Port 24224
|
||||
|
||||
[OUTPUT]
|
||||
Name syslog
|
||||
InstanceName syslog-output
|
||||
Match *
|
||||
Addr syslog.example.com
|
||||
Port 514
|
||||
Cluster ranchers
|
||||
|
||||
---
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: fluentbit-syslog-forwarder
|
||||
namespace: cattle-logging-system
|
||||
labels:
|
||||
output: syslog
|
||||
spec:
|
||||
selector:
|
||||
matchLabels:
|
||||
output: syslog
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
output: syslog
|
||||
spec:
|
||||
containers:
|
||||
- name: fluentbit
|
||||
image: paynejacob/fluent-bit-out-syslog:latest
|
||||
ports:
|
||||
- containerPort: 24224
|
||||
volumeMounts:
|
||||
- mountPath: "/fluent-bit/etc/"
|
||||
name: configuration
|
||||
volumes:
|
||||
- name: configuration
|
||||
secret:
|
||||
secretName: syslog-config
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: syslog-forwarder
|
||||
namespace: cattle-logging-system
|
||||
spec:
|
||||
selector:
|
||||
output: syslog
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 24224
|
||||
targetPort: 24224
|
||||
---
|
||||
apiVersion: logging.banzaicloud.io/v1beta1
|
||||
kind: ClusterFlow
|
||||
metadata:
|
||||
name: all-logs
|
||||
namespace: cattle-logging-system
|
||||
spec:
|
||||
globalOutputRefs:
|
||||
- syslog
|
||||
---
|
||||
apiVersion: logging.banzaicloud.io/v1beta1
|
||||
kind: ClusterOutput
|
||||
metadata:
|
||||
name: syslog
|
||||
namespace: cattle-logging-system
|
||||
spec:
|
||||
forward:
|
||||
servers:
|
||||
- host: "syslog-forwarder.cattle-logging-system"
|
||||
require_ack_response: false
|
||||
ignore_network_errors_at_startup: false
|
||||
```
|
||||
|
||||
Let's break down what is happening here. First, we create a deployment of a container that has the additional `syslog` plugin and accepts logs forwarded from another `fluentd`. Next we create an `Output` configured as a forwarder to our deployment. The deployment `fluentd` will then forward all logs to the configured `syslog` destination.
|
||||
@@ -0,0 +1,28 @@
|
||||
---
|
||||
title: Architecture
|
||||
---
|
||||
|
||||
This section summarizes the architecture of the Rancher logging application.
|
||||
|
||||
For more details about how the Banzai Cloud Logging operator works, see the [official documentation.](https://banzaicloud.com/docs/one-eye/logging-operator/#architecture)
|
||||
|
||||
### How the Banzai Cloud Logging Operator Works
|
||||
|
||||
The Logging operator automates the deployment and configuration of a Kubernetes logging pipeline. It deploys and configures a Fluent Bit DaemonSet on every node to collect container and application logs from the node file system.
|
||||
|
||||
Fluent Bit queries the Kubernetes API and enriches the logs with metadata about the pods, and transfers both the logs and the metadata to Fluentd. Fluentd receives, filters, and transfers logs to multiple `Outputs`.
|
||||
|
||||
The following custom resources are used to define how logs are filtered and sent to their `Outputs`:
|
||||
|
||||
- A `Flow` is a namespaced custom resource that uses filters and selectors to route log messages to the appropriate `Outputs`.
|
||||
- A `ClusterFlow` is used to route cluster-level log messages.
|
||||
- An `Output` is a namespaced resource that defines where the log messages are sent.
|
||||
- A `ClusterOutput` defines an `Output` that is available from all `Flows` and `ClusterFlows`.
|
||||
|
||||
Each `Flow` must reference an `Output`, and each `ClusterFlow` must reference a `ClusterOutput`.
|
||||
|
||||
The following figure from the [Banzai documentation](https://banzaicloud.com/docs/one-eye/logging-operator/#architecture) shows the new logging architecture:
|
||||
|
||||
<figcaption>How the Banzai Cloud Logging Operator Works with Fluentd and Fluent Bit</figcaption>
|
||||
|
||||
|
||||
@@ -0,0 +1,95 @@
|
||||
---
|
||||
title: rancher-logging Helm Chart Options
|
||||
---
|
||||
|
||||
|
||||
### Enable/Disable Windows Node Logging
|
||||
|
||||
You can enable or disable Windows node logging by setting `global.cattle.windows.enabled` to either `true` or `false` in the `values.yaml`.
|
||||
|
||||
By default, Windows node logging will be enabled if the Cluster Dashboard UI is used to install the logging application on a Windows cluster.
|
||||
|
||||
In this scenario, setting `global.cattle.windows.enabled` to `false` will disable Windows node logging on the cluster.
|
||||
When disabled, logs will still be collected from Linux nodes within the Windows cluster.
|
||||
|
||||
:::note
|
||||
|
||||
Currently an [issue](https://github.com/rancher/rancher/issues/32325) exists where Windows nodeAgents are not deleted when performing a `helm upgrade` after disabling Windows logging in a Windows cluster. In this scenario, users may need to manually remove the Windows nodeAgents if they are already installed.
|
||||
|
||||
:::
|
||||
|
||||
### Working with a Custom Docker Root Directory
|
||||
|
||||
If using a custom Docker root directory, you can set `global.dockerRootDirectory` in `values.yaml`.
|
||||
|
||||
This will ensure that the Logging CRs created will use your specified path rather than the default Docker `data-root` location.
|
||||
|
||||
Note that this only affects Linux nodes.
|
||||
|
||||
If there are any Windows nodes in the cluster, the change will not be applicable to those nodes.
|
||||
|
||||
### Adding NodeSelector Settings and Tolerations for Custom Taints
|
||||
|
||||
You can add your own `nodeSelector` settings and add `tolerations` for additional taints by editing the logging Helm chart values. For details, see [this page.](taints-and-tolerations.md)
|
||||
|
||||
### Enabling the Logging Application to Work with SELinux
|
||||
|
||||
:::note Requirements:
|
||||
|
||||
Logging v2 was tested with SELinux on RHEL/CentOS 7 and 8.
|
||||
|
||||
:::
|
||||
|
||||
[Security-Enhanced Linux (SELinux)](https://en.wikipedia.org/wiki/Security-Enhanced_Linux) is a security enhancement to Linux. After being historically used by government agencies, SELinux is now industry standard and is enabled by default on CentOS 7 and 8.
|
||||
|
||||
To use Logging v2 with SELinux, we recommend installing the `rancher-selinux` RPM according to the instructions on [this page.](../../pages-for-subheaders/selinux-rpm.md#installing-the-rancher-selinux-rpm)
|
||||
|
||||
Then, when installing the logging application, configure the chart to be SELinux aware by changing `global.seLinux.enabled` to `true` in the `values.yaml`.
|
||||
|
||||
### Additional Logging Sources
|
||||
|
||||
By default, Rancher collects logs for [control plane components](https://kubernetes.io/docs/concepts/overview/components/#control-plane-components) and [node components](https://kubernetes.io/docs/concepts/overview/components/#node-components) for all cluster types.
|
||||
|
||||
In some cases, Rancher may be able to collect additional logs.
|
||||
|
||||
The following table summarizes the sources where additional logs may be collected for each node types:
|
||||
|
||||
| Logging Source | Linux Nodes (including in Windows cluster) | Windows Nodes |
|
||||
| --- | --- | ---|
|
||||
| RKE | ✓ | ✓ |
|
||||
| RKE2 | ✓ | |
|
||||
| K3s | ✓ | |
|
||||
| AKS | ✓ | |
|
||||
| EKS | ✓ | |
|
||||
| GKE | ✓ | |
|
||||
|
||||
To enable hosted Kubernetes providers as additional logging sources, enable **Enable enhanced cloud provider logging** option when installing or upgrading the Logging Helm chart.
|
||||
|
||||
When enabled, Rancher collects all additional node and control plane logs the provider has made available, which may vary between providers
|
||||
|
||||
If you're already using a cloud provider's own logging solution such as AWS CloudWatch or Google Cloud operations suite (formerly Stackdriver), it is not necessary to enable this option as the native solution will have unrestricted access to all logs.
|
||||
|
||||
### Systemd Configuration
|
||||
|
||||
In Rancher logging, `SystemdLogPath` must be configured for K3s and RKE2 Kubernetes distributions.
|
||||
|
||||
K3s and RKE2 Kubernetes distributions log to journald, which is the subsystem of systemd that is used for logging. In order to collect these logs, the `systemdLogPath` needs to be defined. While the `run/log/journal` directory is used by default, some Linux distributions do not default to this path. For example, Ubuntu defaults to `var/log/journal`. To determine your `systemdLogPath` configuration, see steps below.
|
||||
|
||||
**Steps for Systemd Configuration:**
|
||||
|
||||
* Run `cat /etc/systemd/journald.conf | grep -E ^\#?Storage | cut -d"=" -f2` on one of your nodes.
|
||||
* If `persistent` is returned, your `systemdLogPath` should be `/var/log/journal`.
|
||||
* If `volatile` is returned, your `systemdLogPath` should be `/run/log/journal`.
|
||||
* If `auto` is returned, check if `/var/log/journal` exists.
|
||||
* If `/var/log/journal` exists, then use `/var/log/journal`.
|
||||
* If `/var/log/journal` does not exist, then use `/run/log/journal`.
|
||||
|
||||
:::note Notes:
|
||||
|
||||
If any value not described above is returned, Rancher Logging will not be able to collect control plane logs. To address this issue, you will need to perform the following actions on every control plane node:
|
||||
|
||||
* Set `Storage=volatile` in journald.conf.
|
||||
* Reboot your machine.
|
||||
* Set `systemdLogPath` to `/run/log/journal`.
|
||||
|
||||
:::
|
||||
@@ -0,0 +1,188 @@
|
||||
---
|
||||
title: Migrating to Rancher v2.5 Logging
|
||||
---
|
||||
Starting in v2.5, the logging feature available within Rancher has been completely overhauled. The [logging operator](https://github.com/banzaicloud/logging-operator) from Banzai Cloud has been adopted; Rancher configures this tooling for use when deploying logging.
|
||||
|
||||
Among the many features and changes in the new logging functionality is the removal of project-specific logging configurations. Instead, one now configures logging at the namespace level. Cluster-level logging remains available, but configuration options differ.
|
||||
|
||||
## Installation
|
||||
|
||||
To install logging in Rancher v2.5+, refer to the [installation instructions](../../pages-for-subheaders/logging.md#enabling-logging).
|
||||
|
||||
### Terminology
|
||||
|
||||
In v2.5+, logging configuration in the **Cluster Dashboard**. To configure logging custom resources after the Logging application is installed, go to the left navigation bar and click **Logging**. It is from this menu option that logging for both cluster and namespace is configured.
|
||||
|
||||
:::note
|
||||
|
||||
Logging is installed on a per-cluster basis. You will need to navigate between clusters to configure logging for each cluster.
|
||||
|
||||
:::
|
||||
|
||||
There are four key concepts to understand for v2.5+ logging:
|
||||
|
||||
1. Outputs
|
||||
|
||||
`Outputs` are a configuration resource that determine a destination for collected logs. This is where settings for aggregators such as ElasticSearch, Kafka, etc. are stored. `Outputs` are namespaced resources.
|
||||
|
||||
2. Flows
|
||||
|
||||
`Flows` are a configuration resource that determine collection, filtering, and destination rules for logs. It is within a flow that one will configure what logs to collect, how to mutate or filter them, and which `Outputs` to send the logs to. `Flows` are namespaced resources, and can connect either to an `Output` in the same namespace, or a `ClusterOutput`.
|
||||
|
||||
3. ClusterOutputs
|
||||
|
||||
`ClusterOutputs` serve the same functionality as `Outputs`, except they are a cluster-scoped resource. `ClusterOutputs` are necessary when collecting logs cluster-wide, or if you wish to provide an `Output` to all namespaces in your cluster.
|
||||
|
||||
4. ClusterFlows
|
||||
|
||||
`ClusterFlows` serve the same function as `Flows`, but at the cluster level. They are used to configure log collection for an entire cluster, instead of on a per-namespace level. `ClusterFlows` are also where mutations and filters are defined, same as `Flows` (in functionality).
|
||||
|
||||
## Cluster Logging
|
||||
|
||||
To configure cluster-wide logging for v2.5+ logging, one needs to set up a `ClusterFlow`. This object defines the source of logs, any transformations or filters to be applied, and finally the `Output` (or `Outputs`) for the logs.
|
||||
|
||||
:::note Important:
|
||||
|
||||
`ClusterFlows` must be defined within the `cattle-logging-system` namespace. `ClusterFlows` will not work if defined in any other namespace.
|
||||
|
||||
:::
|
||||
|
||||
In legacy logging, in order to collect logs from across the entire cluster, one only needed to enable cluster-level logging and define the desired `Output`. This basic approach remains in v2.5+ logging. To replicate legacy cluster-level logging, follow these steps:
|
||||
|
||||
1. Define a `ClusterOutput` according to the instructions found under [Output Configuration](#output-configuration)
|
||||
2. Create a `ClusterFlow`, ensuring that it is set to be created in the `cattle-logging-system` namespace
|
||||
1. Remove all _Include_ and _Exclude_ rules from the `Flow` definition. This ensures that all logs are gathered.
|
||||
2. You do not need to configure any filters if you do not wish - default behavior does not require their creation
|
||||
3. Define your cluster `Output` or `Outputs`
|
||||
|
||||
This will result in logs from all sources in the cluster (all pods, and all system components) being collected and sent to the `Output` or `Outputs` you defined in the `ClusterFlow`.
|
||||
|
||||
## Project Logging
|
||||
|
||||
Logging in v2.5+ is not project-aware. This means that in order to collect logs from pods running in project namespaces, you will need to define `Flows` for those namespaces.
|
||||
|
||||
To collect logs from a specific namespace, follow these steps:
|
||||
|
||||
1. Define an `Output` or `ClusterOutput` according to the instructions found under [Output Configuration](#output-configuration)
|
||||
2. Create a `Flow`, ensuring that it is set to be created in the namespace in which you want to gather logs.
|
||||
1. If you wish to define _Include_ or _Exclude_ rules, you may do so. Otherwise, removal of all rules will result in all pods in the target namespace having their logs collected.
|
||||
2. You do not need to configure any filters if you do not wish - default behavior does not require their creation
|
||||
3. Define your outputs - these can be either `ClusterOutput` or `Output` objects.
|
||||
|
||||
This will result in logs from all sources in the namespace (pods) being collected and sent to the `Output` (or `Outputs`) you defined in your `Flow`.
|
||||
|
||||
:::note
|
||||
|
||||
To collect logs from a project, repeat the above steps for every namespace within the project. Alternatively, you can label your project workloads with a common label (e.g. `project=my-project`) and use a `ClusterFlow` to collect logs from all pods matching this label.
|
||||
|
||||
:::
|
||||
|
||||
## Output Configuration
|
||||
In legacy logging, there are five logging destinations to choose from: Elasticsearch, Splunk, Kafka, Fluentd, and Syslog. With the exception of Syslog, all of these destinations are available in logging v2.5+.
|
||||
|
||||
|
||||
### Elasticsearch
|
||||
|
||||
| Legacy Logging | v2.5+ Logging | Notes |
|
||||
|-----------------------------------------------|-----------------------------------|-----------------------------------------------------------|
|
||||
| Endpoint | Target -> Host | Make sure to specify Scheme (https/http), as well as Port |
|
||||
| X-Pack Security -> Username | Access -> User | |
|
||||
| X-Pack Security -> Password | Access -> Password | Password must now be stored in a secret |
|
||||
| SSL Configuration -> Client Private Key | SSL -> Client Key | Key must now be stored in a secret |
|
||||
| SSL Configuration -> Client Certificate | SSL -> Client Cert | Certificate must now be stored in a secret |
|
||||
| SSL Configuration -> Client Key Password | SSL -> Client Key Pass | Password must now be stored in a secret |
|
||||
| SSL Configuration -> Enabled SSL Verification | SSL -> Certificate Authority File | Certificate must now be stored in a secret |
|
||||
|
||||
|
||||
In legacy logging, indices were automatically created according to the format in the "Index Patterns" section. In v2.5 logging, default behavior has been changed to logging to a single index. You can still configure index pattern functionality on the `Output` object by editing as YAML and inputting the following values:
|
||||
|
||||
```yaml
|
||||
...
|
||||
spec:
|
||||
elasticsearch:
|
||||
...
|
||||
logstash_format: true
|
||||
logstash_prefix: <desired prefix>
|
||||
logstash_dateformat: "%Y-%m-%d"
|
||||
```
|
||||
|
||||
Replace `<desired prefix>` with the prefix for the indices that will be created. In legacy logging, this defaulted to the name of the cluster.
|
||||
|
||||
### Splunk
|
||||
|
||||
| Legacy Logging | v2.5+ Logging | Notes |
|
||||
|------------------------------------------|----------------------------------------|----------------------------------------------------------------------------------------|
|
||||
| HEC Configuration -> Endpoint | Target -> Host | Protocol (https/http) and port must be defined separately from the host |
|
||||
| HEC Configuration -> Token | Access -> Token | Token must now be stored as a secret |
|
||||
| HEC Configuration -> Index | Edit as YAML -> `index` | `index` field must be added as YAML key under `spec.splunkHec` |
|
||||
| HEC Configuration -> Source | Edit as YAML -> `source` | `source` field must be added as YAML key under `spec.splunkHec` |
|
||||
| SSL Configuration -> Client Private Key | Edit as YAML -> `client_key` | `client_key` field must be added as YAML key under `spec.splunkHec`. See (1) |
|
||||
| SSL Configuration -> Client Certificate | Edit as YAML -> `client_cert` | `client_cert` field must be added as YAML key under `spec.splunkHec`. See (1) |
|
||||
| SSL Configuration -> Client Key Password | _Not Supported_ | Specifying a password for the client private key is not currently supported. |
|
||||
| SSL Configuration -> SSL Verify | Edit as YAML -> `ca_file` or `ca_path` | `ca_file` or `ca_path` field must be added as YAML key under `spec.splunkHec`. See (2) |
|
||||
|
||||
_(1) `client_key` and `client_cert` values must be paths to the key and cert files, respectively. These files must be mounted into the `rancher-logging-fluentd` pod in order to be used._
|
||||
|
||||
_(2) Users can configure either `ca_file` (a path to a PEM-encoded CA certificate) or `ca_path` (a path to a directory containing CA certificates in PEM format). These files must be mounted into the `rancher-logging-fluentd` pod in order to be used._
|
||||
|
||||
### Kafka
|
||||
|
||||
| Legacy Logging | v2.5+ Logging | Notes |
|
||||
|-----------------------------------------|----------------------------|------------------------------------------------------|
|
||||
| Kafka Configuration -> Endpoint Type | - | Zookeeper is no longer supported as an endpoint type |
|
||||
| Kafka Configuration -> Endpoint | Target -> Brokers | Comma-separated list of brokers (host:port) |
|
||||
| Kafka Configuration -> Topic | Target -> Default Topic | |
|
||||
| SSL Configuration -> Client Private Key | SSL -> SSL Client Cert | Certificate must be stored as a secret |
|
||||
| SSL Configuration -> Client Certificate | SSL -> SSL Client Cert Key | Key must be stored as a secret |
|
||||
| SSL Configuration -> CA Certificate PEM | SSL -> SSL CA Cert | Certificate must be stored as a secret |
|
||||
| SASL Configuration -> Username | Access -> Username | Username must be stored in a secret |
|
||||
| SASL Configuration -> Password | Access -> Password | Password must be stored in a secret |
|
||||
| SASL Configuration -> Scram Mechanism | Access -> Scram Mechanism | Input mechanism as string, e.g. "sha256" or "sha512" |
|
||||
|
||||
### Fluentd
|
||||
|
||||
As of v2.5.2, it is only possible to add a single Fluentd server using the "Edit as Form" option. To add multiple servers, edit the `Output` as YAML and input multiple servers.
|
||||
|
||||
| Legacy Logging | v2.5+ Logging | Notes |
|
||||
|------------------------------------------|-----------------------------------------------------|----------------------------------------------------------------------|
|
||||
| Fluentd Configuration -> Endpoint | Target -> Host, Port | Input the host and port separately |
|
||||
| Fluentd Configuration -> Shared Key | Access -> Shared Key | Shared key must be stored as a secret |
|
||||
| Fluentd Configuration -> Username | Access -> Username | Username must be stored as a secret |
|
||||
| Fluentd Configuration -> Password | Access -> Password | Password must be stored as a secret |
|
||||
| Fluentd Configuration -> Hostname | Edit as YAML -> `host` | `host` field set as YAML key under `spec.forward.servers[n]` |
|
||||
| Fluentd Configuration -> Weight | Edit as YAML -> `weight` | `weight` field set as YAML key under `spec.forward.servers[n]` |
|
||||
| SSL Configuration -> Use TLS | - | Do not need to explicitly enable. Define client cert fields instead. |
|
||||
| SSL Configuration -> Client Private Key | Edit as YAML -> `tls_private_key_path` | Field set as YAML key under `spec.forward`. See (1) |
|
||||
| SSL Configuration -> Client Certificate | Edit as YAML -> `tls_client_cert_path` | Field set as YAML key under `spec.forward`. See (1) |
|
||||
| SSL Configuration -> Client Key Password | Edit as YAML -> `tls_client_private_key_passphrase` | Field set as YAML key under `spec.forward`. See (1) |
|
||||
| SSL Configuration -> SSL Verify | Edit as YAML -> `tls_insecure_mode` | Field set as YAML key under `spec.forward`. Default: `false` |
|
||||
| SSL Configuration -> CA Certificate PEM | Edit as YAML -> `tls_cert_path` | Field set as YAML key under `spec.forward`. See (1) |
|
||||
| Enable Gzip Compression | - | No longer supported in v2.5+ logging |
|
||||
|
||||
_(1) These values are to be specified as paths to files. Those files must be mounted into the `rancher-logging-fluentd` pod in order to be used._
|
||||
|
||||
### Syslog
|
||||
|
||||
As of v2.5.2, syslog is not currently supported for `Outputs` using v2.5+ logging.
|
||||
|
||||
## Custom Log Fields
|
||||
|
||||
In order to add custom log fields, you will need to add the following YAML to your `Flow` configuration:
|
||||
|
||||
```yaml
|
||||
...
|
||||
spec:
|
||||
filters:
|
||||
- record_modifier:
|
||||
records:
|
||||
- foo: "bar"
|
||||
```
|
||||
|
||||
(replace `foo: "bar"` with custom log fields you wish to add)
|
||||
|
||||
## System Logging
|
||||
|
||||
In legacy logging, collecting logs from system components was accomplished by checking a box labeled "Include System Log" when setting up cluster logging. In v2.5+ logging, system logs are gathered in one of two ways:
|
||||
|
||||
1. Gather all cluster logs, not specifying any match or exclusion rules. This results in all container logs from the cluster being collected, which includes system logs.
|
||||
2. Specifically target system logs by adding match rules for system components. Specific match rules depend on the component being collected.
|
||||
@@ -0,0 +1,23 @@
|
||||
---
|
||||
title: Role-based Access Control for Logging
|
||||
---
|
||||
|
||||
Rancher logging has two roles, `logging-admin` and `logging-view`.
|
||||
|
||||
- `logging-admin` gives users full access to namespaced `Flows` and `Outputs`
|
||||
- `logging-view` allows users to *view* namespaced `Flows` and `Outputs`, and `ClusterFlows` and `ClusterOutputs`
|
||||
|
||||
:::note Why choose one role over the other?
|
||||
|
||||
Edit access to `ClusterFlow` and `ClusterOutput` resources is powerful. Any user with it has edit access for all logs in the cluster.
|
||||
|
||||
:::
|
||||
|
||||
In Rancher, the cluster administrator role is the only role with full access to all `rancher-logging` resources. Cluster members are not able to edit or read any logging resources. Project owners and members have the following privileges:
|
||||
|
||||
Project Owners | Project Members
|
||||
--- | ---
|
||||
able to create namespaced `Flows` and `Outputs` in their projects' namespaces | only able to view the `Flows` and `Outputs` in projects' namespaces
|
||||
can collect logs from anything in their projects' namespaces | cannot collect any logs in their projects' namespaces
|
||||
|
||||
Both project owners and project members require at least *one* namespace in their project to use logging. If they do not, then they may not see the logging button in the top nav dropdown.
|
||||
@@ -0,0 +1,65 @@
|
||||
---
|
||||
title: Working with Taints and Tolerations
|
||||
---
|
||||
|
||||
"Tainting" a Kubernetes node causes pods to repel running on that node.
|
||||
|
||||
Unless the pods have a `toleration` for that node's taint, they will run on other nodes in the cluster.
|
||||
|
||||
[Taints and tolerations](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/) can work in conjunction with the `nodeSelector` [field](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#nodeselector) within the `PodSpec`, which enables the *opposite* effect of a taint.
|
||||
|
||||
Using `nodeSelector` gives pods an affinity towards certain nodes.
|
||||
|
||||
Both provide choice for the what node(s) the pod will run on.
|
||||
|
||||
- [Default Implementation in Rancher's Logging Stack](#default-implementation-in-rancher-s-logging-stack)
|
||||
- [Adding NodeSelector Settings and Tolerations for Custom Taints](#adding-nodeselector-settings-and-tolerations-for-custom-taints)
|
||||
|
||||
|
||||
### Default Implementation in Rancher's Logging Stack
|
||||
|
||||
By default, Rancher taints all Linux nodes with `cattle.io/os=linux`, and does not taint Windows nodes.
|
||||
The logging stack pods have `tolerations` for this taint, which enables them to run on Linux nodes.
|
||||
Moreover, most logging stack pods run on Linux only and have a `nodeSelector` added to ensure they run on Linux nodes.
|
||||
|
||||
This example Pod YAML file shows a nodeSelector being used with a toleration:
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: Pod
|
||||
# metadata...
|
||||
spec:
|
||||
# containers...
|
||||
tolerations:
|
||||
- key: cattle.io/os
|
||||
operator: "Equal"
|
||||
value: "linux"
|
||||
effect: NoSchedule
|
||||
nodeSelector:
|
||||
kubernetes.io/os: linux
|
||||
```
|
||||
|
||||
In the above example, we ensure that our pod only runs on Linux nodes, and we add a `toleration` for the taint we have on all of our Linux nodes.
|
||||
|
||||
You can do the same with Rancher's existing taints, or with your own custom ones.
|
||||
|
||||
### Adding NodeSelector Settings and Tolerations for Custom Taints
|
||||
|
||||
If you would like to add your own `nodeSelector` settings, or if you would like to add `tolerations` for additional taints, you can pass the following to the chart's values.
|
||||
|
||||
```yaml
|
||||
tolerations:
|
||||
# insert tolerations...
|
||||
nodeSelector:
|
||||
# insert nodeSelector...
|
||||
```
|
||||
|
||||
These values will add both settings to the `fluentd`, `fluentbit`, and `logging-operator` containers.
|
||||
Essentially, these are global settings for all pods in the logging stack.
|
||||
|
||||
However, if you would like to add tolerations for *only* the `fluentbit` container, you can add the following to the chart's values.
|
||||
|
||||
```yaml
|
||||
fluentbit_tolerations:
|
||||
# insert tolerations list for fluentbit containers only...
|
||||
```
|
||||
Reference in New Issue
Block a user