mirror of
https://github.com/rancher/rancher-docs.git
synced 2026-09-27 05:28:22 +00:00
Remove unneeded intermediate folders
This commit is contained in:
Billy Tat
@@ -0,0 +1,9 @@
|
||||
---
|
||||
title: Custom Resource Configuration
|
||||
weight: 5
|
||||
---
|
||||
|
||||
The following Custom Resource Definitions are used to configure logging:
|
||||
|
||||
- [Flow and ClusterFlow](./flows)
|
||||
- [Output and ClusterOutput](./outputs)
|
||||
@@ -0,0 +1,165 @@
|
||||
---
|
||||
title: Flows and ClusterFlows
|
||||
weight: 1
|
||||
---
|
||||
|
||||
import Tabs from '@theme/Tabs';
|
||||
import TabItem from '@theme/TabItem';
|
||||
|
||||
For the full details on configuring `Flows` and `ClusterFlows`, see the [Banzai Cloud Logging operator documentation.](https://banzaicloud.com/docs/one-eye/logging-operator/configuration/output/)
|
||||
|
||||
- [Configuration](#configuration)
|
||||
- [YAML Example](#yaml-example)
|
||||
|
||||
# Configuration
|
||||
|
||||
<Tabs>
|
||||
<TabItem value="Rancher v2.5.8+">
|
||||
|
||||
- [Flows](#flows-2-5-8)
|
||||
- [Matches](#matches-2-5-8)
|
||||
- [Filters](#filters-2-5-8)
|
||||
- [Outputs](#outputs-2-5-8)
|
||||
- [ClusterFlows](#clusterflows-2-5-8)
|
||||
|
||||
# Changes in v2.5.8
|
||||
|
||||
The `Flows` and `ClusterFlows` can now be configured by filling out forms in the Rancher UI.
|
||||
|
||||
|
||||
<a id="flows-2-5-8"></a>
|
||||
|
||||
# Flows
|
||||
|
||||
A `Flow` defines which logs to collect and filter and which output to send the logs to.
|
||||
|
||||
The `Flow` is a namespaced resource, which means logs will only be collected from the namespace that the `Flow` is deployed in.
|
||||
|
||||
For more details about the `Flow` custom resource, see [FlowSpec.](https://banzaicloud.com/docs/one-eye/logging-operator/configuration/crds/v1beta1/flow_types/)
|
||||
|
||||
|
||||
<a id="matches-2-5-8"></a>
|
||||
|
||||
### Matches
|
||||
|
||||
Match statements are used to select which containers to pull logs from.
|
||||
|
||||
You can specify match statements to select or exclude logs according to Kubernetes labels, container and host names. Match statements are evaluated in the order they are defined and processed only until the first matching select or exclude rule applies.
|
||||
|
||||
Matches can be configured by filling out the `Flow` or `ClusterFlow` forms in the Rancher UI.
|
||||
|
||||
For detailed examples on using the match statement, see the [official documentation on log routing.](https://banzaicloud.com/docs/one-eye/logging-operator/configuration/log-routing/)
|
||||
|
||||
<a id="filters-2-5-8"></a>
|
||||
|
||||
### Filters
|
||||
|
||||
You can define one or more filters within a `Flow`. Filters can perform various actions on the logs, for example, add additional data, transform the logs, or parse values from the records. The filters in the `Flow` are applied in the order in the definition.
|
||||
|
||||
For a list of filters supported by the Banzai Cloud Logging operator, see [this page.](https://banzaicloud.com/docs/one-eye/logging-operator/configuration/plugins/filters/)
|
||||
|
||||
Filters need to be configured in YAML.
|
||||
|
||||
<a id="outputs-2-5-8"></a>
|
||||
|
||||
### Outputs
|
||||
|
||||
This `Output` will receive logs from the `Flow`. Because the `Flow` is a namespaced resource, the `Output` must reside in same namespace as the `Flow`.
|
||||
|
||||
`Outputs` can be referenced when filling out the `Flow` or `ClusterFlow` forms in the Rancher UI.
|
||||
|
||||
<a id="clusterflows-2-5-8"></a>
|
||||
|
||||
# ClusterFlows
|
||||
|
||||
Matches, filters and `Outputs` are configured for `ClusterFlows` in the same way that they are configured for `Flows`. The key difference is that the `ClusterFlow` is scoped at the cluster level and can configure log collection across all namespaces.
|
||||
|
||||
After `ClusterFlow` selects logs from all namespaces in the cluster, logs from the cluster will be collected and logged to the selected `ClusterOutput`.
|
||||
|
||||
|
||||
</TabItem>
|
||||
<TabItem value="Rancher before v2.5.8">
|
||||
|
||||
- [Flows](#flows-2-5-0)
|
||||
- [Matches](#matches-2-5-0)
|
||||
- [Filters](#filters-2-5-0)
|
||||
- [Outputs](#outputs-2-5-0)
|
||||
- [ClusterFlows](#clusterflows-2-5-0)
|
||||
|
||||
|
||||
<a id="flows-2-5-0"></a>
|
||||
|
||||
# Flows
|
||||
|
||||
A `Flow` defines which logs to collect and filter and which `Output` to send the logs to. The `Flow` is a namespaced resource, which means logs will only be collected from the namespace that the `Flow` is deployed in.
|
||||
|
||||
`Flows` need to be defined in YAML.
|
||||
|
||||
For more details about the `Flow` custom resource, see [FlowSpec.](https://banzaicloud.com/docs/one-eye/logging-operator/configuration/crds/v1beta1/flow_types/)
|
||||
|
||||
|
||||
<a id="matches-2-5-0"></a>
|
||||
|
||||
### Matches
|
||||
|
||||
Match statements are used to select which containers to pull logs from.
|
||||
|
||||
You can specify match statements to select or exclude logs according to Kubernetes labels, container and host names. Match statements are evaluated in the order they are defined and processed only until the first matching select or exclude rule applies.
|
||||
|
||||
For detailed examples on using the match statement, see the [official documentation on log routing.](https://banzaicloud.com/docs/one-eye/logging-operator/configuration/log-routing/)
|
||||
|
||||
<a id="filters-2-5-0"></a>
|
||||
|
||||
### Filters
|
||||
|
||||
You can define one or more filters within a `Flow`. Filters can perform various actions on the logs, for example, add additional data, transform the logs, or parse values from the records. The filters in the `Flow` are applied in the order in the definition.
|
||||
|
||||
For a list of filters supported by the Banzai Cloud Logging operator, see [this page.](https://banzaicloud.com/docs/one-eye/logging-operator/configuration/plugins/filters/)
|
||||
|
||||
<a id="outputs-2-5-0"></a>
|
||||
|
||||
### Outputs
|
||||
|
||||
This `Output` will receive logs from the `Flow`.
|
||||
|
||||
Because the `Flow` is a namespaced resource, the `Output` must reside in same namespace as the `Flow`.
|
||||
|
||||
<a id="clusterflows-2-5-0"></a>
|
||||
|
||||
# ClusterFlows
|
||||
|
||||
Matches, filters and `Outputs` are also configured for `ClusterFlows`. The only difference is that the `ClusterFlow` is scoped at the cluster level and can configure log collection across all namespaces.
|
||||
|
||||
`ClusterFlow` selects logs from all namespaces in the cluster. Logs from the cluster will be collected and logged to the selected `ClusterOutput`.
|
||||
|
||||
`ClusterFlows` need to be defined in YAML.
|
||||
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
|
||||
|
||||
# YAML Example
|
||||
|
||||
The following example `Flow` transforms the log messages from the default namespace and sends them to an S3 `Output`:
|
||||
|
||||
```yaml
|
||||
apiVersion: logging.banzaicloud.io/v1beta1
|
||||
kind: Flow
|
||||
metadata:
|
||||
name: flow-sample
|
||||
namespace: default
|
||||
spec:
|
||||
filters:
|
||||
- parser:
|
||||
remove_key_name_field: true
|
||||
parse:
|
||||
type: nginx
|
||||
- tag_normaliser:
|
||||
format: ${namespace_name}.${pod_name}.${container_name}
|
||||
localOutputRefs:
|
||||
- s3-output
|
||||
match:
|
||||
- select:
|
||||
labels:
|
||||
app: nginx
|
||||
```
|
||||
@@ -0,0 +1,348 @@
|
||||
---
|
||||
title: Outputs and ClusterOutputs
|
||||
weight: 2
|
||||
---
|
||||
|
||||
import Tabs from '@theme/Tabs';
|
||||
import TabItem from '@theme/TabItem';
|
||||
|
||||
For the full details on configuring `Outputs` and `ClusterOutputs`, see the [Banzai Cloud Logging operator documentation.](https://banzaicloud.com/docs/one-eye/logging-operator/configuration/output/)
|
||||
|
||||
- [Configuration](#configuration)
|
||||
- [YAML Examples](#yaml-examples)
|
||||
- [Cluster Output to ElasticSearch](#cluster-output-to-elasticsearch)
|
||||
- [Output to Splunk](#output-to-splunk)
|
||||
- [Output to Syslog](#output-to-syslog)
|
||||
- [Unsupported Outputs](#unsupported-outputs)
|
||||
|
||||
# Configuration
|
||||
|
||||
<Tabs>
|
||||
<TabItem value="v2.5.8+">
|
||||
|
||||
- [Outputs](#outputs-2-5-8)
|
||||
- [ClusterOutputs](#clusteroutputs-2-5-8)
|
||||
|
||||
# Changes in v2.5.8
|
||||
|
||||
The `Outputs` and `ClusterOutputs` can now be configured by filling out forms in the Rancher UI.
|
||||
|
||||
|
||||
<a id="outputs-2-5-8"></a>
|
||||
# Outputs
|
||||
|
||||
The `Output` resource defines where your `Flows` can send the log messages. `Outputs` are the final stage for a logging `Flow`.
|
||||
|
||||
The `Output` is a namespaced resource, which means only a `Flow` within the same namespace can access it.
|
||||
|
||||
You can use secrets in these definitions, but they must also be in the same namespace.
|
||||
|
||||
For the details of `Output` custom resource, see [OutputSpec.](https://banzaicloud.com/docs/one-eye/logging-operator/configuration/crds/v1beta1/output_types/)
|
||||
|
||||
The Rancher UI provides forms for configuring the following `Output` types:
|
||||
|
||||
- Amazon ElasticSearch
|
||||
- Azure Storage
|
||||
- Cloudwatch
|
||||
- Datadog
|
||||
- Elasticsearch
|
||||
- File
|
||||
- Fluentd
|
||||
- GCS
|
||||
- Kafka
|
||||
- Kinesis Stream
|
||||
- LogDNA
|
||||
- LogZ
|
||||
- Loki
|
||||
- New Relic
|
||||
- Splunk
|
||||
- SumoLogic
|
||||
- Syslog
|
||||
|
||||
The Rancher UI provides forms for configuring the `Output` type, target, and access credentials if applicable.
|
||||
|
||||
For example configuration for each logging plugin supported by the logging operator, see the [logging operator documentation.](https://banzaicloud.com/docs/one-eye/logging-operator/configuration/plugins/outputs/)
|
||||
|
||||
<a id="clusteroutputs-2-5-8"></a>
|
||||
|
||||
# ClusterOutputs
|
||||
|
||||
`ClusterOutput` defines an `Output` without namespace restrictions. It is only effective when deployed in the same namespace as the logging operator.
|
||||
|
||||
For the details of the `ClusterOutput` custom resource, see [ClusterOutput.](https://banzaicloud.com/docs/one-eye/logging-operator/configuration/crds/v1beta1/clusteroutput_types/)
|
||||
|
||||
</TabItem>
|
||||
<TabItem value="Rancher before v2.5.8">
|
||||
|
||||
- [Outputs](#outputs-2-5-0)
|
||||
- [ClusterOutputs](#clusteroutputs-2-5-0)
|
||||
|
||||
|
||||
<a id="outputs-2-5-0"></a>
|
||||
# Outputs
|
||||
|
||||
The `Output` resource defines where your `Flows` can send the log messages. `Outputs` are the final stage for a logging `Flow`.
|
||||
|
||||
The `Output` is a namespaced resource, which means only a `Flow` within the same namespace can access it.
|
||||
|
||||
You can use secrets in these definitions, but they must also be in the same namespace.
|
||||
|
||||
`Outputs` are configured in YAML. For the details of `Output` custom resource, see [OutputSpec.](https://banzaicloud.com/docs/one-eye/logging-operator/configuration/crds/v1beta1/output_types/)
|
||||
|
||||
For examples of configuration for each logging plugin supported by the logging operator, see the [logging operator documentation.](https://banzaicloud.com/docs/one-eye/logging-operator/configuration/plugins/outputs/)
|
||||
|
||||
<a id="clusteroutputs-2-5-0"></a>
|
||||
|
||||
# ClusterOutputs
|
||||
|
||||
`ClusterOutput` defines an `Output` without namespace restrictions. It is only effective when deployed in the same namespace as the logging operator.
|
||||
|
||||
The Rancher UI provides forms for configuring the `ClusterOutput` type, target, and access credentials if applicable.
|
||||
|
||||
`ClusterOutputs` are configured in YAML. For the details of `ClusterOutput` custom resource, see [ClusterOutput.](https://banzaicloud.com/docs/one-eye/logging-operator/configuration/crds/v1beta1/clusteroutput_types/)
|
||||
|
||||
For example configuration for each logging plugin supported by the logging operator, see the [logging operator documentation.](https://banzaicloud.com/docs/one-eye/logging-operator/configuration/plugins/outputs/)
|
||||
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
|
||||
|
||||
# YAML Examples
|
||||
|
||||
Once logging is installed, you can use these examples to help craft your own logging pipeline.
|
||||
|
||||
- [Cluster Output to ElasticSearch](#cluster-output-to-elasticsearch)
|
||||
- [Output to Splunk](#output-to-splunk)
|
||||
- [Output to Syslog](#output-to-syslog)
|
||||
- [Unsupported Outputs](#unsupported-outputs)
|
||||
|
||||
### Cluster Output to ElasticSearch
|
||||
|
||||
Let's say you wanted to send all logs in your cluster to an `elasticsearch` cluster. First, we create a cluster `Output`.
|
||||
|
||||
```yaml
|
||||
apiVersion: logging.banzaicloud.io/v1beta1
|
||||
kind: ClusterOutput
|
||||
metadata:
|
||||
name: "example-es"
|
||||
namespace: "cattle-logging-system"
|
||||
spec:
|
||||
elasticsearch:
|
||||
host: elasticsearch.example.com
|
||||
port: 9200
|
||||
scheme: http
|
||||
```
|
||||
|
||||
We have created this `ClusterOutput`, without elasticsearch configuration, in the same namespace as our operator: `cattle-logging-system.`. Any time we create a `ClusterFlow` or `ClusterOutput`, we have to put it in the `cattle-logging-system` namespace.
|
||||
|
||||
Now that we have configured where we want the logs to go, let's configure all logs to go to that `ClusterOutput`.
|
||||
|
||||
```yaml
|
||||
apiVersion: logging.banzaicloud.io/v1beta1
|
||||
kind: ClusterFlow
|
||||
metadata:
|
||||
name: "all-logs"
|
||||
namespace: "cattle-logging-system"
|
||||
spec:
|
||||
globalOutputRefs:
|
||||
- "example-es"
|
||||
```
|
||||
|
||||
We should now see our configured index with logs in it.
|
||||
|
||||
|
||||
### Output to Splunk
|
||||
|
||||
What if we have an application team who only wants logs from a specific namespaces sent to a `splunk` server? For this case, we can use namespaced `Outputs` and `Flows`.
|
||||
|
||||
Before we start, let's set up that team's application: `coolapp`.
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: Namespace
|
||||
metadata:
|
||||
name: devteam
|
||||
---
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: coolapp
|
||||
namespace: devteam
|
||||
labels:
|
||||
app: coolapp
|
||||
spec:
|
||||
replicas: 2
|
||||
selector:
|
||||
matchLabels:
|
||||
app: coolapp
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: coolapp
|
||||
spec:
|
||||
containers:
|
||||
- name: generator
|
||||
image: paynejacob/loggenerator:latest
|
||||
```
|
||||
|
||||
With `coolapp` running, we will follow a similar path as when we created a `ClusterOutput`. However, unlike `ClusterOutputs`, we create our `Output` in our application's namespace.
|
||||
|
||||
```yaml
|
||||
apiVersion: logging.banzaicloud.io/v1beta1
|
||||
kind: Output
|
||||
metadata:
|
||||
name: "devteam-splunk"
|
||||
namespace: "devteam"
|
||||
spec:
|
||||
splunkHec:
|
||||
hec_host: splunk.example.com
|
||||
hec_port: 8088
|
||||
protocol: http
|
||||
```
|
||||
|
||||
Once again, let's feed our `Output` some logs:
|
||||
|
||||
```yaml
|
||||
apiVersion: logging.banzaicloud.io/v1beta1
|
||||
kind: Flow
|
||||
metadata:
|
||||
name: "devteam-logs"
|
||||
namespace: "devteam"
|
||||
spec:
|
||||
localOutputRefs:
|
||||
- "devteam-splunk"
|
||||
```
|
||||
|
||||
|
||||
### Output to Syslog
|
||||
|
||||
Let's say you wanted to send all logs in your cluster to an `syslog` server. First, we create a `ClusterOutput`:
|
||||
|
||||
```yaml
|
||||
apiVersion: logging.banzaicloud.io/v1beta1
|
||||
kind: ClusterOutput
|
||||
metadata:
|
||||
name: "example-syslog"
|
||||
namespace: "cattle-logging-system"
|
||||
spec:
|
||||
syslog:
|
||||
buffer:
|
||||
timekey: 30s
|
||||
timekey_use_utc: true
|
||||
timekey_wait: 10s
|
||||
flush_interval: 5s
|
||||
format:
|
||||
type: json
|
||||
app_name_field: test
|
||||
host: syslog.example.com
|
||||
insecure: true
|
||||
port: 514
|
||||
transport: tcp
|
||||
```
|
||||
|
||||
Now that we have configured where we want the logs to go, let's configure all logs to go to that `Output`.
|
||||
|
||||
```yaml
|
||||
apiVersion: logging.banzaicloud.io/v1beta1
|
||||
kind: ClusterFlow
|
||||
metadata:
|
||||
name: "all-logs"
|
||||
namespace: cattle-logging-system
|
||||
spec:
|
||||
globalOutputRefs:
|
||||
- "example-syslog"
|
||||
```
|
||||
|
||||
### Unsupported Outputs
|
||||
|
||||
For the final example, we create an `Output` to write logs to a destination that is not supported out of the box:
|
||||
|
||||
> **Note on syslog** As of Rancher v2.5.4, `syslog` is a supported `Output`. However, this example still provides an overview on using unsupported plugins.
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: syslog-config
|
||||
namespace: cattle-logging-system
|
||||
type: Opaque
|
||||
stringData:
|
||||
fluent-bit.conf: |
|
||||
[INPUT]
|
||||
Name forward
|
||||
Port 24224
|
||||
|
||||
[OUTPUT]
|
||||
Name syslog
|
||||
InstanceName syslog-output
|
||||
Match *
|
||||
Addr syslog.example.com
|
||||
Port 514
|
||||
Cluster ranchers
|
||||
|
||||
---
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: fluentbit-syslog-forwarder
|
||||
namespace: cattle-logging-system
|
||||
labels:
|
||||
output: syslog
|
||||
spec:
|
||||
selector:
|
||||
matchLabels:
|
||||
output: syslog
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
output: syslog
|
||||
spec:
|
||||
containers:
|
||||
- name: fluentbit
|
||||
image: paynejacob/fluent-bit-out-syslog:latest
|
||||
ports:
|
||||
- containerPort: 24224
|
||||
volumeMounts:
|
||||
- mountPath: "/fluent-bit/etc/"
|
||||
name: configuration
|
||||
volumes:
|
||||
- name: configuration
|
||||
secret:
|
||||
secretName: syslog-config
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: syslog-forwarder
|
||||
namespace: cattle-logging-system
|
||||
spec:
|
||||
selector:
|
||||
output: syslog
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 24224
|
||||
targetPort: 24224
|
||||
---
|
||||
apiVersion: logging.banzaicloud.io/v1beta1
|
||||
kind: ClusterFlow
|
||||
metadata:
|
||||
name: all-logs
|
||||
namespace: cattle-logging-system
|
||||
spec:
|
||||
globalOutputRefs:
|
||||
- syslog
|
||||
---
|
||||
apiVersion: logging.banzaicloud.io/v1beta1
|
||||
kind: ClusterOutput
|
||||
metadata:
|
||||
name: syslog
|
||||
namespace: cattle-logging-system
|
||||
spec:
|
||||
forward:
|
||||
servers:
|
||||
- host: "syslog-forwarder.cattle-logging-system"
|
||||
require_ack_response: false
|
||||
ignore_network_errors_at_startup: false
|
||||
```
|
||||
|
||||
Let's break down what is happening here. First, we create a deployment of a container that has the additional `syslog` plugin and accepts logs forwarded from another `fluentd`. Next we create an `Output` configured as a forwarder to our deployment. The deployment `fluentd` will then forward all logs to the configured `syslog` destination.
|
||||
Reference in New Issue
Block a user