mirror of
https://github.com/rancher/rancher-docs.git
synced 2026-09-25 20:48:11 +00:00
Add v2.14 preview docs (#2212)
This commit is contained in:
Lucas Saintarbor
+12
@@ -0,0 +1,12 @@
|
||||
---
|
||||
title: Custom Resource Configuration
|
||||
---
|
||||
|
||||
<head>
|
||||
<link rel="canonical" href="https://ranchermanager.docs.rancher.com/integrations-in-rancher/logging/custom-resource-configuration"/>
|
||||
</head>
|
||||
|
||||
The following Custom Resource Definitions are used to configure logging:
|
||||
|
||||
- [Flow and ClusterFlow](flows-and-clusterflows.md)
|
||||
- [Output and ClusterOutput](outputs-and-clusteroutputs.md)
|
||||
+79
@@ -0,0 +1,79 @@
|
||||
---
|
||||
title: Flows and ClusterFlows
|
||||
---
|
||||
|
||||
<head>
|
||||
<link rel="canonical" href="https://ranchermanager.docs.rancher.com/integrations-in-rancher/logging/custom-resource-configuration/flows-and-clusterflows"/>
|
||||
</head>
|
||||
|
||||
See the [Logging operator documentation](https://kube-logging.github.io/docs/configuration/flow/) for the full details on how to configure `Flows` and `ClusterFlows`.
|
||||
|
||||
See [Rancher Integration with Logging Services: Troubleshooting](../logging.md#The-Logging-Buffer-Overloads-Pods) for how to resolve memory problems with the logging buffer.
|
||||
|
||||
## Flows
|
||||
|
||||
A `Flow` defines which logs to collect and filter and which output to send the logs to.
|
||||
|
||||
The `Flow` is a namespaced resource, which means logs will only be collected from the namespace that the `Flow` is deployed in.
|
||||
|
||||
`Flows` can be configured by filling out forms in the Rancher UI.
|
||||
|
||||
For more details about the `Flow` custom resource, see [FlowSpec.](https://kube-logging.github.io/docs/configuration/crds/v1beta1/flow_types/)
|
||||
|
||||
### Matches
|
||||
|
||||
Match statements are used to select which containers to pull logs from.
|
||||
|
||||
You can specify match statements to select or exclude logs according to Kubernetes labels, container and host names. Match statements are evaluated in the order they are defined and processed only until the first matching select or exclude rule applies.
|
||||
|
||||
Matches can be configured by filling out the `Flow` or `ClusterFlow` forms in the Rancher UI.
|
||||
|
||||
For detailed examples on using the match statement, see the [official documentation on log routing.](https://kube-logging.github.io/docs/configuration/log-routing/)
|
||||
|
||||
### Filters
|
||||
|
||||
You can define one or more filters within a `Flow`. Filters can perform various actions on the logs, such as adding data, transforming the logs, or parsing values from the records. The filters in the `Flow` are applied in the same order they appear in the definition.
|
||||
|
||||
For a list of filters supported by the Logging operator, see [the official documentation on Fluentd filters](https://kube-logging.github.io/docs/configuration/plugins/filters/).
|
||||
|
||||
Filters need to be configured in YAML.
|
||||
|
||||
### Outputs
|
||||
|
||||
This `Output` will receive logs from the `Flow`. Because the `Flow` is a namespaced resource, the `Output` must reside in same namespace as the `Flow`.
|
||||
|
||||
`Outputs` can be referenced when filling out the `Flow` or `ClusterFlow` forms in the Rancher UI.
|
||||
|
||||
## ClusterFlows
|
||||
|
||||
Matches, filters and `Outputs` are configured for `ClusterFlows` in the same way that they are configured for `Flows`. The key difference is that the `ClusterFlow` is scoped at the cluster level and can configure log collection across all namespaces.
|
||||
|
||||
`ClusterFlows` can be configured by filling out forms in the Rancher UI.
|
||||
|
||||
After `ClusterFlow` selects logs from all namespaces in the cluster, logs from the cluster will be collected and logged to the selected `ClusterOutput`.
|
||||
|
||||
## YAML Example
|
||||
|
||||
The following example `Flow` transforms the log messages from the default namespace and sends them to an S3 `Output`:
|
||||
|
||||
```yaml
|
||||
apiVersion: logging.banzaicloud.io/v1beta1
|
||||
kind: Flow
|
||||
metadata:
|
||||
name: flow-sample
|
||||
namespace: default
|
||||
spec:
|
||||
filters:
|
||||
- parser:
|
||||
remove_key_name_field: true
|
||||
parse:
|
||||
type: nginx
|
||||
- tag_normaliser:
|
||||
format: ${namespace_name}.${pod_name}.${container_name}
|
||||
localOutputRefs:
|
||||
- s3-output
|
||||
match:
|
||||
- select:
|
||||
labels:
|
||||
app: nginx
|
||||
```
|
||||
+299
@@ -0,0 +1,299 @@
|
||||
---
|
||||
title: Outputs and ClusterOutputs
|
||||
---
|
||||
|
||||
<head>
|
||||
<link rel="canonical" href="https://ranchermanager.docs.rancher.com/integrations-in-rancher/logging/custom-resource-configuration/outputs-and-clusteroutputs"/>
|
||||
</head>
|
||||
|
||||
See the [Logging operator documentation](https://kube-logging.github.io/docs/configuration/flow/) for the full details on how to configure `Flows` and `ClusterFlows`.
|
||||
|
||||
See [Rancher Integration with Logging Services: Troubleshooting](../logging.md#The-Logging-Buffer-Overloads-Pods) for how to resolve memory problems with the logging buffer.
|
||||
|
||||
## Outputs
|
||||
|
||||
The `Output` resource defines where your `Flows` can send the log messages. `Outputs` are the final stage for a logging `Flow`.
|
||||
|
||||
The `Output` is a namespaced resource, which means only a `Flow` within the same namespace can access it.
|
||||
|
||||
You can use secrets in these definitions, but they must also be in the same namespace.
|
||||
|
||||
`Outputs` can be configured by filling out forms in the Rancher UI.
|
||||
|
||||
For the details of `Output` custom resource, see [OutputSpec.](https://kube-logging.github.io/docs/configuration/crds/v1beta1/output_types/).
|
||||
|
||||
The Rancher UI provides forms for configuring the following `Output` types:
|
||||
|
||||
- Amazon ElasticSearch
|
||||
- Azure Storage
|
||||
- Cloudwatch
|
||||
- Datadog
|
||||
- Elasticsearch
|
||||
- File
|
||||
- Fluentd
|
||||
- GCS
|
||||
- Kafka
|
||||
- Kinesis Stream
|
||||
- LogDNA
|
||||
- LogZ
|
||||
- Loki
|
||||
- New Relic
|
||||
- Splunk
|
||||
- SumoLogic
|
||||
- Syslog
|
||||
|
||||
The Rancher UI provides forms for configuring the `Output` type, target, and access credentials if applicable.
|
||||
|
||||
For example configuration for each logging plugin supported by the logging operator, see the [Logging operator documentation](https://kube-logging.github.io/docs/configuration/plugins/outputs/).
|
||||
|
||||
## ClusterOutputs
|
||||
|
||||
`ClusterOutput` defines an `Output` without namespace restrictions. It is only effective when deployed in the same namespace as the logging operator.
|
||||
|
||||
`ClusterOutputs` can be configured by filling out forms in the Rancher UI.
|
||||
|
||||
For the details of the `ClusterOutput` custom resource, see [ClusterOutput.](https://kube-logging.github.io/docs/configuration/crds/v1beta1/clusteroutput_types/)
|
||||
|
||||
## YAML Examples
|
||||
|
||||
Once logging is installed, you can use these examples to help craft your own logging pipeline.
|
||||
|
||||
- [Cluster Output to ElasticSearch](#cluster-output-to-elasticsearch)
|
||||
- [Output to Splunk](#output-to-splunk)
|
||||
- [Output to Syslog](#output-to-syslog)
|
||||
- [Unsupported Outputs](#unsupported-outputs)
|
||||
|
||||
### Cluster Output to ElasticSearch
|
||||
|
||||
Let's say you wanted to send all logs in your cluster to an `elasticsearch` cluster. First, we create a cluster `Output`.
|
||||
|
||||
```yaml
|
||||
apiVersion: logging.banzaicloud.io/v1beta1
|
||||
kind: ClusterOutput
|
||||
metadata:
|
||||
name: "example-es"
|
||||
namespace: "cattle-logging-system"
|
||||
spec:
|
||||
elasticsearch:
|
||||
host: elasticsearch.example.com
|
||||
port: 9200
|
||||
scheme: http
|
||||
```
|
||||
|
||||
We have created this `ClusterOutput`, without elasticsearch configuration, in the same namespace as our operator: `cattle-logging-system.`. Any time we create a `ClusterFlow` or `ClusterOutput`, we have to put it in the `cattle-logging-system` namespace.
|
||||
|
||||
Now that we have configured where we want the logs to go, let's configure all logs to go to that `ClusterOutput`.
|
||||
|
||||
```yaml
|
||||
apiVersion: logging.banzaicloud.io/v1beta1
|
||||
kind: ClusterFlow
|
||||
metadata:
|
||||
name: "all-logs"
|
||||
namespace: "cattle-logging-system"
|
||||
spec:
|
||||
globalOutputRefs:
|
||||
- "example-es"
|
||||
```
|
||||
|
||||
We should now see our configured index with logs in it.
|
||||
|
||||
|
||||
### Output to Splunk
|
||||
|
||||
What if we have an application team who only wants logs from a specific namespaces sent to a `splunk` server? For this case, we can use namespaced `Outputs` and `Flows`.
|
||||
|
||||
Before we start, let's set up that team's application: `coolapp`.
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: Namespace
|
||||
metadata:
|
||||
name: devteam
|
||||
---
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: coolapp
|
||||
namespace: devteam
|
||||
labels:
|
||||
app: coolapp
|
||||
spec:
|
||||
replicas: 2
|
||||
selector:
|
||||
matchLabels:
|
||||
app: coolapp
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: coolapp
|
||||
spec:
|
||||
containers:
|
||||
- name: generator
|
||||
image: paynejacob/loggenerator:latest
|
||||
```
|
||||
|
||||
With `coolapp` running, we will follow a similar path as when we created a `ClusterOutput`. However, unlike `ClusterOutputs`, we create our `Output` in our application's namespace.
|
||||
|
||||
```yaml
|
||||
apiVersion: logging.banzaicloud.io/v1beta1
|
||||
kind: Output
|
||||
metadata:
|
||||
name: "devteam-splunk"
|
||||
namespace: "devteam"
|
||||
spec:
|
||||
splunkHec:
|
||||
hec_host: splunk.example.com
|
||||
hec_port: 8088
|
||||
protocol: http
|
||||
```
|
||||
|
||||
Once again, let's feed our `Output` some logs:
|
||||
|
||||
```yaml
|
||||
apiVersion: logging.banzaicloud.io/v1beta1
|
||||
kind: Flow
|
||||
metadata:
|
||||
name: "devteam-logs"
|
||||
namespace: "devteam"
|
||||
spec:
|
||||
localOutputRefs:
|
||||
- "devteam-splunk"
|
||||
```
|
||||
|
||||
|
||||
### Output to Syslog
|
||||
|
||||
Let's say you wanted to send all logs in your cluster to an `syslog` server. First, we create a `ClusterOutput`:
|
||||
|
||||
```yaml
|
||||
apiVersion: logging.banzaicloud.io/v1beta1
|
||||
kind: ClusterOutput
|
||||
metadata:
|
||||
name: "example-syslog"
|
||||
namespace: "cattle-logging-system"
|
||||
spec:
|
||||
syslog:
|
||||
buffer:
|
||||
timekey: 30s
|
||||
timekey_use_utc: true
|
||||
timekey_wait: 10s
|
||||
flush_interval: 5s
|
||||
format:
|
||||
type: json
|
||||
app_name_field: test
|
||||
host: syslog.example.com
|
||||
insecure: true
|
||||
port: 514
|
||||
transport: tcp
|
||||
```
|
||||
|
||||
Now that we have configured where we want the logs to go, let's configure all logs to go to that `Output`.
|
||||
|
||||
```yaml
|
||||
apiVersion: logging.banzaicloud.io/v1beta1
|
||||
kind: ClusterFlow
|
||||
metadata:
|
||||
name: "all-logs"
|
||||
namespace: cattle-logging-system
|
||||
spec:
|
||||
globalOutputRefs:
|
||||
- "example-syslog"
|
||||
```
|
||||
|
||||
### Unsupported Outputs
|
||||
|
||||
For the final example, we create an `Output` to write logs to a destination that is not supported out of the box:
|
||||
|
||||
:::note Note on syslog:
|
||||
|
||||
`syslog` is a supported `Output`. However, this example still provides an overview on using unsupported plugins.
|
||||
|
||||
:::
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: syslog-config
|
||||
namespace: cattle-logging-system
|
||||
type: Opaque
|
||||
stringData:
|
||||
fluent-bit.conf: |
|
||||
[INPUT]
|
||||
Name forward
|
||||
Port 24224
|
||||
|
||||
[OUTPUT]
|
||||
Name syslog
|
||||
InstanceName syslog-output
|
||||
Match *
|
||||
Addr syslog.example.com
|
||||
Port 514
|
||||
Cluster ranchers
|
||||
|
||||
---
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: fluentbit-syslog-forwarder
|
||||
namespace: cattle-logging-system
|
||||
labels:
|
||||
output: syslog
|
||||
spec:
|
||||
selector:
|
||||
matchLabels:
|
||||
output: syslog
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
output: syslog
|
||||
spec:
|
||||
containers:
|
||||
- name: fluentbit
|
||||
image: paynejacob/fluent-bit-out-syslog:latest
|
||||
ports:
|
||||
- containerPort: 24224
|
||||
volumeMounts:
|
||||
- mountPath: "/fluent-bit/etc/"
|
||||
name: configuration
|
||||
volumes:
|
||||
- name: configuration
|
||||
secret:
|
||||
secretName: syslog-config
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: syslog-forwarder
|
||||
namespace: cattle-logging-system
|
||||
spec:
|
||||
selector:
|
||||
output: syslog
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 24224
|
||||
targetPort: 24224
|
||||
---
|
||||
apiVersion: logging.banzaicloud.io/v1beta1
|
||||
kind: ClusterFlow
|
||||
metadata:
|
||||
name: all-logs
|
||||
namespace: cattle-logging-system
|
||||
spec:
|
||||
globalOutputRefs:
|
||||
- syslog
|
||||
---
|
||||
apiVersion: logging.banzaicloud.io/v1beta1
|
||||
kind: ClusterOutput
|
||||
metadata:
|
||||
name: syslog
|
||||
namespace: cattle-logging-system
|
||||
spec:
|
||||
forward:
|
||||
servers:
|
||||
- host: "syslog-forwarder.cattle-logging-system"
|
||||
require_ack_response: false
|
||||
ignore_network_errors_at_startup: false
|
||||
```
|
||||
|
||||
Let's break down what is happening here. First, we create a deployment of a container that has the additional `syslog` plugin and accepts logs forwarded from another `fluentd`. Next we create an `Output` configured as a forwarder to our deployment. The deployment `fluentd` will then forward all logs to the configured `syslog` destination.
|
||||
Reference in New Issue
Block a user