Logging

2026-08-20 17:02:16 +00:00 · 2019-03-22 10:23:26 -07:00
parent c0ead959f5
commit 422951d2c8
8 changed files with 233 additions and 21 deletions
@@ -60,14 +60,11 @@ For more information, see [Notifiers]({{< baseurl >}}/rancher/v2.x/en/tools/noti

 Rancher can integrate with popular external services used for event streams, telemetry, or search. Rancher can integrate with the following services:

- Embedded Elasticsearch (experimental)
-
-    >**Note:** This option is available only at the cluster level.
-
 - Elasticsearch
 - splunk
 - kafka
 - syslog
+- fluentd

 These services collect container log events, which are saved to the `/var/log/containers` directory on each of your nodes. The service collects both standard and error events. You can then log into your services to review the events collected, leveraging each service's unique features.

@@ -13,6 +13,7 @@ Rancher supports the following services:
 - [Splunk]({{< baseurl >}}/rancher/v2.x/en/tools/logging/splunk)
 - [Kafka]({{< baseurl >}}/rancher/v2.x/en/tools/logging/kafka)
 - [Syslog]({{< baseurl >}}/rancher/v2.x/en/tools/logging/syslog) 
+- [Fluentd]({{< baseurl >}}/rancher/v2.x/en/tools/logging/fluentd) 

 ## Requirements

@@ -49,7 +50,7 @@ You can configure logging at either cluster or project level.

 - If you're a [project owner or member]({{< baseurl >}}/rancher/v2.x/en/admin-settings/rbac/cluster-project-roles/#project-roles) who works on an application, configure project logging.

-    Project logging writes logs for every pod in the project (`/var/log/containers`).
+    Project logging writes logs for every pod in the project.

 After collection, all logs are stored by your logging service. Log into your service to view them.

@@ -0,0 +1,88 @@
+---
+title: Custom
+weight: 700
+---
+
+You can configure Rancher to send Kubernetes logs to Elasticsearch, Splunk, Kafka, Syslog or Fluentd.
+
+## Configuring Advance Mode
+
+You can configure Rancher to send cluster or project logs to one of the logging targets in advance mode by inputting raw fluentd configuration.
+
+>**Prerequisite:** You must have a one of the logging targets of Elasticsearch, Splunk, Kafka, Syslog and Fluentd server configured.
+
+1. Browse to the cluster or project.
+{{% accordion id="cluster" label="To Configure Cluster Logging:" %}}
+If you're a [cluster owner or member]({{< baseurl >}}/rancher/v2.x/en/admin-settings/rbac/cluster-project-roles/#cluster-roles) who works in operations or security, configure cluster logging.
+
+1. From the **Global** view, open the cluster that you want to configure logging for.
+
+1. From the main menu, select **Tools > Logging**.
+
+{{% /accordion %}}
+{{% accordion id="project" label="To Configure Project Logging:" %}}
+If you're a [project owner or member]({{< baseurl >}}/rancher/v2.x/en/admin-settings/rbac/cluster-project-roles/#project-roles) who works on an application, configure project logging.
+
+1. From the **Global** view, open the project that you want to configure logging for.
+
+1. From the main menu, select **Tools > Logging**. 
+
+{{% /accordion %}}
+
+1. Select one of the logging targets, giving example for **Elasticsearch**.
+
+1. Click the  **Edit as File**.
+
+    1. Input the Fluentd output configuration.
+
+    1. Giving an example for **Elasticsearch** target.
+    
+        ```sh
+        @type elasticsearch
+        include_tag_key  true
+        user elasticsearch
+        password PleaseChaneMe
+        hosts https://rancher.com:9200    
+        logstash_format true
+        logstash_prefix elastic-index
+        logstash_dateformat  %Y-%m-%d
+        type_name  container_log    
+        
+        ```
+
+    1. For Elasticsearch configuration, see [Elasticsearch Documentation](https://github.com/uken/fluent-plugin-elasticsearch) for details.
+
+    1. For Splunk configuration, see [Splunk Documentation](https://github.com/fluent/fluent-plugin-splunk) for details.
+
+    1. For Kafka configuration, see [Kafka Documentation](https://github.com/fluent/fluent-plugin-kafka) for details.
+
+    1. For Syslog configuration, see [Syslog Documentation](https://github.com/dlackty/fluent-plugin-remote_syslog) for details.
+
+    1. For Fluentd configuration, see [Fluentd Documentation](https://docs.fluentd.org/v1.0/articles/out_forward) for details.
+
+1. If your logging server is using TLS, you need to complete the **SSL Configuration** form.
+
+    1. Enter the private key and client certificate. You can either copy and paste them or upload them by **Read from a file**.
+
+        You can use either a self-signed certificate or one provided by a certificate authority.
+
+        You can generate a self-signed certificate using an openssl command. For example:
+            
+            openssl req -x509 -newkey rsa:2048 -keyout myservice.key -out myservice.cert -days 365 -nodes -subj "/CN=myservice.example.com"
+       
+    
+    1. If you are using a self-signed certificate, you need to provide the **CA Certificate PEM** as well.
+
+1. Complete the **Additional Logging Configuration** form.
+
+    1. **Optional:** Use the **Add Field** button to add custom log fields to your logging configuration. These fields are key value pairs (such as `foo=bar`) that you can use to filter the logs from another system.
+    
+    1. Enter a **Flush Interval**. This value determines how often [Fluentd](https://www.fluentd.org/) flushes data to the logging server. Intervals are measured in seconds.
+
+    1. **Include System Log**. The logs from pods in system project and RKE components will be sent to the target. Uncheck it to exclude the system logs.
+
+1. Click **Dry Run**. Rancher calls the fluentd dry run command to validate the configuration.
+
+1. Click **Save**.
+
+**Result:** Rancher is now configured to send logs to your configured server. View your server to see logs for your cluster and containers.
@@ -24,7 +24,7 @@ If you're a [project owner or member]({{< baseurl >}}/rancher/v2.x/en/admin-sett

 1. From the **Global** view, open the project that you want to configure logging for.

-1. From the main menu, select **Resources > Logging**. 
+1. From the main menu, select **Tools > Logging**. 
 {{% /accordion %}}

 1. Select **Elasticsearch**.
@@ -39,7 +39,7 @@ If you're a [project owner or member]({{< baseurl >}}/rancher/v2.x/en/admin-sett

 1. If your instance of Elasticsearch uses SSL, complete the **SSL Configuration** form. 

-    1. Enter a private key and client certificate. Either copy and paste them or browse to them using **Read from a file**. This certificate will be installed on your logging server.
+    1. Enter the private key and client certificate. You can either copy and paste them or upload them by **Read from a file**.

        You can use either a self-signed certificate or one provided by a certificate authority.

@@ -47,10 +47,14 @@ If you're a [project owner or member]({{< baseurl >}}/rancher/v2.x/en/admin-sett
            
            openssl req -x509 -newkey rsa:2048 -keyout myservice.key -out myservice.cert -days 365 -nodes -subj "/CN=myservice.example.com"
       
+    
+    1. If you are using a self-signed certificate, you need to provide the **CA Certificate PEM** as well.

    1. Enter your private key password.

-    1. If you are using a certificate from a certificate authority (and not a self-signed certificate), select the **Enabled - Input trusted server certificate** option and then enter your **Trusted Server Certificate**.
+    1. Enter your ssl version. The default version is tlsv1_2.
+
+    1. Select the **Enabled - Input trusted server certificate** option and enter your **Trusted Server Certificate Chain** if you are using a certificate from a certificate authority.

 1. Complete the **Additional Logging Configuration** form.

@@ -58,6 +62,10 @@ If you're a [project owner or member]({{< baseurl >}}/rancher/v2.x/en/admin-sett

    1. Enter a **Flush Interval**. This value determines how often [Fluentd](https://www.fluentd.org/) flushes data to the logging server. Intervals are measured in seconds.

+    1. **Include System Log**. The logs from pods in system project and RKE components will be sent to the target. Uncheck it to exclude the system logs.
+
+1. Click **Test**. Rancher sends a test log to Elasticsearch.
+
 1. Click **Save**.

 **Result:** Rancher is now configured to send cluster and container logs to Elasticsearch. Log into Elasticsearch or Kibana to view your cluster/project logs.
@@ -0,0 +1,68 @@
+---
+title: Fluentd
+weight: 600
+---
+
+If your organization is using [Fluentd](https://www.fluentd.org/), you can configure Rancher to send logs to Fluentd server.
+
+## Configuring Fluentd Logging
+
+You can configure Rancher to send Kubernetes logs to your Fluentd.
+
+>**Prerequisites:** Configure Fluentd input forward to receive the event stream.
+>
+>See [Fluentd Documentation](https://docs.fluentd.org/v1.0/articles/in_forward) for details.
+
+1. Browse to the cluster or project that you want to log.
+{{% accordion id="cluster" label="To Configure Cluster Logging:" %}}
+If you're a [cluster owner or member]({{< baseurl >}}/rancher/v2.x/en/admin-settings/rbac/cluster-project-roles/#cluster-roles) who works in operations or security, configure cluster logging.
+
+1. From the **Global** view, open the cluster that you want to configure logging for.
+
+1. From the main menu, select **Tools > Logging**.
+{{% /accordion %}}
+{{% accordion id="project" label="To Configure Project Logging:" %}}
+If you're a [project owner or member]({{< baseurl >}}/rancher/v2.x/en/admin-settings/rbac/cluster-project-roles/#project-roles) who works on an application, configure project logging.
+
+1. From the **Global** view, open the project that you want to configure logging for.
+
+1. From the main menu, select **Tools > Logging**. 
+{{% /accordion %}}
+
+1. Select **Fluentd**.
+
+1. Complete the **Fluentd Server Configuration** form.
+
+    1. From the **Endpoint** field, enter the address of you Fluentd instance (i.e. `http://Fluentd-server:24224`).
+
+    1. Enter the **Shared Key** if your Fluentd Server is using shared key for authentication.
+
+    1. Enter the **Username** and **Password** if your Fluentd Server is using username and pasword for authentication.
+
+    1. **Optional:** Enter the **Hostname** of the server.
+
+    1. Enter the load balancing **Weight** of the server. If the weight of one server is 20 and the other server is 30, events will be sent in a 2:3 ratio. The default weight is 60 if you leave this field empty.
+
+    1. Check **Use as Standby Only** if this server is standby. Standby servers will be used when all non-standby servers are down.
+
+    1. You can add multiple Fluentd servers.
+
+1. Select **Enable Gzip Compression**. The transferred payload size will be reduced.
+
+1. If your Fluentd is using SSL, you need to complete the **SSL Configuration** form. 
+
+    1. If you are using a self-signed certificate, you need to provide the **CA Certificate PEM** as well.
+
+1. Complete the **Additional Logging Configuration** form.
+
+    1. **Optional:** Use the **Add Field** button to add custom log fields to your logging configuration. These fields are key value pairs (such as `foo=bar`) that you can use to filter the logs from another system.
+
+    1. Enter a **Flush Interval**. This value determines how often [Fluentd](https://www.fluentd.org/) flushes event data to the logging server. Intervals are measured in seconds.
+
+    1. **Include System Log**. The logs from pods in system project and RKE components will be sent to the target. Uncheck it to exclude the system logs.
+
+1. Click **Test**. Rancher sends a test log to Fluentd with rancher tag.
+
+1. Click **Save**.
+
+**Result:** Rancher is now configured to send logs to Fluentd. Check you Fluentd output events for your cluster and containers.
@@ -22,7 +22,7 @@ If you're a [project owner or member]({{< baseurl >}}/rancher/v2.x/en/admin-sett

 1. From the **Global** view, open the project that you want to configure logging for.

-1. From the main menu, select **Resources > Logging**. 
+1. From the main menu, select **Tools > Logging**. 
 {{% /accordion %}}

 1. Select **Kafka**.
@@ -31,18 +31,43 @@ If you're a [project owner or member]({{< baseurl >}}/rancher/v2.x/en/admin-sett

    1. From **Endpoint Type**, select the type of Kafka server you are using: **Zookeeper** or **Broker**.

-    1. From the **Endpoint** field, enter the IP address and port for your Kafka server.
+    1. From the **Endpoint** field, enter the IP address and the port for your Zookeeper or Broker. For Broker type, you can add multiple endpoints.

-        By default, Kafka uses port `9092`.
+        By default, Zookeeper uses port `2181`, Kafka broker uses port `9092`. Please note that Zookeeper endpoint is for Kafka cluster not enable tls.

    1. From the **Topic** field, enter the name of a Kafka [topic](https://kafka.apache.org/documentation/#basic_ops_add_topic) that your Kubernetes cluster submits logs to.

+1. If your Kafka cluster is using SSL for **Broker**, you need to complete the **SSL Configuration** form. 
+
+    1. Enter the private key and client certificate. You can either copy and paste them or upload them by **Read from a file**.
+
+        You can use either a self-signed certificate or one provided by a certificate authority.
+
+        You can generate a self-signed certificate using an openssl command. For example:
+            
+            openssl req -x509 -newkey rsa:2048 -keyout myservice.key -out myservice.cert -days 365 -nodes -subj "/CN=myservice.example.com"
+       
+    
+    1. If you are using a self-signed certificate, you need to provide the **CA Certificate PEM** as well.
+
+1. If your Kafka cluster is using SASL authentication for **Broker**, you need to complete the **SASL Configuration** form. See [Kafka SASL](https://kafka.apache.org/documentation/#security_sasl) for details.
+
+    1. Enter SASL **Username** and **Passwork**
+
+    1. Select **SASL Type** which your Kafka is using. Please make sure your Kafka cluster is using SSL when you select **Plain**.
+
+    1. If your Kafka is using **Scram**, you need to select which **Scram Mechanism** Kafka is using.
+
 1. Complete the **Additional Logging Configuration** form.

    1. **Optional:** Use the **Add Field** button to add custom log fields to your logging configuration. These fields are key value pairs (such as `foo=bar`) that you can use to filter the logs from another system.
    
    1. Enter a **Flush Interval**. This value determines how often [Fluentd](https://www.fluentd.org/) flushes logs to the logging server. Intervals are measured in seconds.

+    1. **Include System Log**. The logs from pods in system project and RKE components will be sent to the target. Uncheck it to exclude the system logs.    
+
+1. Click **Test**. Rancher sends a test log to Kafka.
+
 1. Click **Save**.

 **Result:** Rancher is now configured to send logs to Kafka. View your Kafka stream to view logs for your cluster and containers.
@@ -31,7 +31,7 @@ If you're a [project owner or member]({{< baseurl >}}/rancher/v2.x/en/admin-sett

 1. From the **Global** view, open the project that you want to configure logging for.

-1. From the main menu, select **Resources > Logging**. 
+1. From the main menu, select **Tools > Logging**. 
 {{% /accordion %}}

 1. Select **Splunk**.
@@ -48,15 +48,33 @@ If you're a [project owner or member]({{< baseurl >}}/rancher/v2.x/en/admin-sett

    1. **Optional:** Enter one or more [index](http://docs.splunk.com/Documentation/Splunk/7.1.2/Indexer/Aboutindexesandindexers) that's allowed for your token.

+1. If your Splunk is using SSL, you need to complete the **SSL Configuration** form. 
+
+    1. Enter the private key and client certificate. You can either copy and paste them or upload them by **Read from a file**.
+
+        You can use either a self-signed certificate or one provided by a certificate authority.
+
+        You can generate a self-signed certificate using an openssl command. For example:
+            
+            openssl req -x509 -newkey rsa:2048 -keyout myservice.key -out myservice.cert -days 365 -nodes -subj "/CN=myservice.example.com"
+
+    1. Enter your private key password.
+
+    1. If you are using a self-signed certificate, you need to select the **Enabled - Input trusted server certificate** option and enter your **CA Certificate PEM**.
+
 1. Complete the **Additional Logging Configuration** form.

    1. **Optional:** Use the **Add Field** button to add custom log fields to your logging configuration. These fields are key value pairs (such as `foo=bar`) that you can use to filter the logs from another system.

    1. Enter a **Flush Interval**. This value determines how often [Fluentd](https://www.fluentd.org/) flushes event data to the logging server. Intervals are measured in seconds.

+    1. **Include System Log**. The logs from pods in system project and RKE components will be sent to the target. Uncheck it to exclude the system logs.
+
+1. Click **Test**. Rancher sends a test log to Splunk.
+
 1. Click **Save**.

-**Result:** Rancher is now configured to send logs to Splunk. Log into your Spunk instance to view events for your cluster and containers.
+**Result:** Rancher is now configured to send logs to Splunk. Log into your Splunk instance to view events for your cluster and containers.

 ## Viewing Logs

@@ -83,4 +101,4 @@ $ curl http://splunk-server:8088/services/collector/event \
 If Splunk is configured correctly, you should receive **json** data returning `success code 0`. You should be able
 to send logging data to HEC.

-If you received an error, check your configuration in Splunk and Rancher.
+If you received an error, check your configuration in Splunk and Rancher.
@@ -25,7 +25,7 @@ If you're a [project owner or member]({{< baseurl >}}/rancher/v2.x/en/admin-sett

 1. From the **Global** view, open the project that you want to configure logging for.

-1. From the main menu, select **Resources > Logging**. 
+1. From the main menu, select **Tools > Logging**. 

 {{% /accordion %}}

@@ -41,25 +41,32 @@ If you're a [project owner or member]({{< baseurl >}}/rancher/v2.x/en/admin-sett

    1. Select a **Log Severity** for events that are logged to the Syslog server. For more information on each severity level, see the [Syslog protocol documentation](https://tools.ietf.org/html/rfc5424#page-11).

-1. If your Syslog server uses **TCP** protocol, complete the **SSL Configuration** form.
+1. If your Syslog is using **TCP** protocol, you need to select **Use TLS** and complete the **SSL Configuration** form.

-    1. Enter a private key and client certificate. Either copy and paste them or browse to them using **Read from a file**. This certificate will be installed on your logging server.
+    1. Enter the private key and client certificate. You can either copy and paste them or upload them by **Read from a file**.

        You can use either a self-signed certificate or one provided by a certificate authority.

        You can generate a self-signed certificate using an openssl command. For example:
-        
            
            openssl req -x509 -newkey rsa:2048 -keyout myservice.key -out myservice.cert -days 365 -nodes -subj "/CN=myservice.example.com"

-    1. If you are using a certificate from a certificate authority (and not a self-signed certificate), select the **Enabled - Input trusted server certificate** option and then enter your **Trusted Server Certificate**.
+    1. Enter your private key password.
+
+    1. If you are using a self-signed certificate, you need to select the **Enabled - Input trusted server certificate** option and enter your **CA Certificate PEM**.
+
+    1. If you are using rsyslog, please make sure your rsyslog authentication mode is **x509/name**.

 1. Complete the **Additional Logging Configuration** form.

    1. **Optional:** Use the **Add Field** button to add custom log fields to your logging configuration. These fields are key value pairs (such as `foo=bar`) that you can use to filter the logs from another system.
    
-     1. Enter a **Flush Interval**. This value determines how often [Fluentd](https://www.fluentd.org/) flushes data to the logging server. Intervals are measured in seconds.
+    1. Enter a **Flush Interval**. This value determines how often [Fluentd](https://www.fluentd.org/) flushes data to the logging server. Intervals are measured in seconds.
+
+    1. **Include System Log**. The logs from pods in system project and RKE components will be sent to the target. Uncheck it to exclude the system logs.
+
+1. Click **Test**. Rancher sends a test log to Syslog.

 1. Click **Save**.

-**Result:** Rancher is now configured to send logs to your Syslog server. View your Syslog stream to view logs for your cluster and containers.
+**Result:** Rancher is now configured to send logs to your Syslog server. View your Syslog stream to view logs for your cluster and containers.